Rmail-mbox branch

Rmail-mbox branch

[Chong Yidong]

Is anyone willing to help finish the work on the rmail-mbox branch?

Re: Rmail-mbox branch

[Paul Michael Reilly]

Chong Yidong wrote:
> Is anyone willing to help finish the work on the rmail-mbox branch?
> 

I've bitten the bullet and started to work on it again.  As I said in
private email to Chong, working on this project in a CVS branch has all
the appeal of trying to convince most of the readers of this list that
vi was the right way after all.  Sure enough, I check out the branch,
do a ./configure run "make bootstrap" and BAM:

...
./temacs --batch --load loadup bootstrap
Loading loadup.el (source)...
Using load-path (/export/projects/gnu/mbox/lisp 
/export/projects/gnu/mbox/lisp/emacs-lisp 
/export/projects/gnu/mbox/lisp/language 
/export/projects/gnu/mbox/lisp/international 
/export/projects/gnu/mbox/lisp/textmodes)
Loading emacs-lisp/byte-run (source)...
Wrong type argument: listp, #<subr defmacro>
make[2]: *** [bootstrap-emacs] Error 255
make[2]: Leaving directory `/export/projects/gnu/mbox/src'
make[1]: *** [bootstrap-build] Error 2
make[1]: Leaving directory `/export/projects/gnu/mbox'
make: *** [bootstrap] Error 2


I'll supply more of the audit trail if anyone has way too much time on
their hands and wants to help figure out why the branch won't build on
my F9 Gnu/Linux system.  My wife is convinced I have way too much time
on MY hands to even consider working on Rmail/mbox again.

In any case, to head off more branch and merge grief, right now I'm
considering taking an approach of doing a minimalist clone of the mbox
code to my CVS trunk based workspace and call it Pmail for lack of a
better name.  The goal is to allow me to work on both Rmail/babyl and
Rmail/mbox simultaneously.  It might seem like more work but I have a
strong hunch that being able to work on both versions side by side
will help increase the chance that this project will actually get
done.

Now if someone else should actually finish this mbox work in the
branch and merge it back then more power to you and you have my
eternal thanks.

And I invite helpful comments, suggestions, and/or encouragement,
especially form both of the other Users who actually still want to use
the Rmail/mbox library. :-)

-pmr

Re: Rmail-mbox branch

[Chong Yidong]

Paul Michael Reilly <pmr@pajato.com> writes:

> I've bitten the bullet and started to work on it again.  As I said in
> private email to Chong, working on this project in a CVS branch has all
> the appeal of trying to convince most of the readers of this list that
> vi was the right way after all.  Sure enough, I check out the branch,
> do a ./configure run "make bootstrap" and BAM:

Most unfortunate.  But shouldn't the relevant file be only rmail.el?
Maybe you could pull in the rest of the CVS trunk keep the version of
rmail.el from the branch.  Or does the mbox branch change other files?

Anyway, thanks for doing this.

Re: Rmail-mbox branch

[Paul Michael Reilly]

Chong Yidong wrote:
> Paul Michael Reilly <pmr@pajato.com> writes:
> 
>> I've bitten the bullet and started to work on it again.  As I said in
>> private email to Chong, working on this project in a CVS branch has all
>> the appeal of trying to convince most of the readers of this list that
>> vi was the right way after all.  Sure enough, I check out the branch,
>> do a ./configure run "make bootstrap" and BAM:
> 
> Most unfortunate.  But shouldn't the relevant file be only rmail.el?

It is the most relevant but not the only one.

> Maybe you could pull in the rest of the CVS trunk keep the version of
> rmail.el from the branch.  Or does the mbox branch change other files?

I'll pull in the files from the branch as they are needed.  The new 
files are of course trivial to merge.  I plan on handling files which 
need to be changed by renaming them initially to p* instead or r* but 
otherwise leave them intact from the branch.  This is part of my goal to 
be able to have side by side instances of Rmail/babyl and Rmail/mbox 
running simultaneously.  Then after a set of key people, probably you, 
rms and Stefan at the least, bless the Rmail/mbox variant, I'll do the 
merge which will be tantamount to combining the p* files into the r* 
files but all done using the trunk.  It might make some sense to resolve 
conflicts as I pull in files from the branch that need to be modified. 
We'll see how it plays out.

-pmr

Re: Rmail-mbox branch

[Miles Bader]

Whoever made the branch didn't make a base tag?

-Miles

-- 
Somebody has to do something, and it's just incredibly pathetic that it
has to be us.  -- Jerry Garcia

Re: Rmail-mbox branch

[Stefan Monnier]

> Whoever made the branch didn't make a base tag?

What else is new?


        Stefan


PS: Welcome to the wonderful world of CVS.

Re: Rmail-mbox branch

[Miles Bader]

On Tue, Jul 1, 2008 at 12:21 AM, Paul Michael Reilly <pmr@pajato.com> wrote:
>> Whoever made the branch didn't make a base tag?
>
> Probably not.  What was he thinking?  Let's find the culprit and shoot him!
>  Or at least pull his check-in privileges!  Can you find out who created the
> branch and take it from there? :-)

Actually I just looked at CVS, and there _does_ seem to be a base-tag
for the rmail branch, called "RMAIL-MBOX-BASE" (in uppercase).

That should make merging much easier... (I'd just use the tag to make
a patch file and apply it to the current trunk).

-Miles

-- 
Steven Wright  - "A lot of people are afraid of heights. Not me, I'm
afraid of widths."

Re: Rmail-mbox branch

[Stefan Monnier]

> In any case, to head off more branch and merge grief, right now I'm
> considering taking an approach of doing a minimalist clone of the mbox
> code to my CVS trunk based workspace and call it Pmail for lack of a
> better name.  The goal is to allow me to work on both Rmail/babyl and
> Rmail/mbox simultaneously.  It might seem like more work but I have a
> strong hunch that being able to work on both versions side by side
> will help increase the chance that this project will actually get
> done.

You mean you want to replace the branch by combining both features so
they coexist in the same branch (e.g. in the trunk)?

That sounds like a good plan.  Even better if you can avoid duplicating
files, using (if rmail-mbox <somecode> <othercode>) where
necessary instead.

As long as the BABYL code is not broken before the mbox code works, and
as long as there is clear understanding that we *will* finish the mbox
support, you could even install this code on the trunk so people can
test it just by (setq rmail-mbox t).


        Stefan

Re: Rmail-mbox branch

[Paul Michael Reilly]

Stefan Monnier wrote:
>> In any case, to head off more branch and merge grief, right now I'm
>> considering taking an approach of doing a minimalist clone of the mbox
>> code to my CVS trunk based workspace and call it Pmail for lack of a
>> better name.  The goal is to allow me to work on both Rmail/babyl and
>> Rmail/mbox simultaneously.  It might seem like more work but I have a
>> strong hunch that being able to work on both versions side by side
>> will help increase the chance that this project will actually get
>> done.
> 
> You mean you want to replace the branch by combining both features so
> they coexist in the same branch (e.g. in the trunk)?

Correct.  Once both Rmail/mbox and Rmail/bably are running in the trunk 
the branch is obsolete.

> That sounds like a good plan.  Even better if you can avoid duplicating
> files, using (if rmail-mbox <somecode> <othercode>) where
> necessary instead.

This type of approach may make it more difficult (but not impossible by 
any stretch) to have both running simultaneously.  I'll keep it in mind. 
The more I think about it the more appealing it is.

> As long as the BABYL code is not broken before the mbox code works, and
> as long as there is clear understanding that we *will* finish the mbox
> support, you could even install this code on the trunk so people can
> test it just by (setq rmail-mbox t).

Agreed.

-pmr

Re: Rmail-mbox branch

[Stefan Monnier]

> This type of approach may make it more difficult (but not impossible by any
> stretch) to have both running simultaneously.

Very easy: run 2 copies of Emacs.
Problem solved,


        Stefan

Re: Rmail-mbox branch

[Paul Michael Reilly]

Stefan Monnier wrote:
>> In any case, to head off more branch and merge grief, right now I'm
>> considering taking an approach of doing a minimalist clone of the mbox
>> code to my CVS trunk based workspace and call it Pmail for lack of a
>> better name.  The goal is to allow me to work on both Rmail/babyl and
>> Rmail/mbox simultaneously.  It might seem like more work but I have a
>> strong hunch that being able to work on both versions side by side
>> will help increase the chance that this project will actually get
>> done.
> 
> You mean you want to replace the branch by combining both features so
> they coexist in the same branch (e.g. in the trunk)?
> 
> That sounds like a good plan.  Even better if you can avoid duplicating
> files, using (if rmail-mbox <somecode> <othercode>) where
> necessary instead.
> 
> As long as the BABYL code is not broken before the mbox code works, and
> as long as there is clear understanding that we *will* finish the mbox
> support, you could even install this code on the trunk so people can
> test it just by (setq rmail-mbox t).

I'm not quite to that point yet but it is worth working towards even if 
difficult.  For now, I just committed a bunch of "pmail*.el" files in 
lisp/mail that will allow developers to test Rmail/mbox using M-x pmail.

My long term goal is to replace my use of Thunderbird with Rmail/mbox as 
the first step.  The next step is to test Rmail/mbox using imap via Gnu 
Mailutils (for the movemail/imap support) so my Rmail/mbox testing will 
be geared very much to my day-day use of email, rather than specific 
test cases.  A good thing, I think.

But the real good news, IMHO, is that we can now work on finishing 
Rmail/mbox in the trunk rather than have to use an evil branch.

-pmr

Re: Rmail-mbox branch

[Miles Bader]

Paul Michael Reilly <pmr@pajato.com> writes:
> For now, I just committed a bunch of "pmail*.el" files in
> lisp/mail that will allow developers to test Rmail/mbox using M-x pmail.

I presume it's a mistake that you also committed a bunch of what seem to
be temporary files ?

[The directories lisp/mail/mbox-trunk-annotations and lisp/mail/mbox-changes]

-Miles

-- 
Non-combatant, n. A dead Quaker.

Re: Rmail-mbox branch

[Paul Michael Reilly]

Miles Bader wrote:
> Paul Michael Reilly <pmr@pajato.com> writes:
>> For now, I just committed a bunch of "pmail*.el" files in
>> lisp/mail that will allow developers to test Rmail/mbox using M-x pmail.
> 
> I presume it's a mistake that you also committed a bunch of what seem to
> be temporary files ?

Nope.  I figured it would make some sense to provide this data, 
especially the log of changes I applied as part of the somewhat 
incomplete merge.  The annotation (temp) files are more of an indulgence 
but, yes, all these pmail*.el, *.changes and *.annotation files will 
eventually be expunged from the repository.

-pmr

Re: Rmail-mbox branch

[Miles Bader]

Paul Michael Reilly <pmr@pajato.com> writes:
>> I presume it's a mistake that you also committed a bunch of what seem to
>> be temporary files ?
>
> Nope.  I figured it would make some sense to provide this data,
> especially the log of changes I applied as part of the somewhat
> incomplete merge.

I'm not sure what you mean -- those files seem to merely contain output
from the "cvs log" command, which could presumably be easily duplicated by
anyone.  Obviously the annotations could as well.  [If you want that info
at a particular point in time, you should use "cvs tag" to add a tag.]

So what exactly is the point of dumping it all into files in the repository?

-Miles

-- 
`Suppose Korea goes to the World Cup final against Japan and wins,' Moon said.
`All the past could be forgiven.'   [NYT]

Re: Rmail-mbox branch

[Paul Michael Reilly]

Miles Bader wrote:
> Paul Michael Reilly <pmr@pajato.com> writes:
>>> I presume it's a mistake that you also committed a bunch of what seem to
>>> be temporary files ?
>> Nope.  I figured it would make some sense to provide this data,
>> especially the log of changes I applied as part of the somewhat
>> incomplete merge.
> 
> I'm not sure what you mean -- those files seem to merely contain output
> from the "cvs log" command, which could presumably be easily duplicated by
> anyone.  Obviously the annotations could as well.  [If you want that info
> at a particular point in time, you should use "cvs tag" to add a tag.]
> 
> So what exactly is the point of dumping it all into files in the repository?

The *.changes files contain notes I added on what was done in the merge 
in case anyone wants to use/check that information in the absence of a 
real merge log entry.  The real merge entry will get generated when the 
pmail*.el files are converted back to rmail*.el files.  The *.annotation 
files were extremely useful during the merge from the branch to the 
pmail*.el files and while I knew/know they can be regenerated it is a 
convenience to have them handy, IMHO.  Again, all of these pmail*.el, 
*.changes and *.annotations files supporting completing the Rmail/mbox 
conversion are temporary and exist in the faint hope that it will make 
it easier to get other developers to help out.  And in no small part 
because I have not worked on this code in years and wanted to have an 
audit trail to help reveal any mistakes I might have made.

-pmr

Re: Rmail-mbox branch

[Glenn Morris]

Where's the ChangeLog for this stuff?

Re: Rmail-mbox branch

[Paul Michael Reilly]

Glenn Morris wrote:
> Where's the ChangeLog for this stuff?

The pmail*.el and related files I created are temporary and pretty much 
only committed to the trunk as a convenience to make Rmail/mbox testable 
without having to use a branch so I didn't create a ChangeLog for these 
files.

After I work out with the maintainers when to inflict Rmail/mbox on all 
Users, the ChangeLog will be created.

My preference is to do this right after the next release.

-pmr

Re: Rmail-mbox branch

[Chong Yidong]

Paul Michael Reilly <pmr@pajato.com> writes:

> The pmail*.el and related files I created are temporary and pretty
> much only committed to the trunk as a convenience to make Rmail/mbox
> testable without having to use a branch so I didn't create a ChangeLog
> for these files.
>
> After I work out with the maintainers when to inflict Rmail/mbox on
> all Users, the ChangeLog will be created.

Is there any reason not to inflict it *now*?

Re: Rmail-mbox branch

[Evil Boris]

Chong Yidong <cyd@stupidchicken.com> writes:

> Paul Michael Reilly <pmr@pajato.com> writes:
>
>> After I work out with the maintainers when to inflict Rmail/mbox on
>> all Users, the ChangeLog will be created.
>
> Is there any reason not to inflict it *now*?

I am a regular user of RMAIL.  I have not had a chance to use the mbox
version, for two reasons.

The main reason is that, as of the last time anything concrete was said
on this mailing list of the state of RMAIL/mbox, it was mentioned that
there was no support for dealing with attachments (and I somehow also
assumed that dealing with non-US character sets and/or UNICODE, but I
may be confused).  I need this functionality for my day-to-day use of
rmail.  "Inflicting" it on users without such support seems a bit
excessive.

Secondly, rmail/mbox converts my mailbox to mbox format.  Which is fine
for a one-time experiment (backup, try it, play with the buttons, etc),
but I would be reluctant to use it day-to-day, until I have some
confidence that newly arriving mail is handled without data loss.  Just
paranoid, a bit.

--Boris

PS. I do not use IMAP at the moment, so this is a secondary issue for
me.  I am not sure if IMAP (not just fetching via IMAP) is consistent
with Rmail's view of the world, by the way.

Re: Rmail-mbox branch

[Eli Zaretskii]

> From: Evil Boris <evilborisnet@netscape.net>
> Date: Sun, 31 Aug 2008 15:15:04 -0400
> 
> 
> The main reason is that, as of the last time anything concrete was said
> on this mailing list of the state of RMAIL/mbox, it was mentioned that
> there was no support for dealing with attachments (and I somehow also
> assumed that dealing with non-US character sets and/or UNICODE, but I
> may be confused).  I need this functionality for my day-to-day use of
> rmail.  "Inflicting" it on users without such support seems a bit
> excessive.

?? The current Rmail, the one which uses Babyl, doesn't support
attachments either.  Or am I misunderstanding what you mean by
``support for dealing with attachments''?

Re: Rmail-mbox branch

[Evil Boris]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Evil Boris <evilborisnet@netscape.net>
>> Date: Sun, 31 Aug 2008 15:15:04 -0400
>> 
>> [...] there was no support for dealing with attachments (and I somehow also
>> assumed that dealing with non-US character sets and/or UNICODE, but I
>> may be confused). [...]
>
> ?? The current Rmail, the one which uses Babyl, doesn't support
> attachments either.  Or am I misunderstanding what you mean by
> ``support for dealing with attachments''?

Now that I sent the msg, I am a bit confused by what I wrote myself. :-)

I am aware that rmail by itself does not deal with attachments (but
apparently forgot about it while writing the above).  I most recently
have been using etach, though I am not completely happy with it.  I have
tried other things in the past (rmime? rmail-mime?), but I do not recall
now.  [I also recall having to save msgs to a file and manually run them
through "metamail -w", which I still do in desperate situations.]

I guess in absense of such support (and hoping etach or equivalent will
magically support mbox format :-), my main concern would be handling of
different character sets (Unicode, ISO-8859-x, KOI8-R, are most popular)
in plain text msgs, so that I can see the text without jumping through
hoops.

--Boris

PS. Reading a multipart/alternative email with a text/plain component
encoded in quoted-printable [in a non-latin-based character set] or
base64 is currently a pain, as I have to detach and then play around
with decoding the character set in the raw RMAIL file...  but this is a
separate story...

Re: Rmail-mbox branch

[Eli Zaretskii]

> From: Evil Boris <evilborisnet@netscape.net>
> Date: Mon, 01 Sep 2008 07:46:23 -0400
> 
> I guess in absense of such support (and hoping etach or equivalent will
> magically support mbox format :-), my main concern would be handling of
> different character sets (Unicode, ISO-8859-x, KOI8-R, are most popular)
> in plain text msgs, so that I can see the text without jumping through
> hoops.

I don't see why the mbox branch would support this any worse than the
trunk.  The decoding stuff is pretty much stable and well understood,
after so many releases.

> PS. Reading a multipart/alternative email with a text/plain component
> encoded in quoted-printable [in a non-latin-based character set] or
> base64 is currently a pain, as I have to detach and then play around
> with decoding the character set in the raw RMAIL file...

??? For me, it's as easy as

  . Type 'e' to make the message editable

  . Make region around the attachment

    . For base64-encoded attachments:

        M-x base64-decode-region RET

    . For quoted-printable: 

        M-: (mail-unquote-printable-region (mark) (point) nil nil t) RET

  . C-c (to exit rmail-edit)

  . M-x rmail-redecode-body RET CHARSET-NAME RET

where CHARSET-NAME is whatever appears in the charset= header of the
attachment, if different from US-ASCII.

(Of course, the above are just the primitives; wrapping them with a
single command is left as an exercise for the interested readers.)

Re: Rmail-mbox branch

[Stefan Monnier]

>> I guess in absense of such support (and hoping etach or equivalent will
>> magically support mbox format :-), my main concern would be handling of
>> different character sets (Unicode, ISO-8859-x, KOI8-R, are most popular)
>> in plain text msgs, so that I can see the text without jumping through
>> hoops.
> I don't see why the mbox branch would support this any worse than the
> trunk.  The decoding stuff is pretty much stable and well understood,
> after so many releases.

IIIUC the differnece is that BABYL is encoded using emacs-mule, whereas
mbox needs to stick to the standard rfc2822+MIME format.  So BABYL can
be decoded in place once and forall, whereas mbox needs to be decoded
upon display, more or less.


        Stefan

Re: Rmail-mbox branch

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Evil Boris <evilborisnet@netscape.net>,  emacs-devel@gnu.org
> Date: Mon, 01 Sep 2008 15:57:50 -0400
> 
> >> I guess in absense of such support (and hoping etach or equivalent will
> >> magically support mbox format :-), my main concern would be handling of
> >> different character sets (Unicode, ISO-8859-x, KOI8-R, are most popular)
> >> in plain text msgs, so that I can see the text without jumping through
> >> hoops.
> > I don't see why the mbox branch would support this any worse than the
> > trunk.  The decoding stuff is pretty much stable and well understood,
> > after so many releases.
> 
> IIIUC the differnece is that BABYL is encoded using emacs-mule, whereas
> mbox needs to stick to the standard rfc2822+MIME format.  So BABYL can
> be decoded in place once and forall, whereas mbox needs to be decoded
> upon display, more or less.

Yes, but that's the same decoding we do today, and the fact mbox
should do it more than once doesn't make it less correct.

Re: Rmail-mbox branch

[Evil Boris]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Evil Boris <evilborisnet@netscape.net>
>> Date: Mon, 01 Sep 2008 07:46:23 -0400
>> 
> I don't see why the mbox branch would support this any worse than the
> trunk.  The decoding stuff is pretty much stable and well understood,
> after so many releases.

I was trying to dig up where I found something that made me suspect the
abitility of Rmail-mbox to deal with non-ASCII, and after much digging
through archives found the following:

>From: Henrik Enberg <henrik.enberg <at> telia.com>
>Subject: Re: status of the rmail-mbox branch?
>Newsgroups: gmane.emacs.devel
>Date: 2007-03-06 14:01:27 GMT (1 year, 25 weeks, 5 days, 3 hours and 36 minutes ago)
>
> Francesco Potorti` <pot <at> gnu.org> writes:
> > What is the current status of the rmail-mbox branch?
> > 
> > Does it have the same functionalities as the regular rmail?
>
> Not entirely.  It is certainly possible to use it to read email, but is
> has some deficencies in the area of coding-systems.  It currently
> doesn't do any decoding from raw-text at all.  The main problem is that
> if we want interoperability with other tools (which I guess is the main
> reason to switch to the mbox format), we can't really encode the
> mailboxes with emacs-mule anymore as is done with BABYL boxes.

It may not be relevant now.

>> PS. Reading a multipart/alternative email with a text/plain component
>> encoded in quoted-printable [in a non-latin-based character set] or
>> base64 is currently a pain, as I have to detach and then play around
>> with decoding the character set in the raw RMAIL file...
>
> ??? For me, it's as easy as
>
>   . Type 'e' to make the message editable
>
>   . Make region around the attachment
>
>     . For base64-encoded attachments:
>
>         M-x base64-decode-region RET
>
>     . For quoted-printable: 
>
>         M-: (mail-unquote-printable-region (mark) (point) nil nil t) RET
>
>   . C-c (to exit rmail-edit)
>
>   . M-x rmail-redecode-body RET CHARSET-NAME RET
>
> where CHARSET-NAME is whatever appears in the charset= header of the
> attachment, if different from US-ASCII.
>
> (Of course, the above are just the primitives; wrapping them with a
> single command is left as an exercise for the interested readers.)

I have been doing this on and off (mostly by hand) and have occasionally
gotten completely incomprehensible gibberish, and a time or two an RMAIL
file that RMAIL would not recognize.  Sorry, no examples lying around
[on a slightly different topic, my RMAIL file is filled with non-ASCII
msgs that used to be perfectly legible and now do not display properly.
I guess this is what happens when one uses cutting edge CVS version, and
cannot affort to do a full "make bootstrap" on every update...
separate story.]

But there is a more general problem with this:  I do not like losing
data or making irreversible changes.  If the said msg contained another
alternative (say, HTML) that I later would discover contained some
information I needed, I am not sure how easy it would be to get it
back.  Maybe I am just being paranoid.

--Boris

Re: Rmail-mbox branch

[Evil Boris]

Evil Boris <evilborisnet@netscape.net> writes:

> But there is a more general problem with this:  I do not like losing
> data or making irreversible changes.  If the said msg contained another
> alternative (say, HTML) that I later would discover contained some
> information I needed, I am not sure how easy it would be to get it
> back.  Maybe I am just being paranoid.

Forgot an important point (of course): After such a manipulation
[undoing quoted-printable- or base64-encoded msg and hand-decoding it],
is it clear that the file is a legitimate mbox file, usable by other
applications?  Or is this not one of the design goals of Rmail-mbox?

--Boris

Re: Rmail-mbox branch

[Eli Zaretskii]

> From: Evil Boris <evilborisnet@netscape.net>
> Date: Mon, 01 Sep 2008 22:38:52 -0400
> 
> Forgot an important point (of course): After such a manipulation
> [undoing quoted-printable- or base64-encoded msg and hand-decoding it],
> is it clear that the file is a legitimate mbox file, usable by other
> applications?

Current Rmail doesn't keep the file in mbox format, but in Babyl.
Decoding as I suggested leaves a valid Babyl file, yes.

Pmail will hold the original mbox (undecoded) format, so such decoding
will not affect it in any way.  The decoded text lives only as long as
the message is displayed.

Re: Rmail-mbox branch

[Evil Boris]

Eli Zaretskii <eliz@gnu.org> writes:

> Current Rmail doesn't keep the file in mbox format, but in Babyl.
> Decoding as I suggested leaves a valid Babyl file, yes.

I guess one can imagine a part whose decoding would introduce Babyl msg
separators, but I digress.

> Pmail will hold the original mbox (undecoded) format, so such decoding
> will not affect it in any way.  The decoded text lives only as long as
> the message is displayed.

I did not realize that a decision was made to separate the presentation
from the mbox format, in Rmail/mbox.  I recall a discussion about this
on emacs-devel, but did not remember the outcome.

This makes at least that objection go away.  Thanks.

--Boris

Re: Rmail-mbox branch

[Paul Michael Reilly]

Evil Boris wrote:
> Eli Zaretskii <eliz@gnu.org> writes:
> 
>> Current Rmail doesn't keep the file in mbox format, but in Babyl.
>> Decoding as I suggested leaves a valid Babyl file, yes.
> 
> I guess one can imagine a part whose decoding would introduce Babyl msg
> separators, but I digress.
> 
>> Pmail will hold the original mbox (undecoded) format, so such decoding
>> will not affect it in any way.  The decoded text lives only as long as
>> the message is displayed.
> 
> I did not realize that a decision was made to separate the presentation
> from the mbox format, in Rmail/mbox.  I recall a discussion about this
> on emacs-devel, but did not remember the outcome.

Me either. :-) But it is becoming more and more clear that there is no
other reasonable way to do Rmail/mbox and maintain the PMAIL file in
mbox format.

> This makes at least that objection go away.  Thanks.

And a few other problems too, like invisible headers becoming visible
when you least expect it (a current bug).

-pmr

Re: Rmail-mbox branch

[Richard M. Stallman]

    Forgot an important point (of course): After such a manipulation
    [undoing quoted-printable- or base64-encoded msg and hand-decoding it],
    is it clear that the file is a legitimate mbox file, usable by other
    applications?

I am not sure.  If it isn't valid as an mbox file, perhaps
it is true that Rmail/mbox needs to do this decoding
each time it displays the message, rather than just once
(as some have claimed before).

Stefan, can you tell Paul Reilly about the new features you told me
ought to make this feasible?

If you edit such a message, how can Rmail/mbox save the edit?  Perhaps
it needs to re-enode the changed message body in the same encoding
that the message used.

Re: Rmail-mbox branch

[Stephen J. Turnbull]

Richard M. Stallman writes:
 >     Forgot an important point (of course): After such a manipulation
 >     [undoing quoted-printable- or base64-encoded msg and hand-decoding it],
 >     is it clear that the file is a legitimate mbox file,

*chortle* mbox is a bastard format.
http://www.jwz.org/doc/content-length.html is the canonical reference
here.

 >     usable by other applications?

Usable, no.  Applications that use mbox will expect the headers to be
100% ASCII; non-ASCII content must be represented as MIME-words.

MIME media-type text bodies may use Content-Transfer-Encoding: 8bit,
which would allow you to save the predecoded text body.  But who
cares?  Nobody sends multimegabyte text/plain messages; decoding per
display is not an efficiency problem.

MIME media-type multipart bodies need to be left as is, in case the
user prefers an alternative decoding in other applications.

Other MIME media-types can't be saved in decoded format, since that is
going to be an Emacs internal data-type (consider image/png).

 > I am not sure.  If it isn't valid as an mbox file, perhaps
 > it is true that Rmail/mbox needs to do this decoding
 > each time it displays the message, rather than just once
 > (as some have claimed before).

s/\(It\|perhaps\) //g

 > If you edit such a message, how can Rmail/mbox save the edit?  Perhaps
 > it needs to re-enode the changed message body in the same encoding
 > that the message used.

Yes.

Re: Rmail-mbox branch

[Richard M. Stallman]

    Other MIME media-types can't be saved in decoded format, since that is
    going to be an Emacs internal data-type (consider image/png).

I don't follow you.

     > I am not sure.  If it isn't valid as an mbox file, perhaps
     > it is true that Rmail/mbox needs to do this decoding
     > each time it displays the message, rather than just once
     > (as some have claimed before).

    s/\(It\|perhaps\) //g

I don't follow you.

Re: Rmail-mbox branch

[Stephen J. Turnbull]

Richard M. Stallman writes:
 > Stephen Turnbull wrote:

 >     Other MIME media-types can't be saved in decoded format, since
 >     that is going to be an Emacs internal data-type (consider
 >     image/png).
 > 
 > I don't follow you.

Media types other than "text" are attachments.  They can be displayed
by Emacs in many cases (eg, images, some XML files).  However, the
decoded format generally has no meaning as mbox contents; it's an
internal Emacs object with no defined serialization into ASCII.

Of course Rmail can simply ignore those attachments, and leave them
as-is in the mbox.  But Rmail will still need to know about MIME
multipart structures to edit any message containing them safely.
Similarly for any message containing non-ASCII text (at minimum it
would need to manipulate Content-Type headers).

 >      > I am not sure.  If it isn't valid as an mbox file, perhaps
 >      > it is true that Rmail/mbox needs to do this decoding
 >      > each time it displays the message, rather than just once
 >      > (as some have claimed before).

The kind of thing that Emacs will display is typically not going to be
a valid RFC 2822 message, and an mbox is simply a sequence of RFC 2822
messages (with empty line separators between messages, and a From line
prepended to each message, and possibly other minor variations
according to which MTA is writing, or which MUA is reading, the file).
Therefore, it is true that Rmail/mbox needs to be prepared to do
decoding each time it displays a message.

The alternative would be to simply declare that Rmail/mbox will *only*
handle multipart and text/plain media, and ignore all the rest.  But
that seems a shame when Emacs is quite capable of handling a wide
variety of media types, including text/html, text/rich-text, image,
audio, and even video (at least with the help of external players).

Re: Rmail-mbox branch

[Richard M. Stallman]

    The alternative would be to simply declare that Rmail/mbox will *only*
    handle multipart and text/plain media, and ignore all the rest.  But
    that seems a shame when Emacs is quite capable of handling a wide
    variety of media types, including text/html, text/rich-text, image,
    audio, and even video (at least with the help of external players).

I am still lost.  You seem to allude to some difference between
text/plain and text/html which I am not aware of.

Re: Rmail-mbox branch

[Stephen J. Turnbull]

Richard M. Stallman writes:
 >     The alternative would be to simply declare that Rmail/mbox will *only*
 >     handle multipart and text/plain media, and ignore all the rest.  But
 >     that seems a shame when Emacs is quite capable of handling a wide
 >     variety of media types, including text/html, text/rich-text, image,
 >     audio, and even video (at least with the help of external players).
 > 
 > I am still lost.  You seem to allude to some difference between
 > text/plain and text/html which I am not aware of.

It's the same difference as between text/plain and any of the other
media types mentioned: you will cannot preserve all the information in
a text/html part while saving it in mbox format.

Rmail/mbox must be prepared to decode a message each time it is
presented.

Re: Rmail-mbox branch

[Paul Michael Reilly]

Stephen J. Turnbull wrote:
> Richard M. Stallman writes:
>  >     The alternative would be to simply declare that Rmail/mbox will *only*
>  >     handle multipart and text/plain media, and ignore all the rest.  But
>  >     that seems a shame when Emacs is quite capable of handling a wide
>  >     variety of media types, including text/html, text/rich-text, image,
>  >     audio, and even video (at least with the help of external players).
>  > 
>  > I am still lost.  You seem to allude to some difference between
>  > text/plain and text/html which I am not aware of.
> 
> It's the same difference as between text/plain and any of the other
> media types mentioned: you will cannot preserve all the information in
> a text/html part while saving it in mbox format.
> 
> Rmail/mbox must be prepared to decode a message each time it is
> presented.
> 

I think that we are saying that the Rmail/mbox message presentation
buffer should decode as much of the MIME pieces as it can.  The task
here is to make the Rmail/mime contributions a first class part of
Rmail.  The developer with the initials "as" started that ball rolling
with (p/r)mailmm.el which is apparently based on Alexander Pohoyda's
work.  This is the direction I plan to take. Comments?

-pmr

Re: Rmail-mbox branch

[Stephen J. Turnbull]

Paul Michael Reilly writes:

 > I think that we are saying that the Rmail/mbox message presentation
 > buffer should decode as much of the MIME pieces as it can.  The task
 > here is to make the Rmail/mime contributions a first class part of
 > Rmail.  The developer with the initials "as" started that ball rolling
 > with (p/r)mailmm.el which is apparently based on Alexander Pohoyda's
 > work.  This is the direction I plan to take. Comments?

I'm not familiar with that file.  However, there are several existing
MUAs with relevant features.  Gnus has been mentioned, VM (but it's
unlikely you'll be able to get Kyle to sign papers), MEW, Wanderlust
are all quite MIME-capable.  You might be able to borrow code,
although as in the case of Gnus you may find that the assumptions
about presentation format is sufficiently different to make that
difficult.

Besides those full-fledged MUAs, tm (tiny MIME) and SEMI are generic
MIME processing packages with add-on glue for Gnus, VM, RMail, and
MH-E (IIRC) that were abandoned in favor of direct support by VM and
Gnus, at least.

VM's internals are poorly documented, too.

Re: Rmail-mbox branch

[Stefan Monnier]

>> >     The alternative would be to simply declare that Rmail/mbox will *only*
>> >     handle multipart and text/plain media, and ignore all the rest.  But
>> >     that seems a shame when Emacs is quite capable of handling a wide
>> >     variety of media types, including text/html, text/rich-text, image,
>> >     audio, and even video (at least with the help of external players).
>> >  > I am still lost.  You seem to allude to some difference between
>> > text/plain and text/html which I am not aware of.
>> 
>> It's the same difference as between text/plain and any of the other
>> media types mentioned: you will cannot preserve all the information in
>> a text/html part while saving it in mbox format.
>> 
>> Rmail/mbox must be prepared to decode a message each time it is
>> presented.

> I think that we are saying that the Rmail/mbox message presentation
> buffer should decode as much of the MIME pieces as it can.  The task
> here is to make the Rmail/mime contributions a first class part of
> Rmail.  The developer with the initials "as" started that ball rolling
> with (p/r)mailmm.el which is apparently based on Alexander Pohoyda's
> work.  This is the direction I plan to take. Comments?

I think that Pmail's MIME support should first limit itself to the small
subset supported by Rmail.  Afterwards, it'd probably be good to try and
make it use Gnus's code (which IIUC is also used y MH-E, so you may
want to talk to the MH-E guy(s) to figure out how that worked).

BTW, as for how to handle the issue of "the current buffer is the mbox
file buffer but I want to display something else in it", you may want to
use the new buffer-swap-text primitive (currently only used in
tar-mode where the same problem appears).


        Stefan

Re: Rmail-mbox branch

[Paul Michael Reilly]

Stefan Monnier wrote:
>>>>     The alternative would be to simply declare that Rmail/mbox will *only*
>>>>     handle multipart and text/plain media, and ignore all the rest.  But
>>>>     that seems a shame when Emacs is quite capable of handling a wide
>>>>     variety of media types, including text/html, text/rich-text, image,
>>>>     audio, and even video (at least with the help of external players).
>>>>  > I am still lost.  You seem to allude to some difference between
>>>> text/plain and text/html which I am not aware of.
>>> It's the same difference as between text/plain and any of the other
>>> media types mentioned: you will cannot preserve all the information in
>>> a text/html part while saving it in mbox format.
>>>
>>> Rmail/mbox must be prepared to decode a message each time it is
>>> presented.
> 
>> I think that we are saying that the Rmail/mbox message presentation
>> buffer should decode as much of the MIME pieces as it can.  The task
>> here is to make the Rmail/mime contributions a first class part of
>> Rmail.  The developer with the initials "as" started that ball rolling
>> with (p/r)mailmm.el which is apparently based on Alexander Pohoyda's
>> work.  This is the direction I plan to take. Comments?
> 
> I think that Pmail's MIME support should first limit itself to the small
> subset supported by Rmail.  Afterwards, it'd probably be good to try and
> make it use Gnus's code (which IIUC is also used y MH-E, so you may
> want to talk to the MH-E guy(s) to figure out how that worked).
> 
> BTW, as for how to handle the issue of "the current buffer is the mbox
> file buffer but I want to display something else in it", you may want to
> use the new buffer-swap-text primitive (currently only used in
> tar-mode where the same problem appears).

I was wondering how I was going to do this.  Very nice solution.

Thanks,

-pmr

Re: Rmail-mbox branch

[Eli Zaretskii]

> Date: Thu, 04 Sep 2008 03:03:41 -0400
> From: Paul Michael Reilly <pmr@pajato.com>
> Cc: monnier@IRO.UMontreal.CA, emacs-devel@gnu.org, rms@gnu.org,
> 	evilborisnet@netscape.net
> 
> I think that we are saying that the Rmail/mbox message presentation
> buffer should decode as much of the MIME pieces as it can.  The task
> here is to make the Rmail/mime contributions a first class part of
> Rmail.  The developer with the initials "as" started that ball rolling
> with (p/r)mailmm.el which is apparently based on Alexander Pohoyda's
> work.  This is the direction I plan to take. Comments?

My comment is that we should first make Rmail/mbox perform as well as
Rmail/Babyl, in this regard.  This means you need to be able to decode
the body of a single-part message and display it correctly.  As soon
as this part is done, we can recommend Rmail users to switch to
Rmail/mbox.  The multi-part/MIME handling can be improved later.

Re: Rmail-mbox branch

[Paul Michael Reilly]

Eli Zaretskii wrote:
>> Date: Thu, 04 Sep 2008 03:03:41 -0400
>> From: Paul Michael Reilly <pmr@pajato.com>
>> Cc: monnier@IRO.UMontreal.CA, emacs-devel@gnu.org, rms@gnu.org,
>> 	evilborisnet@netscape.net
>>
>> I think that we are saying that the Rmail/mbox message presentation
>> buffer should decode as much of the MIME pieces as it can.  The task
>> here is to make the Rmail/mime contributions a first class part of
>> Rmail.  The developer with the initials "as" started that ball rolling
>> with (p/r)mailmm.el which is apparently based on Alexander Pohoyda's
>> work.  This is the direction I plan to take. Comments?
> 
> My comment is that we should first make Rmail/mbox perform as well as
> Rmail/Babyl, in this regard.  This means you need to be able to decode
> the body of a single-part message and display it correctly.  As soon
> as this part is done, we can recommend Rmail users to switch to
> Rmail/mbox.  The multi-part/MIME handling can be improved later.
> 

Sounds unanimous.

-pmr

Re: Rmail-mbox branch

[Richard M. Stallman]

    It's the same difference as between text/plain and any of the other
    media types mentioned: you will cannot preserve all the information in
    a text/html part while saving it in mbox format.

Why not?

Re: Rmail-mbox branch

[Stephen J. Turnbull]

Richard M. Stallman writes:

 >     It's the same difference as between text/plain and any of the other
 >     media types mentioned: you will cannot preserve all the information in
 >     a text/html part while saving it in mbox format.
 > 
 > Why not?

I have no clue what you could have in mind that you would ask that
question.

Re: Rmail-mbox branch

[Richard M. Stallman]

     >     It's the same difference as between text/plain and any of the other
     >     media types mentioned: you will cannot preserve all the information in
     >     a text/html part while saving it in mbox format.
     > 
     > Why not?

    I have no clue what you could have in mind that you would ask that
    question.

I simply do not see why that claim would be true.
I thought you might provide an argument to support it.

Re: Rmail-mbox branch

[Stephen J. Turnbull]

Richard M. Stallman writes:
 >     >     It's the same difference as between text/plain and any of
 >     >     the other media types mentioned: you will cannot preserve
 >     >     all the information in a text/html part while saving it
 >     >     in mbox format.
 >     > 
 >     > Why not?
 > 
 >     I have no clue what you could have in mind that you would ask
 >     that question.
 > 
 > I simply do not see why that claim would be true.
 > I thought you might provide an argument to support it.

Let me try again, this time bluntly.  I cannot determine from your
disyllabic question, nor from the tenfold increased reiteration above,
what claim you think I'm making.  Specifically, I don't know whether

a. you are unaware of the context of your question from my point of
   view and think I'm making a much broader claim than I intend, or

b. you are nearly completely ignorant of the relevant aspects of
   how email works and don't understand the claim, which is obviously
   true to those who work with MIME email, or perhaps

c. both, or even

d. something else.

Now, addressing confusion (a), paraphrased to compress thread context,
I understand the question as "can we achieve interoperability with
other MUAs, yet avoid (MIME-)decoding a message every time we present
it?"  The answer is "only if you're willing to destroy information"
for all MIME types except 'text/plain; charset=us-ascii'.

Addressing lack of knowledge (b), the multimedia functionality common
to Internet MUAs expects to be handed messages in a complex form
defined by RFCs 2821, 2822, 2045-2049, 2231, and several others of
subsidiary interest.  The first two RFCs establish the requirement
that mail shall be transmitted as plain ASCII text.[1]  The rest
provide various conventions for how to serialize quite arbitrary
objects, from non-ASCII text to video clips to binary blobs, as plain
ASCII text.  They are the *only* conventions commonly adopted by MUA
authors as defining non-ASCII-text email.[2]

Therefore, it simply is not possible to save Emacs's presentation of
HTML in *any form other than an HTML MIME body* that can be *expected*
to be operated on usefully by other MUAs.  Except if that form strips
out the HTML-specific formatting information, links, multimedia, etc.,
and turns it into plain ASCII text, which all MUAs must be able to
handle.  Thus, to preserve all information it needs to be left as an
HTML MIME body in the mbox file, and therefore it will need to be
decoded every time it is presented.  It should be clear that the same
rationale applies to any emailed object which is not MIME 'text/plain;
charset=us-ascii'.[3]

I hope that makes it clear that the constraints are on Rmail/mbox are
that you can get the first two but not the third of these desiderata:

1) interoperability with other MUAs (including a future Rmail with
   real MIME capability);

2) preservation of all information in each message in the folder; and

3) saving the decoded form of each message in the folder in lieu of
   the MIME-encoded message.

I advocate abandoning the last, as it's not a perceptible efficiency
gain in the context of Rmail.  Stefan and Paul have both acknowledged
that the runtime cost of decoding is negligible.  It's also safe and
easy to provide a per-message cache in the form of RFC 2822 extension
fields in the message header, with the only constraint being to
respect the ASCII-only restriction of RFC 2822 for header fields.


Footnotes: 
[1]  Indeed there are non-ASCII-text "content transfer encodings", but
they are not considered as reliable, and in my experience they are not
as reliable as the ASCII-based ones.  In general, a private agreement
between mail agents is required for use of the non-ASCII-based ones,
as conformance is not universal.

[2]  With the likely exception of uuencoded file attachments, which I
believe even Microsoft Outlook still recognizes.

[3]  For practical purposes we can assume that message bodies which are
'text/plain; charset=XYZ' for XYZ=iso-8859-1, XYZ=iso-8859-15, and
XYZ=utf-8 will generally "just work", too.  But it's easy to imagine
situations where a MIME-conforming MUA might render such incorrectly
without a MIME Content-Type header to guide it, such as a Latin-2-
using locale.

Re: Rmail-mbox branch

[Francesco Potorti`]

>[3]  For practical purposes we can assume that message bodies which are
>'text/plain; charset=XYZ' for XYZ=iso-8859-1, XYZ=iso-8859-15, and
>XYZ=utf-8 will generally "just work", too.  But it's easy to imagine
>situations where a MIME-conforming MUA might render such incorrectly
>without a MIME Content-Type header to guide it, such as a Latin-2-
>using locale.

One thing that I find useful is searching mbox files using standard Unix
text tools.  This is not possible for base-64 coded messages.  However,
I can edit an mbox file, convert its text from base-64 and edit the MIME
header appropriately, thus still obtaining an RFC-compliant message that
is searchable and that can be displayed without further decoding (it is
smaller, too).  I do this routinely for the three charsets you mentioned
above, but it should work for any charset.  I suggest that this
conversion be made automatically whenever possible and sensible.

Re: Rmail-mbox branch

[Stephen J. Turnbull]

Francesco Potorti` writes:

 > >[3]  For practical purposes we can assume that message bodies which are
 > >'text/plain; charset=XYZ' for XYZ=iso-8859-1, XYZ=iso-8859-15, and
 > >XYZ=utf-8 will generally "just work", too.  But it's easy to imagine
 > >situations where a MIME-conforming MUA might render such incorrectly
 > >without a MIME Content-Type header to guide it, such as a Latin-2-
 > >using locale.
 > 
 > One thing that I find useful is searching mbox files using standard Unix
 > text tools.  This is not possible for base-64 coded messages.  However,
 > I can edit an mbox file, convert its text from base-64 and edit the MIME
 > header appropriately, thus still obtaining an RFC-compliant message that
 > is searchable and that can be displayed without further decoding (it is
 > smaller, too).  I do this routinely for the three charsets you mentioned
 > above, but it should work for any charset.  I suggest that this
 > conversion be made automatically whenever possible and sensible.

This is always trivially possible in Emacs.[1]  The key, as you
mention, is fixing the Content-Transfer-Encoding field to '8bit' after
decoding, and (perhaps) recoding from a legacy charset to UTF-8, in
which case you'll need to fix the Content-Type field's 'charset'
parameter, too.

However, whether it is "sensible" depends on the user's environment,
including what other MUAs and text-processing tools you have
available.  I recommend rather making this a folder-specific user
setting, defaulting to 'ask.  You yourself could set it to 'always
since you have favorable experience with it.  Or perhaps the values
should be something like 'never, 'ask, 'always-content-type (where
this uses the value specified in the 'charset' parameter of the
Content-Type field), and 'always-utf8.

Footnotes: 
[1]  Of course you have to know how to use Mule, which is not yet
fully diffused into the community.  Once you've got that, though, it's
trivial.  Borrow the BASE64 and QP decoders from Gnus, and use
de/en-code-coding-region to fix up the MIME charset.

Re: Rmail-mbox branch

[Stefan Monnier]

> This is always trivially possible in Emacs.[1]  The key, as you
> mention, is fixing the Content-Transfer-Encoding field to '8bit' after
> decoding, and (perhaps) recoding from a legacy charset to UTF-8, in
> which case you'll need to fix the Content-Type field's 'charset'
> parameter, too.

Sadly this cannot be done for headers,


        Stefan

Re: Rmail-mbox branch

[Stephen J. Turnbull]

Stefan Monnier writes:

 > Sadly [saving the decoded version] cannot be done for headers,

Not if you wish to maintain portability to strictly conforming MUAs.
It could be a user option, though.  I suspect that many MUAs (eg mutt)
can be taught to do something graceful with high-bit-set characters in
the headers (like, "cat them to the terminal" ;-).

If interoperability with grep is more important than interoperability
with Outhouse Excess, why not let the user do it?

Also, it would be mildly tedious but certainly easy enough to let your
favorite MUA (eg, Gnus, or nmh from the command line) give you a
summary list from the (undecoded) headers, grep that, and then grep
the bodies.

Don't get me wrong.  Interoperability with other MUAs is extremely
important.  To be taken seriously as a well-behaved MUA, IMO Rmail TNG
*must* be capable of saving messages in MIME form (which is trivially
possible for well-formed messages off the wire, just leave the mbox
alone) and decode them on every presentation.  But if some user
doesn't care about RFC conformance, we need not strangle her with it!

Re: Rmail-mbox branch

[Eli Zaretskii]

> From: "Stephen J. Turnbull" <stephen@xemacs.org>
> Date: Mon, 08 Sep 2008 18:53:41 +0900
> Cc: evilborisnet@netscape.net, monnier@IRO.UMontreal.CA, rms@gnu.org,
> 	emacs-devel@gnu.org
> 
> Borrow the BASE64 and QP decoders from Gnus

No need for this anymore, as  of Emacs 22:  base64-decode-region is a
Lisp primitive and mail-unquote-printable-region is in mail-utils.el.

Re: Rmail-mbox branch

[Richard M. Stallman]

    I understand the question as "can we achieve interoperability with
    other MUAs, yet avoid (MIME-)decoding a message every time we present
    it?"  The answer is "only if you're willing to destroy information"
    for all MIME types except 'text/plain; charset=us-ascii'.

How do you reach that conclusion for text/html?

      The first two RFCs establish the requirement
    that mail shall be transmitted as plain ASCII text.[1]  The rest
    provide various conventions for how to serialize quite arbitrary
    objects, from non-ASCII text to video clips to binary blobs, as plain
    ASCII text.

HTML is made of plain ASCII text.

Re: Rmail-mbox branch

[David De La Harpe Golden]

Richard M. Stallman wrote:

> HTML is made of plain ASCII text.

(1) No it isn't, at least not necessarily.
http://www.w3.org/TR/REC-html40-971218/charset.html#encodings

While HTML numeric/entity refs _allow_ most all >7-bit chars to
make it through an ascii channel, IF someone uses them, AFAIK their use
is _not_ mandated, you can e.g. just write your html in utf-8 and use ☺
characters directly.  i.e. they're a facility a bit like those C
trigraphs (entity refs have a range of other uses in SGML/XML land).


(2) text/html can have a declared charset.  It _may_ be US-ASCII,
but it isn't necessarily the case that it is.
http://www.ietf.org/rfc/rfc2854.txt

"Because of the availability within HTML itself for using character
entity references, documents that use a wide repertoire of characters
may still be represented using the US-ASCII charset and transported
without encoding.  However, transport of text/html using a charset
other than US-ASCII may require base64 or quoted-printable encoding
for 7-bit channels."

When I wrote a mostly-ascii Unicode HTML mail with a capable mailer (not
that I make a habit of HTML mail, bleurgh), it got sent as:

Content-Type: text/html;
  charset="utf-8"
Content-Transfer-Encoding: quoted-printable


At some point, after pasting in a big bunch of unicode squiggles
(presumably some heuristic on proportion of non-7-bit-ascii chars or
encoded length) , it decided to switch to:

Content-Type: text/html;
  charset="utf-8"
Content-Transfer-Encoding: base64

Re: Rmail-mbox branch

[Stephen J. Turnbull]

Richard M. Stallman writes:

 > How do you reach that conclusion for text/html?

I explained that already.

 > HTML is made of plain ASCII text.

You're just being argumentative; that's false on the face of it
("<html>Voilà!</html>"), and beside the point.  The *information* in
an HTML document is not all textual, and if the MIME media type is
text/html, users have a right to expect that that non-textual
information will be used in presentation and preserved in
transmission.

If you have a suggestion for improving Rmail which depends on
text/html being a format that doesn't require MIME encapsulation for
interoperability, please present it, and we can discuss its
feasibility.  Otherwise, we're just wasting each others' time.

Re: Rmail-mbox branch

[Richard M. Stallman]

     > How do you reach that conclusion for text/html?

    I explained that already.

You didn't really try to explain, only to vent contempt.
Someone else did really explain, which I appreciate.

Re: Rmail-mbox branch

[Stephen J. Turnbull]

Richard M. Stallman writes:
 >      > How do you reach that conclusion for text/html?
 > 
 >     I explained that already.
 > 
 > You didn't really try to explain, only to vent contempt.

Wrong on both counts.  You seem to think I was addressing my posts
only to you, but in fact I *succeeded* in explaining (as evidenced by
offline thanks I have received) to others.  The fact that that did not
succeed for you, and further attempts to explain for your personal
benefit met with no help from you, was frustrating.  I admit that.

Especially frustrating since in fact *you* are the most prominent
beneficiary of a better Rmail; most of the world has long since moved
on to more capable MUAs, and has no need of improvements in Rmail.

I do not appreciate your attempt to blame me for your lack of effort
in communicating.  You have your reasons for terseness, I'm aware, but
that is no excuse for this rudeness.

 > Someone else did really explain, which I appreciate.

If you are referring to David de la Harpe's explanation of why
text/html is not "plain ASCII", you are still missing the main point
(though his post was a complete and useful explanation of the
subsidiary point it addresses).

I do not appreciate you putting me down by calling that a "real
explanation" in this context.

You see, "charset=us-ascii" doesn't really matter here.  As I pointed
out to Francesco, there's no reason at all not to transcode body text
to UTF-8, and as I pointed out to Stefan, for many users it would be
very convenient and would cause no harm to transcode the headers too.
(I suspect that *you* are one user who would greatly benefit from that
feature, which will be easy to implement, and I had you in mind when I
exerted the effort to propose it to Stefan.)

The main point, on the other hand, is that for Rmail to be generally
useful and interoperable these days, it needs to support non-text
objects, at least to the extent of not harming them.  The straight-
forward way to avoid loss of information is simply to store messages
containing such objects in their "straight from the wire" form in an
mbox-format file, and decode the parts that Rmail supports for
presentation every time the message is presented.

Re: Rmail-mbox branch

[Paul Michael Reilly]

Stephen J. Turnbull wrote:
> ... and as I pointed out to Stefan, for many users it would be
> very convenient and would cause no harm to transcode the headers too.

I could not chase down that message and I'm a little confused about
transcoding headers to the extent that I understand them to be ASCII
per rfc2822.  About the only way it would make sense to me to talk
about transcoding headers would be if headers are included (also
encoded) in the content encoding, i.e. a message that is text/html
could have html tags in the headers.  Is this the case?  Or are you 
saying something else?

-pmr

Re: Rmail-mbox branch

[tomas]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Wed, Sep 10, 2008 at 07:43:02AM -0400, Paul Michael Reilly wrote:
> Stephen J. Turnbull wrote:
>> ... and as I pointed out to Stefan, for many users it would be
>> very convenient and would cause no harm to transcode the headers too.
>
> I could not chase down that message and I'm a little confused about
> transcoding headers to the extent that I understand them to be ASCII
> per rfc2822.  About the only way it would make sense to me to talk
> about transcoding headers would be if headers are included (also
> encoded) in the content encoding, i.e. a message that is text/html
> could have html tags in the headers.  Is this the case?  Or are you saying 
> something else?

They may be quoted-printable. AFAIK this is the only allowed way of
transcending US-ASCII in headers (but I don't know very "far").

Regards
- -- tomás
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFIx7woBcgs9XrR2kYRAvt5AJ9yL4DHsK3sqewuoVuvxAK+WUXqOgCdFEhp
SmGl5U9jiHqnvJuvZQFMSXc=
=DMEW
-----END PGP SIGNATURE-----

Re: Rmail-mbox branch

[Stefan Monnier]

>> ... and as I pointed out to Stefan, for many users it would be
>> very convenient and would cause no harm to transcode the headers too.

> I could not chase down that message and I'm a little confused about
> transcoding headers to the extent that I understand them to be ASCII
> per rfc2822.  About the only way it would make sense to me to talk

He was simply suggesting to do away with strict rfc2822 compliance.
After all, it's common for email headers to contain latin-1 text without
encoding it with QP as it should.  Most MTAs will preserve this 8bit
info, and most MUAs will try and do something useful with it (like
assume it's encoded in the locale's favorite encoding).

It'd most likely be controlled by a config var, since it's useful but
somewhat dangerous.


        Stefan

Re: Rmail-mbox branch

[Stephen J. Turnbull]

Paul Michael Reilly writes:

 > Stephen J. Turnbull wrote:
 > > ... and as I pointed out to Stefan, for many users it would be
 > > very convenient and would cause no harm to transcode the headers too.
 > 
 > I could not chase down that message and I'm a little confused about
 > transcoding headers to the extent that I understand them to be ASCII
 > per rfc2822.

Stefan is correct: it will NOT be ASCII per RFC 2822; it will no
longer be proper mbox format.  Many tools (eg, most versions of
Python's email module) will immediately fatal error on it.  *But* it
would be useful to Rmail (which would need very little extra coding to
handle it, I believe) and to plain-text tools such as grep.

For users (Francesco seems to be one, and I bet Richard is, too) who
do not plan to use other MUAs, the extra convenience of this extension
to mbox format would be very useful.

Note that the reverse translation is trivial, too, since the functions
needed to MIME-encode the headers are easily available (probably
already in Rmail?, certainly in Gnus).

Re: Rmail-mbox branch

[David De La Harpe Golden]

Stephen J. Turnbull wrote:

> If you are referring to David de la Harpe's explanation of why

No particular way you'd have to know this in advance, but for future
reference, if you're contracting my (admittedly peculiar) surname,
"David Golden" is preferable. (Though bearing in mind I'm not the
well-known Perl programmer David Golden).

Re: Rmail-mbox branch

[Stefan Monnier]

> MIME media-type text bodies may use Content-Transfer-Encoding: 8bit,
> which would allow you to save the predecoded text body.  But who
> cares?  Nobody sends multimegabyte text/plain messages; decoding per
> display is not an efficiency problem.

Indeed, decoding upon display is not a problem.  The problem that's new
for Pmail is that Rmail allows the user to edit a message, so Pmail will
need to be able to re-encode a displayed message.


        Stefan

Re: Rmail-mbox branch

[Paul Michael Reilly]

Stefan Monnier wrote:
>> MIME media-type text bodies may use Content-Transfer-Encoding: 8bit,
>> which would allow you to save the predecoded text body.  But who
>> cares?  Nobody sends multimegabyte text/plain messages; decoding per
>> display is not an efficiency problem.
> 
> Indeed, decoding upon display is not a problem.  The problem that's new
> for Pmail is that Rmail allows the user to edit a message, so Pmail will
> need to be able to re-encode a displayed message.

Or tell the User that that particular encoding is not handled yet.  But 
I take your point.

-pmr

Re: Rmail-mbox branch

[Kenichi Handa]

In article <jwviqtdz008.fsf-monnier+emacs@gnu.org>, Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

> > MIME media-type text bodies may use Content-Transfer-Encoding: 8bit,
> > which would allow you to save the predecoded text body.  But who
> > cares?  Nobody sends multimegabyte text/plain messages; decoding per
> > display is not an efficiency problem.

> Indeed, decoding upon display is not a problem.  The problem that's new
> for Pmail is that Rmail allows the user to edit a message, so Pmail will
> need to be able to re-encode a displayed message.

Isn't it possible to use some of Gnus' functionality to compose a
MIME-correct mail?

---
Kenichi Handa
handa@ni.aist.go.jp

Re: Rmail-mbox branch

[Stefan Monnier]

>> > MIME media-type text bodies may use Content-Transfer-Encoding: 8bit,
>> > which would allow you to save the predecoded text body.  But who
>> > cares?  Nobody sends multimegabyte text/plain messages; decoding per
>> > display is not an efficiency problem.

>> Indeed, decoding upon display is not a problem.  The problem that's new
>> for Pmail is that Rmail allows the user to edit a message, so Pmail will
>> need to be able to re-encode a displayed message.

> Isn't it possible to use some of Gnus' functionality to compose a
> MIME-correct mail?

Of course, there is, in the sense that several pieces of the puzzle are
already written, but there's still a fair bit of glue to write.  For one
thing, Gnus's mime-composition uses a different input format than Gnus's
mime-display output-format.

Then of course, there's the issue of making sure that the encoding is
done reliably without any loss of information, which probably means that
the decoding itself should be careful to preserve all the information.


        Stefan

Re: Rmail-mbox branch

[Stephen J. Turnbull]

Stefan Monnier writes:

 > > MIME media-type text bodies may use Content-Transfer-Encoding: 8bit,
 > > which would allow you to save the predecoded text body.  But who
 > > cares?  Nobody sends multimegabyte text/plain messages; decoding per
 > > display is not an efficiency problem.
 > 
 > Indeed, decoding upon display is not a problem.  The problem that's new
 > for Pmail is that Rmail allows the user to edit a message, so Pmail will
 > need to be able to re-encode a displayed message.

Yup.

Continuing to maintain Rmail, even with a more-commonly-used mailbox
format, is going to be an ongoing headache IMO.

Re: Rmail-mbox branch

[Paul Michael Reilly]

Stephen J. Turnbull wrote:
> Stefan Monnier writes:
> 
>  > > MIME media-type text bodies may use Content-Transfer-Encoding: 8bit,
>  > > which would allow you to save the predecoded text body.  But who
>  > > cares?  Nobody sends multimegabyte text/plain messages; decoding per
>  > > display is not an efficiency problem.
>  > 
>  > Indeed, decoding upon display is not a problem.  The problem that's new
>  > for Pmail is that Rmail allows the user to edit a message, so Pmail will
>  > need to be able to re-encode a displayed message.
> 
> Yup.
> 
> Continuing to maintain Rmail, even with a more-commonly-used mailbox
> format, is going to be an ongoing headache IMO.

Tell me about it!  My headache is already pounding.  After using 
Rmail/VM/Eudora/Gnus/Evolution/Pine/Netscape Mail/Kmail/Thunderbird and 
probably a few other clients, I always come back to the conclusion that 
Rmail was the most satisfying for me (except for IMAP support, which is 
my long term goal and a whole different topic that will start to get 
addressed when Rmail/mbox has replaced Rmail/babyl).  Your comments have 
been helpful.  Keep 'em coming.  It's aspirin for sure.

-pmr

Re: Rmail-mbox branch

[Eli Zaretskii]

> From: Evil Boris <evilborisnet@netscape.net>
> Date: Mon, 01 Sep 2008 19:46:46 -0400
> 
> But there is a more general problem with this:  I do not like losing
> data or making irreversible changes.  If the said msg contained another
> alternative (say, HTML) that I later would discover contained some
> information I needed, I am not sure how easy it would be to get it
> back.

That's exactly the kind of problems that the mbox branch is supposed
to solve: since the original message is left in the RMAIL file, you
can try decoding it any way any number of times without losing the
original.

Re: Rmail-mbox branch

[Richard M. Stallman]

What Rmail support for attachments do you use?

I have some ideas for how Rmail support for attachments should work,
in regard to UI.  If someone has time and interest to work on it,
please write to me and I will write them down.

Re: Rmail-mbox branch

[Francesco Potorti`]

>What Rmail support for attachments do you use?

It was not me, but I use Rmail together with the very old Rmime, patched
by me to work with current Rmail.  I have also Etach installed, but I
never got down to using it.

By the way, I also use Mailcrypt with Rmail.

Re: Rmail-mbox branch

[Evil Boris]

Francesco Potorti` <pot@gnu.org> writes:

>>What Rmail support for attachments do you use?
>
> It was not me, but I use Rmail together with the very old Rmime, patched
> by me to work with current Rmail.  I have also Etach installed, but I
> never got down to using it.
>
> By the way, I also use Mailcrypt with Rmail.

I use etach.

Re: Rmail-mbox branch

[Paul Michael Reilly]

Chong Yidong wrote:
> Paul Michael Reilly <pmr@pajato.com> writes:
> 
>> The pmail*.el and related files I created are temporary and pretty
>> much only committed to the trunk as a convenience to make Rmail/mbox
>> testable without having to use a branch so I didn't create a ChangeLog
>> for these files.
>>
>> After I work out with the maintainers when to inflict Rmail/mbox on
>> all Users, the ChangeLog will be created.
> 
> Is there any reason not to inflict it *now*?

Only that there will very likely be a raft of messages to the affect
of "What the fuck happened to Rmail..."  My oft stated opinion is that
we should commit the Rmail/mbox code immediately after a release thus
giving us the maximum amount of time to fix Rmail/mbox bugs.
Admittedly Rmail/mbox has been less buggy than I had expected it would
be in the week or so that I've been using it.  Richard has found a
couple of bugs, as have I, one of which was the same bug.  So for me
the answer to your question depends on when the next release of Emacs
is targeted.  If it is imminent, like in weeks or a few months, then I
would advise waiting.  If it won't happen for many months, then now is
as good a time as any.

The only area I know is incomplete is confirming that spam filtering
is up to date.  I think it is but there is a few commits to
Rmail/babyl that I'm not sure are relevant to Rmail/mbox.

Also, there is one suggestion that Richard made that will facilitate
automatic conversion of Babyl formatted file to mbox format.  This one
should probably be completed before we make the switch.

Hope that helps,

-pmr

Re: Rmail-mbox branch

[Chong Yidong]

Paul Michael Reilly <pmr@pajato.com> writes:

>> Is there any reason not to inflict it *now*?
>
> Only that there will very likely be a raft of messages to the affect
> of "What the fuck happened to Rmail..."  My oft stated opinion is that
> we should commit the Rmail/mbox code immediately after a release thus
> giving us the maximum amount of time to fix Rmail/mbox bugs.
> Admittedly Rmail/mbox has been less buggy than I had expected it would
> be in the week or so that I've been using it.  Richard has found a
> couple of bugs, as have I, one of which was the same bug.  So for me
> the answer to your question depends on when the next release of Emacs
> is targeted.  If it is imminent, like in weeks or a few months, then I
> would advise waiting.  If it won't happen for many months, then now is
> as good a time as any.

The Emacs 23 release is several months away.  I would suggest making the
switch now, so that it can receive as much testing as possible.

Could you do that?

Re: Rmail-mbox branch

[Stefan Monnier]

> Only that there will very likely be a raft of messages to the affect
> of "What the fuck happened to Rmail..."  My oft stated opinion is that
> we should commit the Rmail/mbox code immediately after a release thus
> giving us the maximum amount of time to fix Rmail/mbox bugs.
> Admittedly Rmail/mbox has been less buggy than I had expected it would
> be in the week or so that I've been using it.  Richard has found a
> couple of bugs, as have I, one of which was the same bug.  So for me
> the answer to your question depends on when the next release of Emacs
> is targeted.  If it is imminent, like in weeks or a few months, then I
> would advise waiting.  If it won't happen for many months, then now is
> as good a time as any.

> The only area I know is incomplete is confirming that spam filtering
> is up to date.  I think it is but there is a few commits to
> Rmail/babyl that I'm not sure are relevant to Rmail/mbox.

> Also, there is one suggestion that Richard made that will facilitate
> automatic conversion of Babyl formatted file to mbox format.  This one
> should probably be completed before we make the switch.

Installing pmail on the trunk as it is now only makes/made sense on the
assumption that it's temporary and will be moved to rmail before
the release.  "The" release here is 23, since the trunk holds the code for
Emacs-23.
Emacs-23 is in feature freeze but hasn't started pretesting.
You could call it something like `alpha' or `beta' quality.

The sooner you can move Pmail to Rmail the better (it's the one
remaining feature that hasn't yet been installed, so it's an exception
to the feature freeze).

I don't use Rmail myself, so I can't really comment concretely, but as
soon as Pmail's feature-set is equal to (or a superset of) Rmail and
those features are mostly working, we should move Pmail to Rmail.
From what you say, it seems that we're really close now.


        Stefan

Re: Rmail-mbox branch

[Richard M. Stallman]

Pmail has several bugs, which I am reporting to Paul.
I can manage to use it, but it is not ready for release now.
It needs work first.

Re: Rmail-mbox branch

[Eli Zaretskii]

> From: "Richard M. Stallman" <rms@gnu.org>
> Date: Mon, 01 Sep 2008 21:09:52 -0400
> Cc: pmr@pajato.com, cyd@stupidchicken.com, emacs-devel@gnu.org, rgm@gnu.org
> 
> Pmail has several bugs, which I am reporting to Paul.
> I can manage to use it, but it is not ready for release now.

Emacs 23 isn't ready for release, either.  So Pmail will not delay its
release in any way, I think.

Re: Rmail-mbox branch

[Richard M. Stallman]

    Emacs 23 isn't ready for release, either.  So Pmail will not delay its
    release in any way, I think.

I hope that is true, but it's a different question.  Rmail in Emacs 23
does work, and Pmail doesn't work well enough yet.  I can say that
with assurance because I just tried it for a couple of days.

People who want to try Pmail can already do so.
It should not replace Rmail yet.

With luck in a few of weeks it will be much better than it is now.

Re: Rmail-mbox branch

[Eli Zaretskii]

> From: "Richard M. Stallman" <rms@gnu.org>
> CC: monnier@iro.umontreal.ca, pmr@pajato.com, cyd@stupidchicken.com,
> 	emacs-devel@gnu.org, rgm@gnu.org
> Date: Tue, 02 Sep 2008 22:41:18 -0400
> 
> Rmail in Emacs 23 does work, and Pmail doesn't work well enough yet.
> I can say that with assurance because I just tried it for a couple
> of days.

What were the problems?  Can you please publish at least their
summary?

Re: Rmail-mbox branch

[Paul Michael Reilly]

Eli Zaretskii wrote:
>> From: "Richard M. Stallman" <rms@gnu.org>
>> CC: monnier@iro.umontreal.ca, pmr@pajato.com, cyd@stupidchicken.com,
>> 	emacs-devel@gnu.org, rgm@gnu.org
>> Date: Tue, 02 Sep 2008 22:41:18 -0400
>>
>> Rmail in Emacs 23 does work, and Pmail doesn't work well enough yet.
>> I can say that with assurance because I just tried it for a couple
>> of days.
> 
> What were the problems?  Can you please publish at least their
> summary?

You make a persuasive case for using the bug tracking system.  I don't 
suppose someone could create a "category" for lack of a better term to 
keep bug reports on Pmail, or tell me how I should do it if I even can?

In a nutshell, and off the of my head, the problems have been: 
pmail-output will not output to an Rmail/babyl file (not fixed); expunge 
would lose track of which message was the current message (fixed); 
message decoding works in Rmail but not in Pmail (this is the current 
big issue); reply did not correctly hide the X-BABYL... header (fixed); 
  invisible headers become visible in unexpected ways, like when using 
regions operations on a message (not fixed) and various byte compiler 
issues (fixed).  I'm sure that a few have been forgotten already but 
that should give you a feel for the issues.

By all means give it a try.  You should be able to do so and seamlessly 
switch back to Rmail when you find a deal breaker, just make sure you 
preserve your inboxes when using Pmail.

-pmr

Re: Rmail-mbox branch

[Chong Yidong]

Paul Michael Reilly <pmr@pajato.com> writes:

> invisible headers become visible in unexpected ways, like when using
> regions operations on a message (not fixed)

If you want to keep invisible headers hidden on yanking, one way to do
that is to set a buffer-local `buffer-substring-filters'.

Re: Rmail-mbox branch

[Glenn Morris]

Paul Michael Reilly wrote:

> You make a persuasive case for using the bug tracking system.  I don't 
> suppose someone could create a "category" for lack of a better term to 
> keep bug reports on Pmail, or tell me how I should do it if I even can?

Send a bug report as you normally would, except make the first line of
the body:

Package: emacs,pmail

(probably followed by a blank line).

That's it. Packages do not have to be defined to be used.

Other things are possible - see admin/notes/bugtracker.

Re: Rmail-mbox branch

[Richard M. Stallman]

I have reported the problems to Paul.

Re: Rmail-mbox branch

[Paul Michael Reilly]

Paul Michael Reilly wrote:
> Chong Yidong wrote:
...
> Also, there is one suggestion that Richard made that will facilitate
> automatic conversion of Babyl formatted file to mbox format.  This one
> should probably be completed before we make the switch.

It turns out this is already in Pmail.  In order to use Pmail without 
worrying about inflicting any damage to Rmail (RMAIL) do the following:

Copy RMAIL to PMAIL and then invoke M-x pmail with 'pmail-preserve-inbox 
set to t and everything else being equal, i.e. use the same setup for 
Pmail that you would for Rmail.

This will convert your PMAIL file to mbox format automagically and allow 
you to test Pmail and switch back to Rmail at any point without any loss 
of messages.

-pmr

Re: Rmail-mbox branch

[Glenn Morris]

Paul Michael Reilly wrote:

> The pmail*.el and related files I created are temporary and pretty much 
> only committed to the trunk as a convenience to make Rmail/mbox testable 
> without having to use a branch so I didn't create a ChangeLog for these 
> files.
>
> After I work out with the maintainers when to inflict Rmail/mbox on all 
> Users, the ChangeLog will be created.

This all seems rather odd. I assume you took the rmail-mbox-branch
(With some extra changes? It's unclear...), extracted the files that
actually differed from the trunk, and installed them into the trunk as
pmail*. Fair enough, this might indeed make it easier for people to
try out.

I don't see why you wouldn't just take the existing ChangeLog from
that branch and install as ChangeLog.pmail, rather than instead
creating two odd directories with CVS info that I doubt anyone will
ever read...

But I don't use rmail, so my opinion's irrelevant. :)

Re: Rmail-mbox branch

[Paul Michael Reilly]

Glenn Morris wrote:
> Paul Michael Reilly wrote:
> 
>> The pmail*.el and related files I created are temporary and pretty much 
>> only committed to the trunk as a convenience to make Rmail/mbox testable 
>> without having to use a branch so I didn't create a ChangeLog for these 
>> files.
>>
>> After I work out with the maintainers when to inflict Rmail/mbox on all 
>> Users, the ChangeLog will be created.
> 
> This all seems rather odd. I assume you took the rmail-mbox-branch
> (With some extra changes? It's unclear...), extracted the files that
> actually differed from the trunk, and installed them into the trunk as
> pmail*. Fair enough, this might indeed make it easier for people to
> try out.

That certainly was the goal.  Richard made a suggestion which I will 
implement shortly that automagically check for an existing Rmail/babyl 
formatted PMAIL file which it will then convert to mbox format.  This 
will make trying out Pmail event easier.

> I don't see why you wouldn't just take the existing ChangeLog from
> that branch and install as ChangeLog.pmail, rather than instead
> creating two odd directories with CVS info that I doubt anyone will
> ever read...

Good idea. Done.  I committed the info directories just because 
Rmail/mbox has a history of people working on it and then disappearing 
(including myself) for years at a time.  So I wanted to have some 
mechanism, albeit temporary, to capture what was done and not done to 
bring Rmail/mbox up to date.

Thanks,

-pmr

Re: Rmail-mbox branch

[Chong Yidong]

Paul Michael Reilly <pmr@pajato.com> writes:

> For now, I just committed a bunch of "pmail*.el" files in lisp/mail
> that will allow developers to test Rmail/mbox using M-x pmail.
>
> My long term goal is to replace my use of Thunderbird with Rmail/mbox
> as the first step.  The next step is to test Rmail/mbox using imap via
> Gnu Mailutils (for the movemail/imap support) so my Rmail/mbox testing
> will be geared very much to my day-day use of email, rather than
> specific test cases.  A good thing, I think.

Thanks for working on this.

It may not be feasible to get imap working with rmail in time for Emacs
23.1; better to finish up the mbox support, to give enough time for
testing/bugfixes.  How long do you want to wait before replacing the old
rmail?  Say, a couple of weeks?

I would suggest making the switch sooner rather than later, so that it
gets as much testing as possible.

Re: Rmail-mbox branch

[Paul Michael Reilly]

Chong Yidong wrote:
> Paul Michael Reilly <pmr@pajato.com> writes:
> 
>> For now, I just committed a bunch of "pmail*.el" files in lisp/mail
>> that will allow developers to test Rmail/mbox using M-x pmail.
>>
>> My long term goal is to replace my use of Thunderbird with Rmail/mbox
>> as the first step.  The next step is to test Rmail/mbox using imap via
>> Gnu Mailutils (for the movemail/imap support) so my Rmail/mbox testing
>> will be geared very much to my day-day use of email, rather than
>> specific test cases.  A good thing, I think.
> 
> Thanks for working on this.
> 
> It may not be feasible to get imap working with rmail in time for Emacs
> 23.1; better to finish up the mbox support, to give enough time for
> testing/bugfixes.  How long do you want to wait before replacing the old
> rmail?  Say, a couple of weeks?

Here's my conundrum.  I'm married to IMAP, for better or worse.  I have 
my mail on a very reliable RAID5, backed up, fast server.  One under my 
control.  So my day-day use of email revolves around using Thunderbird 
to access this mail.  Even though I map Thunderbird editing to Emacs, I 
much prefer the Rmail interface.  That's why I'm putting the effort into 
Rmail/mbox.  It's a step to having email my way.

If I'm using Rmail/mbox to access my daily dose of email, it's getting a 
lot of testing and bug fixing from me, for sure.  If not, then not.  I 
could respond to bug reports and dribble out test cases as needed but 
that will, IMHO, prolong the drip, drip, drip of the Rmail/mbox saga.

> I would suggest making the switch sooner rather than later, so that it
> gets as much testing as possible.

This is your call.  Richard seems to feel it is low risk, i.e. that 
Rmail/mbox is basically there.  I was aghast when I executed M-x pmail 
and it actually worked.  Imagine.  So Richard could well be right.  He 
has a knack for that.

But my considered opinion is that switching BEFORE some small number of 
developers (could be 1) is using Rmail/mbox as their production mail 
client is fraught with the risk of introducing lots of bugs.  I'm more 
than willing to be one of those developers but not without basic access 
to IMAP to fetch the mail.  I don't need extensive support, especially 
committed (as in checked-in) support, i.e. using both Thunderbird and 
Rmail simultaneously works for me.  I think getting that basic support 
is as simple as building MailUtils movemail and configuring it for use 
with Rmail as an inbox.  Two days if that.  Then a couple of weeks of 
intensive testing (daily use) makes all the sense in the world to me. 
At that point my comfort level is high to make the switch.  Sound like a 
plan to you?

The parts of the merge that were deferred were mostly around spam 
filtering and Babyl<-->mbox conversion.  I'll revisit those deferred 
changes this week.  The lisp/mail/mbox-changes/*.changes files contain 
more details on what was deferred.

-pmr

Re: Rmail-mbox branch

[Stefan Monnier]

> Here's my conundrum.  I'm married to IMAP, for better or worse.  I have my
> mail on a very reliable RAID5, backed up, fast server.  One under my

Adding proper IMAP support to Rmail is between difficult and impossible,
but if you limit yourself to using the IMAP protocol in order to
download your messages (i.e. as if it were just POP), then it should be
pretty simple and indeed using GNU mailutils's code may give you this
feature for free.

We do want to bring Emacs's and mailutils's movemail closer.


        Stefan

Re: Rmail-mbox branch

[Paul Michael Reilly]

Stefan Monnier wrote:
>> Here's my conundrum.  I'm married to IMAP, for better or worse.  I have my
>> mail on a very reliable RAID5, backed up, fast server.  One under my
> 
> Adding proper IMAP support to Rmail is between difficult and impossible,

Distressing but not surprising to hear.  I'm guessing there has been 
discussions around this topic but I can't find any trace.  Do you have a 
reference or can you elaborate?

> but if you limit yourself to using the IMAP protocol in order to
> download your messages (i.e. as if it were just POP), then it should be
> pretty simple and indeed using GNU mailutils's code may give you this
> feature for free.

I am getting my messages now using the mailutil movemail which will 
allow me to use both Rmail/mbox and Thunderbird on a daily basis.

> We do want to bring Emacs's and mailutils's movemail closer.

This is very good to hear.  What else can you tell me?  Is someone 
working on it?  Have constraints been established?  Having movemail 
provide a "synchronize" operation for IMAP seems to make some sense to 
me but in due time I suppose.

-pmr

Re: Rmail-mbox branch

[Stefan Monnier]

>>> Here's my conundrum.  I'm married to IMAP, for better or worse.  I have my
>>> mail on a very reliable RAID5, backed up, fast server.  One under my
>> Adding proper IMAP support to Rmail is between difficult and impossible,

> Distressing but not surprising to hear.  I'm guessing there has been
> discussions around this topic but I can't find any trace.  Do you have
> a reference or can you elaborate?

No, it's just my own assessment based on my knwoledge of how IMAP and
Rmail work.

>> but if you limit yourself to using the IMAP protocol in order to
>> download your messages (i.e. as if it were just POP), then it should be
>> pretty simple and indeed using GNU mailutils's code may give you this
>> feature for free.

> I am getting my messages now using the mailutil movemail which will allow me
> to use both Rmail/mbox and Thunderbird on a daily basis.

Good.

>> We do want to bring Emacs's and mailutils's movemail closer.
> This is very good to hear.  What else can you tell me?  Is someone working
> on it?  Have constraints been established?

Not that I know.  It's just generally desirable to reuse other
people's work.

> Having movemail provide a "synchronize" operation for IMAP seems to
> make some sense to me but in due time I suppose.

That would probably be the easiest way to get Rmail to "properly"
support IMAP.  "Easiest" does not necessarily mean "easy", of course.
Especially if you want it to work reliably.


        Stefan