Re: RMAIL, MIME-related bug

Re: RMAIL, MIME-related bug

[Alexander Pohoyda]

> > I have found a variable called rmail-enable-mime, and some void
> > variables of type rmail-*-mime-* which don't fit into my
> > implementation and I don't understand why they are here anyway.
> 
> IIRC, this is for add-on packages that add MIME support to RMAIL.

Do you know of such a package? I was not able to find any.
If there exist some, I don't want to break it. Otherwise I don't
believe those variables are useful.

> > 2) If I save an email to the file, I want it to be the same message
> >    as RMail read from my mailbox. I want to be able to read it from
> >    another mail client, for example.
> 
> Someone wrote code to do that, but it's currently on a separate
> branch of the Emacs CVS.  I suggest to leave this issue alone, not to
> mix it with the other changes you did.

Actually, I ment rmail-output function from rmailout.el file, but
after consideration I agree that I used that functionality so often
only because there was no better way to get attached files from the
email. Normally I don't need to export emails to files.

> > 3) I agree that some action (transfer-/charset decoding, PGP signature
> >    verification, etc.) is OK to do once in function called from
> >    rmail-convert-to-babyl-format, but some other (picture previews,
> >    big attachement hiding, etc.) should be done by every (?)
> >    rmail-show-message call.
> 
> I suggest to discuss this (on emacs-devel).  There are trade-offs here
> that IMHO are not so trivial to resolve.

OK, this is how I see it.

When we convert the email to babyl format, we can do some MIME-related
processing on it, for example: all text/* bodyparts may be
transfer-encoding (quoted-printable and base64) decoded, PGP/GPG
signatures verified, PGP/GPG decryption done, header fields of type
=?...?B?...?= decoded (RFC 2047) and unfolded (RFC 2822), etc.
This needs to be done once. No information is lost in this step. No
structure information is lost either.

When we show the message, I'm not interested in following things:
* MIME-related boundaries;
* Header fields of most entities;
* Entities of some types (multipart/alternative(text/html),
pgp-signature, etc);
* Entities of types image, audio, video, application. Instead, I would
like to see a button [Save as...] which will save an entity to file;
* Entities of types image/jpeg, image/png, etc. may be displayed right
here in the buffer.

Granted, we can delete boundaries and header field we don't want to
see, but what to do with big bodyparts like attached images? Where to
store them so that the user does not normally see them, but can
access them anytime in the future?

Also, please note that if we delete any MIME-related information, we
will not be able to parse the message again later. Well, we will need
some other parser. This way we invent another format. I don't like
this, that's why I don't agree that we should delete something we
don't want to see on the screen.
As I proposed before, I would like to hide the information using the
"invisible" property. I'm open to suggestions.


Another problem is that we currently brake the email by inserting
"simplified" header between original header and the message body (in
rmail-reformat-message function). After this is done, MIME-parsing
function does not work anymore.

I would very like to hear your ideas.
Thanks to anybody in advance.


-- 
Alexander Pohoyda <alexander.pohoyda@gmx.net>
PGP Key fingerprint: 7F C9 CC 5A 75 CD 89 72  15 54 5F 62 20 23 C6 44

Re: RMAIL, MIME-related bug

[Kenichi Handa]

[-- Attachment #1: Type: text/plain, Size: 1228 bytes --]

In article <200310121947.h9CJlhKH006102@oak.pohoyda.family>, Alexander Pohoyda <alexander.pohoyda@gmx.net> writes:
>>  > I have found a variable called rmail-enable-mime, and some void
>>  > variables of type rmail-*-mime-* which don't fit into my
>>  > implementation and I don't understand why they are here anyway.
>>  
>>  IIRC, this is for add-on packages that add MIME support to RMAIL.

> Do you know of such a package? I was not able to find any.

Morioka-san developped such a package.  The several
rmail-*-mime-*-function are introduced for that package.
I'll attach it at the tail.  But, unfortunately, before
releasing that package officially, he moved to another
organization and got very busy.  It decodes a message into
another buffer (rmail-view-buffer) without modifying RMAIL
buffer itself.  As far as I know, one problem with this
package is that it's very slow in rmail-search[-backward].

> If there exist some, I don't want to break it. Otherwise I don't
> believe those variables are useful.

It seems that we can't expect Morioka-san to work on
rmail-mime package, at least, for a while.  So, if keeping
those variables is very difficult, I think you can delete
them.

---
Ken'ichi HANDA
handa@m17n.org


[-- Attachment #2: rmail-mime.tar.gz --]
[-- Type: application/octet-stream, Size: 193573 bytes --]

[-- Attachment #3: Type: text/plain, Size: 141 bytes --]

_______________________________________________
Emacs-devel mailing list
Emacs-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/emacs-devel

Re: RMAIL, MIME-related bug

[Robert J. Chassell]

    >>  IIRC, this is for add-on packages that add MIME support to RMAIL.
    > Do you know of such a package? I was not able to find any.

Take a look at rmime.el.  I have been using it with no trouble.

Here is how rmime.el describes itself:

    ;;; rmime.el --- read MIME messages

    ;; Author: Ray Moody <moody@cray.com>
    ;; Version: $Id: rmime.el,v 1.2 1996/05/30 02:24:58 moody Exp $
    ;; Charset support version 0.3 by J. Chroboczek <jec@dcs.ed.ac.uk>
    ;; Keywords: MIME, mail

    ;;; Commentary:

    ;; RMIME provides MIME support for several Emacs message reading
    ;; packages.  RMIME has been designed with RMAIL in mind, but it has
    ;; also been tested with mh-e and VM.  It should work with most other
    ;; major modes as well.

Before loading rmime.el, you need to load charset.el, which is in a
library from 1996 written by Juliusz Chroboczek <jec@dcs.ed.ac.uk>.

The rmime.el file says to put this into your .emacs file:

    ;;	(add-hook 'rmail-show-message-hook 'rmime-format)
    ;;	(add-hook 'rmail-edit-mode-hook    'rmime-cancel)
    ;;	(autoload 'rmime-format "rmime" "" nil)

but I do this:

    (load "charset")
    (load "rmime")
    (set-face-foreground 'rmime-markup-face "cyan")
    (define-key rmail-mode-map [f7] 'rmime-mode) ; fn key F7

(For creating MIME messages, I use Marc Andreessen's mime-compose.el,
which he wrote in 1992 before he started Netscape.)

Also, you need to have installed mimencode and then, since the names
have changed, to symbollically link mmencode to mimencode:

    ln -s mimencode mmencode

This works in today's CVS snapshot, Tue, 2003 Oct 14  12:16 UTC
GNU Emacs 21.3.50.118 (i686-pc-linux-gnu, GTK+ Version 2.2.1)

-- 
    Robert J. Chassell                         Rattlesnake Enterprises
    http://www.rattlesnake.com                  GnuPG Key ID: 004B4AC8
    http://www.teak.cc                             bob@rattlesnake.com

Re: RMAIL, MIME-related bug

[Richard Stallman]

    Before loading rmime.el, you need to load charset.el, which is in a
    library from 1996 written by Juliusz Chroboczek <jec@dcs.ed.ac.uk>.

We don't have legal papers from Chroboczek.

    (For creating MIME messages, I use Marc Andreessen's mime-compose.el,
    which he wrote in 1992 before he started Netscape.)

We don't have papers for that either.

We would need to find them and ask them for papers
if we are to use this code.  Nonetheless, to compare
this and Morioka's package would be a useful activity.

Re: RMAIL, MIME-related bug

[Peter S Galbraith]

Robert J. Chassell <bob@rattlesnake.com> wrote:

>     >>  IIRC, this is for add-on packages that add MIME support to RMAIL.
>     > Do you know of such a package? I was not able to find any.
> 
> Take a look at rmime.el.  I have been using it with no trouble.
> 
> Here is how rmime.el describes itself:
> 
>     ;;; rmime.el --- read MIME messages
> 
>     ;; Author: Ray Moody <moody@cray.com>
>     ;; Version: $Id: rmime.el,v 1.2 1996/05/30 02:24:58 moody Exp $
>     ;; Charset support version 0.3 by J. Chroboczek <jec@dcs.ed.ac.uk>
>     ;; Keywords: MIME, mail
> 
>     ;;; Commentary:
> 
>     ;; RMIME provides MIME support for several Emacs message reading
>     ;; packages.  RMIME has been designed with RMAIL in mind, but it has
>     ;; also been tested with mh-e and VM.  It should work with most other
>     ;; major modes as well.

Note: It probably doesn't work with modern MH-E as it handles MIME
natively.

Another avenue for rmail developers is to borrow some MIME support code
from MH-E which was borrowed from gnus.  (It might actually be easier to
isolate in the MH-E code.)

Re: RMAIL, MIME-related bug

[Eli Zaretskii]

> Date: Sun, 12 Oct 2003 21:47:43 +0200 (CEST)
> From: Alexander Pohoyda <alexander.pohoyda@gmx.net>
> 
> When we convert the email to babyl format, we can do some MIME-related
> processing on it, for example: all text/* bodyparts may be
> transfer-encoding (quoted-printable and base64) decoded, PGP/GPG
> signatures verified, PGP/GPG decryption done, header fields of type
> =?...?B?...?= decoded (RFC 2047) and unfolded (RFC 2822), etc.
> This needs to be done once. No information is lost in this step. No
> structure information is lost either.

I agree that these are best done while converting to Babyl. 

> Granted, we can delete boundaries and header field we don't want to
> see, but what to do with big bodyparts like attached images?

We could replace them with some icon, and give the user a way to
decode and display them with some key binding.

> Also, please note that if we delete any MIME-related information, we
> will not be able to parse the message again later.

You don't need to delete it, just to hide it.  RMAIL already hides
some headers, so the same machinery could be used here.

Re: RMAIL, MIME-related bug

[Alexander Pohoyda]

> > Granted, we can delete boundaries and header field we don't want to
> > see, but what to do with big bodyparts like attached images?
> 
> We could replace them with some icon, and give the user a way to
> decode and display them with some key binding.

Replace on the screen, not changing the content of the mail, right?
That's what I would like to do. The problem is that we have to do this
after the mail is reformated, but there is no valid MIME message after
the rmail-reformat-message call: the message is broken into two parts
by the EOOH marker.
To be able to MIME-parse it, I have to glue the message back and
delete the "simplified" header in between.

> > Also, please note that if we delete any MIME-related information, we
> > will not be able to parse the message again later.
> 
> You don't need to delete it, just to hide it.  RMAIL already hides
> some headers, so the same machinery could be used here.

If you mean rmail-clear-headers function, then it does not hide headers,
but deletes them.

That's what I'm trying to hear in this discussion.
Can't we just hide header fields instead of creating another "simplified"
header in between of the message and thus braking it?

-- 
Alexander Pohoyda <alexander.pohoyda@gmx.net>
PGP Key fingerprint: 7F C9 CC 5A 75 CD 89 72
 15 54 5F 62 20 23 C6 44

Re: RMAIL, MIME-related bug

[Thien-Thi Nguyen]

"Alexander Pohoyda" <alexander.pohoyda@gmx.net> writes:

   That's what I'm trying to hear in this discussion.  Can't we just
   hide header fields instead of creating another "simplified" header in
   between of the message and thus braking it?

you have the right idea.  hiding is the desired functionality, and
currently, deleting is the implementation.  it's probably ok to keep the
functionality and choose a more desirable implementation (overlays, for
example).  then when someone complains about the change breaking their
code you can say "that was an implementation detail".  then, they will
of course ask you for your high-level design and API, in order to avoid
being burned in the future.  then, you will know you have thought about
things in the right way from the beginning.

thi

Re: RMAIL, MIME-related bug

[Thien-Thi Nguyen]

Thien-Thi Nguyen <ttn@glug.org> writes:

   currently, deleting is the implementation.
              ^^^^^^^^

oops, this should be some other gerund, like "copying".
i hope you understand anyway.

thi

Re: RMAIL, MIME-related bug

[Eli Zaretskii]

> Date: Thu, 16 Oct 2003 09:41:26 +0200 (MEST)
> From: "Alexander Pohoyda" <alexander.pohoyda@gmx.net>
> 
> The problem is that we have to do this
> after the mail is reformated, but there is no valid MIME message after
> the rmail-reformat-message call: the message is broken into two parts
> by the EOOH marker.
> To be able to MIME-parse it, I have to glue the message back and
> delete the "simplified" header in between.

I don't understand why is it important to have the message in its
original pre-`rmail-reformat-message' form.  All the necessary MIME
information is still there, part of it before EOOH, part of it
after.  So you ought to be able to figure out how to process the
message, right?  What am I missing?

Re: RMAIL, MIME-related bug

[Alexander Pohoyda]

> > The problem is that we have to do this
> > after the mail is reformated, but there is no valid MIME message after
> > the rmail-reformat-message call: the message is broken into two parts
> > by the EOOH marker.
> > To be able to MIME-parse it, I have to glue the message back and
> > delete the "simplified" header in between.
> 
> I don't understand why is it important to have the message in its
> original pre-`rmail-reformat-message' form. 

Because, after reformatting it is not a MIME message anymore!

You agreed that we can do some mail processing before reformatting
and you agree that we have to do even more processing after reformatting.
Thus, if we want to use the same functions to do the job, we have to
teach them to ignore this "simplified" header. Now, MIME is recursive and
this means that we have to do this ignoring in every media type handler.
Is there anything we gain with this implementation?

OK, this may not be that important. I just don't see a reason why do we
have to reformat the message just to ignore this reformatting later.
Reformatting only introduces an obstacle. Why to do that?

And, as I mentioned before, I cannot re-use the old machinery (namely:
deleting header fields), thus header-field hiding will be necessary anyway.
So why not to hide header fields everywhere instead of deleting them
and reformatting the original message?


> All the necessary MIME information is still there, part of it before
> EOOH, part of it after. 

Yes, all information is there, but the structure is not.
The structure is broken. There is no MIME message anymore.

> So you ought to be able to figure out how to process the
> message, right?  What am I missing?

Right, everything is possible. The only question is: does it make sense?

In the end, if there is no other acceptable solution, I will have to do it
this way.

-- 
Alexander Pohoyda <alexander.pohoyda@gmx.net>
PGP Key fingerprint: 7F C9 CC 5A 75 CD 89 72
 15 54 5F 62 20 23 C6 44

Re: RMAIL, MIME-related bug

[Alexander Pohoyda]

>    That's what I'm trying to hear in this discussion.  Can't we just
>    hide header fields instead of creating another "simplified" header in
>    between of the message and thus braking it?
> 
> you have the right idea.  hiding is the desired functionality, and
> currently, deleting is the implementation.  it's probably ok to keep the
> functionality and choose a more desirable implementation (overlays, for
> example).  then when someone complains about the change breaking their
> code you can say "that was an implementation detail".  then, they will
> of course ask you for your high-level design and API, in order to avoid
> being burned in the future.  then, you will know you have thought about
> things in the right way from the beginning.

Sorry, but I do not understand what you propose here.
Not to change anything because it may brake some code?
I'm afraid you are right. Too much code may be based on current
implementation.

-- 
Alexander Pohoyda <alexander.pohoyda@gmx.net>
PGP Key fingerprint: 7F C9 CC 5A 75 CD 89 72
 15 54 5F 62 20 23 C6 44

Re: RMAIL, MIME-related bug

[Thien-Thi Nguyen]

   From: "Alexander Pohoyda" <alexander.pohoyda@gmx.net>
   Date: Thu, 16 Oct 2003 15:27:40 +0200 (MEST)

   Sorry, but I do not understand what you propose here.
   Not to change anything because it may brake some code?

no, that is an extreme position that is recognizably unworkable.
i am, instead, proposing you analyze the code from a two-level
perspective, the design level (high) and the implementation
level (low), and change the implementation while keeping the
design.

here, the design is relatively (in my mind as an RMAIL user)
straightforward: allow customization of which headers are
visible and their ordering.  to date, the implementation has
been a re-rendering (copying) of the desired-visible headers to
another block followed by narrowing to that block.  what new
implementation you choose should support the design as well as
any additional criteria that have arisen since the original
rmail.el was written (e.g., encoding issues).

   Too much code may be based on current implementation.

this is something we cannot know a priori.  but if we know the
nature of the changes to the implementation layer, we can help
fix those pieces of code that rely on it.

we can do this by either retargeting the broken code to the new
implementation or by retargeting the broken code to the design.
the latter is preferable since, if we do a good job, the design
will not change so much in the future whereas the implementation
may (or may not ;-).

thi

Re: RMAIL, MIME-related bug

[Ehud Karni]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Thu, 16 Oct 2003 13:20:48 +0200, Eli Zaretskii <eliz@elta.co.il> wrote:
>
> I don't understand why is it important to have the message in its
> original pre-`rmail-reformat-message' form.  All the necessary MIME
> information is still there, part of it before EOOH, part of it
> after.  So you ought to be able to figure out how to process the
> message, right?  What am I missing?

It is important if you wish to run external command on it. I use
metamail frequently on the mail buffer. With the  current `rmail' I
use the `rmail-edit-current-message' function to restore (almost) the
original message (it has the `x-Coding-System' line added and the
Mail-from: pseudo header prefixed).

Also, you need the mail in its "undisturbed" form, if you want to
resend it like a mail-relay (another function that I use a lot).

Ehud.


- --
 Ehud Karni           Tel: +972-3-7966-561  /"\
 Mivtach - Simon      Fax: +972-3-7966-667  \ /  ASCII Ribbon Campaign
 Insurance agencies   (USA) voice mail and   X   Against   HTML   Mail
 http://www.mvs.co.il  FAX:  1-815-5509341  / \
 GnuPG: 98EA398D <http://www.keyserver.net/>    Better Safe Than Sorry
-----BEGIN PGP SIGNATURE-----
Comment: use http://www.keyserver.net/ to get my key (and others)

iD8DBQE/jqzpLFvTvpjqOY0RAsaZAJ9fQjCX/EfJiUj5XUGVOwqrw1MpCgCfQEEV
dRsVCG8KWmgC2B+h9fGicq8=
=5G08
-----END PGP SIGNATURE-----

Re: RMAIL, MIME-related bug

[Eli Zaretskii]

> Date: Thu, 16 Oct 2003 15:14:55 +0200 (MEST)
> From: "Alexander Pohoyda" <alexander.pohoyda@gmx.net>
> 
> Because, after reformatting it is not a MIME message anymore!

So?  We are not sending that text as-is to any MTA, are we?  We are
just processing the text locally.  So we don't need to comply to MIME
standards, do we?

> OK, this may not be that important. I just don't see a reason why do we
> have to reformat the message just to ignore this reformatting later.
> Reformatting only introduces an obstacle. Why to do that?

I think it eases further processing elsewhere in RMAIL.  IIRC, many
RMAIL functions need to access the headers and the body separately.

Of course, it is also possible to avoid that reformatting, but then
your job will become much bigger, I think.

Re: RMAIL, MIME-related bug

[Eli Zaretskii]

> Date: Thu, 16 Oct 2003 16:36:26 +0200
> From: "Ehud Karni" <ehud@unix.mvs.co.il>
> 
> It is important if you wish to run external command on it. I use
> metamail frequently on the mail buffer. With the  current `rmail' I
> use the `rmail-edit-current-message' function to restore (almost) the
> original message (it has the `x-Coding-System' line added and the
> Mail-from: pseudo header prefixed).

It is also decoded into the Emacs internal representation of
characters, so you don't have the original message anyway (unless it's
pure ASCII).

> Also, you need the mail in its "undisturbed" form, if you want to
> resend it like a mail-relay (another function that I use a lot).

That's why in the past discussions of this issue someone suggested to
redesign RMAIL along the lines of Gnus: leave the original message
alone, and produce the reformatted one in a separate buffer.  This is
actually the main reason I asked Alexander to take this discussion
here.  However, it sounds like Alexander has no intent to follow Gnus,
and no one else raised that problem either (until now).

Re: RMAIL, MIME-related bug

[Stefan Monnier]

> When we convert the email to babyl format, we can do some MIME-related
> processing on it, for example: all text/* bodyparts may be
> transfer-encoding (quoted-printable and base64) decoded, PGP/GPG
> signatures verified, PGP/GPG decryption done, header fields of type
> =?...?B?...?= decoded (RFC 2047) and unfolded (RFC 2822), etc.
> This needs to be done once. No information is lost in this step. No
> structure information is lost either.

In order not to lose information, you need to keep the GPG signatures you
have just verified since you can't re-create them yourself.  Of course, you
also need to keep the signed-text unless you're super-extra-careful to make
sure that you can re-create the exact same byte-sequence from the rest of
the data, which is rather unlikely.

Why do people even consider doing any processing at that point?
The processing should be done for display and nothing else, because any
other option will make you lose information at some point and also forces
you to have code to re-create the original format, whereas the "display-side
processing" approach only needs to parse MIME but never needs to
re-construct it.


        Stefan

Re: RMAIL, MIME-related bug

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Date: 16 Oct 2003 12:54:24 -0400
> 
> Why do people even consider doing any processing at that point?

Because doing that when the user wants to display a message means that
(a) displaying a message will be slow, and (b) you will need to do
this kind of processing every time you display a message.  In
contrast, the current design of RMAIL does the processing only once
for each message.

> any other option will make you lose information at some point and
> also forces you to have code to re-create the original format

The question is, do we care that we lose information, and do we need
to recreate the original format?  That's exactly what I was asking
Alexander to explain.

Re: RMAIL, MIME-related bug

[David Kastrup]

"Eli Zaretskii" <eliz@elta.co.il> writes:

> > Date: Thu, 16 Oct 2003 15:14:55 +0200 (MEST)
> > From: "Alexander Pohoyda" <alexander.pohoyda@gmx.net>
> > 
> > Because, after reformatting it is not a MIME message anymore!
> 
> So?  We are not sending that text as-is to any MTA, are we?

How about forwarding?

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: RMAIL, MIME-related bug

[Stefan Monnier]

> The question is, do we care that we lose information, and do we need
> to recreate the original format?  That's exactly what I was asking
> Alexander to explain.

I need to be able to forward/resend a GPG-signed message (or a message with
a GPG-signed subpart), so at least in those cases we need to preserve all
the information.


        Stefan


PS: All the MUAs I know do the processing upon display, so I don't
    understand why it would pose any performance problem.

Re: RMAIL, MIME-related bug

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Date: 16 Oct 2003 14:49:13 -0400
> 
> I need to be able to forward/resend a GPG-signed message (or a message with
> a GPG-signed subpart), so at least in those cases we need to preserve all
> the information.

Is forwarding such a frequent operation?

> PS: All the MUAs I know do the processing upon display, so I don't
>     understand why it would pose any performance problem.

It simply sounds sub-optimal to do the decoding every time the message
is displayed.

Re: RMAIL, MIME-related bug

[Ehud Karni]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 16 Oct 2003 14:49:13 -0400, Stefan Monnier <monnier@IRO.UMontreal.CA> wrote:
>
> > The question is, do we care that we lose information, and do we need
> > to recreate the original format?  That's exactly what I was asking
> > Alexander to explain.
>
> I need to be able to forward/resend a GPG-signed message (or a message with
> a GPG-signed subpart), so at least in those cases we need to preserve all
> the information.

I second Stefan opinion. I just checked current RMAIL behavior. When
you do rmail edit (i.e. rmail-edit-current-message) it fully restores
the original mail including the original character encoding (I checked
it on a problematic Hebrew encoded in utf-8).

> PS: All the MUAs I know do the processing upon display, so I don't
>     understand why it would pose any performance problem.

I agree on this point too (only few mails are read more than once).

Ehud.


- --
 Ehud Karni           Tel: +972-3-7966-561  /"\
 Mivtach - Simon      Fax: +972-3-7966-667  \ /  ASCII Ribbon Campaign
 Insurance agencies   (USA) voice mail and   X   Against   HTML   Mail
 http://www.mvs.co.il  FAX:  1-815-5509341  / \
 GnuPG: 98EA398D <http://www.keyserver.net/>    Better Safe Than Sorry
-----BEGIN PGP SIGNATURE-----
Comment: use http://www.keyserver.net/ to get my key (and others)

iD8DBQE/ju4ZLFvTvpjqOY0RAjgkAJ99A7/T4aSpAFQLeFzsmjoD7dbLaACfdSYc
SUy/Z7U3LG+mHarpzQoG4TE=
=/Ytc
-----END PGP SIGNATURE-----

Re: RMAIL, MIME-related bug

[Stefan Monnier]

>> I need to be able to forward/resend a GPG-signed message (or a message with
>> a GPG-signed subpart), so at least in those cases we need to preserve all
>> the information.
> Is forwarding such a frequent operation?

The question is not even relevant: it needs to be possible to do it,
so you need to keep the info, just in case.

>> PS: All the MUAs I know do the processing upon display, so I don't
>> understand why it would pose any performance problem.
> It simply sounds sub-optimal to do the decoding every time the message
> is displayed.

It might be suboptimal in some cases, but in terms of programmer resources,
it's pretty close to optimal.  Admittedly, it's not clear whether programmer
resources is the bottleneck w.r.t MIME decoding given the number of
different MIME-decoding package have been written in elisp.  Or maybe it is
the bottleneck and it's a shame that we would write yet-another one rather
than use a pre-existing one such as the one in Gnus (wisely used by MH-E).


        Stefan

Re: RMAIL, MIME-related bug

[Ehud Karni]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Thu, 16 Oct 2003 20:58:28 +0200, Eli Zaretskii <eliz@elta.co.il> wrote:
>
> > I need to be able to forward/resend a GPG-signed message (or a message with
> > a GPG-signed subpart), so at least in those cases we need to preserve all
> > the information.
>
> Is forwarding such a frequent operation?

Yes, I do 2-5 resends a day. I even wrote my own resend function to do
it like a mail relay, so only 1 received header is added and if the
new recipient mail transport agent will return the mail, it will go to
the originator of the mail.

Ehud.


- --
 Ehud Karni           Tel: +972-3-7966-561  /"\
 Mivtach - Simon      Fax: +972-3-7966-667  \ /  ASCII Ribbon Campaign
 Insurance agencies   (USA) voice mail and   X   Against   HTML   Mail
 http://www.mvs.co.il  FAX:  1-815-5509341  / \
 GnuPG: 98EA398D <http://www.keyserver.net/>    Better Safe Than Sorry
-----BEGIN PGP SIGNATURE-----
Comment: use http://www.keyserver.net/ to get my key (and others)

iD8DBQE/jvKCLFvTvpjqOY0RAm7+AJ9jZoiSBEjGUJ5pm78eU4ZmmiXM7ACfTl/U
VGRgqh93ZwcfF9TmLk7Pygs=
=ASP5
-----END PGP SIGNATURE-----

Re: RMAIL, MIME-related bug

[Alexander Pohoyda]

"Eli Zaretskii" <eliz@elta.co.il> writes:

> > Because, after reformatting it is not a MIME message anymore!
> 
> So?  We are not sending that text as-is to any MTA, are we?

No, I have never done that, but remember the problem with exporting
an email to the Unix mail file. Who knows how else we could use the
email would it be not reformatted.


> We are just processing the text locally. So we don't need to comply
> to MIME standards, do we?

Right, we don't have to, but why not to if that significantly
simplifies the processing?


> > OK, this may not be that important. I just don't see a reason why do we
> > have to reformat the message just to ignore this reformatting later.
> > Reformatting only introduces an obstacle. Why to do that?
> 
> I think it eases further processing elsewhere in RMAIL.

Who may know that for sure?


>                                                         IIRC, many
> RMAIL functions need to access the headers and the body separately.

No problems with that. I am not going to change it.


> Of course, it is also possible to avoid that reformatting, but then
> your job will become much bigger, I think.

Yes, I will have to modify some Rmail functions, but the result, I'm
sure, is well worth the effort.


-- 
Alexander Pohoyda <alexander.pohoyda@gmx.net>
PGP Key fingerprint: 7F C9 CC 5A 75 CD 89 72  15 54 5F 62 20 23 C6 44

Re: RMAIL, MIME-related bug

[Alexander Pohoyda]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

> > When we convert the email to babyl format, we can do some
> > MIME-related processing on it, for example: all text/* bodyparts
> > may be transfer-encoding (quoted-printable and base64) decoded,
> > PGP/GPG signatures verified, PGP/GPG decryption done, header
> > fields of type =?...?B?...?= decoded (RFC 2047) and unfolded (RFC
> > 2822), etc. This needs to be done once. No information is lost in
> > this step. No structure information is lost either.
> 
> In order not to lose information, you need to keep the GPG
> signatures you have just verified since you can't re-create them
> yourself.

Sure, I'm just hiding that bodypart at display time.

>                                                             Of
> course, you also need to keep the signed-text unless you're
> super-extra-careful to make sure that you can re-create the exact
> same byte-sequence from the rest of the data, which is rather
> unlikely.

OK, let's keep the signed text un-changed, but I would prefer to do
the decoding on normal messages. As Eli points out, this is logical
to do once on message arrival.


> Why do people even consider doing any processing at that point?

Just to optimise the process. Doing it every time in show-message is
even simplier from the programmer's point of view.


> The processing should be done for display and nothing else, because
> any other option will make you lose information at some point and
> also forces you to have code to re-create the original format,
> whereas the "display-side processing" approach only needs to parse
> MIME but never needs to re-construct it.

You are right, but let's focus on the need not to invent the
RMAIL-MIME, no additional header in between the message. Do you agree
that hiding header fields is preferred to having two message headers?


-- 
Alexander Pohoyda <alexander.pohoyda@gmx.net>
PGP Key fingerprint: 7F C9 CC 5A 75 CD 89 72  15 54 5F 62 20 23 C6 44

Re: RMAIL, MIME-related bug

[Alexander Pohoyda]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

> > When we convert the email to babyl format, we can do some
> > MIME-related processing on it, for example: all text/* bodyparts
> > may be transfer-encoding (quoted-printable and base64) decoded,
> > PGP/GPG signatures verified, PGP/GPG decryption done, header
> > fields of type =?...?B?...?= decoded (RFC 2047) and unfolded (RFC
> > 2822), etc. This needs to be done once. No information is lost in
> > this step. No structure information is lost either.
> 
> In order not to lose information, you need to keep the GPG
> signatures you have just verified since you can't re-create them
> yourself.

Sure, I'm just hiding that bodypart at display time.

>                                                             Of
> course, you also need to keep the signed-text unless you're
> super-extra-careful to make sure that you can re-create the exact
> same byte-sequence from the rest of the data, which is rather
> unlikely.

OK, let's keep the signed text un-changed, but I would prefer to do
the decoding on normal messages. As Eli points out, this is logical
to do once on message arrival.


> Why do people even consider doing any processing at that point?

Just to optimise the process. Doing it every time in show-message is
even simplier from the programmer's point of view.


> The processing should be done for display and nothing else, because
> any other option will make you lose information at some point and
> also forces you to have code to re-create the original format,
> whereas the "display-side processing" approach only needs to parse
> MIME but never needs to re-construct it.

You are right, but let's focus on the need not to invent the
RMAIL-MIME, no additional header in between the message. Do you agree
that hiding header fields is preferred to having two message headers?


-- 
Alexander Pohoyda <alexander.pohoyda@gmx.net>
PGP Key fingerprint: 7F C9 CC 5A 75 CD 89 72  15 54 5F 62 20 23 C6 44

Re: RMAIL, MIME-related bug

[Alexander Pohoyda]

"Ehud Karni" <ehud@unix.mvs.co.il> writes:

> > > The question is, do we care that we lose information, and do we need
> > > to recreate the original format?  That's exactly what I was asking
> > > Alexander to explain.
> >
> > I need to be able to forward/resend a GPG-signed message (or a message with
> > a GPG-signed subpart), so at least in those cases we need to preserve all
> > the information.
> 
> I second Stefan opinion. I just checked current RMAIL behavior. When
> you do rmail edit (i.e. rmail-edit-current-message) it fully restores
> the original mail including the original character encoding (I checked
> it on a problematic Hebrew encoded in utf-8).

And if the message was sent base64 or quoted-printable -encoded?


> > PS: All the MUAs I know do the processing upon display, so I don't
> >     understand why it would pose any performance problem.
> 
> I agree on this point too (only few mails are read more than once).

I disagree here. I store emails and sometimes have to read them many
times. Not to mention summary-navigation.


-- 
Alexander Pohoyda <alexander.pohoyda@gmx.net>
PGP Key fingerprint: 7F C9 CC 5A 75 CD 89 72  15 54 5F 62 20 23 C6 44

Re: RMAIL, MIME-related bug

[Stefan Monnier]

>> Of course, you also need to keep the signed-text unless you're
>> super-extra-careful to make sure that you can re-create the exact same
>> byte-sequence from the rest of the data, which is rather unlikely.

> OK, let's keep the signed text un-changed, but I would prefer to do
> the decoding on normal messages. As Eli points out, this is logical
> to do once on message arrival.

I see nothing logical about it.  AFAICT, the message will be saved somehow
in a text file, so you'll need to use some kind of unambiguous textual
representation of the various parts of the message.  Of course, you can
re-invent the wheel or use another existing wheel (the builtin lisp
printer), but using the original MIME representation is a pretty
natural choice.

>> Why do people even consider doing any processing at that point?
> Just to optimise the process. Doing it every time in show-message is
> even simplier from the programmer's point of view.

I'd first concentrate on getting something working correctly.
Especially since there is no evidence that performance will be a problem,
let alone that a pre-processed form will be more efficient.

> You are right, but let's focus on the need not to invent the
> RMAIL-MIME, no additional header in between the message. Do you agree
> that hiding header fields is preferred to having two message headers?

I don't really know the details of what this is referring to.
I generally like approaches based on adding text-properties and overlays
rather than modifying buffer text, but modifying buffer text can be much
simpler, with fewer surprises and much more flexible (I like my headers to
be sorted for example, which can't be done just by hiding undesirable
header fields).


        Stefan

Re: RMAIL, MIME-related bug

[Ehud Karni]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 16 Oct 2003 23:56:50 +0200, Alexander Pohoyda <alexander.pohoyda@gmx.net> wrote:
>
> > I second Stefan opinion. I just checked current RMAIL behavior. When
> > you do rmail edit (i.e. rmail-edit-current-message) it fully restores
> > the original mail including the original character encoding (I checked
> > it on a problematic Hebrew encoded in utf-8).
>
> And if the message was sent base64 or quoted-printable -encoded?

Neither. It was sent as 8 bit text. Here are the headers:

  Content-Type: text/plain; charset=UTF-8; format=flowed
  Content-Transfer-Encoding: 8bit


> > I agree on this point too (only few mails are read more than once).
>
> I disagree here. I store emails and sometimes have to read them many
> times. Not to mention summary-navigation.

I store email too, but still I say that only few are read more than
once. What summary navigation has to with it ? On other mail readers
the summary (index) is stored on a separate file altogether.

Ehud.


- --
 Ehud Karni           Tel: +972-3-7966-561  /"\
 Mivtach - Simon      Fax: +972-3-7966-667  \ /  ASCII Ribbon Campaign
 Insurance agencies   (USA) voice mail and   X   Against   HTML   Mail
 http://www.mvs.co.il  FAX:  1-815-5509341  / \
 GnuPG: 98EA398D <http://www.keyserver.net/>    Better Safe Than Sorry
-----BEGIN PGP SIGNATURE-----
Comment: use http://www.keyserver.net/ to get my key (and others)

iD8DBQE/jxnpLFvTvpjqOY0RAoxkAJoDcIN11nTbQ+CgDBygSbJjKy+2VQCeJHeX
7Ff6oTbSRF6rWCFE2VbcVTQ=
=qE/4
-----END PGP SIGNATURE-----

Re: RMAIL, MIME-related bug

[Richard Stallman]

    > When we convert the email to babyl format,

Paul Riley and I have made a version of Rmail that doesn't use Babyl
format.  We want to get rid of it entirely.  I think it is currently
available in a branch called rmail-mbox-branch.

So please don't think in terms of conversion to Babyl
format when designing future Rmail changes.

						 we can do some MIME-related
    > processing on it, for example: all text/* bodyparts may be
    > transfer-encoding (quoted-printable and base64) decoded, PGP/GPG
    > signatures verified, PGP/GPG decryption done, header fields of type
    > =?...?B?...?= decoded (RFC 2047) and unfolded (RFC 2822), etc.
    > This needs to be done once. No information is lost in this step. No
    > structure information is lost either.

    I agree that these are best done while converting to Babyl. 

I disagree.  I don't want any decryption done and saved on my disk.
(You won't be able to do the decryption without asking me for my
passphrase, and I would not give it.)

Decoding of quoted-printable and base64 may cause problems too.  The
point of these encodings is so that the data can be unambiguously
included in a message.  If you decode it and store the result in the
Rmail file, that brings back the ambiguities these encodings were
designed to avoid.

    > All the necessary MIME information is still there, part of it before
    > EOOH, part of it after. 

    Yes, all information is there, but the structure is not.
    The structure is broken. There is no MIME message anymore.

It is not a good idea to make such drastic changes to the message as
it is saved on disk.  They should be done as part of displaying the
message, when the user selects it.  The results can be cached to make
moving between messages faster, if that's desirable.

Re: RMAIL, MIME-related bug

[Richard Stallman]

    If you mean rmail-clear-headers function, then it does not hide headers,
    but deletes them.

In the version of Rmail that uses mbox format, I believe it
makes unwanted header fields invisible.

Re: RMAIL, MIME-related bug

[Luc Teirlinck]

Ehud Karni wrote:

   I store email too, but still I say that only few are read more than
   once. What summary navigation has to with it ?

In RMAIL, with a summary buffer and an RMAIL buffer in another window,
C-p and C-n in the summary buffer update the message shown in the
RMAIL buffer.  The MIME decoding should be fast enough not to
excessively slow down C-n and C-p in the summary buffer.  It should
also be fast enough not to excessively slow down `n' and `p' in the
RMAIL buffer itself, which I would guess to be pretty much equivalent
(in that sense, "_summary_ navigation" is not the main problem,
navigation _in whatever form_ is.)

Whether I use C-n or C-p in the summary buffer or `n' or `p' in RMAIL,
I would wind up MIME decoding every message I do not immediately
delete several times, just by navigation.  That makes speed of MIME
decoding more relevant than it would otherwise be. As Richard pointed
out, however:

   It is not a good idea to make such drastic changes to the message as
   it is saved on disk.  They should be done as part of displaying the
   message, when the user selects it.  The results can be cached to make
   moving between messages faster, if that's desirable.

Whether caching would be desirable depends, of course, on how fast
MIME decoding without caching winds up being.

Sincerely,

Luc.

Re: RMAIL, MIME-related bug

[Kenichi Handa]

In article <200310162348.h9GNmZm25160@raven.dms.auburn.edu>, Luc Teirlinck <teirllm@dms.auburn.edu> writes:
> Ehud Karni wrote:
>    I store email too, but still I say that only few are read more than
>    once. What summary navigation has to with it ?

> In RMAIL, with a summary buffer and an RMAIL buffer in another window,
> C-p and C-n in the summary buffer update the message shown in the
> RMAIL buffer.  The MIME decoding should be fast enough not to
> excessively slow down C-n and C-p in the summary buffer.  It should
> also be fast enough not to excessively slow down `n' and `p' in the
> RMAIL buffer itself, which I would guess to be pretty much equivalent
> (in that sense, "_summary_ navigation" is not the main problem,
> navigation _in whatever form_ is.)

One more point about efficiency is rmail-search and
rmail-search-backwards.  I can wait for a while just after I
type M-x rmail or g in RMAIL buffer.  But, I won't be able
to stand slowdown of those search operations.

I think this kind of method is ideal.

(1) Read RMAIL file without decoding into some hidden source
    buffer (unibyte).  It may be ok to process only
    Content-Transfer-Encoding.
(2) Prepare a view buffer.
(3) Insert the current message in the view buffer after decoding it.
(4) A background process (run by idle timer?) decodes not
    yet decoded message into the view buffer (like
    jit-lock-stealth-fontify).

While decoding, keep such big embeded data as images and
*.tar.gz in the source buffer, and embed buttons or
something in the view buffer.  Ignoreable headers are as
well.  Then, although we have double buffers source and
view, the latter won't become that big because it contains
only text.

We don't have to decode spam-like messages (detected
somehow) correctly, instead just insert "Perhaps this
message is a spam" in the view buffer, and allow users to
click it if they really want to see a decoded message.

But, it seems that it is a faily complicated work to
implement such a mechanism.

---
Ken'ichi HANDA
handa@m17n.org

Re: RMAIL, MIME-related bug

[Peter S Galbraith]

Eli Zaretskii <eliz@elta.co.il> wrote:

> > From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> > Date: 16 Oct 2003 12:54:24 -0400
> > 
> > Why do people even consider doing any processing at that point?
> 
> Because doing that when the user wants to display a message means that
> (a) displaying a message will be slow, and (b) you will need to do
> this kind of processing every time you display a message.  In
> contrast, the current design of RMAIL does the processing only once
> for each message.

Excuse the intrusion of a lurker...

Why would it be slow?  Try MH-E.  It re-does all MIME processing at
every display and it's quite fast.
 
> > any other option will make you lose information at some point and
> > also forces you to have code to re-create the original format
> 
> The question is, do we care that we lose information, and do we need
> to recreate the original format?  That's exactly what I was asking
> Alexander to explain.

For one, you can't check a GPG signature if you change one bit.
Secondly, it's much easier to convert your mail to some other MUA if it
left it intact.

We came up against similar questions working on MH-E, so I'm speaking
from some experience.

Best regards,
Peter

Re: RMAIL, MIME-related bug

[Alexander Pohoyda]

Richard Stallman <rms@gnu.org> writes:

> Paul Riley and I have made a version of Rmail that doesn't use Babyl
> format.  We want to get rid of it entirely.  I think it is currently
> available in a branch called rmail-mbox-branch.
> 
> So please don't think in terms of conversion to Babyl
> format when designing future Rmail changes.

Great! I'll be learning rmail-mbox-branch now and will come back
with new questions later.

Thanks to everybody for your ideas!


-- 
Alexander Pohoyda <alexander.pohoyda@gmx.net>
PGP Key fingerprint: 7F C9 CC 5A 75 CD 89 72  15 54 5F 62 20 23 C6 44

Re: RMAIL, MIME-related bug

[Eli Zaretskii]

> Date: Thu, 16 Oct 2003 21:33:22 +0200
> From: "Ehud Karni" <ehud@unix.mvs.co.il>
> >
> > Is forwarding such a frequent operation?
> 
> Yes, I do 2-5 resends a day.

I almost never forward a message, let alone one with MIME attachments.

Re: RMAIL, MIME-related bug

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Date: 16 Oct 2003 15:21:40 -0400
> 
> > Is forwarding such a frequent operation?
> 
> The question is not even relevant: it needs to be possible to do it,
> so you need to keep the info, just in case.

If the operation is not frequent, the information can be kept in some
form that doesn't allow an efficient implementation.  So the question
is relevant.

> It might be suboptimal in some cases, but in terms of programmer resources,
> it's pretty close to optimal.  Admittedly, it's not clear whether programmer
> resources is the bottleneck w.r.t MIME decoding given the number of
> different MIME-decoding package have been written in elisp.  Or maybe it is
> the bottleneck and it's a shame that we would write yet-another one rather
> than use a pre-existing one such as the one in Gnus (wisely used by MH-E).

Actually, I think it's a shame that every add-on mail package invented
its own way of doing things instead of contributing the infrastructure
to Emacs.  If they did it the Right Way, we would not need to have
this discussion now, just point Alexander to the existing functions.

Re: RMAIL, MIME-related bug

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Date: Thu, 16 Oct 2003 19:06:49 -0400
> 
>     > When we convert the email to babyl format,
> 
> Paul Riley and I have made a version of Rmail that doesn't use Babyl
> format.  We want to get rid of it entirely.  I think it is currently
> available in a branch called rmail-mbox-branch.
> 
> So please don't think in terms of conversion to Babyl
> format when designing future Rmail changes.

There still is a phase where the messages are processed for display,
right?  At the very least, the modified RMAIL should find the
boundaries of each message and narrow the buffer to those boundaries,
right?  That's the phase I was referring to, the "conversion to Babyl"
is just a convenient name for it, nothing more.

Re: RMAIL, MIME-related bug

[Eli Zaretskii]

> From: Alexander Pohoyda <alexander.pohoyda@gmx.net>
> Date: 16 Oct 2003 23:18:39 +0200
> 
> > > OK, this may not be that important. I just don't see a reason why do we
> > > have to reformat the message just to ignore this reformatting later.
> > > Reformatting only introduces an obstacle. Why to do that?
> > 
> > I think it eases further processing elsewhere in RMAIL.
> 
> Who may know that for sure?

I'd say look in the sources, it's quite clear from that.

Re: RMAIL, MIME-related bug

[Stephen J. Turnbull]

>>>>> "Eli" == Eli Zaretskii <eliz@elta.co.il> writes:

    Eli> Actually, I think it's a shame that every add-on mail package
    Eli> invented its own way of doing things instead of contributing
    Eli> the infrastructure to Emacs.  If they did it the Right Way,
    Eli> we would not need to have this discussion now, just point
    Eli> Alexander to the existing functions.

What do you think tm and SEMI are?


-- 
Institute of Policy and Planning Sciences     http://turnbull.sk.tsukuba.ac.jp
University of Tsukuba                    Tennodai 1-1-1 Tsukuba 305-8573 JAPAN
               Ask not how you can "do" free software business;
              ask what your business can "do for" free software.

Re: RMAIL, MIME-related bug

[Peter S Galbraith]

Eli Zaretskii <eliz@elta.co.il> wrote:

> > From: Stefan Monnier <monnier@IRO.UMontreal.CA>

> > It might be suboptimal in some cases, but in terms of programmer
> > resources, it's pretty close to optimal.  Admittedly, it's not clear
> > whether programmer resources is the bottleneck w.r.t MIME decoding
> > given the number of different MIME-decoding package have been
> > written in elisp.  Or maybe it is the bottleneck and it's a shame
> > that we would write yet-another one rather than use a pre-existing
> > one such as the one in Gnus (wisely used by MH-E).
> 
> Actually, I think it's a shame that every add-on mail package invented
> its own way of doing things instead of contributing the infrastructure
> to Emacs. 

I'm not sure what you say is so true.

 wanderlust -> uses SEMI or tm.
 mew -> ?
 gnus -> implemented their own.
 MH-E -> uses gnus.

I see two major implemenations, and from the little I know the gnus and
FLIM/SEMI camps have a complicated history.  There are still many files
in common between the two.

When we started doing MIME in MH-E, we briefly looked at semi.  I had
trouble making heads or tails of it (my own limitations; the Japanese
web sites didn't help since I couldn't understand them) and since it's
not part of Emacs we would have had to get it to be part of Emacs before
we could use in MH-E.  So we took the gnus route.  We were lucky that
Satyaki Das was familiar with gnus, and he did the vast majority of the
MIME work.

>            If they did it the Right Way, we would not need to have
> this discussion now, just point Alexander to the existing functions.

The only MIME functions currently in Emacs are from gnus, so point him
at those.  Then all three MUAs in Emacs (rmail, gnus, MH-E) will be
using the same thing.  Or you drop rmail and start using MH-E (Okay,
just kidding!).

Peter

-- 
Peter S. Galbraith, MH-E developer  <p.galbraith@globetrotter.net>
GPG key 1024/D2A913A1 - 97CE 866F F579 96EE  6E68 8170 35FF 799E
6623'rd GNU/Linux user at the Counter - http://counter.li.org/

Re: RMAIL, MIME-related bug

[Stefan Monnier]

>> Actually, I think it's a shame that every add-on mail package invented
>> its own way of doing things instead of contributing the infrastructure
>> to Emacs. 

> I'm not sure what you say is so true.

>  wanderlust -> uses SEMI or tm.
>  mew -> ?
>  gnus -> implemented their own.
>  MH-E -> uses gnus.

He was probably also thinking of rmime.el, etach.el, VM, ...


        Stefan

Re: RMAIL, MIME-related bug

[Peter S Galbraith]

Stefan Monnier <monnier@IRO.UMontreal.CA> wrote:

> >> Actually, I think it's a shame that every add-on mail package invented
> >> its own way of doing things instead of contributing the infrastructure
> >> to Emacs. 
> 
> > I'm not sure what you say is so true.
> 
> >  wanderlust -> uses SEMI or tm.
> >  mew -> ?
> >  gnus -> implemented their own.
> >  MH-E -> uses gnus.
> 
> He was probably also thinking of rmime.el, etach.el, VM, ...
> 
> 
>         Stefan

Ah, forgive my ignorance.  I knew of VM (but have never tried it)
and have never heard of the others.  I guess as a MH-E developer I
should try the competition to see what MH-E lacks...  But it doesn't
lack MIME!  :-)

Peter

Re: RMAIL, MIME-related bug

[Eli Zaretskii]

> From: "Stephen J. Turnbull" <stephen@xemacs.org>
> Date: Fri, 17 Oct 2003 20:54:34 +0900
> 
> What do you think tm and SEMI are?

I don't know.  They aren't part of Emacs, so I couldn't possibly mean
them.

Re: RMAIL, MIME-related bug

[Eli Zaretskii]

> Date: Fri, 17 Oct 2003 11:39:24 -0400
> From: Peter S Galbraith <p.galbraith@globetrotter.net>
> 
> >            If they did it the Right Way, we would not need to have
> > this discussion now, just point Alexander to the existing functions.
> 
> The only MIME functions currently in Emacs are from gnus, so point him
> at those.

They should be in lisp/mail/, not in lisp/gnus/.  In other words, they
should be in common Emacs infrastructure, documented in
lispref/*.texi, and all the Emacs mail packages should use them.

Re: RMAIL, MIME-related bug

[Peter S Galbraith]

Eli Zaretskii <eliz@elta.co.il> wrote:

> > Date: Fri, 17 Oct 2003 11:39:24 -0400
> > From: Peter S Galbraith <p.galbraith@globetrotter.net>
> > 
> > >            If they did it the Right Way, we would not need to have
> > > this discussion now, just point Alexander to the existing functions.
> > 
> > The only MIME functions currently in Emacs are from gnus, so point him
> > at those.
> 
> They should be in lisp/mail/, not in lisp/gnus/.  In other words, they
> should be in common Emacs infrastructure, documented in
> lispref/*.texi, and all the Emacs mail packages should use them.

That sounds like a pretty good idea to me.

The drawback for us is the very slow release cycle of Emacs (e.g. the
released version of MH-E with Emacs is two years old).  Currently, some
fancier capabilities of MH-E are only availble if the user has also
installed a separate recent version of gnus since the Emacs version is
too old.  I suppose we would solve that problem by packaging a current
snapshot of the Emacs MIME package (if it existed), but we would have to
maintain a fork if you used code that is not backward compatible with
older versions of Emacs.

-- 
Peter S. Galbraith, MH-E developer  <p.galbraith@globetrotter.net>
GPG key 1024/D2A913A1 - 97CE 866F F579 96EE  6E68 8170 35FF 799E
6623'rd GNU/Linux user at the Counter - http://counter.li.org/

Re: RMAIL, MIME-related bug

[Satyaki Das]

Eli Zaretskii <eliz@elta.co.il> writes:

> Actually, I think it's a shame that every add-on mail package invented
> its own way of doing things instead of contributing the infrastructure
> to Emacs.  If they did it the Right Way, we would not need to have
> this discussion now, just point Alexander to the existing functions.

I don't know about "Right Way", but one way would be to call the
`mm-dissect-buffer' function found in mm-decode.el.

Satyaki

Re: RMAIL, MIME-related bug

[Lars Magne Ingebrigtsen]

"Eli Zaretskii" <eliz@elta.co.il> writes:

> Actually, I think it's a shame that every add-on mail package invented
> its own way of doing things instead of contributing the infrastructure
> to Emacs.  If they did it the Right Way, we would not need to have
> this discussion now, just point Alexander to the existing functions.

The MIME library developed for Gnus was designed as a general-purpose
Emacs MIME library.  (Which is why it's documented in the Emacs-MIME
manual.) 

Some Gnus stuff has crept into the libraries, but that should be
fixable.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: RMAIL, MIME-related bug

[Richard Stallman]

I have already decided to switch to the new version of Rmail
that uses mbox format instead of Babyl format.  It is waiting
for more testing and debugging.  New MIME support needs to be
designed to work with mbox format.

Preserving mime messages in their original transferred format is the
only acceptable way to handle them.  Machines now so fast that it
should not take any substantial time to decode base64 or quoted
printables, so we will do that when displaying a message.

So my decision is that Rmail will continue to store MIME
contents in its undecoded form.  Please do not keep arguing
for other methods, because my mind is made up.

Re: RMAIL, MIME-related bug

[Stefan Monnier]

> that uses mbox format instead of Babyl format.  It is waiting
> for more testing and debugging.

Maybe installing on the CVS trunk would result in more testing (and
debugging) ?

BTW, when's the feature freeze ?


        Stefan

Re: RMAIL, MIME-related bug

[Eli Zaretskii]

> From: Lars Magne Ingebrigtsen <larsi@gnus.org>
> Date: Fri, 17 Oct 2003 20:57:46 +0200
> 
> The MIME library developed for Gnus was designed as a general-purpose
> Emacs MIME library.

With all due respect, ``designed as'' is not the same as ``is''.  As
long as a library is not sent upstream to the Emacs maintainers and is
not described in the ELisp manual, it cannot be considered part of
Emacs infrastructure, in my opinion.

> (Which is why it's documented in the Emacs-MIME manual.)

That's a separate manual, not part of the ELisp manual.  Therefore, a
programmer who is looking for infrastructure that's part of Emacs will
not find this library.

In other words, a general-purpose library should not be part of a
package that is maintained outside Emacs.  IMHO, of course; YMMV.

Re: RMAIL, MIME-related bug

[Richard Stallman]

    With all due respect, ``designed as'' is not the same as ``is''.  As
    long as a library is not sent upstream to the Emacs maintainers and is
    not described in the ELisp manual, it cannot be considered part of
    Emacs infrastructure, in my opinion.

That is not our policy.  Something can be in a separate manual
and be a recommended part of Emacs facilities.

We typically do not document specialized Emacs libraries in the Emacs
Lisp manual, because it isn't useful to print the info about them on
paper.  We certainly would not document a MIME library in the Emacs
Lisp manual, because it is useful only for mail readers, and that is
too specialized.  That doesn't mean it isn't part of the standard
Emacs infrastructure.

However, it needs to be maintained in the Emacs CVS repository
if we are going to depend on it.

Re: RMAIL, MIME-related bug

[Richard Stallman]

    > that uses mbox format instead of Babyl format.  It is waiting
    > for more testing and debugging.

    Maybe installing on the CVS trunk would result in more testing (and
    debugging) ?

It certainly would, but I think it needs a bit more testing first.
Could someone please try it out?

Re: RMAIL, MIME-related bug

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Date: Sat, 18 Oct 2003 19:05:29 -0400
> 
> Something can be in a separate manual and be a recommended part of
> Emacs facilities.

I don't think it ever worked in practice (not the recommendation
part--that's easy--but the actual use part).  Do you have any
examples?

> We certainly would not document a MIME library in the Emacs Lisp
> manual, because it is useful only for mail readers, and that is too
> specialized.

It is no less specialized than primitives like base64-decode-region.

Re: RMAIL, MIME-related bug

[Alexander Pohoyda]

Richard Stallman <rms@gnu.org> writes:

>     > that uses mbox format instead of Babyl format.  It is waiting
>     > for more testing and debugging.
> 
>     Maybe installing on the CVS trunk would result in more testing (and
>     debugging) ?
> 
> It certainly would, but I think it needs a bit more testing first.
> Could someone please try it out?

Yes, I'm going to do that.

-- 
Alexander Pohoyda <alexander.pohoyda@gmx.net>
PGP Key fingerprint: 7F C9 CC 5A 75 CD 89 72  15 54 5F 62 20 23 C6 44

Re: RMAIL, MIME-related bug

[Alex Schroeder]

Eli Zaretskii <eliz@elta.co.il> writes:

> I don't think it ever worked in practice (not the recommendation
> part--that's easy--but the actual use part).  Do you have any
> examples?

I've used the widget library before (ie. outside of custom).

Alex.
-- 
http://www.emacswiki.org/alex/
There is no substitute for experience.

Re: RMAIL, MIME-related bug

[Juri Linkov]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:
> Why do people even consider doing any processing at that point?
> The processing should be done for display and nothing else, because any
> other option will make you lose information at some point and also forces
> you to have code to re-create the original format, whereas the "display-side
> processing" approach only needs to parse MIME but never needs to
> re-construct it.

I will not argue about the default mbox format (it's pretty obvious that
messages should be preserved in their original transferred format).
But I'll tell about functionality still missing in Emacs.

1. Often there is a need to grep the mail archive for some old message.
This is impossible if message is encoded in base64/quoted-printable.

2. Sometimes there is a need to make a bookmark to the message to be
able to quickly return to it later.  Hopeless.  Only one way to fetch
old message is to start MUA and browse through dozens of folders and
hundreds of messages in every folder.

I see one solution to these problems - to export entire mbox to the
file system, i.e. to save MIME parts to separate files, decode
base64/quoted-printable regions, convert the charset from Content-Type
line to Emacs encoding, add corresponding -*- coding: -*- line, and
save fully decoded message to a separate file.

Note that this don't contradict the decision to store messages in mbox
in their undecoded form.  The proposed solution duplicates the content
of messages in the decoded form.  Hard disks now so cheap that having
two copies of the same message is not a problem.  The decoded messages
could also be used as a cache to make displaying messages faster,
whereas undecoded messages will be available to resend forwards, where
no original information should be lost.  There should be some way to
correlate between them, i.e. to find corresponding decoded message
from undecoded one, and vice versa.

-- 
http://www.jurta.org/emacs/

Re: RMAIL, MIME-related bug

[Richard Stallman]

    > We certainly would not document a MIME library in the Emacs Lisp
    > manual, because it is useful only for mail readers, and that is too
    > specialized.

    It is no less specialized than primitives like base64-decode-region.

Those are primitives, and they are simple to document.

Anyway, I'm just telling you what our policy is.  Not every Emacs Lisp
facility has to be documented in the Emacs Lisp manual.

Re: RMAIL, MIME-related bug

[Kenichi Handa]

Juri Linkov <juri@jurta.org> writes:
> I see one solution to these problems - to export entire mbox to the
> file system, i.e. to save MIME parts to separate files, decode
> base64/quoted-printable regions, convert the charset from Content-Type
> line to Emacs encoding, add corresponding -*- coding: -*- line, and
> save fully decoded message to a separate file.

> Note that this don't contradict the decision to store messages in mbox
> in their undecoded form.  The proposed solution duplicates the content
> of messages in the decoded form.  Hard disks now so cheap that having
> two copies of the same message is not a problem.  The decoded messages
> could also be used as a cache to make displaying messages faster,
> whereas undecoded messages will be available to resend forwards, where
> no original information should be lost.  There should be some way to
> correlate between them, i.e. to find corresponding decoded message
> from undecoded one, and vice versa.

This method is very similar to what I proposed in my
previous mail.

Kenichi Handa <handa@m17n.org> writes:
> (1) Read RMAIL file without decoding into some hidden source
>     buffer (unibyte).  It may be ok to process only
>     Content-Transfer-Encoding.
> (2) Prepare a view buffer.
> (3) Insert the current message in the view buffer after decoding it.
> (4) A background process (run by idle timer?) decodes not
>     yet decoded message into the view buffer (like
>     jit-lock-stealth-fontify).

Mine doesn't save the view buffer in a file, yours does.

---
Ken'ichi HANDA
handa@m17n.org

Re: RMAIL, MIME-related bug

[Juri Linkov]

Kenichi Handa <handa@m17n.org> writes:
> Juri Linkov <juri@jurta.org> writes:
> > I see one solution to these problems - to export entire mbox to the
> > file system, i.e. to save MIME parts to separate files, decode
> > base64/quoted-printable regions, convert the charset from Content-Type
> > line to Emacs encoding, add corresponding -*- coding: -*- line, and
> > save fully decoded message to a separate file.
> 
> > Note that this don't contradict the decision to store messages in mbox
> > in their undecoded form.  The proposed solution duplicates the content
> > of messages in the decoded form.  Hard disks now so cheap that having
> > two copies of the same message is not a problem.  The decoded messages
> > could also be used as a cache to make displaying messages faster,
> > whereas undecoded messages will be available to resend forwards, where
> > no original information should be lost.  There should be some way to
> > correlate between them, i.e. to find corresponding decoded message
> > from undecoded one, and vice versa.
> 
> This method is very similar to what I proposed in my
> previous mail.
> 
> Kenichi Handa <handa@m17n.org> writes:
> > (1) Read RMAIL file without decoding into some hidden source
> >     buffer (unibyte).  It may be ok to process only
> >     Content-Transfer-Encoding.
> > (2) Prepare a view buffer.
> > (3) Insert the current message in the view buffer after decoding it.
> > (4) A background process (run by idle timer?) decodes not
> >     yet decoded message into the view buffer (like
> >     jit-lock-stealth-fontify).
> 
> Mine doesn't save the view buffer in a file, yours does.

Yes, this method is very similar to yours.  However, I added saving of
decoded messages in files to solve the mentioned problems:

1. Mail archives with fully decoded messages can be grepped, if they are
   stored in files;

2. It's possible to refer to separate messages as to ordinary files.

-- 
http://www.jurta.org/emacs/

Re: RMAIL, MIME-related bug

[Alexander Pohoyda]

Kenichi Handa <handa@m17n.org> writes:

> > I see one solution to these problems - to export entire mbox to
> > the file system, i.e. to save MIME parts to separate files, decode
> > base64/quoted-printable regions, convert the charset from
> > Content-Type line to Emacs encoding, add corresponding -*- coding:
> > -*- line, and save fully decoded message to a separate file.
> 
> > Note that this don't contradict the decision to store messages in
> > mbox in their undecoded form.  The proposed solution duplicates
> > the content of messages in the decoded form.  Hard disks now so
> > cheap that having two copies of the same message is not a problem.
> > The decoded messages could also be used as a cache to make
> > displaying messages faster, whereas undecoded messages will be
> > available to resend forwards, where no original information should
> > be lost.  There should be some way to correlate between them,
> > i.e. to find corresponding decoded message from undecoded one, and
> > vice versa.
 
> This method is very similar to what I proposed in my
> previous mail.
> 
> Kenichi Handa <handa@m17n.org> writes:
> > (1) Read RMAIL file without decoding into some hidden source
> >     buffer (unibyte).  It may be ok to process only
> >     Content-Transfer-Encoding.
> > (2) Prepare a view buffer.
> > (3) Insert the current message in the view buffer after decoding
> >     it. 
> > (4) A background process (run by idle timer?) decodes not
> >     yet decoded message into the view buffer (like
> >     jit-lock-stealth-fontify).
> 
> Mine doesn't save the view buffer in a file, yours does.

I like these ideas very much and I'll keep them in my mind.
Thank you very much!


-- 
Alexander Pohoyda <alexander.pohoyda@gmx.net>
PGP Key fingerprint: 7F C9 CC 5A 75 CD 89 72  15 54 5F 62 20 23 C6 44

Re: RMAIL, MIME-related bug

[Alexander Pohoyda]

Richard Stallman <rms@gnu.org> writes:

>     > We certainly would not document a MIME library in the Emacs Lisp
>     > manual, because it is useful only for mail readers, and that is too
>     > specialized.
> 
>     It is no less specialized than primitives like base64-decode-region.
> 
> Those are primitives, and they are simple to document.
> 
> Anyway, I'm just telling you what our policy is.  Not every Emacs Lisp
> facility has to be documented in the Emacs Lisp manual.

After I implemented it myself, I have found 2 more implementations of
quoted-printable primitives in Emacs/lisp tree.
Guess why? Because "Describe function" had no idea that they are
already implemented and I haven't found them in Emacs Lisp manual.

Would I even found them at the beginning, I would not dare to use the
one from Gnus, because:
* it belongs to different project in my eyes.
* gnus/qp.el requires mm-util, which I neither need nor want right now.

base64 functions are built-in, though.


2Lars: Only first 7 lines of the References header field are
highlighted in Gnus v5.9.0 and sorry, if this is already fixed.

-- 
Alexander Pohoyda <alexander.pohoyda@gmx.net>
PGP Key fingerprint: 7F C9 CC 5A 75 CD 89 72  15 54 5F 62 20 23 C6 44

Re: RMAIL, MIME-related bug

[Lars Magne Ingebrigtsen]

Alexander Pohoyda <alexander.pohoyda@gmx.net> writes:

> After I implemented it myself, I have found 2 more implementations of
> quoted-printable primitives in Emacs/lisp tree.

Where did the second implementation live?

> Guess why? Because "Describe function" had no idea that they are
> already implemented and I haven't found them in Emacs Lisp manual.

Well, ok, but I find it somewhat puzzling that anybody would think
that Emacs didn't have something as basic as qp en/decoding somewhere
in its vast mass.

> Would I even found them at the beginning, I would not dare to use the
> one from Gnus, because:
> * it belongs to different project in my eyes.
> * gnus/qp.el requires mm-util, which I neither need nor want right now.

I think a better solution than writing yet another implementation
would be to move gnus/qp.el to mime/qp.el, and remove the
dependencies on mm-util (which is basically a compatibility library
between a bewildering array of X/Emacs versions).

> 2Lars: Only first 7 lines of the References header field are
> highlighted in Gnus v5.9.0 and sorry, if this is already fixed.

You mean in Message?  That's handled by font-lock, which has a
tendency to stop highlighting extremely long lines after a certain
point.  I'm sure it's customizable.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: RMAIL, MIME-related bug

[Lars Magne Ingebrigtsen]

Juri Linkov <juri@jurta.org> writes:

> I see one solution to these problems - to export entire mbox to the
> file system, i.e. to save MIME parts to separate files, decode
> base64/quoted-printable regions, convert the charset from Content-Type
> line to Emacs encoding, add corresponding -*- coding: -*- line, and
> save fully decoded message to a separate file.

And all that just to be able to use grep?  Why not have the mail
reader be able to do searches for you instead?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: RMAIL, MIME-related bug

[Juri Linkov]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
> Juri Linkov <juri@jurta.org> writes:
> > I see one solution to these problems - to export entire mbox to the
> > file system, i.e. to save MIME parts to separate files, decode
> > base64/quoted-printable regions, convert the charset from Content-Type
> > line to Emacs encoding, add corresponding -*- coding: -*- line, and
> > save fully decoded message to a separate file.
> 
> And all that just to be able to use grep?  Why not have the mail
> reader be able to do searches for you instead?

Searches done by mail reader are not effective (or I'd rather say they are
"effective" with the first letter "d", i.e. "defective" :-), because
mail reader needs to parse MIME parts and decode base64/qp regions on
every of hundreds messages, and repeat all this again on every new search.

-- 
http://www.jurta.org/emacs/

Re: RMAIL, MIME-related bug

[Lars Magne Ingebrigtsen]

Juri Linkov <juri@jurta.org> writes:

>> And all that just to be able to use grep?  Why not have the mail
>> reader be able to do searches for you instead?
>
> Searches done by mail reader are not effective (or I'd rather say they are
> "effective" with the first letter "d", i.e. "defective" :-), because
> mail reader needs to parse MIME parts and decode base64/qp regions on
> every of hundreds messages, and repeat all this again on every new search.

Perhaps, perhaps not.  Some mailers (I've been told) generate indexes
when they receive messages.  When you do searches, the mailer looks
stuff up in the index.

Grepping several GBs worth of mail isn't very efficient.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: RMAIL, MIME-related bug

[Richard Stallman]

    > They should be in lisp/mail/, not in lisp/gnus/.  In other words, they
    > should be in common Emacs infrastructure, documented in
    > lispref/*.texi, and all the Emacs mail packages should use them.

If they are a general facility, they should definitely be moved;
the main repository for them should be the Emacs repository.

      Currently, some
    fancier capabilities of MH-E are only availble if the user has also
    installed a separate recent version of gnus since the Emacs version is
    too old.

If you distribute MH-E outside of Emacs, you can distribute whatever
you like with it.

Re: RMAIL, MIME-related bug

[Robert J. Chassell]

   Juri Linkov <juri@jurta.org> writes:
   > I see one solution to these problems - to export entire mbox to the
   > file system, i.e. to save MIME parts to separate files, ....

   > Note that this don't contradict the decision to store messages in mbox
   > in their undecoded form.  ....

It is useful to use the terms `deep representation' and `surface
expressions' to distinguish between the various formats.

Think of the mbox as the `deep representation' of the set of messages.
Much of it is readable using `find-file-literally' as it should be.
But various `surface expressions' make reading the file easier, and
decode images or base64 encoded material as needed.  `Surface
expressions' can be cached -- that can be an option.

And, of course,  a particular  `surface expression', such as a Babyl
file format, can be a  `deep representation' for further processing,
such as one to show an image.

In any event, a `surface expression' may be suited to audio, using
Emacspeak, or suited to rmail, or gnus, or...

This method is similar to Kenichi Handa's proposal, excepting that he
speaks of a `view buffer', which is a good phrase but does not
persuade programmers to think of the blind.  

(Emacspeak may well use a `view buffer' -- and not deal with images in
it.  The idea is to apply to mbox format the existing way in which
people think about Texinfo, in which the `deep representation', the
Texinfo source, has many `surface expressions', such as Info, DVI,
HTML, PostScript, and plain text.)

The `deep representation/surface expressions' way of thinking goes
against traditional Emacs style:  in that style, the `surface
expression' of a buffer was very closely linked to its `deep
representation'.  The characters you saw on the screen represented the
bytes in the file.

It is vital to keep this ability.  Just yesterday, I had to look at a
mail file using `find-file-literally' to discover what was giving me
trouble.  But mostly, I do not use `find-file-literally' but look at
the `surface expression' produced by rmail.  Obviously, this `surface
expression' is not hugely different from the mail file's `deep
representation', but it is somewhat different.

-- 
    Robert J. Chassell                         Rattlesnake Enterprises
    http://www.rattlesnake.com                  GnuPG Key ID: 004B4AC8
    http://www.teak.cc                             bob@rattlesnake.com

Re: RMAIL, MIME-related bug

[Paul Michael Reilly]

Lars Magne Ingebrigtsen wrote:

>Juri Linkov <juri@jurta.org> writes:
>
>  
>
>>I see one solution to these problems - to export entire mbox to the
>>file system, i.e. to save MIME parts to separate files, decode
>>base64/quoted-printable regions, convert the charset from Content-Type
>>line to Emacs encoding, add corresponding -*- coding: -*- line, and
>>save fully decoded message to a separate file.
>>    
>>
>
>And all that just to be able to use grep?  Why not have the mail
>reader be able to do searches for you instead?
>
That's the right answer.  Using grep is fine to the extent that one 
cannot really expect grep to grok the internals of a mail application.  
IMAP provides a decent model for mail searching.  Gnus might as well.  
I'm just not as familiar with Gnus as I am with IMAP.

-pmr

Re: RMAIL, MIME-related bug

[Satyaki Das]

Juri Linkov <juri@jurta.org> writes:

> 1. Often there is a need to grep the mail archive for some old message.
> This is impossible if message is encoded in base64/quoted-printable.

I am a huge fan of mail indexing programs like: swish++, mairix,
glimpse, namazu etc. Many of these will decode base64 and
quoted-printable parts. Also they are much faster than grep can
ever hope to be.

Many Emacs based email clients (like Gnus, MH-E) have interfaces
to these programs. May be rmail can be improved to support these
programs?

Satyaki

Re: RMAIL, MIME-related bug

[Alexander Pohoyda]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> > After I implemented it myself, I have found 2 more implementations
> > of quoted-printable primitives in Emacs/lisp tree.
> Where did the second implementation live?

mail/rmail.el, rmail-decode-quoted-printable function.

I'd be very thankful if you merge them. Also, could we please agree
that these functions should return the size of en/de-coded text, the
same way as their base64-... counterparts?


> > Guess why? Because "Describe function" had no idea that they are
> > already implemented and I haven't found them in Emacs Lisp
> > manual. 
> Well, ok, but I find it somewhat puzzling that anybody would think
> that Emacs didn't have something as basic as qp en/decoding
> somewhere in its vast mass.

I was easily able to locate base64, thus the absence of
quoted-printable in base installation made me believe that it was
intentionally left out.


> > Would I even found them at the beginning, I would not dare to use
> > the one from Gnus, because:
> > * it belongs to different project in my eyes.
> > * gnus/qp.el requires mm-util, which I neither need nor want right
> > now. 
> I think a better solution than writing yet another implementation
> would be to move gnus/qp.el to mime/qp.el, and remove the

Why not mail/qp.el instead? The mime/ directory is going to be quite
empty :-)


> > 2Lars: Only first 7 lines of the References header field are
> > highlighted in Gnus v5.9.0 and sorry, if this is already fixed.
> You mean in Message?  That's handled by font-lock, which has a
> tendency to stop highlighting extremely long lines after a certain
> point.  I'm sure it's customizable.  :-)

Indeed. Thank you! :-)


-- 
Alexander Pohoyda <alexander.pohoyda@gmx.net>
PGP Key fingerprint: 7F C9 CC 5A 75 CD 89 72  15 54 5F 62 20 23 C6 44

Re: RMAIL, MIME-related bug

[Peter S Galbraith]

Richard Stallman <rms@gnu.org> wrote:

>     > They should be in lisp/mail/, not in lisp/gnus/.  In other words, they
>     > should be in common Emacs infrastructure, documented in
>     > lispref/*.texi, and all the Emacs mail packages should use them.
> 
> If they are a general facility, they should definitely be moved;
> the main repository for them should be the Emacs repository.
> 
>       Currently, some
>     fancier capabilities of MH-E are only availble if the user has also
>     installed a separate recent version of gnus since the Emacs version is
>     too old.
> 
> If you distribute MH-E outside of Emacs, you can distribute whatever
> you like with it.

We distribure a few contrib files that are not part of MH-E proper, but
we let users fetch gnus for themselves.  (On Debian systems, the mh-e
package simply recommends the gnus package.)

Peter

Re: RMAIL, MIME-related bug

[Richard Stallman]

    1. Often there is a need to grep the mail archive for some old message.
    This is impossible if message is encoded in base64/quoted-printable.

It just means it might be useful for the program that searches to know
how to decode attachments.

    2. Sometimes there is a need to make a bookmark to the message to be
    able to quickly return to it later.

That could mean various things--I don't understand why you think it is
so hard.

       Only one way to fetch
    old message is to start MUA and browse through dozens of folders and
    hundreds of messages in every folder.

That is certainly not the only way.  The way I normally do this is
with grep.  It works pretty well, but it could be better.

I think we could use a program designed to search through mail folders
in mbox format.  Such a program could be convenient in many ways.  For
instance, it could have a feature to decode base64 and
quoted-printable encodings.  It could also have the ability to search
for matches against specific header fields, or against the body, and
handle boolean combinations of such criteria.

It should output the messages that meet the specified criterion, and
add to each message something in the form of a header field that says
which file it came from and which line number.  That way another
program could look at the output and find the relevant messages in the
files where they originally came from.

If the output gives you too many messages, you could apply the search
program to the output of the previous run, to filter it further.

It would not be a very big job to write this program by adapting grep.
Would anyone like to try?

Re: RMAIL, MIME-related bug

[Richard Stallman]

    > Would I even found them at the beginning, I would not dare to use the
    > one from Gnus, because:
    > * it belongs to different project in my eyes.
    > * gnus/qp.el requires mm-util, which I neither need nor want right now.

    I think a better solution than writing yet another implementation
    would be to move gnus/qp.el to mime/qp.el,

We already have functions to do this in mailutils, so we
should standardize on those, unless the other set has some
advantage that would be an argument for preferring them.

Re: RMAIL, MIME-related bug

[Kevin Rodgers]

Juri Linkov wrote:

> 2. Sometimes there is a need to make a bookmark to the message to be
> able to quickly return to it later.  Hopeless.  Only one way to fetch
> old message is to start MUA and browse through dozens of folders and
> hundreds of messages in every folder.

I did a little work on this issue about 2 1/2 years ago.  My approach
was to define a bookmark as a URL whose protocol is "mailmessage", so
that I could use Emacs' url.el library to parse it and ange-ftp.el to
access remote //[USER@]HOST folders.  However, I wrote it as an add-on
to VM, so the code that actually views the folder and selects the
message would have to be converted to Rmail.  And it would have to be
extended to handle IMAP folders (cf. file folders).

-- 
Kevin Rodgers

Re: RMAIL, MIME-related bug

[Stephen J. Turnbull]

>>>>> "Eli" == Eli Zaretskii <eliz@elta.co.il> writes:

    >> From: "Stephen J. Turnbull" <stephen@xemacs.org> Date: Fri, 17
    >> Oct 2003 20:54:34 +0900

    >> What do you think tm and SEMI are?

    Eli> I don't know.  They aren't part of Emacs, so I couldn't
    Eli> possibly mean them.

I know.  The question is why aren't they?  SEMI is semi-encumbered
(the same legal issues that prevented Mule from being assigned to the
FSF), but tm (SEMI's predecessor) was written when Tomohiko Morioka
was a student, dating back to 1995 or so.  He told me that he
certainly hoped they would be distributed with Emacs and XEmacs.  I
don't know why Emacs never bothered to pick them up; I'm sure Tomo
would have signed an assignment, and SEMI could have been picked up
more or less "for free" along with Mule since it was done by Tomo
while he was employed at ETL AFAIK.

The VM libraries were written at a time when Emacs didn't support
images or audio, and Mule had been explicitly rejected.  It simply
wasn't possible to contribute them to Emacs, as they would have been
meaningless.  (Of course Kyle would never sign an assignment, but even
if Kyle would, it would have been useless effort to support FSF GNU
Emacs then.  NB: There were GNU Emacsen being distributed with the
Mule patch, and vm supported those, as it supports GNU Emacs today.)

w3.el's MIME support was also explicitly oriented toward Emacsen with
internal multimedia support.

So we come to Gnus.  The Gnus libraries were written post-Mule, with
generality clearly in mind, and will be included with Emacs in the
future.  MH-E clearly made the right decision in picking them as the
basis for its own MIME support.

So I don't see what your problem is.  Emacs's lack of a generic MIME
library seems to be just another of the tradeoffs that have to be made
in world with limited resources.  I think it was a mistake, but that's
20-20 hindsight.  In any case the lack of a generic MIME library will
soon be corrected, except that vm probably will continue to use its
idiosyncratic libraries for its own idiosyncratic reasons.

-- 
Institute of Policy and Planning Sciences     http://turnbull.sk.tsukuba.ac.jp
University of Tsukuba                    Tennodai 1-1-1 Tsukuba 305-8573 JAPAN
               Ask not how you can "do" free software business;
              ask what your business can "do for" free software.

Re: RMAIL, MIME-related bug

[Simon Josefsson]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

>>>>>> "Eli" == Eli Zaretskii <eliz@elta.co.il> writes:
>
>     >> From: "Stephen J. Turnbull" <stephen@xemacs.org> Date: Fri, 17
>     >> Oct 2003 20:54:34 +0900
>
>     >> What do you think tm and SEMI are?
>
>     Eli> I don't know.  They aren't part of Emacs, so I couldn't
>     Eli> possibly mean them.
>
> I know.  The question is why aren't they?

Wasn't tm abandoned 1998 or so?  I hope that some parts of SEMI that
we managed to get copyright papers for, will be moving into Emacs as
part of Gnus.  For example PGG (part of Gnus 5.10) and SASL (hopefully
will be part of the next development cycle of Gnus).  The main problem
with SEMI code is that it is assumes APEL.

Generally, I'm not sure there is much in tm/SEMI that is not already
supported by Emacs, via Gnus or some other package.  Do you have
example of some practical benefit (i.e., feature-wise, visible to
users) adding SEMI to Emacs actually would have?

Re: RMAIL, MIME-related bug

[Karl Eichwalder]

Richard Stallman <rms@gnu.org> writes:

> It should output the messages that meet the specified criterion, and
> add to each message something in the form of a header field that says
> which file it came from and which line number.

As a Gnus user I'd like to see results end up in a virtual Group
(folder).

> If the output gives you too many messages, you could apply the search
> program to the output of the previous run, to filter it further.

Gnus already offers useful search capabilities :)

> It would not be a very big job to write this program by adapting grep.

To match against header fields and similar elements you can use
http://www.cs.helsinki.fi/~jjaakkol/sgrep.html

-- 
                                                         |      ,__o
                                                         |    _-\_<,
http://www.gnu.franken.de/ke/                            |   (*)/'(*)

Re: RMAIL, MIME-related bug

[Juri Linkov]

Kevin Rodgers <ihs_4664@yahoo.com> writes:
> Juri Linkov wrote:
> > 2. Sometimes there is a need to make a bookmark to the message to be
> > able to quickly return to it later.  Hopeless.  Only one way to fetch
> > old message is to start MUA and browse through dozens of folders and
> > hundreds of messages in every folder.
> 
> I did a little work on this issue about 2 1/2 years ago.  My approach
> was to define a bookmark as a URL whose protocol is "mailmessage", so
> that I could use Emacs' url.el library to parse it and ange-ftp.el to
> access remote //[USER@]HOST folders.  However, I wrote it as an add-on
> to VM, so the code that actually views the folder and selects the
> message would have to be converted to Rmail.  And it would have to be
> extended to handle IMAP folders (cf. file folders).

Using URL for references to messages is a good solution, but instead
of inventing ad hoc protocol name, it's better to use standard name
"mid" defined by http://www.ietf.org/rfc/rfc2392.txt.

Then, for example, opening the link <mid:3F9471FB.4000609@yahoo.com>
should start a mail reader, find corresponding message by its
Message-ID, prepare article buffer and switch to it (e.g. in case of
Gnus to call the function `gnus-summary-goto-article'; other mail
readers should have a similar function).

I'm not sure yet, where this could be implemented: url.el is not part
of Emacs (BTW: Why?); thingatpt.el mentions this protocol name, but
that's all; perhaps this should be done partially in ffap.el.

-- 
http://www.jurta.org/emacs/

Re: RMAIL, MIME-related bug

[Stephen J. Turnbull]

>>>>> "Simon" == Simon Josefsson <jas@extundo.com> writes:

    Simon> Wasn't tm abandoned 1998 or so?

Yes.

    Simon> Do you have example of some practical benefit (i.e.,
    Simon> feature-wise, visible to users) adding SEMI to Emacs
    Simon> actually would have?

Not any more.  Eli was remarking that a lot of duplicate effort could
have been avoided if there were a standard MIME library.  AFAIK, tm
was on offer in 1996-97, but nobody did anything about it.  Ditto SEMI
in '97-99.  Unfortunate lack of foresight, that's all.

-- 
Institute of Policy and Planning Sciences     http://turnbull.sk.tsukuba.ac.jp
University of Tsukuba                    Tennodai 1-1-1 Tsukuba 305-8573 JAPAN
               Ask not how you can "do" free software business;
              ask what your business can "do for" free software.

Re: RMAIL, MIME-related bug

[Lars Magne Ingebrigtsen]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> Not any more.  Eli was remarking that a lot of duplicate effort could
> have been avoided if there were a standard MIME library.  AFAIK, tm
> was on offer in 1996-97, but nobody did anything about it.  Ditto SEMI
> in '97-99.  Unfortunate lack of foresight, that's all.

Well, yes and no.  When I started working on the MIME library for
Gnus, I had intended using SEMI.  However, that proved rather
difficult due to how SEMI was designed: It isn't (or wasn't at that
time) structured anything like a library.  Instead, it worked by
redefining functions/hooks in the packages it supported.  You'd
basically require the relevant SEMI packages, and Gnus/rmail/whatever
would suddenly become MIME capable.

That's a perfectly valid approach if you intend being an add-on
package with no cooperation from the packages you support, but it
makes it rather difficult to maintain things and integrate things in
a more user-friendly (and customizable) fashion.

I spent about a week looking at SEMI and trying to pick stuff out of
it, but in the end I gave up and started writing things (basically)
from scratch.

I may be misremembering, but I think Kyle went through a similar
process about the same time with vm. 

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: RMAIL, MIME-related bug

[Richard Stallman]

    I am a huge fan of mail indexing programs like: swish++, mairix,
    glimpse, namazu etc.

I seem to recall that glimpse is or was non-free.  I don't know about
the others.  Can you check which of these are non-free?  We don't
want Gnus or MH-E to refer to them.

Re: RMAIL, MIME-related bug

[Satyaki Das]

Richard Stallman <rms@gnu.org> writes:

>     I am a huge fan of mail indexing programs like: swish++, mairix,
>     glimpse, namazu etc.
> 
> I seem to recall that glimpse is or was non-free.  I don't know about
> the others.  Can you check which of these are non-free?  We don't
> want Gnus or MH-E to refer to them.

MH-E supports the use of swish++, swish-e, mairix, namazu and
glimpse. Except for glimpse the rest are all free and distributed
under the GPL.

We only have a customizable option that allows the user to choose
to use glimpse.

Satyaki

Re: RMAIL, MIME-related bug

[Richard Stallman]

    I know.  The question is why aren't they?  SEMI is semi-encumbered
    (the same legal issues that prevented Mule from being assigned to the
    FSF),

What specifically is the problem?  If it is work done at ETL, that
may not really be a problem.  Since we accepted their Mule code,
we can accept some more with no difficulty.

I think we intended to install SEMI, but the work of integrating
it never got done.

Now we have the option of using SEMI or adopting the Gnus code.  To
make that decision, someone who knows Emacs Lisp well and is not
already a partisan of either one should study them both and compare
them.  Is there someone who would like to do that?

Re: RMAIL, MIME-related bug

[Richard Stallman]

    > It should output the messages that meet the specified criterion, and
    > add to each message something in the form of a header field that says
    > which file it came from and which line number.

    As a Gnus user I'd like to see results end up in a virtual Group
    (folder).

The program I envision is a shell command.  How Gnus might invoke it
is a separate issue.  Since I don't know Gnus, I won't say anything
about that.

Re: RMAIL, MIME-related bug

[Stephen J. Turnbull]

>>>>> "rms" == Richard Stallman <rms@gnu.org> writes:

    rms> What specifically is the problem?  If it is work done at ETL,
    rms> that may not really be a problem.  Since we accepted their
    rms> Mule code, we can accept some more with no difficulty.

That is exactly what I meant; it's just that at the time that SEMI was
being developed, Mule had not yet been accepted into Emacs.  I don't
think there are any problems now (except that Morioka-san's current
employer, the University of Kyoto, probably has an interest in more
recently developed code).


-- 
Institute of Policy and Planning Sciences     http://turnbull.sk.tsukuba.ac.jp
University of Tsukuba                    Tennodai 1-1-1 Tsukuba 305-8573 JAPAN
               Ask not how you can "do" free software business;
              ask what your business can "do for" free software.

Re: RMAIL, MIME-related bug

[Kenichi Handa]

In article <E1ACFFL-0004gm-Pm@fencepost.gnu.org>, Richard Stallman <rms@gnu.org> writes:
>     I know.  The question is why aren't they?  SEMI is semi-encumbered
>     (the same legal issues that prevented Mule from being assigned to the
>     FSF),

> What specifically is the problem?  If it is work done at ETL, that
> may not really be a problem.  Since we accepted their Mule code,
> we can accept some more with no difficulty.

Although Morioka-san, the main programmer of SEMI, stayed at
ETL for one year, ETL is not a copyright holder of SEMI.  We
already have assignment papers from the developpers, but
that was several years ago.  I don't know how many other
programers contributed code to the current SEMI, FLIM, and
APEL (three cooperating packages).

I included Morioka-san's address in CC:.  Morioka-san,
please give us the information about the current status of
those packages.  Does the assignment papers already sent to
FSF cover the whole codes (except tiny changes)?  If not, is
it possible to get papers from the other programmers?  FYI,
you can read previous mails of this thread at:
	http://mail.gnu.org/archive/html/emacs-devel/

---
Ken'ichi HANDA
handa@m17n.org

> I think we intended to install SEMI, but the work of integrating
> it never got done.

> Now we have the option of using SEMI or adopting the Gnus code.  To
> make that decision, someone who knows Emacs Lisp well and is not
> already a partisan of either one should study them both and compare
> them.  Is there someone who would like to do that?

Re: RMAIL, MIME-related bug

[Alexander Pohoyda]

Richard Stallman <rms@gnu.org> writes:

> Now we have the option of using SEMI or adopting the Gnus code.  To
> make that decision, someone who knows Emacs Lisp well and is not
> already a partisan of either one should study them both and compare
> them.  Is there someone who would like to do that?

Where can I get SEMI?


-- 
Alexander Pohoyda <alexander.pohoyda@gmx.net>
PGP Key fingerprint: 7F C9 CC 5A 75 CD 89 72  15 54 5F 62 20 23 C6 44

Re: RMAIL, MIME-related bug

[Lars Magne Ingebrigtsen]

Alexander Pohoyda <alexander.pohoyda@gmx.net> writes:

> Where can I get SEMI?

<URL: http://emacs-21.ki.nu/application/semi.html> has the latest
versions of SEMI/APEL/FLIM, I think, but I might be wrong about that. 

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: RMAIL, MIME-related bug

[Kenichi Handa]

In article <87smllb370.fsf@oak.pohoyda.family>, Alexander Pohoyda <alexander.pohoyda@gmx.net> writes:

> Where can I get SEMI?

Please try these ftp directories.

ftp://ftp.m17.org/pub/mule/semi/
ftp://ftp.m17.org/pub/mule/flim/
ftp://ftp.m17.org/pub/mule/apel/

---
Ken'ichi HANDA
handa@m17n.org

Re: RMAIL, MIME-related bug

[Kevin Rodgers]

Juri Linkov wrote:

> Kevin Rodgers <ihs_4664@yahoo.com> writes:
>>I did a little work on this issue about 2 1/2 years ago.  My approach
>>was to define a bookmark as a URL whose protocol is "mailmessage", so
>>that I could use Emacs' url.el library to parse it and ange-ftp.el to
>>access remote //[USER@]HOST folders.  However, I wrote it as an add-on
>>to VM, so the code that actually views the folder and selects the
>>message would have to be converted to Rmail.  And it would have to be
>>extended to handle IMAP folders (cf. file folders).
> 
> Using URL for references to messages is a good solution, but instead
> of inventing ad hoc protocol name, it's better to use standard name
> "mid" defined by http://www.ietf.org/rfc/rfc2392.txt.


Thanks for the pointer, but I find that document somewhat disappointing.

For one thing, I was confused by this:

	A "cid" URL is converted to the corresponding Content-ID message
	header [MIME] by removing the "cid:" prefix, converting the % encoded
	character to their equivalent US-ASCII characters, and enclosing the
	remaining parts with an angle bracket pair, "<" and ">".  For example,
	"cid:foo4%25foo1@bar.net" corresponds to

	    Content-ID: <foo4%25foo1@bar.net>

	Reversing the process and converting URL special characters to their
	% encodings produces the original cid.

	A "mid" URL is converted to a Message-ID or Message-ID/Content-ID
	pair in a similar fashion.

Shouldn't "%25" have been converted to "%":

	Content-ID: <foo4%foo1@bar.net>

But more importantly, the mid and cid URLs do not actually specify a
LOCATION: that makes them more like a URN than a URL.


> Then, for example, opening the link <mid:3F9471FB.4000609@yahoo.com>
> should start a mail reader, find corresponding message by its
> Message-ID, prepare article buffer and switch to it (e.g. in case of
> Gnus to call the function `gnus-summary-goto-article'; other mail
> readers should have a similar function).


Shouldn't the cid and mid schemes specify a location via the Common
Internet Scheme Syntax defined in RFC 1738?  That way, you could specify
a folder file, or a POP or IMAP server (and user and password).


> I'm not sure yet, where this could be implemented: url.el is not part
> of Emacs (BTW: Why?); thingatpt.el mentions this protocol name, but
> that's all; perhaps this should be done partially in ffap.el.

As long as it respects the read-mail-command variable.  To do that
correctly, I think we would need to define an interface like compose-mail
that would accept all the arguments that might be necessary to specify a
message (folder file and format; or server, protocol, user, and password;
plus message id); then a new utility define-read-mail-command (analogous to
define-mail-user-agent) that would associate each read-mail-command to a
function that supports that interface.


Otherwise, how would Emacs know where to search for the Message-Id?

-- 
Kevin Rodgers

Re: RMAIL, MIME-related bug

[Kevin Rodgers]

Kevin Rodgers wrote:

> Juri Linkov wrote:
>> Using URL for references to messages is a good solution, but instead
>> of inventing ad hoc protocol name, it's better to use standard name
>> "mid" defined by http://www.ietf.org/rfc/rfc2392.txt.
> 
> Thanks for the pointer, but I find that document somewhat disappointing.
...
> But more importantly, the mid and cid URLs do not actually specify a
> LOCATION: that makes them more like a URN than a URL.


And it makes them similar to the news:<message-id> URL scheme.  But
there is also a nntp://<host>:<port>/<newsgroup>/<article> URL scheme,
which I think is closer to what we need.


> Shouldn't the cid and mid schemes specify a location via the Common
> Internet Scheme Syntax defined in RFC 1738?  That way, you could specify
> a folder file, or a POP or IMAP server (and user and password).

We could define separate URL schemes for mbox, POP, and IMAP access, but
I think for now I'm going to pursue my origial idea, but rename it to
conform to the standard for experimental schemes:

x-mid:[//[user@]host]/path/to/folder#message-id

-- 
Kevin Rodgers

Re: RMAIL, MIME-related bug

[Bill Wohler]

Richard Stallman <rms@gnu.org> writes:

>     I am a huge fan of mail indexing programs like: swish++, mairix,
>     glimpse, namazu etc.
>
> I seem to recall that glimpse is or was non-free.  I don't know about
> the others.  Can you check which of these are non-free?  We don't
> want Gnus or MH-E to refer to them.

Glimpse support was dropped from CVS MH-E today.

A poll of MH-E users revealed one glimpse user who was going to switch
to another indexer anyway, and another who was happy to switch, so there
wasn't much pain involved. I use swish++ myself.

As Satyaki mentioned, Juri's concerns are unfounded with modern indexers
like swish that do base64 and qp decoding. MH-E can find text in a
base64 message somewhere in all of your mail and display it in a second.

We also support a default search engine called grep which does suffer
Yuri's concerns however.

Note that the Emacs mailcrypt package does have the option to write a
decrypted mail message back to the disk, which I find to be a useful
option since typing the pass phrase over and over again is a pain and
most times, encryption is used to protect the message in transport, not
on my disk.

I also don't think there would be any harm in saving the decoded base64
and qp body parts, just as long you left the body parts intact and set
the Content-Transfer-Encoding as necessary (8bit, for example) so that
these messages could be recoded if forwarded. I don't see any benefit
either.

-- 
Bill Wohler <wohler@newt.com>  http://www.newt.com/wohler/  GnuPG ID:610BD9AD
Maintainer of comp.mail.mh FAQ and MH-E. Vote Libertarian!
If you're passed on the right, you're in the wrong lane.

Re: RMAIL, MIME-related bug

[守岡知彦 / MORIOKA Tomohiko]

>>>>> In <200310221052.TAA08319@etlken.m17n.org> 
>>>>>	"Kenichi" = Kenichi Handa <handa@m17n.org> wrote:

Kenichi> I included Morioka-san's address in CC:.  Morioka-san,
Kenichi> please give us the information about the current status of
Kenichi> those packages.  Does the assignment papers already sent to
Kenichi> FSF cover the whole codes (except tiny changes)?  If not, is
Kenichi> it possible to get papers from the other programmers?  FYI,
Kenichi> you can read previous mails of this thread at:
Kenichi> 	http://mail.gnu.org/archive/html/emacs-devel/

I've rearranged APEL/FLIM/SEMI to merge into GNU Emacs in 2000--2002.
The result is available via anonymous CVS:

    % cvs -d :pserver:anonymous@cvs.m17n.org:/cvs/root login
    CVS password: [CR] # NULL string 

    % cvs -z9 -d :pserver:anonymous@cvs.m17n.org:/cvs/root co lemi

SEMI related codes included in LEMI is already covered by assignment
papers.

-- 
tomo.