using movemail directly in .emacs

using movemail directly in .emacs

[Hikaru Ichijyo]

This is a newbie Emacs question.  If the answers are obvious or the 
intent misguided, let me know.

I'd like to start handling my email in Emacs, probably in VM.  But I 
have seen that all Emacs mailreaders like to move the spool file into 
the home directory before operating, because none of them can do locking 
on a mailbox in place while the system mailer daemon is trying to append 
new messages.  (This is not as outrageous of an expectation as it 
sounds.  Mutt, Pine/Alpine, and KMail can all do this, and they all 
provide code examples for ways it could be done.)

I'd like to keep my inbox in the system spool where other email programs 
expect it, so I can freely go back and forth between various email 
programs while I'm in the process of getting myself moved over to an 
Emacs way of thinking.  I realize it is possible to setup Alpine and 
Mutt to work the other way -- to read their inbox from the home 
directory, in effect, making them work with Emacs way of doing things.  
I'd rather do the opposite:

The Lisp program movemail, bundled with Emacs, seems to have the ability 
to do mailbox locking so that RMAIL can safely move mailboxes into the 
home directory.  (I think VM uses it also.)

Would it be possible to directly call movemail from the .emacs file and 
make it do the following?  This is what I want to do:

- When Emacs launches, I want my .emacs file to run movemail, and put my 
system spool inbox into my home directory in a place that VM expects it.

- It can be safely assumed that so long as Emacs is running, I'm not 
going to run Alpine or Mutt or anything similar -- if Emacs is 
available, I'll use Emacs for mail, and it won't matter that my inbox is 
in my home directory at that time.

- When Emacs shuts down, I'd like to use its shutdown hooks to append 
any newly received messages in the system spool to the inbox in my home 
directory, and then use movemail to put the whole home directory inbox 
back into /var/spool/mail before Emacs quits itself.

Is any of this impossible or misguided?  I'd just strongly prefer my 
mailbox in the system spool area where most UNIX tools expect it to be.

-- 
He that would make his own liberty secure must guard even his enemy from
oppression; for if he violates this duty, he establishes a precedent
that will reach to himself.
					--Thomas Paine

Re: using movemail directly in .emacs

[W. Greenhouse]

Hello Hikaru,

Hikaru Ichijyo <ichijyo-P3s0QLKLlZzR7TO4fIE1SA@public.gmane.org> writes:

> Is any of this impossible or misguided?  I'd just strongly prefer my 
> mailbox in the system spool area where most UNIX tools expect it to be.

This sounds vaguely possible but probably misguided. If the goal is
simply to leave mail sitting at the spool, a simpler place to start is

(setq rmail-preserve-inbox t)

which will prevent Movemail from emptying the spool as it delivers to
rmail. I would not think it wise to copy back the RMAIL file to your
system mail spool, because rmail adds its own headers to track
flagged/read/replied/forwarded state and user-generated labels. Ideally,
other mbox-reading clients will just ignore these additional headers,
but rmail is mutating the messages it stores, not just moving or
deleting them, so some potential for problems exists.

As an alternative to flinging an mbox back and forth, you may be able to
have your system MTA deliver your mail directly to your home directory
in the more modern, non-blocking Maildir format, which several clients
can read and edit simultaneously even as mail is being delivered. For
example, at sites where Procmail controls local delivery from the MTA,
you can have a one-line ~/.procmailrc like

DEFAULT=Maildir/

to trigger delivery of all new mail to ~/Maildir/, as a mailbox name
ending in / is interpreted by procmail as a Maildir instead. With
Maildir you could use Mutt, KMail, Emacs's Gnus, and many others to
operate on the same mail store simultaneously, if you wanted. Alpine
requires a patch to support Maildir, and many other older clients such
as BSD mailx do not support it at all, but it is the emerging standard
over the past 10-15 years for UNIX mailclients that operate on a local
store, as well as for mail indexing programs and IMAP servers. And
message metadata such as flagged/read status is stored in the message
filenames themselves, and so is interoperable between clients.

--
Best,
WGG

Replies to list only, please.

Re: using movemail directly in .emacs

[Hikaru Ichijyo]

W. Greenhouse <wgreenhouse@riseup.net> wrote:

In article <mailman.634.1399184068.1147.help-gnu-emacs@gnu.org> you 
wrote:

> This sounds vaguely possible but probably misguided. If the goal is
> simply to leave mail sitting at the spool, a simpler place to start is
>
> (setq rmail-preserve-inbox t)
>
> which will prevent Movemail from emptying the spool as it delivers to
> rmail. I would not think it wise to copy back the RMAIL file to your
> system mail spool, because rmail adds its own headers to track
> flagged/read/replied/forwarded state and user-generated labels.
> Ideally, other mbox-reading clients will just ignore these additional
> headers, but rmail is mutating the messages it stores, not just moving
> or deleting them, so some potential for problems exists.

I only mentioned RMAIL because I think the movemail Lisp program is
provided for its sake in Emacs.  I'd probably be using VM though.  While
I'm using Emacs, probably there will be new emails received, and I may
delete messages.  Whatever has happened while I'm working on my mailbox
in Emacs will need to be restored to the system spool afterward.  Do you
think VM will put metadata in my mailbox that other mailers couldn't
deal with?

> As an alternative to flinging an mbox back and forth, you may be able
> to have your system MTA deliver your mail directly to your home
> directory in the more modern, non-blocking Maildir format, which
> several clients can read and edit simultaneously even as mail is being
> delivered. For example, at sites where Procmail controls local
> delivery from the MTA, you can have a one-line ~/.procmailrc like
>
> DEFAULT=Maildir/
>
> to trigger delivery of all new mail to ~/Maildir/, as a mailbox name
> ending in / is interpreted by procmail as a Maildir instead. With
> Maildir you could use Mutt, KMail, Emacs's Gnus, and many others to
> operate on the same mail store simultaneously, if you wanted. Alpine
> requires a patch to support Maildir, and many other older clients such
> as BSD mailx do not support it at all, but it is the emerging standard
> over the past 10-15 years for UNIX mailclients that operate on a local
> store, as well as for mail indexing programs and IMAP servers. And
> message metadata such as flagged/read status is stored in the message
> filenames themselves, and so is interoperable between clients.

Sorry, I appreciate the advantages of Maildir, but I'm really trying to 
keep a regular Berkeley mbox file in my /var/spool/mail.

If I wanted to manually invoke movemail in my .emacs as part of startup 
and shutdown hooks, what would that look like?

-- 
He that would make his own liberty secure must guard even his enemy from
oppression; for if he violates this duty, he establishes a precedent
that will reach to himself.
					--Thomas Paine

Re: using movemail directly in .emacs

[Robert Thorpe]

Hikaru Ichijyo <ichijyo@macross.sdf.jp> writes:

I think it's possible to do what you want.  It'll be quite a fragile
setup though.

Movemail isn't a Lisp program, it's a C program.  There are two versions
of it, the simple one the ships with Emacs and a more complete one
that's part of the GNU Mailutils package.

You run it like this:
movemail -p mbox://var/spool/mail/smith ~/to.mbox
The "-p" switch preserves the contents of the the mailbox you're taking
the mail from.  You can run it from .emacs using something like
"shell-command".  Check out the Mailutils manual at gnu.org.

As W. Greenhouse mentions Rmail and similar mailers put info in the mbox
file that may make it incompatible with other mailers.  I have no idea
if VM does this, check it's manual.

Here's an alternative way to change mailer that may be easier.  Start up
another email account.  Send stuff like this list and other email lists
to it.  Get used to your chosen Emacs mailer using that new account.
Then, later on use Emacs for both your regular account and that one.
Most Emacs mailers allow you to pick up mail from multiple accounts.
That may interfer with your use of the mail spool though.

BR,
Robert Thorpe

Re: using movemail directly in .emacs

[lee]

Hikaru Ichijyo <ichijyo@macross.sdf.jp> writes:

> I'd like to start handling my email in Emacs, probably in VM.  But I 
> have seen that all Emacs mailreaders like to move the spool file into 
> the home directory before operating, because none of them can do locking 
> on a mailbox in place while the system mailer daemon is trying to append 
> new messages.  (This is not as outrageous of an expectation as it 
> sounds.  Mutt, Pine/Alpine, and KMail can all do this, and they all 
> provide code examples for ways it could be done.)

Gnus can probably do what you want.  However, I wouldn`t recommend
operating on the mail spool file or otherwise using a single file to
store emails.

> I'd like to keep my inbox in the system spool where other email programs 
> expect it, so I can freely go back and forth between various email 
> programs while I'm in the process of getting myself moved over to an 
> Emacs way of thinking.

IMAP might be the best solution for this.

> Would it be possible to directly call movemail from the .emacs file and 
> make it do the following?  This is what I want to do:

Gnus can do mail splitting on it`s own.

> - When Emacs launches, I want my .emacs file to run movemail, and put my 
> system spool inbox into my home directory in a place that VM expects it.

You could start gnus from within your .emacs.

> - It can be safely assumed that so long as Emacs is running, I'm not 
> going to run Alpine or Mutt or anything similar -- if Emacs is 
> available, I'll use Emacs for mail, and it won't matter that my inbox is 
> in my home directory at that time.

When you have your MTA deliver in maildir format, you can run all your
MUAs at the same time.

> - When Emacs shuts down, I'd like to use its shutdown hooks to append 
> any newly received messages in the system spool to the inbox in my home 
> directory, and then use movemail to put the whole home directory inbox 
> back into /var/spool/mail before Emacs quits itself.

You would need to disable mail delivery to the spool file (lock the
spool file) during this operation.  And I wouldn`t want to copy a file
of currently over 4GB back and forth every time I start or exit emacs.

> Is any of this impossible or misguided?  I'd just strongly prefer my 
> mailbox in the system spool area where most UNIX tools expect it to
> be.

It`s not impossible, yet it doesn`t make much sense.  Using a single
file like the spool file is an anachronism (and apparently doesn`t work
too well over NFS).  A single file is very awkward to work with: When
you delete an email somewhere within the file or set a flag (like read,
answered, etc.), the whole file needs to be rewritten.  That wouldn`t be
ideal with a file of over 4GB.

In any case, the file can easily be damaged, in which case you might
loose all your email.  What happens when your computer crashes while
it`s copying or rewriting your 4GB+ spool file?

Do you really need an assortment of various tools that work with the
spool file?  I doubt it, especially not when you use gnus.  If you do
need to use many different MUAs, IMAP may be a good idea.  When you use
maildir or IMAP, you can use both gnus and mutt and any other MUA that
supports maildir.

Other than that, just use gnus with it`s native nnml.  While getting
used to gnus, just back up your email frequently.

In any case, do not store all your email in a single file.  It`s one
file per email since file systems don`t have issues with many files in a
single directory anymore.


-- 
Knowledge is volatile and fluid.  Software is power.

Re: using movemail directly in .emacs

[Robert Thorpe]

lee <lee@yun.yagibdah.de> writes:

> Hikaru Ichijyo <ichijyo@macross.sdf.jp> writes:
>> Is any of this impossible or misguided?  I'd just strongly prefer my 
>> mailbox in the system spool area where most UNIX tools expect it to
>> be.
>
> It`s not impossible, yet it doesn`t make much sense.  Using a single
> file like the spool file is an anachronism (and apparently doesn`t work
> too well over NFS).  A single file is very awkward to work with: When
> you delete an email somewhere within the file or set a flag (like read,
> answered, etc.), the whole file needs to be rewritten.  That wouldn`t be
> ideal with a file of over 4GB.
>
> In any case, the file can easily be damaged, in which case you might
> loose all your email.  What happens when your computer crashes while
> it`s copying or rewriting your 4GB+ spool file?

Spool files are a temporary storage area for email.  A user reads their
mail using a program (a "Mail User Agent" or MUA) which takes the mail
from the spool file and stores it elsewhere.  For most MUAs all mail is
moved from the spool file to somewhere in the user's home directory.
For some MUAs the "inbox" is the spool file, normally for those the user
is expect to move stuff out into other files.

The problem here isn't that spool files are an anarchonism, it's that
they're not made for storing large amounts of mail.  It may now work on
some system too because they have scripts that delete very large spool
files.  (Notice I'm not saying that spool files aren't an anachonism.
Multi-user *nix computers that are never switched off are going the way
of dinosaurs and spool files with them.)

BR,
Robert Thorpe

Re: using movemail directly in .emacs

[lee]

Robert Thorpe <rt@robertthorpeconsulting.com> writes:

> lee <lee@yun.yagibdah.de> writes:
>
>> It`s not impossible, yet it doesn`t make much sense.  Using a single
>> file like the spool file is an anachronism
>
> [...]
>
> The problem here isn't that spool files are an anarchonism, it's that
> they're not made for storing large amounts of mail.

The OP seemed to want to keep all their email in their spool file. Doing
that, and keeping many emails in the same file (other than for
archiving), is the anachronism, not the spool file itself --- I should
have said it this way.

> (Notice I'm not saying that spool files aren't an
> anachonism. Multi-user *nix computers that are never switched off are
> going the way of dinosaurs and spool files with them.)

Hm, I`m using a spool file, and gnus takes the mail from there.  It`s
simple and works fine when used like that.


-- 
Knowledge is volatile and fluid.  Software is power.

Re: using movemail directly in .emacs

[Hikaru Ichijyo]

Robert Thorpe <rt@robertthorpeconsulting.com> wrote:

> I think it's possible to do what you want.  It'll be quite a fragile 
> setup though.
> 
> Movemail isn't a Lisp program, it's a C program.  There are two 
> versions of it, the simple one the ships with Emacs and a more 
> complete one that's part of the GNU Mailutils package.
> 
> You run it like this:
> movemail -p mbox://var/spool/mail/smith ~/to.mbox
> The "-p" switch preserves the contents of the the mailbox you're taking
> the mail from.  You can run it from .emacs using something like
> "shell-command".  Check out the Mailutils manual at gnu.org.

What I was really wondering (as an Emacs newbie) was what would be the 
most "correct" way to run it from within the .emacs file, using Emacs' 
startup and shutdown hooks?  Since I'm just doing local file access and 
no IMAP or anything like that, I'd like to run the shipped version, 
which is at /usr/libexec/emacs/24.3/x86_64-pc-linux-gnu/movemail on my 
system.  Is there some special syntax that Emacs uses for running 
programs that are inside its tree like this, or do you use the whole 
path?  What does it look like in the .emacs to run something like this 
on startup and shutdown?

-- 
He that would make his own liberty secure must guard even his enemy from
oppression; for if he violates this duty, he establishes a precedent
that will reach to himself.
					--Thomas Paine

Re: using movemail directly in .emacs

[Hikaru Ichijyo]

Robert Thorpe <rt@robertthorpeconsulting.com> wrote:
> lee <lee@yun.yagibdah.de> writes:
> 
>> Hikaru Ichijyo <ichijyo@macross.sdf.jp> writes:
>>> Is any of this impossible or misguided?  I'd just strongly prefer my 
>>> mailbox in the system spool area where most UNIX tools expect it to
>>> be.
>>
>> It`s not impossible, yet it doesn`t make much sense.  Using a single
>> file like the spool file is an anachronism (and apparently doesn`t work
>> too well over NFS).  A single file is very awkward to work with: When
>> you delete an email somewhere within the file or set a flag (like read,
>> answered, etc.), the whole file needs to be rewritten.  That wouldn`t be
>> ideal with a file of over 4GB.
>>
>> In any case, the file can easily be damaged, in which case you might
>> loose all your email.  What happens when your computer crashes while
>> it`s copying or rewriting your 4GB+ spool file?
> 
> Spool files are a temporary storage area for email.  A user reads 
> their mail using a program (a "Mail User Agent" or MUA) which takes 
> the mail from the spool file and stores it elsewhere.  For most MUAs 
> all mail is moved from the spool file to somewhere in the user's home 
> directory. For some MUAs the "inbox" is the spool file, normally for 
> those the user is expect to move stuff out into other files.

The main reason I initially wanted to go down this path is because Mutt 
and (Al)pine handle it gracefully, and because with their code being 
freely available, I couldn't see a good reason why a Lisp-based 
mailreader couldn't be coded to do the same thing.

Somehow, Mutt and Alpine are both able to negotiate locking on the spool 
file such that you can safely write and expunge messages while the 
system MDA is doing delivery without any fear of corruption.  Alpine 
actually handles this a little better, since whereas Mutt will handle a 
situation where you wanted to make a change on disk at the same time the 
MDA delivered a message by beeping, *not* making your change, and 
updating the display to show the new mails that just came in, Alpine's 
code seems to be able to pull the tablecloth out while the diners are 
eating -- it effects whatever change you made, *and* it updates the 
display to show the new mail.  In decades of using Pine and Alpine, I've 
never seen it screw this up.  If anyone wants a code example for how 
this could work, Alpine is under the Apache license.

I finally gave up though and changed my setup to what Emacs expects, 
since Mutt and Alpine can both be setup to do that also (they just don't 
have to be).  All my mailreaders (VM, Alpine, Mutt...) now move mail 
from the system spool to ~/mbox and work on it there...so Emacs is happy 
now.

> The problem here isn't that spool files are an anarchonism, it's that 
> they're not made for storing large amounts of mail.

It seems like most users let gigabytes of mail accumulate in their 
inbox.  The only messages I leave in mine are active items that still 
need attention.  Everything else gets moved to a folder as soon as 
possible.  I never have more than 400 messages in my inbox at its worst.

> It may now work on some system too because they have scripts that 
> delete very large spool files.  (Notice I'm not saying that spool 
> files aren't an anachonism. Multi-user *nix computers that are never 
> switched off are going the way of dinosaurs and spool files with 
> them.)

Not anywhere around me they aren't.  :)

Granted, we now store the central user databases in some kind of 
directory on a central server with something like LDAP rather than 
having a humongous /etc/passwd, but I work in academic computing, and 
universities are where big multiuser UNIX-based Internet sites have deep 
roots, not likely to go away any time soon.

-- 
He that would make his own liberty secure must guard even his enemy from
oppression; for if he violates this duty, he establishes a precedent
that will reach to himself.
					--Thomas Paine

Re: using movemail directly in .emacs

[lee]

Hikaru Ichijyo <ichijyo@macross.sdf.jp> writes:

>> The problem here isn't that spool files are an anarchonism, it's that 
>> they're not made for storing large amounts of mail.
>
> It seems like most users let gigabytes of mail accumulate in their 
> inbox.

Most of the mail here is from mailing lists.  It never gets into the
inbox because gnus splits it.

Do you seriously want to store over 120k mails from a mailing list in a
single (mbox) file?  Rewrite the whole file *every time* you flag a
message as "read" or "answered"?

> The only messages I leave in mine are active items that still 
> need attention.  Everything else gets moved to a folder as soon as 
> possible.  I never have more than 400 messages in my inbox at its worst.

The anachronism is storing many emails in one file.

That was done in the past because file systems were used the performance
of which would be severely impacted when you had many files in the same
directory.  Without this limitation, it doesn`t make sense to put many
emails into a single file, even if it`s only 10 or 5, other than for
special purposes.  It is a bad idea to do that because the single file
is prone to locking problems and corruption, and it can easily become
unwieldy and very slow to work with.  Using a number of single files
("folders") doesn`t really make it better.

IIRC, maildir supports flags (like read, answered) in the file names,
with, besides efficiency, the advantage that every MUA which supports
maildir can understand the flags.  When you use mbox, you either have to
use some sort of index file to store the flags, or you need to re-write
the whole file when a flag changes, and each MUA may use their own way
for this, not understood by others.  How efficient is that?

Using mbox is as much an anachronism as using a spool file for storing
email:  It`s still a single file.

> Granted, we now store the central user databases in some kind of 
> directory on a central server with something like LDAP rather than 
> having a humongous /etc/passwd, but I work in academic computing, and 
> universities are where big multiuser UNIX-based Internet sites have deep 
> roots, not likely to go away any time soon.

They are not using IMAP on top of maildir, or cyrus?  Do they make the
users` spool files available over the network?


-- 
Knowledge is volatile and fluid.  Software is power.

Re: using movemail directly in .emacs

[Robert Thorpe]

lee <lee@yun.yagibdah.de> writes:
> The anachronism is storing many emails in one file.
>
> That was done in the past because file systems were used the performance
> of which would be severely impacted when you had many files in the same
> directory.  Without this limitation, it doesn`t make sense to put many
> emails into a single file, even if it`s only 10 or 5, other than for
> special purposes.  It is a bad idea to do that because the single file
> is prone to locking problems and corruption, and it can easily become
> unwieldy and very slow to work with.  Using a number of single files
> ("folders") doesn`t really make it better.
>
> IIRC, maildir supports flags (like read, answered) in the file names,
> with, besides efficiency, the advantage that every MUA which supports
> maildir can understand the flags.  When you use mbox, you either have to
> use some sort of index file to store the flags, or you need to re-write
> the whole file when a flag changes, and each MUA may use their own way
> for this, not understood by others.  How efficient is that?
>
> Using mbox is as much an anachronism as using a spool file for storing
> email:  It`s still a single file.

I agree that storing gigabytes of email in a single file is unwise.
Keeping it in the spool file of a multi-user installation is a even
worse since some sysadmins delete spool files that are too large.

In general though storing lots of emails in a file isn't really a
problem.  I've used email clients that do that for many years and I
haven't had any problems.  I have gigabytes of mail in mbox files, but
it's not all in one file.  I file things and I that keeps the largest
file smaller than ~150MB, most are much smaller.  I don't have a very
fast computer, I bought it four years ago, but even so the delay in
writing out files isn't very noticable.  That's because when the program
writes the file out it goes into the disk cache first and it's only
written to hard drive later on.  I use Rmail at present, though I have
used other MUAs.  Rmail saves a backup file for every mbox file.  Even
so I've never had a problem with corruption, using Rmail or any other
program.  Mbox files are very simple, it's hard to get writing to them
wrong.

You're right about flags though.  Mbox files aren't very portable
between mailers for that reason.  Another, bigger, problem is coding
systems.  Thunderbird (for example) treats mbox files as ASCII [1].  If
you get a UTF-8 email in Thunderbird then it inserts it as base64 (well
I assume it's base64) into the mbox.  On the other hand in Rmail seems to
inserts it as UTF-8.  Both work correctly but in their own way.

I'm not saying that it's best to use mbox files, but the problems with
them aren't large.  Large directory structures have other problems.

[1] - I think it's ASCII, it may be Latin-1 or something.

BR,
Robert Thorpe

Re: using movemail directly in .emacs

[lee]

Robert Thorpe <rt@robertthorpeconsulting.com> writes:

> lee <lee@yun.yagibdah.de> writes:
>> The anachronism is storing many emails in one file.
>
> I agree that storing gigabytes of email in a single file is unwise.
> [...]
>
> In general though storing lots of emails in a file isn't really a
> problem.
> [...]
> Mbox files are very simple, it's hard to get writing to them
> wrong.

That goes only as long as everything works as intended.  Have a power
failure or a yet-unnoticed disk failure, have your MUA crash due to some
bug or because the system kills it because it`s out of memory, have your
computer crash or freeze, have some issue with a network file system or
other things that don`t come to mind atm --- and your whole file with
all the mails may be gone.

Deal with a single (relatively small) file, and chances are that only
this single file is affected.

> You're right about flags though.  Mbox files aren't very portable
> between mailers for that reason.  Another, bigger, problem is coding
> systems.  Thunderbird (for example) treats mbox files as ASCII [1].  If
> you get a UTF-8 email in Thunderbird then it inserts it as base64 (well
> I assume it's base64) into the mbox.  On the other hand in Rmail seems to
> inserts it as UTF-8.  Both work correctly but in their own way.

Hm, so you need to somehow identify such base64 encoded files and decode
them when searching ... that`s awkward.

> I'm not saying that it's best to use mbox files, but the problems with
> them aren't large.  Large directory structures have other problems.

Problems like?


> [1] - I think it's ASCII, it may be Latin-1 or something.


-- 
Knowledge is volatile and fluid.  Software is power.

Re: using movemail directly in .emacs

[Robert Thorpe]

lee <lee@yun.yagibdah.de> writes:

> Robert Thorpe <rt@robertthorpeconsulting.com> writes:
>> In general though storing lots of emails in a file isn't really a
>> problem.
>> [...]
>> Mbox files are very simple, it's hard to get writing to them
>> wrong.
>
> That goes only as long as everything works as intended.  Have a power
> failure or a yet-unnoticed disk failure, have your MUA crash due to some
> bug or because the system kills it because it`s out of memory, have your
> computer crash or freeze, have some issue with a network file system or
> other things that don`t come to mind atm --- and your whole file with
> all the mails may be gone.

MUAs and MTAs have been using mbox files for a long time, they take
precautions against this.  Rmail, for example, uses Emacs' default
backup file system.  For each file MBOX there's a MBOX~ that can be used
for recovery.  Other mailers do different things.  As I understand it,
MTAs only delete emails from their internal cache when the email has
been written to the users spool file.

I've never lost any email because of this type of problem.  (I have lost
it using Microsoft PST files though).

> Deal with a single (relatively small) file, and chances are that only
> this single file is affected.
>
>> You're right about flags though.  Mbox files aren't very portable
>> between mailers for that reason.  Another, bigger, problem is coding
>> systems.  Thunderbird (for example) treats mbox files as ASCII [1].  If
>> you get a UTF-8 email in Thunderbird then it inserts it as base64 (well
>> I assume it's base64) into the mbox.  On the other hand in Rmail seems to
>> inserts it as UTF-8.  Both work correctly but in their own way.
>
> Hm, so you need to somehow identify such base64 encoded files and decode
> them when searching ... that`s awkward.

I just use Thunderbird for those ones.  It's an issue though.  I may
recode those one day.

>> I'm not saying that it's best to use mbox files, but the problems with
>> them aren't large.  Large directory structures have other problems.
>
> Problems like?

It asks a lot of the filesystem.  Some filesystems can't handle long
path and some can't handle certain characters in filenames.  Some behave
quite slowly if a lot of directories are being checked.  The minimum
file size is quite large on many systems, so if you have lots of small
emails then it wastes a lot of space.  If I keep backup files that waste
more space, but I can always delete a few of them if file length is an
issue.  I can just copy a mbox file to a USB key, or download one from a
website (such as the GNU mailing list archives).


However, I don't think that Maildirs or MH are bad systems.  My mail
point here is that mbox files aren't bad either.  The whole debate is
about a few details that aren't really that important.

BR,
Robert Thorpe

Re: using movemail directly in .emacs

[lee]

Robert Thorpe <rt@robertthorpeconsulting.com> writes:

> lee <lee@yun.yagibdah.de> writes:
>
>> Robert Thorpe <rt@robertthorpeconsulting.com> writes:
>>> In general though storing lots of emails in a file isn't really a
>>> problem.
>>> [...]
>>> Mbox files are very simple, it's hard to get writing to them
>>> wrong.
>>
>> That goes only as long as everything works as intended.  Have a power
>> failure or a yet-unnoticed disk failure, have your MUA crash due to some
>> bug or because the system kills it because it`s out of memory, have your
>> computer crash or freeze, have some issue with a network file system or
>> other things that don`t come to mind atm --- and your whole file with
>> all the mails may be gone.
>
> MUAs and MTAs have been using mbox files for a long time, they take
> precautions against this.  Rmail, for example, uses Emacs' default
> backup file system.  For each file MBOX there's a MBOX~ that can be used
> for recovery.

So you have like 1GB mbox and another 1GB mbox~ and copy that every time
you change a flag, like read, and perhaps you do that over the network
on a centralised storage or over a VPN connection using a modem.  In any
case, it`s horribly inefficient.

> [...]
> I've never lost any email because of this type of problem.  (I have lost
> it using Microsoft PST files though).

Those are particularly awful.  Microsoft doesn`t have any MUA which
could handle more than a handful of mails per day, if even that much.


> [...]
>>> I'm not saying that it's best to use mbox files, but the problems with
>>> them aren't large.  Large directory structures have other problems.
>>
>> Problems like?
>
> It asks a lot of the filesystem.

It merely asks of it what it was made for.

> Some filesystems can't handle long path and some can't handle certain
> characters in filenames.

Characters in file names aren`t a problem with maildir.

> Some behave quite slowly if a lot of directories are being checked.

Like which ones?  IIRC, extfs would slow down considerably with many
files in a directory, ext2 didn`t.  Using ancient file systems with mbox
doesn`t seem to be a good idea either, considering the increase of the
amounts of data they were designed to deal with and the amounts of data
we are dealing with nowadays.

> The minimum file size is quite large on many systems, so if you have
> lots of small emails then it wastes a lot of space.

I`ve investigated this problem quite a while ago with the actual mail
storage I have.  I found that most of the mails are between a little
over 2kB and below 4kB, so a minimum size of 4kB is fine.  If you use
anything larger for minimum size, you probably have rather special
requirements which may be suited better by a different file system.

> If I keep backup files that waste more space, but I can always delete
> a few of them if file length is an issue.  I can just copy a mbox file
> to a USB key, or download one from a website (such as the GNU mailing
> list archives).

You can always turn a maildir into a single file with tools like tar, or
copy it to an USB stick.  For archives, mbox can be a good thing to use.

> However, I don't think that Maildirs or MH are bad systems.  My mail
> point here is that mbox files aren't bad either.  The whole debate is
> about a few details that aren't really that important.

Well, I find reliability and efficiency important, and maildir and nnml
have advantages in that.


-- 
Knowledge is volatile and fluid.  Software is power.

Re: using movemail directly in .emacs

[Robert Thorpe]

lee <lee@yun.yagibdah.de> writes:

> So you have like 1GB mbox and another 1GB mbox~ and copy that every time
> you change a flag, like read, and perhaps you do that over the network
> on a centralised storage or over a VPN connection using a modem.  In any
> case, it`s horribly inefficient.

I don't have any 1GB mbox files.  I file emails by subject.  The biggest
one I have is ~150MB.  I'll probably archive that one and start another
one soon.

I expect if I was using a VPN connection I would have a problem as you
say, but I don't.  (If I did have to use remote access I'd do it another
way).  Writing a big mbox file isn't really a problem because the OS
caches writes.  The delay of reading it is the real problem.

>> I've never lost any email because of this type of problem.  (I have lost
>> it using Microsoft PST files though).
>
> Those are particularly awful.  Microsoft doesn`t have any MUA which
> could handle more than a handful of mails per day, if even that much.

Outlook can handle far more than that, at a previous job I used to get
hundreds per day sometimes.  PST files aren't good though, they're pretty
much impossible to fix once broken.

>> It asks a lot of the filesystem.
>
> It merely asks of it what it was made for.

That's a rather philosophical point of view.  Practical filesystems have
limitations and sometimes we have to live with them.

>> Some filesystems can't handle long path and some can't handle certain
>> characters in filenames.
>
> Characters in file names aren`t a problem with maildir.

The colon ':' is on MS Windows because it doesn't support colons in filenames.

>> Some behave quite slowly if a lot of directories are being checked.
>
> Like which ones?  IIRC, extfs would slow down considerably with many
> files in a directory, ext2 didn`t.  Using ancient file systems with mbox
> doesn`t seem to be a good idea either, considering the increase of the
> amounts of data they were designed to deal with and the amounts of data
> we are dealing with nowadays.

I don't know much about the subject.  But, when I read about Maildirs
online there's often lots of discussion about the right filesystem to
use, and setting up it's parameters.

>> The minimum file size is quite large on many systems, so if you have
>> lots of small emails then it wastes a lot of space.
>
> I`ve investigated this problem quite a while ago with the actual mail
> storage I have.  I found that most of the mails are between a little
> over 2kB and below 4kB, so a minimum size of 4kB is fine.  If you use
> anything larger for minimum size, you probably have rather special
> requirements which may be suited better by a different file system.

You've persuaded me on that.

>> If I keep backup files that waste more space, but I can always delete
>> a few of them if file length is an issue.  I can just copy a mbox file
>> to a USB key, or download one from a website (such as the GNU mailing
>> list archives).
>
> You can always turn a maildir into a single file with tools like tar, or
> copy it to an USB stick.  For archives, mbox can be a good thing to use.

Yes.

>> However, I don't think that Maildirs or MH are bad systems.  My mail
>> point here is that mbox files aren't bad either.  The whole debate is
>> about a few details that aren't really that important.
>
> Well, I find reliability and efficiency important, and maildir and nnml
> have advantages in that.

I agree that maildirs can be reliable and efficient.  What I don't see
are the problems you bring up with mboxes.  If a user were to keep all
their mail in one big file then I'd agree.  Generally though people need
to file things.

Anyway, this is no longer very related to Emacs, so I'm going to stop now.

BR,
Robert Thorpe