Gnus maildir backend issues

Gnus maildir backend issues

[Sascha Wilde]

Hi *,

there are two issues with the Gnus maildir backend (nnmaildir) which I
would like to discuss:

1. the backend creates empty files in .nnmaildir/num for each mail seen
   and never deletes them.  This results in thousands and thousands of
   (empty) files (last time I looked in a regularly used folder there
   were more than 35000 files in the .nnmaildir directory.

   I don't understand why these files are needed and I definitely
   consider it a bug, that they are never removed.

2. the backend moves mail from new to cur whenever mails/news are
   fetched, that means that mails are removed from the new folder,
   without being noticed in any way by the user.

   It would be great to have an option, that says: only move mails to
   cur when the folders summary was displayed.  (mutt has an option like
   this and the use case is, that more than one person accesses the
   maildir folder)

cheers
sascha

PS. I'm not subscribed to ding@gnus.org, please CC me in any reply.

-- 
"Anyone who slaps a 'this page is best viewed with Browser X' label on a Web 
page appears to be yearning for the bad old days, before the Web, when you had 
very little chance of reading a document written on another computer, another 
word processor, or another network." -- Tim Berners-Lee, July 1996

Re: Gnus maildir backend issues

[Paul Jarc]

Sascha Wilde <wilde@sha-bang.de> wrote:
> 1. the backend creates empty files in .nnmaildir/num for each mail seen
>    and never deletes them.  This results in thousands and thousands of
>    (empty) files

Not quite.  There are thousands and thousands of links to a small
number of empty files (often just one per maildir), so the amount of
disk space used is very small.

>    I don't understand why these files are needed and I definitely
>    consider it a bug, that they are never removed.

They're used to keep track of article number assignments.  If you
remove them, you may cause some data corruption.

Is this causing any actual problems for you, or is it an aesthetic
objection?

>    It would be great to have an option, that says: only move mails to
>    cur when the folders summary was displayed.

I can see how that might be useful, but it would take a significant
amount of work.


paul

Re: Gnus maildir backend issues

[Sascha Wilde]

prj@po.cwru.edu (Paul Jarc) wrote:

> Sascha Wilde <wilde@sha-bang.de> wrote:
>> 1. the backend creates empty files in .nnmaildir/num for each mail seen
>>    and never deletes them.  This results in thousands and thousands of
>>    (empty) files
>
> Not quite.  There are thousands and thousands of links to a small
> number of empty files (often just one per maildir), so the amount of
> disk space used is very small.

I wasn't worrying about disc space, but a bit about inodes, but as they
are mainly links it turns out my concern wasn't justified.  :)

>>    I don't understand why these files are needed and I definitely
>>    consider it a bug, that they are never removed.
>
> They're used to keep track of article number assignments.  If you
> remove them, you may cause some data corruption.

Hu?  I removed the entire .nnmaildir sometimes and had no problems.
Even more, I wouldn't expect to encounter an problems as I would expect
the original maildir to be self contained.  Actually for most other MUAs
it is, isn't it?

> Is this causing any actual problems for you, or is it an aesthetic
> objection?

Well..  First of all I consider it problematic in general, to store meta
data forever.  Why would one need any information on mails deleted years
ago?  (You might call this "aesthetic".)

The actual problem is:  We are having a bunch of maildir folders on an
nfs volume and sometimes want to do operations like find-greps on all of
them.  In this case all the thousands of files significantly slow down
operations -- or excluding all the .nnmaildir directories makes entering
the commands more cumbersome.  Not a big issue, but annoying.

>>    It would be great to have an option, that says: only move mails to
>>    cur when the folders summary was displayed.
>
> I can see how that might be useful, but it would take a significant
> amount of work.

While it would be very helpful for our current use case (being two, soon
three admins working together on one shared maildir spam folder, me
using gnus and my coworker using mutt), I have to admit, that it
probably isn't worth much afford.

cheers
sascha
-- 
Sascha Wilde : "Ist es nicht schon schlimm genug, dass ICH hier rumtrolle?"
             : (Henning Leise in d.o.c.)

Re: Gnus maildir backend issues

[Paul Jarc]

Sascha Wilde <wilde@sha-bang.de> wrote:
> prj@po.cwru.edu (Paul Jarc) wrote:
>> [.nmaildir/num/ files are] used to keep track of article number
>> assignments.  If you remove them, you may cause some data
>> corruption.
>
> Hu?  I removed the entire .nnmaildir sometimes and had no problems.

It depends on what Gnus features you use.  Removing those files will
make nnmaildir reassign article numbers, which may be different from
the article numbers that were assigned the first time.  If you use any
features that store article numbers outside the backend, such as the
cache or agent, then you'll have trouble.

Also, deleting just some of the num/* files, and leaving the rest of
.nnmaildir intact, will definitely cause trouble, even for just
gathering new messages.

> Even more, I wouldn't expect to encounter an problems as I would expect
> the original maildir to be self contained.  Actually for most other MUAs
> it is, isn't it?

Could be; I'm not familiar with any others enough to say.  It's
certainly possible to design a MUA from scratch so that all metadata
can be regenerated at any time.  nnmaildir does that as much as it can
(removing the nov/* files is safe), but Gnus requires numeric IDs
assigned to articles, and some features require those IDs to never
change.  So deleting the information of what numbers have been
assigned will screw up those features.

> First of all I consider it problematic in general, to store meta
> data forever.  Why would one need any information on mails deleted
> years ago?

As much of the metadata as possible is deleted along with the article
itself.  All that is kept is the fact that the article number has
already been assigned, so it won't be reused.

> excluding all the .nnmaildir directories makes entering the commands
> more cumbersome.  Not a big issue, but annoying.

If I had to do that often, I'd make a small script or some other kind
of shortcut.


paul