Avoiding the "huge INBOX of death"

Avoiding the "huge INBOX of death"

[Raphaël Fournier-S'niehotta]

Hello fellow notmuchers,

I am a longtime "classic" mutt user, willing to switch to a notmuch-based
solution (probably neomutt). However, as far as I understand, I will have to
deal with a "huge INBOX of death", as put by Anarcat in a recent blog post [1],
that when all emails remain in a single maildir, which may the become very
large. 

I would like to avoid this, for quota reasons on my work imap server first, and
also because I sometimes have to use webmail access, with poor searching
capabilities.

Thus, I think I need to have my emails sorted into maildir folders according to
the notmuch tags. Namely, emails with the "inbox" tag should stay in the INBOX
maildir, those with "archive" should be moved to Archives (all synchronized with
the server). As a bonus, it would be great to have a few more features, like
"mails in Archives older than 6 months are moved to Archives/<year>" (kept
locally and not stored on the server).

As far as I understand, "afew" and its MailMover feature could be useful for my
case, but it does not seem currently maintained (plus, very few mentions of it
on the notmuch archives) and Michael Turquette said it's buggy [2], so I would
like more feedback to be sure before letting it mess with my maildir.

Another tool that could be useful is the "maildir-notmuch-sync" script written
by Ethan Schoonover and adapted by Michael Turquette [3]. However, it seems
dedicated to handling the synchronization of Gmail labels, not a generic imap
server hierarchy.

Perhaps there is another solution to deal with this in a notmuch-based setup, I
would very much like any of your feedback.

[1]: https://anarc.at/blog/2016-05-12-email-setup/
[2]: http://deferred.io/2016/01/17/state-of-the-dotfiles.html
[3]: http://deferred.io/2016/01/18/notmuch-tags-gmail-labels-bidirectional-sync.html

-- 
Raphaël Fournier-S'niehotta
http://raphael.fournier-sniehotta.fr/apropos

Re: Avoiding the "huge INBOX of death"

[David Bremner]

[-- Attachment #1: Type: text/plain, Size: 1737 bytes --]

Raphaël Fournier-S'niehotta <raphael@raphaelfournier.net> writes:

> Hello fellow notmuchers,
>
> I am a longtime "classic" mutt user, willing to switch to a notmuch-based
> solution (probably neomutt). However, as far as I understand, I will have to
> deal with a "huge INBOX of death", as put by Anarcat in a recent blog post [1],
> that when all emails remain in a single maildir, which may the become very
> large. 
>

TBH, I don't understand what specific problem anarcat is having, it
sounds like a workflow issue.  I currently have 1074 messages tagged
inbox, out of about 450K messages.

Notmuch doesn't care how you organize your mail files on disk, as long
as they are one message per file, and one tree of messages. So feel free
to store them e.g. by date using procmail if you are worried about
performance, or in some other set of folders with semantic 

For performance I'd also say measure before you optimize. I currently
have about 77K messages in one of my maildirs, with no noticable
performance hit.  Of course you want e.g. directory indexing on ext3,
but that is default on anything recent.

> Thus, I think I need to have my emails sorted into maildir folders according to
> the notmuch tags. Namely, emails with the "inbox" tag should stay in the INBOX
> maildir, those with "archive" should be moved to Archives (all synchronized with
> the server). As a bonus, it would be great to have a few more features, like
> "mails in Archives older than 6 months are moved to Archives/<year>" (kept
> locally and not stored on the server).

Generally, you should search directly by folder, rather than trying to
sync tags and folders.

I'm not familiar with afew as a user.

d



[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 647 bytes --]

Re: Avoiding the "huge INBOX of death"

[Vladimir Marek]

I can't speak for everyone, but here's what I am using.

I do have 'messages' directory which contains 'imap' and 'archive'.

messages
 \-imap
 \-archive

The imap directory is being synchronized via offlineimap to the server.

In the offlineimap postsync hook I call a script which does some
bookkeeping on the newly received messages and it also marks old and
already read and not flagged emails by the 'old-tread' tag.

N=notmuch
while : ; do
   AGE=100d # 100 days
   # Find all threads containing old messages
   OLD_MESSAGES=$( $N search --output=threads not tag:archive and date:..$AGE and not tag:unread and not tag:flagged )
   [ -n "$OLD_MESSAGES" ] || break
   # Find threads containing newer messages
   NEW_MESSAGES=$( $N search --output=threads not tag:archive and not '(' date:..$AGE or tag:unread or tag:flagged ')' )
   # remove the threads containing newer messages
   OLD_THREADS=$( /usr/xpg4/bin/fgrep -v -x -f <(echo "$NEW_MESSAGES") < <(echo "$OLD_MESSAGES") )
   [ -n "$OLD_THREADS" ] || break
   $N tag +old-thread -- $OLD_THREADS
   break
done

# If old thread received newer message, unmark it too
$N tag -old-thread -- tag:old-thread and '(' not date:..$AGE or tag:unread or tag:flagged or tag:archive ')'


Once in a while I review the old-thread messages and assign them by hand
'archive' flag. The postsync hook is looking for this flag and moves all
messages to be archived into archive folder. You can list files to be
archived by

$N search --output=files 'path:imap/**' tag:archive

I won't list the commands to move files from messages/imap to
messages/archive since I have a bit more complex scenario. But it should
be straightforward.

On next run offlineimap sees that the messages disappeared from 'imap'
folder and removes them from imap server.

$ notmuch count '*'
734346
$ notmuch count 'path:imap/**'
18419


In my case I am server side filtering the emails into several folders so
that I have at least some order in emails when reading email via phone,
so 'path:imap/**' contains all the messages in whole imap tree.

$ notmuch count 'path:imap/** and tag:inbox and tag:unread'
11

I'm trying to have my virtual INBOX empty most of the time.

I'm interested what tricks others use!
Cheers
-- 
	Vlad

Re: Avoiding the "huge INBOX of death"

[Olivier Schwander]

Le 19 Jul 2016 11:42, Raphaël Fournier-S'niehotta a écrit:
> Hello fellow notmuchers,
> 
> I am a longtime "classic" mutt user, willing to switch to a notmuch-based
> solution (probably neomutt). However, as far as I understand, I will have to
> deal with a "huge INBOX of death", as put by Anarcat in a recent blog post [1],
> that when all emails remain in a single maildir, which may the become very
> large. 
> 
> I would like to avoid this, for quota reasons on my work imap server first, and
> also because I sometimes have to use webmail access, with poor searching
> capabilities.
> 
> Thus, I think I need to have my emails sorted into maildir folders according to
> the notmuch tags. Namely, emails with the "inbox" tag should stay in the INBOX
> maildir, those with "archive" should be moved to Archives (all synchronized with
> the server). As a bonus, it would be great to have a few more features, like
> "mails in Archives older than 6 months are moved to Archives/<year>" (kept
> locally and not stored on the server).
> 
> As far as I understand, "afew" and its MailMover feature could be useful for my
> case, but it does not seem currently maintained (plus, very few mentions of it
> on the notmuch archives) and Michael Turquette said it's buggy [2], so I would
> like more feedback to be sure before letting it mess with my maildir.
> 
> Another tool that could be useful is the "maildir-notmuch-sync" script written
> by Ethan Schoonover and adapted by Michael Turquette [3]. However, it seems
> dedicated to handling the synchronization of Gmail labels, not a generic imap
> server hierarchy.
> 
> Perhaps there is another solution to deal with this in a notmuch-based setup, I
> would very much like any of your feedback.
> 
> [1]: https://anarc.at/blog/2016-05-12-email-setup/
> [2]: http://deferred.io/2016/01/17/state-of-the-dotfiles.html
> [3]: http://deferred.io/2016/01/18/notmuch-tags-gmail-labels-bidirectional-sync.html

You could use fdm [1] to store messages into a folder hierarchy based on
the date in the messages. In the past, I used something like that to
periodically move messages in my inbox to sub-folder with the year:

account "archives" disabled maildirs {
   "${mailpath}INBOX"
}
action "archive" maildir "${mailpath}%[maildir]/%[mail_year]"
match account "archives" {
    match age > 6 months action "archive"
    match all action keep
}

Nowadays, I use a custom script [2] to mirror some notmuch queries to
maildirs (ran periodically in my crontab), such like:

notmuch-imap.sh ~/Maildir/ NOT tag:archive AND NOT tag:killed AND NOT tag:spam AND tag:inbox
notmuch-imap.sh ~/Maildir/.todo NOT tag:archive AND NOT tag:killed AND NOT tag:spam AND tag:todo

Both solutions works server-side, since I control my own mail server but
it may probably used in an offlineimap hook.

Best,

Olivier

[1]: https://github.com/nicm/fdm
[2]: https://chadok.info/~oschwand/poubelle/e53f1d9ac30909fd17ca98f4160375bbc9a6aca5/notmuch-imap.sh.html

Re: Avoiding the "huge INBOX of death"

[fauno]

[-- Attachment #1: Type: text/plain, Size: 402 bytes --]

Raphaël Fournier-S'niehotta <raphael@raphaelfournier.net> writes:

> Hello fellow notmuchers,
> Perhaps there is another solution to deal with this in a notmuch-based setup, I
> would very much like any of your feedback.

i made toomuch[^0] to archive email by year-month and it works really
well for me :P

[^0]: https://github.com/fauno/toomuch

-- 
http://utopia.partidopirata.com.ar/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 584 bytes --]