Initial tagging

Initial tagging

[James Vasile]

I'm slowly groping my way to using the notmuch emacs client as my
routine MUA.  As I coerce it into tagging and displaying the way I want,
the next big question was automatically tagging things and getting them
in to notmuch.

I'm curious as to what people are doing in this regard.

My solution involves cron running a sync_email script.  Sync_email does
the correct dance to make sure it only ever runs one instance at a time.
It also logs to syslog.  The script runs offlineimap, a mail_filter
script that sorts mail in maildirs (for wanderlust, the MUA I'm hoping
to leave behind), and then finally a shell script to do notmuch new and
initial tagging.

The tagging script uses the inbox tag to identify new mail, tags it
according to criteria, then removes the inbox tag from anything it found
a match for.  Uncategorized mail keeps the inbox tag so I can inspect it
later and make rules for it (or tag it manually).

Also, prepending "tag:inbox and" to search criteria restricts the
tagging to a small subset of the db, which makes the tagging script run
fairly quickly.  My unexpurgated tagging script has almost 100 rules for
tagging, and I expect it to grow over time.

################## notmuch-tag.sh ################
#!/bin/bash

bin=/usr/local/bin/notmuch

function notmuch {    
    echo $1
    while [ 1 -gt 0 ]; do
	result=`$bin $1 2>&1` 
	regex="already locked"

	if [[ $result =~ $regex ]]; then
	    echo "Xapian DB busy.  Retrying in 2 seconds"
	else
	    if [ -n "$result" ]; then
		echo $result
	    fi
	    return
	fi

	sleep 2
    done
}

function tag_new { notmuch "tag $1 tag:inbox and ($2)"; }
function blacklist { tag_new "-inbox -unread +delete" $1; }

notmuch new

blacklist "from:xxx@example.com or from yyy@example.com"

# voicemail
tag_new "-inbox +voicemail"  "from:ast@example.com"

# friends
tag_new "+friend +mathieu" "mathieu or ejm2106 or emily@example.com"
tag_new "+friend +balktick" "balktick"

# open community services
tag_new "+ocs" "open community services or opencommunityservices"
   
# okos
tag_new "+okos" "jim and glaser and not LinkedIn"
tag_new "+okos" "joshlevy.ny@example.com"
tag_new "+okos" "enright@example.com"

# book liberator
tag_new "+bklib" "wnf@example.com or bkrpr"

# joomla
tag_new "+osm" "from:waring or to:waring"
tag_new "+osm" "from:dave.huelsmann@example.com or to:dave.huelsmann@example.com"
tag_new "+osm" "james.vasile@example.com"
tag_new "+osm" "joomla"

#lists
tag_new "+list +notmuch" "to:notmuchmail.org or notmuch"
tag_new "+list +stumpwm" "to:stumpwm-devel@nongnu.org or stumpwm"
tag_new "+list +bklib" "to:bklib@googlegroups.com"
   
## Catchalls for sflc, hv, etc.
tag_new "+sflc" "not tag:list and not tag:friend and softwarefreedom.org and not tag:osm"
tag_new "+sflc" "to:firm@example.com"
tag_new "+hv" "hackervisions.org and not tag:list and not tag:friend"
tag_new "+gmail" "(to:jvasile@example.com or from:jvasile@example.com) and not tag:list and not tag:friend"

## Mark mine unread
tag_new "-unread" "from:james@example.com"
tag_new "-unread" "from:vasile@example.com"
tag_new "-unread" "from:james.vasile@example.com"

## Remove inbox tag
tag_new "-inbox" "tag:sflc or tag:hv or tag:list or tag:osm or tag:okos or tag:friend or tag:bklib"


############# sync_email #########################
#!/bin/sh

## Sync email unless we're already in the process of syncing.

SCRIPTNAME=`basename $0`
PIDDIR=/home/vasile/var/run/${SCRIPTNAME}
PIDFILE=${PIDDIR}/${SCRIPTNAME}.pid

## Do the double-lock with a dir and a pid file
if ! mkdir ${PIDDIR} 2>/dev/null; then

   sleep 3 # give the other process time to write its pid

   if [ -f ${PIDFILE} ]; then
      #verify if the process is actually still running under this pid
      OLDPID=`cat ${PIDFILE}`
      RESULT=`ps -ef | grep ${OLDPID} | grep ${SCRIPTNAME}`  

      if [ -n "${RESULT}" ]; then
        logger -s ${SCRIPTNAME} already running! Exiting
        exit 255
      fi
   fi
fi

## Update pid file
PID=`ps -ef | grep ${SCRIPTNAME} | head -n1 | awk ' {print $2;} '`
echo ${PID} > ${PIDFILE}

logger -s filter done, starting offlineimap
offlineimap -l /home/vasile/.offlineimap/log
logger -s offlineimap done, starting mail filter
mairix --unlock
/home/vasile/bin/mail_filter.py
logger -s mail filter done, starting notmuch tagger
/home/vasile/bin/notmuch-tag.sh > /home/vasile/var/log/notmuch
logger -s notmuch tagger done sync_email finished

## clean up pid file and dir
if [ -f ${PIDFILE} ]; then
    rm -rf ${PIDDIR}
fi

Re: Initial tagging

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 6658 bytes --]

On Thu, 25 Feb 2010 16:25:16 -0500, James Vasile <james@hackervisions.org> wrote:
> I'm curious as to what people are doing in this regard.

I'm currently using a script that does tagging not entirely unlike
yours, (though my script isn't clever to retry a call to
notmuch---believe it or not I'm just calling it manually still, not from
cron).

> The tagging script uses the inbox tag to identify new mail, tags it
> according to criteria, then removes the inbox tag from anything it found
> a match for.  Uncategorized mail keeps the inbox tag so I can inspect it
> later and make rules for it (or tag it manually).

One distinction is that I have all of my "notmuch tag" commands operate
globally rather than just on new messages. One of the things that really
annoyed be about sup was that the support for automatic tagging worked
as a hook on messages as they were processed. So I couldn't use any of
the tags for searches prior to the time that I had added a particular
tag rule. I definitely didn't want to replicate that bug.

I also don't remove the inbox tag from matched mail, since I have a
"notmuch-folders" configuration that takes advantage of the matched tags
in concert with the inbox tag.

I've included my current auto-tagging script (notmuch-poll) below, as
well as the snippet of my .emacs that sets my notmuch-folders variable.

> Also, prepending "tag:inbox and" to search criteria restricts the
> tagging to a small subset of the db, which makes the tagging script run
> fairly quickly.  My unexpurgated tagging script has almost 100 rules for
> tagging, and I expect it to grow over time.

To do my "global" searches quickly, I do a similar subsetting, but it's
much simpler. If I'm adding the "notmuch" tag I do "and not
tag:notmuch". We've even had the proposal of making "notmuch tag" do
that automatically.

Meanwhile, I'm planning on eventually moving entirely away from any tags
that are driven entirely by searches like this. Instead, I'd like to
just have good support for "saved searches" where we have some syntax to
perform string expansion on configured search terms. So where I'm
currently doing:

	notmuch search tag:notmuch

I could instead do:

	notmuch search query:notmuch

with a configuration associating "notmuch" to "to:notmuchmail.org" and
the above would expand to:

	notmuch search '(' to:notmuchmail.org ')'

We've been talking about these saved searches for a while, but we
haven't implemented them yet, (nor decided firmly on the syntax we
want). I just noticed today that sup recently added a similar
saved-search feature with the following syntax:

	notmuch search {notmuch}

I think I might like something like that with more custom syntax, (since
the existing "prefix:" syntax has a fairly standard meaning that doesn't
fit well with this new feature.

With saved-search support in notmuch, I could get rid of my notmuch-poll
script almost entirely, (I'd still want something to automatically
remove the -inbox tag from some messages, but that's about it).

-Carl

##### notmuch-poll #####
echo "Importing new mail"
notmuch new

echo "Running global tag additions to tag new mail"

# Tag bug-mail first, (since we use the bugs tag below)
notmuch tag +bugs from:bugzilla-daemon and not tag:bugs

# Note mail sent specifically to me (excluding bug mail)
notmuch tag +to-me to:cworth@cworth.org and not tag:to-me and not tag:bugs
notmuch tag +to-me to:carl@theworths.org and not tag:to-me and not tag:bugs
notmuch tag +to-me to:carl.d.worth@intel.com and not tag:to-me and not tag:bugs

# And note all mail sent from me
notmuch tag +sent from:cworth@cworth.org and not tag:sent
notmuch tag +sent from:carl@theworths.org and not tag:sent
notmuch tag +sent from:carl.d.worth@intel.com and not tag:sent

# Intel mail deserves some distinction
notmuch tag +intel to:carl.d.worth@intel.com and not tag:intel
notmuch tag +intel from:intel.com and not tag:intel
notmuch tag +intel to:intel-gfx and not tag:intel
notmuch tag +intel to:linux-gfx and not tag:intel

# Next, various free-software projects tagged by recipient
notmuch tag +cairo to:cairographics.org and not tag:cairo
notmuch tag +cairo to:pixman@lists.freedesktop.org and not tag:cairo
notmuch tag +xorg-board to:foundation.x.org and not tag:xorg-board
notmuch tag +xorg to:xorg and not tag:xorg
notmuch tag +xorg to:xcb and not tag:xorg
notmuch tag +announce to:announce@lists.debian.org and not tag:announce
notmuch tag +debian to:lists.debian.org and not tag:debian
notmuch tag +sup to:sup-devel@rubyforge.org and not tag:sup
notmuch tag +lca to:lists.lca2010.org.nz and not tag:lca
notmuch tag +notmuch to:notmuchmail.org and not tag:notmuch
notmuch tag +nickle to:nickle.org and not tag:nickle
notmuch tag +meego '( to:dev@moblin.org or to:dev@lists.moblin.org or to:meego-dev@meego.com )' and not tag:meego

# Finally, a few subject-based tags
notmuch tag +intel subject:xf86-video-intel and not tag:intel
notmuch tag +sup subject:sup-talk and not tag:sup
notmuch tag +pdx subject:pdx and not tag:pdx

# And some sender-based tags
notmuch tag +family from:theworths.org and not tag:family'

# Keep most Debian bug traffic out of my inbox
notmuch tag +debbugs '(' to:bugs.debian.org or from:bugs.debian.org ')' and not tag:debbugs
notmuch tag -inbox tag:inbox and tag:debbugs and not '(' '"Package: cairo"' or '"Package: notmuch"' ')'

echo "Done."

##### a snippet of .emacs #####
(setq notmuch-folders '(("inbox" . "tag:inbox")
		      	("tome" . "tag:inbox and (tag:to-me or tag:family)")
		      	("tome-select" . "tag:inbox and (tag:to-me or tag:family) and not tag:notmuch and not tag:cairo")
			("other" . "tag:inbox AND not (tag:to-me or tag:announce or tag:notmuch or tag:intel or tag:bugs or tag:cairo or tag:xorgboard or tag:xorg or tag:debian or tag:sup or tag:lca or tag:meego or tag:nickle or tag:pdx)")
			("announce" . "tag:inbox AND tag:announce")
			("notmuch" . "tag:inbox AND tag:notmuch")
			("intel" . "tag:inbox AND tag:intel")
			("bugs" . "tag:inbox AND tag:bugs")
			("cairo" . "tag:inbox AND tag:cairo")
			("xorgboard" . "tag:inbox AND tag:xorg-board")
			("xorg" . "tag:inbox AND tag:xorg")
			("debian" . "tag:inbox AND tag:debian")
			("sup" . "tag:inbox AND tag:sup")
			("lca" . "tag:inbox AND tag:lca")
			("meego" . "tag:inbox AND tag:meego")
			("nickle" . "tag:inbox AND tag:nickle")
			("pdx" . "tag:inbox AND tag:pdx")
			("todo" . "tag:todo and not tag:cairo and not tag:notmuch and not tag:intel")
			("inteltodo" . "(tag:todo and tag:intel) or tag:todo-intel")
			("notmuchtodo" . "(tag:todo and tag:notmuch) or tag:todo-notmuch")
			("cairotodo" . "(tag:todo and tag:cairo) or tag:todo-cairo")
			))

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Initial tagging

[James Vasile]

On Thu, 25 Feb 2010 17:02:08 -0800, Carl Worth <cworth@cworth.org> wrote:
> One distinction is that I have all of my "notmuch tag" commands operate
> globally rather than just on new messages. One of the things that really
> annoyed be about sup was that the support for automatic tagging worked
> as a hook on messages as they were processed. So I couldn't use any of
> the tags for searches prior to the time that I had added a particular
> tag rule. I definitely didn't want to replicate that bug.

Sometimes I manually override a rule-based tag.  If automatic tagging
operated on old messages, overrides could be superseded.  When I add
automatic tag rules, I first execute them manually and globally from the
command line.

> To do my "global" searches quickly, I do a similar subsetting, but it's
> much simpler. If I'm adding the "notmuch" tag I do "and not
> tag:notmuch". We've even had the proposal of making "notmuch tag" do
> that automatically.

That sounds like a good idea.  What happened with that proposal?

> Meanwhile, I'm planning on eventually moving entirely away from any tags
> that are driven entirely by searches like this. Instead, I'd like to
> just have good support for "saved searches" where we have some syntax to
> perform string expansion on configured search terms. 

Saved searches is a very good idea.  I look forward to it.

Re: Initial tagging

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 1307 bytes --]

> Sometimes I manually override a rule-based tag.  If automatic tagging
> operated on old messages, overrides could be superseded.  When I add
> automatic tag rules, I first execute them manually and globally from the
> command line.

Right. With my global automatic tags, I don't want to be able to
override them, (and the fact that I *can* by adding/removing tags is a
bug---since, as you point out, the next run of my script will destroy my
changes). That's one of the reasons I want to switch to saved searches
for my own use of automatic tags.

Meanwhile, Martin has talked about having some machine-learning
algorithm apply tags, and have it notice when the user adds/removes
for feedback to improve its learning. So that kind of use case is
something that definitely isn't covered by saved searches.

> > To do my "global" searches quickly, I do a similar subsetting, but it's
> > much simpler. If I'm adding the "notmuch" tag I do "and not
> > tag:notmuch". We've even had the proposal of making "notmuch tag" do
> > that automatically.
> 
> That sounds like a good idea.  What happened with that proposal?

Like many other proposals, it's simply waiting for someone to implement
it.

> Saved searches is a very good idea.  I look forward to it.

Me too! :-)

-Carl


[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Initial tagging

[Jameson Rollins]

[-- Attachment #1: Type: text/plain, Size: 312 bytes --]

On Fri, 26 Feb 2010 01:13:52 -0800, Carl Worth <cworth@cworth.org> wrote:
> > Saved searches is a very good idea.  I look forward to it.
> 
> Me too! :-)

Isn't that what "folders" are?  At least that's what they seem to me.  I
define a specific search, give it a name, and save it as a folder.

jamie.

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: Initial tagging

[martin f krafft]

[-- Attachment #1: Type: text/plain, Size: 795 bytes --]

also sprach Jameson Rollins <jrollins@finestructure.net> [2010.02.26.1600 +0100]:
> > Me too! :-)
> 
> Isn't that what "folders" are?  At least that's what they seem to me.  I
> define a specific search, give it a name, and save it as a folder.

There's a 1:∞ relationship between folders and messages. I might
want a message associated with multiple contexts.

-- 
 .''`.   martin f. krafft <madduck@d.o>      Related projects:
: :'  :  proud Debian developer               http://debiansystem.info
`. `'`   http://people.debian.org/~madduck    http://vcs-pkg.org
  `-  Debian - when you have better things to do than fixing systems
 
"you know you're a hopeless geek when you misspell 'date' as 'data'"
                                                   -- branden robinson

[-- Attachment #2: Digital signature (see http://martin-krafft.net/gpg/) --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Initial tagging

[micah anderson]

[-- Attachment #1: Type: text/plain, Size: 1789 bytes --]

On Thu, 25 Feb 2010 16:25:16 -0500, James Vasile <james@hackervisions.org> wrote:

> I'm curious as to what people are doing in this regard.

I am pretty good at evangelizing notmuch and converted another
person[0]... in doing so I realized I had to walk through a few issues
to get them started. 

The initial documentation for notmuch is really great, but the other
magical bits are missing, like "holy cow all my email is in my inbox
now!"; "how do I deal with folders?"; "do i run notmuch new every
time?"; "do you auto-tag things?"; "how do you deal with attachments?";
"how do you read encrypted email"? etc. etc. all of these getting
started with notmuch things... including the script that James just
posted, the getmail tagging stuff that was posted a little while back,
etc. 

This stuff should go up on the wiki! There is a "Tips and Tricks for
using notmuch with Emacs" section, but more general things like James'
script and other people's scripts really should be put up there so
people who are new can figure out how to get started faster.

So this is just a call out to try and encourage people to put these
useful things that they have sent out to the list up onto the wiki! Most
people don't read all the past archives of a list when they first join,
but do check out the website for info on how to get going. I'm not
saying don't send things to the list either, just don't let the list be
the sink where these great tips go to die.

micah


0. indirectly I believe am also responsible, through 3rd party
evangelizing, for James entering the scene as well, and I'm happy about
that because I have been quite pleased with his enhancements, thanks
james!. now I just need to buy him a drink and talk him into fixing the
openpgp bits ;)

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: Initial tagging

[micah anderson]

[-- Attachment #1: Type: text/plain, Size: 1789 bytes --]


Hey james,

i really like your notmuch-retry bits, I've taken that shell script and
integrated it into my tagging script, thanks! I'm curious if you are
going to update your patch for notmuch.el ("Calls to notmuch get queued
and executed asynchronously") to use this, or if you are going to have
those two continue to be separate ways of retrying?

On Thu, 25 Feb 2010 16:25:16 -0500, James Vasile <james@hackervisions.org> wrote:

> The tagging script uses the inbox tag to identify new mail, tags it
> according to criteria, then removes the inbox tag from anything it found
> a match for.  Uncategorized mail keeps the inbox tag so I can inspect it
> later and make rules for it (or tag it manually).

Why do you tag new mail with tags, rather than construct folder searches
to create views? 

For example, rather than adding more and more data to the database every
time your script runs and does something like the following:

tag_new "+list +notmuch" "to:notmuchmail.org or notmuch"

Why don't you instead create a folder in your .emacs like this:

(setq notmuch-folders '(("inbox" . "tag:inbox")
                        ("unread" . "tag:inbox AND tag:unread")
                        ("notmuch" . "to:notmuchmail.org")))

That way, if you want to read the notmuch mailing list, you would simply
open that folder, which would present you with the results of that
search. 

You could do different things with the tag:unread and tag:inbox flags,
depending on what you wanted to accomplish. Namely, you may wish to only
have a view of all the unread notmuch mailing list messages, in which
case you would make that query "tag:unread AND to:notmuchmail.org".

I'm just curious about the rationale for doing it one way, over the
other. 

micah

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: Initial tagging

[James Vasile]

On Fri, 26 Feb 2010 12:14:45 -0500, micah anderson <micah@riseup.net> wrote:
> On Thu, 25 Feb 2010 16:25:16 -0500, James Vasile <james@hackervisions.org> wrote:
> This stuff should go up on the wiki! There is a "Tips and Tricks for
> using notmuch with Emacs" section, but more general things like James'
> script and other people's scripts really should be put up there so
> people who are new can figure out how to get started faster.

Good point.  I will put some stuff on the wiki.

> 0. indirectly I believe am also responsible, through 3rd party
> evangelizing, for James entering the scene as well, and I'm happy about
> that because I have been quite pleased with his enhancements, thanks
> james!. now I just need to buy him a drink and talk him into fixing the
> openpgp bits ;)

Yes, you are my upstream evangelist.  You put the bug in Karl Fogel's
ear (although he hasn't actually tried it) and he put it in mine.  Glad
you're liking my elisp.

I've spent a bunch of time thinking about mail setups (I actually
sketched out what I'd need to do to mutt to get virtual folders there),
so it's great to see people making serious strides toward getting it
right.

Notmuch isn't a mail client so much as a mail kit.  I'm happy to be
hacking on it and making my mail work the way I want.

-J

Re: Initial tagging

[James Vasile]

On Fri, 26 Feb 2010 20:33:08 -0500, micah anderson <micah@riseup.net> wrote:
> 
> Hey james,
> 
> i really like your notmuch-retry bits, I've taken that shell script and
> integrated it into my tagging script, thanks! I'm curious if you are
> going to update your patch for notmuch.el ("Calls to notmuch get queued
> and executed asynchronously") to use this, or if you are going to have
> those two continue to be separate ways of retrying?

Ah, I updated my local version and am putting it through its paces, but
it's already so much better that I should just send in the revised
patch.  I'll clean it up and do that tomorrow.

> Why do you tag new mail with tags, rather than construct folder searches
> to create views? 

I do a lot of on-the-fly searching and don't want to have to remember
notmuch's email address.  I suppose I could go to my notmuch folder and
then filter from there, but right now I'm much more likely to do M-x
notmuch-search/tag:notmuch and micah.

Re: Initial tagging

[James Vasile]

On Fri, 26 Feb 2010 23:03:34 -0500, James Vasile <james@hackervisions.org> wrote:
> On Fri, 26 Feb 2010 20:33:08 -0500, micah anderson <micah@riseup.net> wrote:
> > 
> > Hey james,
> > 
> > i really like your notmuch-retry bits, I've taken that shell script and
> > integrated it into my tagging script, thanks! I'm curious if you are
> > going to update your patch for notmuch.el ("Calls to notmuch get queued
> > and executed asynchronously") to use this, or if you are going to have
> > those two continue to be separate ways of retrying?
> 
> Ah, I updated my local version and am putting it through its paces, but
> it's already so much better that I should just send in the revised
> patch.  I'll clean it up and do that tomorrow.

Insomnia: when it comes to sleep, my loss is your gain. 

I opened a github account and the updated bits are in the retry branch
at git@github.com:jvasile/notmuch.git

Carl, if you want the retry functionality, feel free to pull from there
or request I post those patches to this list.

Note that the retry branch depends on notmuch-retry being in your path,
but 'make install' does not know about notmuch-retry.

Next step for this branch is to have all errors go to *Notmuch errors*
rather than error buffers named for the tag commands.