(auto-)tagging sent messages

(auto-)tagging sent messages

[Florian Friesdorf]

[-- Attachment #1: Type: text/plain, Size: 934 bytes --]


With my current setup (implying that it might be already possible with
notmuch but I am not aware how), when sending a message it is put in
"/home/cfl/mail/sent" via Fcc.

In order for it to be seen by notmuch I then need to run:
notmuch new

and for it to be tagged as sent:
notmuch tag +sent -- not tag:sent and from:flo@chaoflow.net

Should / how could this be automated?

Further, for certain mails I sent (like this one ) I would like a
WAITING tag (or similar) in order to indicate that I am waiting for an
answer. Currently I set this manually. Could this be achieved through
some indicators via message mode or similar means? e.g.:

<#notmuch tag=WAITING> analog to <#secure method=pgpmime mode=sign>

regards
florian
-- 
Florian Friesdorf <flo@chaoflow.net>
  GPG FPR: 7A13 5EEE 1421 9FC2 108D  BAAF 38F8 99A3 0C45 F083
Jabber/XMPP: flo@chaoflow.net
IRC: chaoflow on freenode,ircnet,blafasel,OFTC

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: (auto-)tagging sent messages

[Pieter Praet]

On Thu, 14 Apr 2011 10:03:12 +0200, Florian Friesdorf <flo@chaoflow.net> wrote:
> 
> With my current setup (implying that it might be already possible with
> notmuch but I am not aware how), when sending a message it is put in
> "/home/cfl/mail/sent" via Fcc.

This can be configured with the variable notmuch-fcc-dirs,
eg. disable it with:  (setq notmuch-fcc-dirs nil)

> In order for it to be seen by notmuch I then need to run:
> notmuch new
> 
> and for it to be tagged as sent:
> notmuch tag +sent -- not tag:sent and from:flo@chaoflow.net
> 
> Should / how could this be automated?

For this to be possible, "notmuch new" needs the ability to selectively
index individual files/folders, which it currently does not.

See this (non-)discussion:
id:"AANLkTikddX=fYiDmnN2Jx3ERcJndw=80+5_07=M45MQ=@mail.gmail.com"

> Further, for certain mails I sent (like this one ) I would like a
> WAITING tag (or similar) in order to indicate that I am waiting for an
> answer. Currently I set this manually. Could this be achieved through
> some indicators via message mode or similar means? e.g.:
> 
> <#notmuch tag=WAITING> analog to <#secure method=pgpmime mode=sign>

No idea how this works since I don't follow Jameson's crypto branch,
but it's probably not a good idea for this use case since (I presume)
the indicator will become permanently lodged in the content of the
mail, which isn't where highly volatile metadata belongs.

> regards
> florian
> -- 
> Florian Friesdorf <flo@chaoflow.net>
>   GPG FPR: 7A13 5EEE 1421 9FC2 108D  BAAF 38F8 99A3 0C45 F083
> Jabber/XMPP: flo@chaoflow.net
> IRC: chaoflow on freenode,ircnet,blafasel,OFTC
Non-text part: application/pgp-signature
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch

Peace

-Pieter

Re: (auto-)tagging sent messages

[Tom Prince]

> > Further, for certain mails I sent (like this one ) I would like a
> > WAITING tag (or similar) in order to indicate that I am waiting for an
> > answer. Currently I set this manually. Could this be achieved through
> > some indicators via message mode or similar means? e.g.:
> > 
> > <#notmuch tag=WAITING> analog to <#secure method=pgpmime mode=sign>
> 
> No idea how this works since I don't follow Jameson's crypto branch,
> but it's probably not a good idea for this use case since (I presume)
> the indicator will become permanently lodged in the content of the
> mail, which isn't where highly volatile metadata belongs.

I think the idea isn't to save it in the message, but to postprocess the
message when sending, remove the marker and add the tag instead.

  Tom

Re: (auto-)tagging sent messages

[Pieter Praet]

On Sat, 16 Apr 2011 15:23:44 -0400, Tom Prince <tom.prince@ualberta.net> wrote:
> > > Further, for certain mails I sent (like this one ) I would like a
> > > WAITING tag (or similar) in order to indicate that I am waiting for an
> > > answer. Currently I set this manually. Could this be achieved through
> > > some indicators via message mode or similar means? e.g.:
> > > 
> > > <#notmuch tag=WAITING> analog to <#secure method=pgpmime mode=sign>
> > 
> > No idea how this works since I don't follow Jameson's crypto branch,
> > but it's probably not a good idea for this use case since (I presume)
> > the indicator will become permanently lodged in the content of the
> > mail, which isn't where highly volatile metadata belongs.
> 
> I think the idea isn't to save it in the message, but to postprocess the
> message when sending, remove the marker and add the tag instead.
> 
>   Tom

Seems like a Catch-22 to me.

The marker needs to remain there long enough for Notmuch to index and
tag the message appropriately (which happens long after message-mode has
left the building) so it's up to Notmuch itself (as opposed to a
message-mode hook) to do the pre-/postprocessing.

Yet it remains Notmuch's policy to *never* alter your mailstore, aside
from Maildir flags.

This would also invite outsiders to sprinkle their mails with "<#notmuch
tag=WAITING>", messing up you workflow.


Peace

-Pieter

Re: (auto-)tagging sent messages

[Tom Prince]

On 2011-04-17, Pieter Praet wrote:
> On Sat, 16 Apr 2011 15:23:44 -0400, Tom Prince <tom.prince@ualberta.net> wrote:
> > > > Further, for certain mails I sent (like this one ) I would like a
> > > > WAITING tag (or similar) in order to indicate that I am waiting for an
> > > > answer. Currently I set this manually. Could this be achieved through
> > > > some indicators via message mode or similar means? e.g.:
> > > > 
> > > > <#notmuch tag=WAITING> analog to <#secure method=pgpmime mode=sign>
> > > 
> > > No idea how this works since I don't follow Jameson's crypto branch,
> > > but it's probably not a good idea for this use case since (I presume)
> > > the indicator will become permanently lodged in the content of the
> > > mail, which isn't where highly volatile metadata belongs.
> > 
> > I think the idea isn't to save it in the message, but to postprocess the
> > message when sending, remove the marker and add the tag instead.
> > 
> >   Tom
> 
> Seems like a Catch-22 to me.
> 
> The marker needs to remain there long enough for Notmuch to index and
> tag the message appropriately (which happens long after message-mode has
> left the building) so it's up to Notmuch itself (as opposed to a
> message-mode hook) to do the pre-/postprocessing.

Not if message mode post-processing is hooked up to something like
notmuch-deliver, that indexes and tags the message on send.

  Tom

Re: (auto-)tagging sent messages

[Pieter Praet]

On Sun, 17 Apr 2011 19:17:40 -0400, Tom Prince <tom.prince@ualberta.net> wrote:
> On 2011-04-17, Pieter Praet wrote:
> > On Sat, 16 Apr 2011 15:23:44 -0400, Tom Prince <tom.prince@ualberta.net> wrote:
> > > > > Further, for certain mails I sent (like this one ) I would like a
> > > > > WAITING tag (or similar) in order to indicate that I am waiting for an
> > > > > answer. Currently I set this manually. Could this be achieved through
> > > > > some indicators via message mode or similar means? e.g.:
> > > > > 
> > > > > <#notmuch tag=WAITING> analog to <#secure method=pgpmime mode=sign>
> > > > 
> > > > No idea how this works since I don't follow Jameson's crypto branch,
> > > > but it's probably not a good idea for this use case since (I presume)
> > > > the indicator will become permanently lodged in the content of the
> > > > mail, which isn't where highly volatile metadata belongs.
> > > 
> > > I think the idea isn't to save it in the message, but to postprocess the
> > > message when sending, remove the marker and add the tag instead.
> > > 
> > >   Tom
> > 
> > Seems like a Catch-22 to me.
> > 
> > The marker needs to remain there long enough for Notmuch to index and
> > tag the message appropriately (which happens long after message-mode has
> > left the building) so it's up to Notmuch itself (as opposed to a
> > message-mode hook) to do the pre-/postprocessing.
> 
> Not if message mode post-processing is hooked up to something like
> notmuch-deliver, that indexes and tags the message on send.
> 
>   Tom

Hadn't looked into notmuch-deliver yet.

Now that I did, I stand corrected, good sir!


Peace

-Pieter

Re: (auto-)tagging sent messages

[Jameson Graef Rollins]

[-- Attachment #1: Type: text/plain, Size: 2582 bytes --]

On Sat, 16 Apr 2011 14:57:07 +0200, Pieter Praet <pieter@praet.org> wrote:
> > Further, for certain mails I sent (like this one ) I would like a
> > WAITING tag (or similar) in order to indicate that I am waiting for an
> > answer. Currently I set this manually. Could this be achieved through
> > some indicators via message mode or similar means? e.g.:
> > 
> > <#notmuch tag=WAITING> analog to <#secure method=pgpmime mode=sign>
> 
> No idea how this works since I don't follow Jameson's crypto branch,
> but it's probably not a good idea for this use case since (I presume)
> the indicator will become permanently lodged in the content of the
> mail, which isn't where highly volatile metadata belongs.

Hey, Pieter.  This doesn't actually have anything to do with the crypto
branch.  These indicators are how emacs handles signing/encryption of
outgoing messages in emacs message-mode, as well as other things like
adding attachments.  It's not actually a part of notmuch.

When using emacs message-mode to compose a message, you can add tags
like these to the message body.  When you then tell emacs to send the
message, emacs reads these tags and performs the specified operation,
like signing the message or adding an attachment.  Emacs then removes
the tags so they're never actually stored in the message.  These tags
are also not interpreted when reading a message, only when composing.

I don't think it's a bad idea to extend this system to benefit notmuch,
by instructing emacs message mode to add tags to outgoing messages.
Like someone suggested, though, it should really be done by having
notmuch index the outgoing message at the time of sending, which would
best be handled by telling notmuch to index an individual message.

Having notmuch index a single message would I think be incredibly
useful, for a lot of different reasons.  I think it would enable a lot
of nice integration with other mail handling tools.  I can imagine this
being done in a couple of ways:

By giving "notmuch new" a path to a message in the store:

  notmuch new /path/to/message

By feeding "notmuch new" a message on stdin, and then having it write
the message to a specified location:

  notmuch new /path/to/maildir <message

You could then add the ability to add tags at the time of index:

  notmuch new +foo +bar /path/to/message

I really think it would not be hard to enable notmuch to do this.  I
wonder what other things.  I think it would make things much easier for
interaction with procmail and spam filters.

jamie.

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: (auto-)tagging sent messages

[Austin Clements]

On Sun, Apr 17, 2011 at 2:23 PM, Jameson Graef Rollins
<jrollins@finestructure.net> wrote:
> By giving "notmuch new" a path to a message in the store:
>
>  notmuch new /path/to/message
>
> By feeding "notmuch new" a message on stdin, and then having it write
> the message to a specified location:
>
>  notmuch new /path/to/maildir <message
>
> You could then add the ability to add tags at the time of index:
>
>  notmuch new +foo +bar /path/to/message

I think passing specific messages or directories to notmuch new would
enable some nice features; beyond immediate indexing any tagging of
sent mail, it could be combined with inotify to immediately index any
new mail (something I've been meaning to play with but haven't yet).

However, concurrency and atomicity have to be considered carefully
with any proposal like this.  For example,
  notmuch new +foo +bar /path/to/message
has concurrency hazards: the caller has to store the message before
invoking notmuch new, but if the machine crashes between these two
operations, the message will be indexed with default tags on the next
notmuch new.  Similarly, a chance unrelated invocation of notmuch new
immediately after the message is stored would result in the new
message being indexed with default tags.

Re: (auto-)tagging sent messages

[servilio]

On 18 April 2011 16:26, Austin Clements <amdragon@mit.edu> wrote:
> On Sun, Apr 17, 2011 at 2:23 PM, Jameson Graef Rollins
> <jrollins@finestructure.net> wrote:
>> By giving "notmuch new" a path to a message in the store:
>>
>>  notmuch new /path/to/message
>>
>> By feeding "notmuch new" a message on stdin, and then having it write
>> the message to a specified location:
>>
>>  notmuch new /path/to/maildir <message
>>
>> You could then add the ability to add tags at the time of index:
>>
>>  notmuch new +foo +bar /path/to/message

I like more the idea of using a CLI option, e.g.:

notmuch new --tag=foo --tag=bar /path/to/message

> I think passing specific messages or directories to notmuch new would
> enable some nice features; beyond immediate indexing any tagging of
> sent mail, it could be combined with inotify to immediately index any
> new mail

I would add "efficiently", it can be done right now, but it would scan
the whole mail sub-tree everytime.

> However, concurrency and atomicity have to be considered carefully
> with any proposal like this.  For example,
>  notmuch new +foo +bar /path/to/message
> has concurrency hazards: the caller has to store the message before
> invoking notmuch new, but if the machine crashes between these two
> operations, the message will be indexed with default tags on the next
> notmuch new.  Similarly, a chance unrelated invocation of notmuch new
> immediately after the message is stored would result in the new
> message being indexed with default tags.

One solution to both issues would be to add the feature to "notmuch
new" of taking the initial tags from a custom header (e.g.:
X-Notmuchmail-Agent-Initial-Tags), and use
message-ignored-mail-headers[1] to filter this header out the sent
message.

Servilio

[1] http://www.gnu.org/software/emacs/manual/html_mono/message.html#Mail-Headers

Re: (auto-)tagging sent messages

[Michael Radziej]

[-- Attachment #1: Type: text/plain, Size: 674 bytes --]

Hi Florian!

On Thu, 14 Apr 2011 10:03:12 +0200, Florian Friesdorf <flo@chaoflow.net> wrote:
> Further, for certain mails I sent (like this one ) I would like a
> WAITING tag (or similar) in order to indicate that I am waiting for an
> answer. Currently I set this manually. Could this be achieved through
> some indicators via message mode or similar means? e.g.:

I use a X-Wait header (like X-Wait: 2, meaning to wait for 2 days). If
there is no activity in the thread within time, it receives additional tags
"late" and "inbox". If an answer to a waiting mail arrives, it receives a tag "expected".

Here's some python code that does the tagging (I haven't tested it):


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: python code --]
[-- Type: text/x-python, Size: 1436 bytes --]

import notmuch, re, time

MY_ADDR = 'my email address'

db =  notmuch.Database(mode=notmuch.Database.MODE.READ_WRITE)

def query(qstring):
    return db.create_query(qstring).search_messages()

def query_threads(qstring):
    return db.create_query(qstring).search_threads()

def timeout(msg):
    try:
        wait_days = msg.get_header('X-Wait')
        if wait_days:
            wait_days = int(wait_days)
            return time.time() > msg.get_date() + 24 * 60 * 60 * int(wait_days)
    except ValueError:
        return False

def run():
    new_thread_ids = set(t.get_thread_id() for t in query_threads('tag:new'))
    new_thread_clause = '(%s)' % " or ".join("thread:%s" % t for t in new_thread_ids)

    # checking all "wait" and "late" threads
    for msg in query('tag:wait'):
        thread_id = msg.get_thread_id()
        if thread_id in new_thread_ids:
            msg.remove_tag('wait')
            msg.remove_tag('late')
            for new_msg in query('thread:%s and tag:new' % thread_id):
                new_msg.add_tag('expected')
        else:
            if 'late' not in msg.get_tags() and timeout(msg):
                msg.add_tag('late')
                msg.add_tag('inbox')
                msg.remove_tag('wait')

    # handle sent messages
    for msg in query('tag:new and (tag:sent or from:%s)' % MY_ADDR):
        msg.add_tag('sent')
        if msg.get_header('X-Wait'):
            msg.add_tag('wait')

[-- Attachment #3: Type: text/plain, Size: 32 bytes --]



Kind regards

Michael Radziej

Re: (auto-)tagging sent messages

[Xavier Maillard]

On Sun, 17 Apr 2011 00:44:06 +0200, Michael Radziej <mir@spieleck.de> wrote:
> Hi Florian!
> 
> On Thu, 14 Apr 2011 10:03:12 +0200, Florian Friesdorf <flo@chaoflow.net> wrote:
> > Further, for certain mails I sent (like this one ) I would like a
> > WAITING tag (or similar) in order to indicate that I am waiting for an
> > answer. Currently I set this manually. Could this be achieved through
> > some indicators via message mode or similar means? e.g.:
> 
> I use a X-Wait header (like X-Wait: 2, meaning to wait for 2 days). If
> there is no activity in the thread within time, it receives additional tags
> "late" and "inbox". If an answer to a waiting mail arrives, it receives a tag "expected".

I like your workflow very much. 

Do you think this could be done directly in a SIEVE script ? The more I
use notmuch, the more I think many things could be done directly at the
IMAP/SIEVE stage. Am I alone thinking this way ? I mean, today, I put
things in specific folders, etc. Notmuch is almost exclusively used with
search phrases like: folder:notmuch and tag:unread.

I am not SIEVE expert but I am pretty sure it can do way more things
than just dispatch messages into folders.

/Xavier

Re: (auto-)tagging sent messages

[Michael Radziej]

On Mon, 18 Apr 2011 15:22:37 +0200, Xavier Maillard <xavier@maillard.im> wrote:
> Do you think this could be done directly in a SIEVE script ? 

Probably not, but you could with procmail. SIEVE does not allow
execution of user provided binaries.


Kind regards

Michael

Re: (auto-)tagging sent messages

[Xavier Maillard]

On Mon, 18 Apr 2011 20:44:35 +0200, Michael Radziej <mir@spieleck.de> wrote:
> On Mon, 18 Apr 2011 15:22:37 +0200, Xavier Maillard <xavier@maillard.im> wrote:
> > Do you think this could be done directly in a SIEVE script ? 
> 
> Probably not, but you could with procmail. SIEVE does not allow
> execution of user provided binaries.

Thank you for your answer. Dumb question: are procmail and IMAP "closed
to friends" ? I would think yes but only on server side, right or wrong ?

/Xavier

Re: (auto-)tagging sent messages

[Michael Radziej]

On Mon, 18 Apr 2011 21:29:53 +0200, Xavier Maillard <xma@gnu.org> wrote:
> Thank you for your answer. Dumb question: are procmail and IMAP "closed
> to friends" ? I would think yes but only on server side, right or
> wrong ?

"Closed to friends?" No idea what you mean.

Can you please describe your mail flow? 


Kind regards 

Michael

Re: (auto-)tagging sent messages

[Xavier Maillard]

On Tue, 19 Apr 2011 12:18:15 +0200, Michael Radziej <mir@spieleck.de> wrote:
> On Mon, 18 Apr 2011 21:29:53 +0200, Xavier Maillard <xma@gnu.org> wrote:
> > Thank you for your answer. Dumb question: are procmail and IMAP "closed
> > to friends" ? I would think yes but only on server side, right or
> > wrong ?
> 
> "Closed to friends?" No idea what you mean.

I mean, is it something usual to have both procmail and IMAP in a "mail
flow" ?
 
> Can you please describe your mail flow? 

Mine is currently quite simple:

postfix/dovecot/sieve on a remote machine <-> offlineimap/notmuch (plus
a tagging script).

My goal is to use notmuch remotely directly on my IMAP server but there
are problems when doing notmuch new in the dovecot tree (dovecot index
files if I recall correctly).

/Xavier

Re: (auto-)tagging sent messages

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 415 bytes --]

> My goal is to use notmuch remotely directly on my IMAP server but there
> are problems when doing notmuch new in the dovecot tree (dovecot index
> files if I recall correctly).

Can you describe what the specific problems are here? It would be great
to fix notmuch to work well in this scenario.

Perhaps we would simply need a list of filename patterns to ignore?

-Carl

-- 
carl.d.worth@intel.com

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: (auto-)tagging sent messages

[Tim Gray]

[-- Attachment #1: Type: text/plain, Size: 449 bytes --]

On Apr 25, 2011 at 01:27 PM -0700, Carl Worth wrote:
>Can you describe what the specific problems are here? It would be great
>to fix notmuch to work well in this scenario.

That sounds like a sensible fix to me.  But I am running notmuch on a 
maildir tree that does have dovecot files floating around in it (I use 
sieve to deliver my mail to folders) and notmuch doesn't seem to have 
much of a problem automatically ignoring said files.

[-- Attachment #2: Type: application/pgp-signature, Size: 881 bytes --]

Re: (auto-)tagging sent messages

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 854 bytes --]

On Mon, 25 Apr 2011 16:56:31 -0400, Tim Gray <tgray@protozoic.com> wrote:
> That sounds like a sensible fix to me.  But I am running notmuch on a 
> maildir tree that does have dovecot files floating around in it (I use 
> sieve to deliver my mail to folders) and notmuch doesn't seem to have 
> much of a problem automatically ignoring said files.

Right. Notmuch does already automatically detect files that are
obviously not email and ignore them.

I just wondered if perhaps there was a performance problem from notmuch
having to keep looking at these non-mail files over and over again.

Though, even if that were the only problem, we could just teach notmuch
to remember that particular files were detected as not-email to fix the
performance problem without requiring any user configuration.

-Carl

-- 
carl.d.worth@intel.com

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

tracking non-mail files [was: Re: (auto-)tagging sent messages]

[Jameson Graef Rollins]

[-- Attachment #1: Type: text/plain, Size: 801 bytes --]

On Mon, 25 Apr 2011 14:41:28 -0700, Carl Worth <cworth@cworth.org> wrote:
> Though, even if that were the only problem, we could just teach notmuch
> to remember that particular files were detected as not-email to fix the
> performance problem without requiring any user configuration.

This is actually related to an issue that I have that I wanted to bring
up at some point.  For some reason I occasionally receive mail (spam)
that is not properly formatted but that my MTA delivers anyway that
notmuch consequently ignores.  I would like to be able to purge these
files from my mail store eventually, though.  If notmuch could track
these files (and ignore them on subsequent runs of notmuch new), but
then be able to return them upon request (notmuch search :notmail?) that
would be nice.

jamie.

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: tracking non-mail files [was: Re: (auto-)tagging sent messages]

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 783 bytes --]

On Mon, 25 Apr 2011 14:59:06 -0700, Jameson Graef Rollins <jrollins@finestructure.net> wrote:
> This is actually related to an issue that I have that I wanted to bring
> up at some point.  For some reason I occasionally receive mail (spam)
> that is not properly formatted but that my MTA delivers anyway that
> notmuch consequently ignores.  I would like to be able to purge these
> files from my mail store eventually, though.  If notmuch could track
> these files (and ignore them on subsequent runs of notmuch new), but
> then be able to return them upon request (notmuch search :notmail?) that
> would be nice.

That's another good reason for the feature. I've added the feature (with
both implications) to the TODO list now.

-Carl

-- 
carl.d.worth@intel.com

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: (auto-)tagging sent messages

[Xavier Maillard]

Hi Carl,

On Mon, 25 Apr 2011 13:27:26 -0700, Carl Worth <cworth@cworth.org> wrote:
> > My goal is to use notmuch remotely directly on my IMAP server but there
> > are problems when doing notmuch new in the dovecot tree (dovecot index
> > files if I recall correctly).
> 
> Can you describe what the specific problems are here? It would be great
> to fix notmuch to work well in this scenario.
> 
> Perhaps we would simply need a list of filename patterns to ignore?

I just tested it again.

My IMAP server tree is like this:

On 'notmuch new', it works as expected -i.e. ignoring non-mail files
such as index files.

Alas, in the end it fails with an error message:

Error opening /exter..../.dovecot.sieve.log: Permission denied
Internal error: add_message returned unexpected value: 4 (notmuch-new.c:448)

I hope it will help.

/Xavier