Notmuch scripts

Notmuch scripts

[ccx]

[-- Attachment #1: Type: text/plain, Size: 2549 bytes --]

Hello list!

As some of you know I have written several scripts that aid using
notmuch, including an alternative to the notmuch-mutt perl script.

Since Carl Worth consented to include them into official distribution
I am now cleaning them up, writing docs and extending it so it covers
all features notmuch-mutt currently has.

They can currently be obtained by:
	bzr branch http://webprojekty.cz/ccx/bzr/zmuch
Or you can browse the code at:
	http://webprojekty.cz/ccx/loggerhead/zmuch/files

During the process of polishing the scripts I thought about few things
that may be interesting to other people integrating notmuch, so let me
hear your opinions. :-)

$NOTMUCH and proxies:
There are applications (whether full-blown interfaces or simple scripts)
that use notmuch. There are scrips that work as proxies to actual
notmuch, eg. by invoking ssh to a machine where the mails are, or
processing the argument list and expanding saved searches. Therefore it
would be very practical if the path to actual executable would be
configurable. Thus I propose:

  * Every application that does not act as a proxy should use
    environment variable NOTMUCH to find the actual notmuch executable.
	If not defined or empty, just execute 'notmuch' as usual.

  * Every application that acts as a proxy should ignore the NOTMUCH
	variable, instead it should be configurable in other way
	(configuration file or something easily changeable). This way
	chaining of proxies will be possible.

Configuration and temporary files:
I like XDG specification. I think it's bit unnecessary to have to have
config files that belong only to few scripts littered all around my
homedir. Also I think it's reasonable that user would be able to specify
where to put the temporary files. That said atm I have ~/.zmuch as the
only config file, as it felt bit weird to create new directory for just
one file. But as I'm preparing the scripts I see more and more things
that are specific to my setup and I would like them to be configurable.
These are:

  * Spam filter. Do you guys use any? What does it's interface look like?
    I currently use bsfilter which I've found does it's job pretty well.

  * Colors. I use bright fg on dark bg, but I understand somebody won't
    be happy with this choice.

  * New message processing. Currently I check for spam and I mute
    selected threads. I can see this can be made quite configurable.
	Maybe create procmail equivalent for notmuch? :-)

Thanks for notmuch! I look forward to your responses.
	Jan Pobrislo (ccxCZ@freenode)

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: Notmuch scripts

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 7191 bytes --]

On Fri, 24 Jun 2011 13:28:21 +0200, ccx@te2000.cz wrote:
> As some of you know I have written several scripts that aid using
> notmuch, including an alternative to the notmuch-mutt perl script.

Thanks for sharing these, Jan!

> Since Carl Worth consented to include them into official distribution
> I am now cleaning them up, writing docs and extending it so it covers
> all features notmuch-mutt currently has.
...
> Or you can browse the code at:
> 	http://webprojekty.cz/ccx/loggerhead/zmuch/files

It's been interesting for me to read through the README here.

So much of the new functionality here consists of things I'd love to
have implemented in the core command-line interface of notmuch, (I
always have wanted it to be useful for direct command-line interaction
by a human). Some of these features I think have even be implemented
previously by Austin through his custom query parser.

The features I see that I'd really like to see in the core notmuch
command-line tool are:

	* Configurable "saved searches", (a syntax for expanding aliases
          for often-repeated search specifications).

	  That's an idea we've had for a while. What's new with the
	  zmuch implementation is the proposal of ":alias" for the
	  syntax. I think I might like that quite a bit. It looks a bit
	  easier to read (and type) than the previously-proposed
	  "{alias}".

	* Delivery of search results to a maildir of symlinks

	  The zmuch implementation has this functionality intertwined
	  with something that also invokes mutt. Obviously, people using
	  other MUAs might like to access this feature independently.

	  I think I'd like to see this as:

		notmuch search --format=maildir:/path/to/directory

	* Operations on files matching search terms (move, remove,
          xargs)

	  This isn't an operation I'd previously considered including in
	  notmuch, but it does seem generally quite useful.

	  Should we consider doing something like git does and allow
	  something like "notmuch xargs" simply find and invoke a shell
	  script named notmuch-xargs?

	  Doing that could let us get a bunch of this functionality in
	  place in the "core" sooner than if we waited for it to be
	  re-implemented in C.

	  Though if we did this, I think I'd be highly inclined to port
	  the scripts from zsh to bash or even POSIX sh. How hard would
	  that be?

	* Better date syntax for search specifications

	  That's something that's obviously been missing from notmuch
	  core from the beginning. And there have been several proposals
	  with patches to do this in various ways.

	* Implicit concatenation of search terms with OR

	  This seems like something easy to do with a command-line
	  arguemnt. Perhaps "notmuch search --or ..." ?

If we got all that into the core, then what would be left here would be:

	notmuch-mailvars.sh
	notmuch-mutt.sh

		These would provide integration of notmuch with mutt.

	notmuch-spam.sh
	notmuch-unspam.sh

		These would provide integration of notmuch with
		bsfilter, (and perhaps should be named to make that more
		clear---or generalized to justify the current name).

	notmuch-pager.sh

		I haven't looked at this to see what the colorization
		actually looks like, (I'm not always a huge fan of lots
		of color in my terminals). It seems that this would be
		more cleanly implemented as notmuch-colorize.sh or so
		and leave the pagination separate.

If we had that, I'd feel really comfortable having each of those in
contrib. I think contrib should be restricted to things which provide
integration of notmuch with some external tool, (and should make that
obvious by having a name like notmuch-<tool> or notmuch-<tool>.sh or
whatever).

All in all, there's definitely some very interesting functionality here,
and I definitely appreciate you sharing it. Let's figure out the best
way to get it all integrated into notmuch.

Maybe in the meantime we throw everything into contrib even if some of
it is seen as just proposals for better interfaces in the core tool? I
don't know.

>   * Every application that does not act as a proxy should use
>     environment variable NOTMUCH to find the actual notmuch executable.
> 
>   * Every application that acts as a proxy should ignore the NOTMUCH
> 	variable

That sounds reasonable enough to me. Perhaps these rules could go into a
new contrib/README that would set out some guidelines for writing
contrib tools, (such as notmuch-<tool> which I mentioned above).

> Configuration and temporary files:
> I like XDG specification.

I'm missing some context to know what you're suggesting here.

> I think it's bit unnecessary to have to have
> config files that belong only to few scripts littered all around my
> homedir.

We should be able to put configuration for contrib tools into the main
notmuch configuration file. If your tools don't want to read that file
directly, they should be able to get by with the interfaces provided by
"notmuch config set" and "notmuch config get". Obviously, each tool
should create its own section in the configuration file.

Is that an insane plan?

>   * Spam filter. Do you guys use any? What does it's interface look like?
>     I currently use bsfilter which I've found does it's job pretty
>     well.

I've currently got amavis and spamassassin adding extra headers, (and
below a certain threshold I've got maildrop delivering detected spam to
a separate maildir).

Currently, notmuch never sees the detected spam. Ever since we got
folder: support I've been meaning to let notmuch see it so that I can
use notmuch to dig into my spam when I suspect something got
mis-detected.

I don't currently have any system for getting user-provided feedback
into my spam filtering. Do you get that with bsfilter?

>   * Colors. I use bright fg on dark bg, but I understand somebody won't
>     be happy with this choice.

I'm pretty-much black-on-white only. I really want a similar experience
with my computer that I get from books. (Though I'm still waiting for
much better contrast from my computer displays—e-ink definitely helps a
lot for the very static use cases).

>   * New message processing. Currently I check for spam and I mute
>     selected threads. I can see this can be made quite configurable.
> 	Maybe create procmail equivalent for notmuch? :-)

I think lots of us have various hand-written scripts that call out to
"notmuch tag" a bunch. It's definitely a common idiom to have "notmuch
new" add a new tag, have the new-mail-processing script operate on
tag:new, and then have that script remove tag:new from the things it
processed.

An alternative approach has been proposed to make "notmuch new" able to
act on specified messages, (and accept an explicit list of tags to
add). That would make it much easier to actually use existing tools like
procmail directly with notmuch. Some people are currently using the
notmuch-deliver.sh script in use cases like this. (And that script is
another existing candidate for contrib.)

-Carl

-- 
carl.d.worth@intel.com

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: Notmuch scripts

[ccx]

[-- Attachment #1: Type: text/plain, Size: 11494 bytes --]

On Fri, Jun 24, 2011 at 11:29:21AM -0700, Carl Worth wrote:
> So much of the new functionality here consists of things I'd love to
> have implemented in the core command-line interface of notmuch

That's what I thought. I think it's actually beneficial to go the
prototyping route here and start with implementations that can be
written and altered quickly.

> The features I see that I'd really like to see in the core notmuch
> command-line tool are:
> 
> 	* Configurable "saved searches", (a syntax for expanding aliases
>           for often-repeated search specifications).
> 
> 	  That's an idea we've had for a while. What's new with the
> 	  zmuch implementation is the proposal of ":alias" for the
> 	  syntax. I think I might like that quite a bit. It looks a bit
> 	  easier to read (and type) than the previously-proposed
> 	  "{alias}".

Glad you like it. :-)

> 	* Delivery of search results to a maildir of symlinks
> 
> 	  The zmuch implementation has this functionality intertwined
> 	  with something that also invokes mutt. Obviously, people using
> 	  other MUAs might like to access this feature independently.

That's one of reasons I'm rewriting it. What I'm planning to have is:

  * notmuch-maildir-create -- make the symlinks, also save metadata in
                              the maildir

  * notmuch-maildir-merge -- reflect the changes on the original files

  * notmuch-maildir-refresh -- merge and recreate with same query

  * notmuch-maildir-mua -- create temp directory, invoke MUA (mutt) on
                           it, merge back

  * notmuch-maildir-search -- merge if old maildir is there, prompt for
                              query, create new maildir

The last one should be sufficient replacement for what notmuch-mutt
currently does, provided one has set up to run merge after mutt instance
exits. I'm not very satisfied with naming on the last two, so
suggestions are welcome :-)

> 	* Operations on files matching search terms (move, remove,
>           xargs)
> 
> 	  This isn't an operation I'd previously considered including in
> 	  notmuch, but it does seem generally quite useful.

These are just handy shorthands, the actual command could still fit on
one line.

> 	  Should we consider doing something like git does and allow
> 	  something like "notmuch xargs" simply find and invoke a shell
> 	  script named notmuch-xargs?

I thought of that also, but I wouldn't do that yet. Not until the
scripts are bit more polished and tested.

> 	  Doing that could let us get a bunch of this functionality in
> 	  place in the "core" sooner than if we waited for it to be
> 	  re-implemented in C.
> 
> 	  Though if we did this, I think I'd be highly inclined to port
> 	  the scripts from zsh to bash or even POSIX sh. How hard would
> 	  that be?

Not much of a problem for simpler scripts, but for the big ones like
saved searches or maildir merging it would mean a lot of added
boilerplate. That would reduce the readability and defeat the purpose of
prototyping.

I could rewrite the more complex ones in python, but on the other hand,
the zsh package is just 6MiB here including debugging symbols. I don't
think there will be many who will find it problematic to install.

> 	* Better date syntax for search specifications
> 
> 	  That's something that's obviously been missing from notmuch
> 	  core from the beginning. And there have been several proposals
> 	  with patches to do this in various ways.

I think it would be neat to steal this from the 'at' command.
By the way, this is another use-case for notmuch proxy.

> 	* Implicit concatenation of search terms with OR
> 
> 	  This seems like something easy to do with a command-line
> 	  arguemnt. Perhaps "notmuch search --or ..." ?

What would I personally like would be to use [] to mean the same as (),
just with implicit ORs inside. For example: "[ foo bar ( spam eggs ) ]"
would expand to "foo or bar or ( spam and eggs )". This way we can have
long sequences of both ORs and ANDs in one query.

 
> If we got all that into the core, then what would be left here would be:
> 
> 	notmuch-mailvars.sh
> 	notmuch-mutt.sh
> 
> 		These would provide integration of notmuch with mutt.

Actually I wish to make it generic enough that it can be used with
anything that can read a maildir.

> 	notmuch-spam.sh
> 	notmuch-unspam.sh
> 
> 		These would provide integration of notmuch with
> 		bsfilter, (and perhaps should be named to make that more
> 		clear---or generalized to justify the current name).

I'd love to make it generic, that's why I asked what people use.

> 	notmuch-pager.sh
> 
> 		I haven't looked at this to see what the colorization
> 		actually looks like, (I'm not always a huge fan of lots
> 		of color in my terminals). It seems that this would be
> 		more cleanly implemented as notmuch-colorize.sh or so
> 		and leave the pagination separate.

Actually I would like to have some common configuration somewhere, so
people can attach colors to certain tags and any app would be able to
utilize that.

By the way the flags for less I use make it behave like it was not there
at all if the contents fits on one screen or the output is not a
terminal.

> If we had that, I'd feel really comfortable having each of those in
> contrib. I think contrib should be restricted to things which provide
> integration of notmuch with some external tool, (and should make that
> obvious by having a name like notmuch-<tool> or notmuch-<tool>.sh or
> whatever).

I think contrib can serve also for experimental extensions and
interfaces, unless you want to move them somewhere else.

> All in all, there's definitely some very interesting functionality here,
> and I definitely appreciate you sharing it. Let's figure out the best
> way to get it all integrated into notmuch.

Thanks. I'm gonna alter most of the scripts so they better match unix
conventions and probably even make manpages for them. So don't take the
current state of the repo as final, I just wanted to give you heads up.

> Maybe in the meantime we throw everything into contrib even if some of
> it is seen as just proposals for better interfaces in the core tool? I
> don't know.

That's what I plan to do.

> >   * Every application that does not act as a proxy should use
> >     environment variable NOTMUCH to find the actual notmuch executable.
> > 
> >   * Every application that acts as a proxy should ignore the NOTMUCH
> > 	variable
> 
> That sounds reasonable enough to me. Perhaps these rules could go into a
> new contrib/README that would set out some guidelines for writing
> contrib tools, (such as notmuch-<tool> which I mentioned above).

Great. Should I write that too? :-) 

> > Configuration and temporary files:
> > I like XDG specification.
> 
> I'm missing some context to know what you're suggesting here.
> 
> > I think it's bit unnecessary to have to have
> > config files that belong only to few scripts littered all around my
> > homedir.
> 
> We should be able to put configuration for contrib tools into the main
> notmuch configuration file. If your tools don't want to read that file
> directly, they should be able to get by with the interfaces provided by
> "notmuch config set" and "notmuch config get". Obviously, each tool
> should create its own section in the configuration file.

I'm not so sure INI format is a good choice for every tool. It's not
conveniently parseable by a shell script and sometimes you need more
than just get/set functions. For example for saved searches I don't know
the key names beforehand and I need to retrieve them in defined order.

> Is that an insane plan?
I think nice thing to have common configuration. I'm unsure this one
size fits all approach is the best way, but I'm kind of okay with it.

> >   * Spam filter. Do you guys use any? What does it's interface look like?
> >     I currently use bsfilter which I've found does it's job pretty
> >     well.
> 
> I've currently got amavis and spamassassin adding extra headers, (and
> below a certain threshold I've got maildrop delivering detected spam to
> a separate maildir).

bsfilter can add headers too, though I don't use that feature now. I
didn't know how would offlineimap or any other mail-sync solution behave
if I went around changing actual content of the messages.

> Currently, notmuch never sees the detected spam. Ever since we got
> folder: support I've been meaning to let notmuch see it so that I can
> use notmuch to dig into my spam when I suspect something got
> mis-detected.
> 
> I don't currently have any system for getting user-provided feedback
> into my spam filtering. Do you get that with bsfilter?

Yes, bsfilter is short for bayesian filter, so it kind of needs sets of
ham and spam messages to learn from. That's why I have the scripts to
handle that.

> >   * Colors. I use bright fg on dark bg, but I understand somebody won't
> >     be happy with this choice.
> 
> I'm pretty-much black-on-white only. I really want a similar experience
> with my computer that I get from books. (Though I'm still waiting for
> much better contrast from my computer displays—e-ink definitely helps a
> lot for the very static use cases).

You'd definitely wouldn't like my cyan for unseen messages then. :-)
That's why I'd like to get that in globally configurable, see above on
notmuch-pager

> >   * New message processing. Currently I check for spam and I mute
> >     selected threads. I can see this can be made quite configurable.
> > 	Maybe create procmail equivalent for notmuch? :-)
> 
> I think lots of us have various hand-written scripts that call out to
> "notmuch tag" a bunch. It's definitely a common idiom to have "notmuch
> new" add a new tag, have the new-mail-processing script operate on
> tag:new, and then have that script remove tag:new from the things it
> processed.

I actually don't do much tagging, as saved searches seem more flexible
to me. The current implementation does three things (apart from having
nice progress meter):
  * applies 'sent' tag to emails from addresses in your config file.
    This can be accomplished by saved searches too, bot it was more
	convenient to me to implement it here.
  * runs spam check on the mail
  * makes 'mute' tag span whole threads

The last thing is not that easy to do in pure shell script.

I'm not sure if some domain-specific language would be that big
improvement over status quo, that it would be worth implementing.
It would be first necessary to asses which features are useful and what
people need. Apart from what it currently does I do think that it would
be useful if it could move messages around .

> An alternative approach has been proposed to make "notmuch new" able to
> act on specified messages, (and accept an explicit list of tags to
> add). That would make it much easier to actually use existing tools like
> procmail directly with notmuch. Some people are currently using the
> notmuch-deliver.sh script in use cases like this. (And that script is
> another existing candidate for contrib.)

I am actually considering using something like that together with
dovecot on my server, I want to get some experience with using notmuch
first.

Thanks for your elaborate response. :-)

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: Notmuch scripts

[Ben Gamari]

On Fri, 24 Jun 2011 11:29:21 -0700, Carl Worth <cworth@cworth.org> wrote:
> >   * Spam filter. Do you guys use any? What does it's interface look like?
> >     I currently use bsfilter which I've found does it's job pretty
> >     well.
> 
> I've currently got amavis and spamassassin adding extra headers, (and
> below a certain threshold I've got maildrop delivering detected spam to
> a separate maildir).
> 
> Currently, notmuch never sees the detected spam. Ever since we got
> folder: support I've been meaning to let notmuch see it so that I can
> use notmuch to dig into my spam when I suspect something got
> mis-detected.
> 
> I don't currently have any system for getting user-provided feedback
> into my spam filtering. Do you get that with bsfilter?
> 
I use bogofilter along with a python cron job to bring my tags and the
spam filter into sync. I use a tag (junk) to mark mail as spam and
two other tags (.bf_ham and .bf_spam) to indicate that the message has been added to
bogofilter. The script's task is then to,

 * Unregister as spam, register as ham messages with tag:.bf_spam and not tag:junk with bogofilter
 * Register as spam, unregister as ham messages with tag:junk and not tag:.bf_spam

This is pretty straightforward to implement and is quite effective. It
would be nice, however, if the emacs interface supported hiding tags
matching certain patterns (say /\..+/).

- Ben

Re: Notmuch scripts

[Pieter Praet]

On Mon, 27 Jun 2011 21:31:26 -0400, Ben Gamari <bgamari.foss@gmail.com> wrote:
> [SNIP] It
> would be nice, however, if the emacs interface supported hiding tags
> matching certain patterns (say /\..+/).

This should be possible as of last month (eb4e0ea2),
by courtesy of Daniel Schoepe [1].

> 
> - Ben
> 
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch


Peace

-- 
Pieter


[1] id:"87fwoath2s.fsf@gilead.home.box"

Re: Notmuch scripts

[Ben Gamari]

On Tue, 28 Jun 2011 10:36:59 +0200, Pieter Praet <pieter@praet.org> wrote:
> On Mon, 27 Jun 2011 21:31:26 -0400, Ben Gamari <bgamari.foss@gmail.com> wrote:
> > [SNIP] It
> > would be nice, however, if the emacs interface supported hiding tags
> > matching certain patterns (say /\..+/).
> 
> This should be possible as of last month (eb4e0ea2),
> by courtesy of Daniel Schoepe [1].
> 
Thanks for your note, althought it appears this only appears to apply to
the notmuch-hello all-tags list. I was hoping one could hide tag
patterns from all lists (or at least search results). This is definitely
a good first step, however.

Cheers,

- Ben

Re: Notmuch scripts

[David Bremner]

[-- Attachment #1: Type: text/plain, Size: 735 bytes --]

On Fri, 24 Jun 2011 13:28:21 +0200, ccx@te2000.cz wrote:
> As some of you know I have written several scripts that aid using
> notmuch, including an alternative to the notmuch-mutt perl script.
> 
> Since Carl Worth consented to include them into official distribution
> I am now cleaning them up, writing docs and extending it so it covers
> all features notmuch-mutt currently has.

Sorry if I missed this somewhere, but how does the functionality of your
scripts compare to those of Stefano Zacchiroli

        http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=628018

At least I think the mutt-notmuch referred to here is a third
alternative compared to your scripts and those of wart_ (Peter John Hartman?)

David

[-- Attachment #2: Type: application/pgp-signature, Size: 315 bytes --]