Re: Thoughts on notmuch and Lua

Re: Thoughts on notmuch and Lua

[Sebastian]

> We've talked about switching from default tags of "inbox" and "unread"
> to simply having new mail tagged with a "new" tag.

Ohh, please make the standard set of tags for new mail configurable in the config file. I might want to add different tags for special purposes, like "not_virus_scanned_yet" or what not by default. Having to run a script just to add those tags to any "new" mails sounds a bit complicated and error-prone to me.

spaetz

Thoughts on notmuch and Lua

[Ali Polatel]

[-- Attachment #1: Type: text/plain, Size: 1186 bytes --]

Before trying to implement anything I decided to send a mail to the list
to ask people's opinion.

What's the problem?
===================
Notmuch isn't very configurable.

How can Lua integration solve this?
===================================
Here are initial thoughts on how to integrate Lua with notmuch.
Any comments appreciated.

1. Configuration file:
The configuration file can be a Lua script that allows more dynamic
configuration. Here's an example:

# notmuch configuration file:
config = {}
config.dbpath = "/path/to/maildir"
config.exclude = function (maildir)
    return not string.match(maildir, ".*Trash.*")
    end
...

2. Hooks:
This is a feature I really miss having switched from sup.
There can be many hooks, a hook that formats search output,
a hook that is called before adding messages to the database which may
be used to add initial tags depending on headers etc.

Why Lua?
========
Lua has many advantages over other scripting languages when it comes to
integration with a C program. It has a very clean and easy C API, the
overhead of running Lua scripts is not noticable among other things.

-- 
Regards,
Ali Polatel

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: Thoughts on notmuch and Lua

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 3365 bytes --]

On Thu, 14 Jan 2010 10:47:13 +0200, Ali Polatel <alip@exherbo.org> wrote:
> Before trying to implement anything I decided to send a mail to the list
> to ask people's opinion.

Hi Ali, welcome to notmuch!

I appreciate you soliciting opinions, but I hope that my answer won't
discourage you. By all means, please feel free to experiment!

> What's the problem?
> ===================
> Notmuch isn't very configurable.

I'll grant that. And as can be seen in TODO and in code comments, we
definitely want to fix that.

> 1. Configuration file:
> The configuration file can be a Lua script that allows more dynamic
> configuration. Here's an example:
> 
> # notmuch configuration file:
> config = {}
> config.dbpath = "/path/to/maildir"
> config.exclude = function (maildir)
>     return not string.match(maildir, ".*Trash.*")
>     end
> ...

That doesn't look very compelling to me.

I'd much rather have:

	[database]
	path=/home/cworth/mail
	maildir_exclude=.*Trash.*

with the exact same functionality.

Granted, having a full programming language in the configuration file
makes thing much more dynamic, but it also makes it much harder for the
user to read, edit, and ensure the syntax is correct.

> 2. Hooks:
> This is a feature I really miss having switched from sup.
> There can be many hooks, a hook that formats search output,
> a hook that is called before adding messages to the database which may
> be used to add initial tags depending on headers etc.

I understand that some people really like their hooks. They let users
invent all kinds of interesting, custom functionality.

But I think hooks also have problems. Sometimes the most interesting
functionality has to be pieced together by every user going to a wiki
page and finding the "standard" hooks. I'd much rather avoid that by
getting the most useful functionality into the program in the first
place.

Hooks also impose a particular amount of maintenance burden on the
software. And they are often implemented in a way that makes them very
hard to be discovered.

I wrote a message to the sup mailing list describing some of these
issues. The context there was a patch I wrote adding a configuration
option, (and the sup maintainer preferring it be added as a patch
instead):

	id:1254417826-sup-6584@yoom.home.cworth.org
	Subject: Re: [sup-talk] [PATCH] Add new :crypto_default configuration option.

I did find out later that the sup hooks were more self-documenting than
I had understood. (There was a sup command-line option that printed
documentation for all available hooks.) Something like that is
definitely a requirement for providing hooks.

So I'm not entirely opposed to the idea of adding hooks to notmuch, but
I'll definitely need to be convinced that any particular functionality
can't be better integrated without the hook.

> Why Lua?
> ========
> Lua has many advantages over other scripting languages when it comes to
> integration with a C program. It has a very clean and easy C API, the
> overhead of running Lua scripts is not noticable among other things.

I've definitely heard lots of good things about "lua embedability". So
if we do decide to provide hooks, then lua would seem like a logical
option to look at first. I've never looked at it closely myself though.

-Carl

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Thoughts on notmuch and Lua

[martin f krafft]

[-- Attachment #1: Type: text/plain, Size: 1149 bytes --]

also sprach Carl Worth <cworth@cworth.org> [2010.01.15.1200 +1300]:
> > Lua has many advantages over other scripting languages when it
> > comes to integration with a C program. It has a very clean and
> > easy C API, the overhead of running Lua scripts is not noticable
> > among other things.
> 
> I've definitely heard lots of good things about "lua
> embedability". So if we do decide to provide hooks, then lua would
> seem like a logical option to look at first. I've never looked at
> it closely myself though.

Lua for hooks has the advantage that the hooks can be executed in
the context of manipulateable objects. On the other hand, hooks in
the style of run-parts directories are more flexible and accessible,
and could always be invoked as filters for the manipulateable data.

-- 
martin | http://madduck.net/ | http://two.sentenc.es/
 
"imagine if every thursday your shoes exploded if you
 tied them the usual way. this happens to us all the time
 with computers, and nobody thinks of complaining."
                                                        -- jeff raskin
 
spamtraps: madduck.bogus@madduck.net

[-- Attachment #2: Digital signature (see http://martin-krafft.net/gpg/) --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Thoughts on notmuch and Lua

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 1018 bytes --]

On Fri, 15 Jan 2010 13:16:00 +1300, martin f krafft <madduck@madduck.net> wrote:
> Lua for hooks has the advantage that the hooks can be executed in
> the context of manipulateable objects. On the other hand, hooks in
> the style of run-parts directories are more flexible and accessible,
> and could always be invoked as filters for the manipulateable data.

Good point.

One thing that notmuch has in its favor already is that there's a
command-line interface that provides all of the interesting
functionality. So it's already a fairly trivial matter to "hook" things
like "notmuch new" by writing a script in whatever language you prefer
to invoke things like "notmuch search" and "notmuch tag".

And that's what I'm doing already, (with a shell script), to tag
newly-arrived messages.

The only part that's really missing here is a directory where scripts
could be dropped in and get automatically invoked.

What do you think, Ali? Would an approach like that satisfy the things
you had in mind for hooks?

-Carl

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Thoughts on notmuch and Lua

[Ali Polatel]

[-- Attachment #1: Type: text/plain, Size: 2418 bytes --]

Carl Worth yazmış:
> On Fri, 15 Jan 2010 13:16:00 +1300, martin f krafft <madduck@madduck.net> wrote:
> > Lua for hooks has the advantage that the hooks can be executed in
> > the context of manipulateable objects. On the other hand, hooks in
> > the style of run-parts directories are more flexible and accessible,
> > and could always be invoked as filters for the manipulateable data.
> 
> Good point.
> 
> One thing that notmuch has in its favor already is that there's a
> command-line interface that provides all of the interesting
> functionality. So it's already a fairly trivial matter to "hook" things
> like "notmuch new" by writing a script in whatever language you prefer
> to invoke things like "notmuch search" and "notmuch tag".
> 
> And that's what I'm doing already, (with a shell script), to tag
> newly-arrived messages.
> 
> The only part that's really missing here is a directory where scripts
> could be dropped in and get automatically invoked.
> 
> What do you think, Ali? Would an approach like that satisfy the things
> you had in mind for hooks?
> 

It might, here are some thoughts and questions to help you elaborate:

- How will these scripts manipulate data?
  e.g.: A "tagger" script may get the new mail from stdin and print out
  the new tags which notmuch will read and apply to the message.
  A "search-filter" script may get search results from stdin and filter
  them. Just my initial thoughts.

This script approach has both upsides and downsides. I have some
experience about this because I wrote a tool called mpdcron¹ which
executes hooks based on mpd events. First I thought it would be cool to
execute scripts based on events by passing data using environment
variables. However this approach didn't give hooks enough power to save
state information etc. so I decided to add module support through
GModule.

I may be going a bit off topic here but what I'm trying to explain is
scripts may not give power-users enough options and is practically
slower than having Lua, or another scripting language, support. In my
humble opinion the former is just a hack while the latter is the proper
way to do it.

P.S.: As I stated in the beginning you could have elaborated what you
meant by script support and I'm just commenting on what I've understood.

¹: http://alip.github.com/mpdcron/

> -Carl

-- 
Regards,
Ali Polatel

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: Thoughts on notmuch and Lua

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 4716 bytes --]

On Fri, 15 Jan 2010 23:09:34 +0200, Ali Polatel <alip@exherbo.org> wrote:
> Carl Worth yazmış:
> > What do you think, Ali? Would an approach like that satisfy the things
> > you had in mind for hooks?
> 
> It might, here are some thoughts and questions to help you elaborate:
> 
> - How will these scripts manipulate data?
>   e.g.: A "tagger" script may get the new mail from stdin and print out
>   the new tags which notmuch will read and apply to the message.

This can happen with current notmuch entirely with no real "hook"
support.

We've talked about switching from default tags of "inbox" and "unread"
to simply having new mail tagged with a "new" tag. So a "tagger" script
could operate simply by doing a "notmuch search" for messages with the
"new" tag and could iterate over the filenames to process actual
messages. (We don't have support now for emitting just filenames from a
"notmuch search", but we have patches for that, and I'll be applying one
soon.)

So that's a taste for the "scriptability" I see in the current notmuch
system that makes it really much more flexible than any "hooks"
system. Additional flexibility comes from:

  * User can run a script like this at any point---not merely when
    messages are added.

  * User script isn't restricted to dealing only with "new" messages,
    but can act on any set of messages based on any search constraint,
    (or even all messages in the database).

The results can then be applied by simply calling "notmuch tag" as
needed.

And if there are any performance problems there we can fix them, (such
as, perhaps we'll end up wanting this script to be able to invoke a
single process for all of its tagging rather than calling "notmuch tag"
over and over).

>   A "search-filter" script may get search results from stdin and filter
>   them. Just my initial thoughts.

And how would this search functionality and filtering be different than
the search functionality provided by notmuch itself?

I can think of at least a couple of ways it might be different:

  1. It would be nice to be able to filter based on tags that are
     present in a thread, though perhaps not present in any message
     matching the original search.

     An obvious application of this is the "thread muting" feature,
     where once a message is tagged as "muted", no messages delivered to
     that thread in the future will appear in the inbox.

     This is a feature I'd like to put into the core of notmuch such
     that one passes a query to match messages and then also a second
     query to filter based on the collected tags in threads. Something
     like:

	notmuch search tag:inbox --filter="not tag:muted"

 2. There are other details available at the thread level that are not
    available at the level at which message-based searching happens.

    A simple example of this would be the ability to search for threads
    with a single message, (perhaps checking to ensure that all requests
    had gotten at least one reply). But one can imagine more complex
    things as well, "Show me all threads where ImportantPerson sent a
    message and where I never replied in the thread."

    For this kind of thing, I think we simply want to build on the
    output of "notmuch search". The current output isn't very usable for
    this, but with things like the structured json output, etc. (which,
    again, I hope to be merging soon), it would be quite easy to write
    new tools that accept that output and provide additional searching
    and filtering, etc. And that tool could provide lua-based scripting
    or whatever else is desired.

So my feeling is that if anything can live outside of notmuch, then it
should, and should simply build on top of notmuch output. (And we should
fix notmuch output to support that well.)

And anything that must live within notmuch (or is best supported there),
we should see if we can't just make that a core part of notmuch itself,
(such as the --filter option I showed above).

I'm *still* not wanting to squelch any experimentation with embedding
scripting languages inside notmuch, or anything else. I'm just still not
seeing anything that requires this.

Look at the amount of emacs-lisp code we've written, for example, and
the various things it does, (hiding away citations, etc.). That's all
"scripting code", but that sits easily on top of the existing notmuch
command-line tool.

I think I'd prefer to keep that nice clean boundary until we find
something that really requires changing that. But, show me something
really cool that requires it, and you might convince me. :-)

-Carl

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Thoughts on notmuch and Lua

[Ali Polatel]

[-- Attachment #1: Type: text/plain, Size: 5544 bytes --]

Carl Worth yazmış:
> On Fri, 15 Jan 2010 23:09:34 +0200, Ali Polatel <alip@exherbo.org> wrote:
> > Carl Worth yazmış:
> > > What do you think, Ali? Would an approach like that satisfy the things
> > > you had in mind for hooks?
> > 
> > It might, here are some thoughts and questions to help you elaborate:
> > 
> > - How will these scripts manipulate data?
> >   e.g.: A "tagger" script may get the new mail from stdin and print out
> >   the new tags which notmuch will read and apply to the message.
> 
> This can happen with current notmuch entirely with no real "hook"
> support.
> 
> We've talked about switching from default tags of "inbox" and "unread"
> to simply having new mail tagged with a "new" tag. So a "tagger" script
> could operate simply by doing a "notmuch search" for messages with the
> "new" tag and could iterate over the filenames to process actual
> messages. (We don't have support now for emitting just filenames from a
> "notmuch search", but we have patches for that, and I'll be applying one
> soon.)
> 

This is one of the reasons I wanted hook support in the beginning.
I'm looking forward to seeing this in notmuch.

> So that's a taste for the "scriptability" I see in the current notmuch
> system that makes it really much more flexible than any "hooks"
> system. Additional flexibility comes from:
> 
>   * User can run a script like this at any point---not merely when
>     messages are added.
> 
>   * User script isn't restricted to dealing only with "new" messages,
>     but can act on any set of messages based on any search constraint,
>     (or even all messages in the database).
> 
> The results can then be applied by simply calling "notmuch tag" as
> needed.
> 

+1. I totally agree.

> And if there are any performance problems there we can fix them, (such
> as, perhaps we'll end up wanting this script to be able to invoke a
> single process for all of its tagging rather than calling "notmuch tag"
> over and over).
> 
> >   A "search-filter" script may get search results from stdin and filter
> >   them. Just my initial thoughts.
> 
> And how would this search functionality and filtering be different than
> the search functionality provided by notmuch itself?
> 

I accept, this search-filter idea is kinda stupid now that I think about
it.

> I can think of at least a couple of ways it might be different:
> 
>   1. It would be nice to be able to filter based on tags that are
>      present in a thread, though perhaps not present in any message
>      matching the original search.
> 
>      An obvious application of this is the "thread muting" feature,
>      where once a message is tagged as "muted", no messages delivered to
>      that thread in the future will appear in the inbox.
> 
>      This is a feature I'd like to put into the core of notmuch such
>      that one passes a query to match messages and then also a second
>      query to filter based on the collected tags in threads. Something
>      like:
> 
> 	notmuch search tag:inbox --filter="not tag:muted"
> 

Looks like a good idea indeed.

>  2. There are other details available at the thread level that are not
>     available at the level at which message-based searching happens.
> 
>     A simple example of this would be the ability to search for threads
>     with a single message, (perhaps checking to ensure that all requests
>     had gotten at least one reply). But one can imagine more complex
>     things as well, "Show me all threads where ImportantPerson sent a
>     message and where I never replied in the thread."
> 
>     For this kind of thing, I think we simply want to build on the
>     output of "notmuch search". The current output isn't very usable for
>     this, but with things like the structured json output, etc. (which,
>     again, I hope to be merging soon), it would be quite easy to write
>     new tools that accept that output and provide additional searching
>     and filtering, etc. And that tool could provide lua-based scripting
>     or whatever else is desired.
> 

Makes sense. Structured output would make things really simpler.

> So my feeling is that if anything can live outside of notmuch, then it
> should, and should simply build on top of notmuch output. (And we should
> fix notmuch output to support that well.)
> 

Works for me, as long as it solves my problems and as I stated above I
think it will.

> And anything that must live within notmuch (or is best supported there),
> we should see if we can't just make that a core part of notmuch itself,
> (such as the --filter option I showed above).
> 
> I'm *still* not wanting to squelch any experimentation with embedding
> scripting languages inside notmuch, or anything else. I'm just still not
> seeing anything that requires this.
> 
> Look at the amount of emacs-lisp code we've written, for example, and
> the various things it does, (hiding away citations, etc.). That's all
> "scripting code", but that sits easily on top of the existing notmuch
> command-line tool.
> 
> I think I'd prefer to keep that nice clean boundary until we find
> something that really requires changing that. But, show me something
> really cool that requires it, and you might convince me. :-)
> 

I don't have anything in mind right now but when I do we can talk
further :)

Thanks for the descriptive response, I really appreciate it.

> -Carl

-- 
Regards,
Ali Polatel

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]