On properties and the notmuch CLI

On properties and the notmuch CLI

[Matt Armstrong]

I see that notmuch supports a tag-like concept called a property.  They
seem to differ from tags in three ways:

 (a) they aren't shown as user visible "tags", so they can be set on
     messages without confusing users with message specific metadata
     that is not related to how they want to classify their mail.

 (b) they are key/value pairs, not merely a tag.

 (c) it is not possible to set or modify properties from the command
 line.

First, a confirming question: it is possible to efficiently search for
all messages that set a given property 'P' with a search for:

    property:P=

True?  Are queries like these any more efficient than tag doing a prefix
regex search over tags?  By this I mean, does the Xapian side keep an
efficient index of all messages that have at least one property "P"?

Second, a question: is there any possibility of relaxing restriction (c)
above?  My rationale is this: I'd like it to be possible to write
notmuch "extensions" without binding to the notmuch C API (which feels
quite low level, and an installation point of friction for those who
might want to use the extension).  I'd like, for example, to be able to
write fully capable extension logic in Emacs itself (let's ignore Emacs
C "modules" for now).  Currently, I am writing code that interacts with
notmuch from Go, and while there are some Go bindings, using them is an
extra level of complexity I'd rather avoid.

Concrete use case ideas:

 - were someone to write a fully functional Gnus backend (nnnotmuch?),
   it would be useful to store Gnus metadata directly in notmuch itself
   (e.g. the message numbers knows to Gnus itself).  I don't have a full
   design for this, nor do I intend to do it, but the *idea* of doing
   this feels very much like what Free Software is all about enabling.

 - were someone (me) writing a Go program that used the GMail API to
   sync messages to notmuch, I'd love to be able to set properties on
   messages at "notmuch insert" time.  This would let me leverage the
   notmuch database instead of having to resort to hacks (like avoiding
   "notmuch insert" entirely, and playing games with filename encoding
   schemes to encode the same data with the message).

As far as the risk of property name collisions, perhaps encourage an
x-<project>- prefix convention for all properties not registered with
the notmuch project?  That seems to have served well enough for email
headers.

Re: On properties and the notmuch CLI

[VA]

Le 17/02/2019 à 20:35, Matt Armstrong a écrit :
> As far as the risk of property name collisions, perhaps encourage an
> x-<project>- prefix convention for all properties not registered with
> the notmuch project?  That seems to have served well enough for email
> headers.

I'm interested in the answer. I'm writing a graphical MUA (alpha stage) 
and am using property "x-lierre-excerpt" to store a pure-text excerpt of 
each email, so it can be displayed in the messages list without having 
to open the whole email file.

Re: On properties and the notmuch CLI

[rhn]

On Sun, 17 Feb 2019 21:14:33 +0100
VA <dev+notmuch@indigo.re> wrote:

> Le 17/02/2019 à 20:35, Matt Armstrong a écrit :
> > As far as the risk of property name collisions, perhaps encourage an
> > x-<project>- prefix convention for all properties not registered with
> > the notmuch project?  That seems to have served well enough for email
> > headers.  
> 
> I'm interested in the answer. I'm writing a graphical MUA (alpha stage) 
> and am using property "x-lierre-excerpt" to store a pure-text excerpt of 
> each email, so it can be displayed in the messages list without having 
> to open the whole email file.

I'm with the others; writing a GUI client [0] I found that querying would be much more aligned with what I need if I could search arbitrary data (headers or notes).

Cheers,
rhn

[0] https://gitlab.com/rhn_mk1/quemail/

Re: On properties and the notmuch CLI

[David Bremner]

Matt Armstrong <marmstrong@google.com> writes:

>  (c) it is not possible to set or modify properties from the command
>  line.

>
> True?  Are queries like these any more efficient than tag doing a prefix
> regex search over tags?  By this I mean, does the Xapian side keep an
> efficient index of all messages that have at least one property "P"?

It's pretty much the same (or would be, if implimented).  Properties are
stored as xapian terms attached to mail document.  This means there is a
sorted list of P=a, P=b, etc..., we can jump efficiently to the first
one, and construct a query that matches them all. It does mean if you
have an enormous number of values for one property it will be
expensive.  Xapian has support for "wildcard" terms ending in "*".
If I remember correctly this is not supported for boolean ("exact")
prefixes like property: and tag: ; internally this does something
similar, although quite possibly a bit faster since it's internal to
Xapian (and written by Xapian experts).

> Second, a question: is there any possibility of relaxing restriction
> (c)

It's possible. It would require a bit of coding (roughly duplicating
notmuch-tag.c, but with a simpler command line syntax, I suppose). It
could be a bit of a footgun since e.g. notmuch is using some properties
to maintain encryption state. But people can already make themselves sad
with mass tag operations.

> As far as the risk of property name collisions, perhaps encourage an
> x-<project>- prefix convention for all properties not registered with
> the notmuch project?  That seems to have served well enough for email
> headers.

BTW, notmuch is currently using . as a namespace separator, so this
would look more like "x-project." .  Your proposal seems plausible to
me; if others agree we can document it.