[PATCH] Index Content-Type of attachments with a contenttype prefix

[PATCH] Index Content-Type of attachments with a contenttype prefix

[Todd]

I wanted to tag messages with calendar invitations, but couldn't as
the information wasn't indexed.

This patch allows for queries for like:

Find calendar invites
- contenttype:text/calendar or contenttype:applicaton/ics

Find any image attachments
- contenttype:image

Find all patches
- contenttype:text/x-patch


- Todd

---
 NEWS                               |  6 ++++++
 completion/notmuch-completion.bash |  2 +-
 doc/man7/notmuch-search-terms.rst  |  6 ++++++
 emacs/notmuch.el                   |  2 +-
 lib/database.cc                    |  1 +
 lib/index.cc                       |  5 +++++
 test/T190-multipart.sh             | 32 ++++++++++++++++++++++++++++++++
 7 files changed, 52 insertions(+), 2 deletions(-)

diff --git a/NEWS b/NEWS
index 44e8d05..5f4622c 100644
--- a/NEWS
+++ b/NEWS
@@ -15,6 +15,12 @@ keyboard shortcuts to saved searches.
 Command-Line Interface
 ----------------------

+There is a new `contenttype:` search prefix
+
+  The new `contenttype:` search prefix allows searching for the
+  content-type of attachments, which is now indexed by `notmuch
+  insert`. See the `notmuch-search-terms` manual page for details.
+
 Stopped `notmuch dump` failing if someone writes to the database

   The dump command now takes the write lock when running. This
diff --git a/completion/notmuch-completion.bash b/completion/notmuch-completion.bash
index d58dc8b..05b5969 100644
--- a/completion/notmuch-completion.bash
+++ b/completion/notmuch-completion.bash
@@ -61,7 +61,7 @@ _notmuch_search_terms()
 		sed "s|^$path/||" | grep -v "\(^\|/\)\(cur\|new\|tmp\)$" ) )
 	    ;;
 	*)
-	    local search_terms="from: to: subject: attachment: tag: id: thread: folder: path: date:"
+	    local search_terms="from: to: subject: attachment: contenttype: tag: id: thread: folder: path: date:"
 	    compopt -o nospace
 	    COMPREPLY=( $(compgen -W "${search_terms}" -- ${cur}) )
 	    ;;
diff --git a/doc/man7/notmuch-search-terms.rst b/doc/man7/notmuch-search-terms.rst
index 1acdaa0..d126ce6 100644
--- a/doc/man7/notmuch-search-terms.rst
+++ b/doc/man7/notmuch-search-terms.rst
@@ -40,6 +40,8 @@ indicate user-supplied values):

 -  attachment:<word>

+-  contenttype:<word>
+
 -  tag:<tag> (or is:<tag>)

 -  id:<message-id>
@@ -66,6 +68,10 @@ by including quotation marks around the phrase, immediately following
 The **attachment:** prefix can be used to search for specific filenames
 (or extensions) of attachments to email messages.

+The **contenttype:** prefix can be used to search for specific
+content-types of attachments to email messages (as specified by the
+sender).
+
 For **tag:** and **is:** valid tag values include **inbox** and
 **unread** by default for new messages added by **notmuch new** as well
 as any other tag values added manually with **notmuch tag**.
diff --git a/emacs/notmuch.el b/emacs/notmuch.el
index 218486a..702700c 100644
--- a/emacs/notmuch.el
+++ b/emacs/notmuch.el
@@ -858,7 +858,7 @@ PROMPT is the string to prompt with."
   (lexical-let
       ((completions
 	(append (list "folder:" "path:" "thread:" "id:" "date:" "from:" "to:"
-		      "subject:" "attachment:")
+		      "subject:" "attachment:" "contenttype:")
 		(mapcar (lambda (tag)
 			  (concat "tag:" (notmuch-escape-boolean-term tag)))
 			(process-lines notmuch-command "search" "--output=tags" "*")))))
diff --git a/lib/database.cc b/lib/database.cc
index 3601f9d..a7a64c9 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -254,6 +254,7 @@ static prefix_t PROBABILISTIC_PREFIX[]= {
     { "from",			"XFROM" },
     { "to",			"XTO" },
     { "attachment",		"XATTACHMENT" },
+    { "contenttype",		"XCONTENTTYPE"},
     { "subject",		"XSUBJECT"},
 };

diff --git a/lib/index.cc b/lib/index.cc
index 1a2e63d..c3f7c6b 100644
--- a/lib/index.cc
+++ b/lib/index.cc
@@ -346,6 +346,11 @@ _index_mime_part (notmuch_message_t *message,
 	return;
     }

+    GMimeContentType*  content_type = g_mime_object_get_content_type(part);
+    if (content_type) {
+	_notmuch_message_gen_terms (message, "contenttype", g_mime_content_type_to_string(content_type));
+    }
+
     if (GMIME_IS_MESSAGE_PART (part)) {
 	GMimeMessage *mime_message;

diff --git a/test/T190-multipart.sh b/test/T190-multipart.sh
index 85cbf67..e3270a7 100755
--- a/test/T190-multipart.sh
+++ b/test/T190-multipart.sh
@@ -104,6 +104,30 @@ Content-Transfer-Encoding: base64
 7w0K
 --==-=-=--
 EOF
+
+cat <<EOF > content_types
+From: Todd <todd@electricoding.com>
+To: todd@electricoding.com
+Subject: odd content types
+Date: Fri, 05 Jan 2001 15:42:57 +0000
+User-Agent: Notmuch/0.5 (http://notmuchmail.org) Emacs/23.3.1 (i486-pc-linux-gnu)
+Message-ID: <87liy5ap01.fsf@yoom.home.cworth.org>
+MIME-Version: 1.0
+Content-Type: multipart/alternative; boundary="==-=-=="
+
+--==-=-==
+Content-Type: application/unique_identifier
+
+<p>This is an embedded message, with a multipart/alternative part.</p>
+
+--==-=-==
+Content-Type: text/some_other_identifier
+
+This is an embedded message, with a multipart/alternative part.
+
+--==-=-==--
+EOF
+cat content_types >> ${MAIL_DIR}/odd_content_type
 notmuch new > /dev/null

 test_begin_subtest "--format=text --part=0, full message"
@@ -727,4 +751,12 @@ test_begin_subtest "html parts included"
 notmuch show --format=json --include-html id:htmlmessage > OUTPUT
 test_expect_equal_json "$(cat OUTPUT)" "$(cat EXPECTED.withhtml)"

+test_begin_subtest "indexes content-type"
+output=$(notmuch search contenttype:application/unique_identifier | notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2001-01-05 [1/1] Todd; odd content types (inbox unread)"
+
+output=$(notmuch search contenttype:text/some_other_identifier | notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2001-01-05 [1/1] Todd; odd content types (inbox unread)"
+
+
 test_done
--
1.9.1

Re: [PATCH] Index Content-Type of attachments with a contenttype prefix

[David Bremner]

Todd <todd@electricoding.com> writes:

> I wanted to tag messages with calendar invitations, but couldn't as
> the information wasn't indexed.
>
> This patch allows for queries for like:
>
> Find calendar invites
> - contenttype:text/calendar or contenttype:applicaton/ics
>
> Find any image attachments
> - contenttype:image
>
> Find all patches
> - contenttype:text/x-patch

The main issue,  that I won't really address in this message (because I'm
hoping Austin finds time to comment) is upgrading the database. The short
version is that a new "database feature" needs to be created.

> --- a/NEWS
> +++ b/NEWS
> @@ -15,6 +15,12 @@ keyboard shortcuts to saved searches.
>  Command-Line Interface
>  ----------------------

Minor point, 0.19 was released, you should start a new NEWS section for
0.20 with date UNRELEASED

> --- a/completion/notmuch-completion.bash
> +++ b/completion/notmuch-completion.bash

If you can without too much suffering, it would be nice to update the zsh completion at the same
time.


> +The **contenttype:** prefix can be used to search for specific
> +content-types of attachments to email messages (as specified by the
> +sender).
> +

I'm not 100% sure, but I did wonder if the docs should mention MIME
somewhere, for people searching.
>      { "attachment",		"XATTACHMENT" },
> +    { "contenttype",		"XCONTENTTYPE"},
>      { "subject",		"XSUBJECT"},

I didn't work through all the details, but I did wonder if it was in
some sense redundant to be indexing contenttype and also the existing
attachement and encrypted pseudotags.  I guess this might be one of
those cases where we are stuck with the extra indexing for now, until we
sort out some query parsing issues.

d

Re: [PATCH] Index Content-Type of attachments with a contenttype prefix

[Jani Nikula]

On Sat, 10 Jan 2015, Todd <todd@electricoding.com> wrote:
> I wanted to tag messages with calendar invitations, but couldn't as
> the information wasn't indexed.
>
> This patch allows for queries for like:
>
> Find calendar invites
> - contenttype:text/calendar or contenttype:applicaton/ics
>
> Find any image attachments
> - contenttype:image
>
> Find all patches
> - contenttype:text/x-patch
>
>
> - Todd
>
> ---
>  NEWS                               |  6 ++++++
>  completion/notmuch-completion.bash |  2 +-
>  doc/man7/notmuch-search-terms.rst  |  6 ++++++
>  emacs/notmuch.el                   |  2 +-
>  lib/database.cc                    |  1 +
>  lib/index.cc                       |  5 +++++
>  test/T190-multipart.sh             | 32 ++++++++++++++++++++++++++++++++

IMO these could be split into several patches.

>  7 files changed, 52 insertions(+), 2 deletions(-)
>
> diff --git a/NEWS b/NEWS
> index 44e8d05..5f4622c 100644
> --- a/NEWS
> +++ b/NEWS
> @@ -15,6 +15,12 @@ keyboard shortcuts to saved searches.
>  Command-Line Interface
>  ----------------------
>
> +There is a new `contenttype:` search prefix
> +
> +  The new `contenttype:` search prefix allows searching for the
> +  content-type of attachments, which is now indexed by `notmuch
> +  insert`. See the `notmuch-search-terms` manual page for details.
> +

Admittedly I did not have the time to dig into details, but I think
"attachment" is misleading, as it's really all mime parts, right?

Will this also index the Content-Type: header of the message itself,
regardless of whether it has mime structure or not? Maybe it should?

>  Stopped `notmuch dump` failing if someone writes to the database
>
>    The dump command now takes the write lock when running. This
> diff --git a/completion/notmuch-completion.bash b/completion/notmuch-completion.bash
> index d58dc8b..05b5969 100644
> --- a/completion/notmuch-completion.bash
> +++ b/completion/notmuch-completion.bash
> @@ -61,7 +61,7 @@ _notmuch_search_terms()
>  		sed "s|^$path/||" | grep -v "\(^\|/\)\(cur\|new\|tmp\)$" ) )
>  	    ;;
>  	*)
> -	    local search_terms="from: to: subject: attachment: tag: id: thread: folder: path: date:"
> +	    local search_terms="from: to: subject: attachment: contenttype: tag: id: thread: folder: path: date:"
>  	    compopt -o nospace
>  	    COMPREPLY=( $(compgen -W "${search_terms}" -- ${cur}) )
>  	    ;;
> diff --git a/doc/man7/notmuch-search-terms.rst b/doc/man7/notmuch-search-terms.rst
> index 1acdaa0..d126ce6 100644
> --- a/doc/man7/notmuch-search-terms.rst
> +++ b/doc/man7/notmuch-search-terms.rst
> @@ -40,6 +40,8 @@ indicate user-supplied values):
>
>  -  attachment:<word>
>
> +-  contenttype:<word>
> +
>  -  tag:<tag> (or is:<tag>)
>
>  -  id:<message-id>
> @@ -66,6 +68,10 @@ by including quotation marks around the phrase, immediately following
>  The **attachment:** prefix can be used to search for specific filenames
>  (or extensions) of attachments to email messages.
>
> +The **contenttype:** prefix can be used to search for specific
> +content-types of attachments to email messages (as specified by the
> +sender).
> +
>  For **tag:** and **is:** valid tag values include **inbox** and
>  **unread** by default for new messages added by **notmuch new** as well
>  as any other tag values added manually with **notmuch tag**.
> diff --git a/emacs/notmuch.el b/emacs/notmuch.el
> index 218486a..702700c 100644
> --- a/emacs/notmuch.el
> +++ b/emacs/notmuch.el
> @@ -858,7 +858,7 @@ PROMPT is the string to prompt with."
>    (lexical-let
>        ((completions
>  	(append (list "folder:" "path:" "thread:" "id:" "date:" "from:" "to:"
> -		      "subject:" "attachment:")
> +		      "subject:" "attachment:" "contenttype:")
>  		(mapcar (lambda (tag)
>  			  (concat "tag:" (notmuch-escape-boolean-term tag)))
>  			(process-lines notmuch-command "search" "--output=tags" "*")))))
> diff --git a/lib/database.cc b/lib/database.cc
> index 3601f9d..a7a64c9 100644
> --- a/lib/database.cc
> +++ b/lib/database.cc
> @@ -254,6 +254,7 @@ static prefix_t PROBABILISTIC_PREFIX[]= {
>      { "from",			"XFROM" },
>      { "to",			"XTO" },
>      { "attachment",		"XATTACHMENT" },
> +    { "contenttype",		"XCONTENTTYPE"},
>      { "subject",		"XSUBJECT"},

Is the use of probabilistic prefix intentional? I think it's probably
the right thing to do, but just checking.

BR,
Jani.

>  };
>
> diff --git a/lib/index.cc b/lib/index.cc
> index 1a2e63d..c3f7c6b 100644
> --- a/lib/index.cc
> +++ b/lib/index.cc
> @@ -346,6 +346,11 @@ _index_mime_part (notmuch_message_t *message,
>  	return;
>      }
>
> +    GMimeContentType*  content_type = g_mime_object_get_content_type(part);
> +    if (content_type) {
> +	_notmuch_message_gen_terms (message, "contenttype", g_mime_content_type_to_string(content_type));
> +    }
> +
>      if (GMIME_IS_MESSAGE_PART (part)) {
>  	GMimeMessage *mime_message;
>
> diff --git a/test/T190-multipart.sh b/test/T190-multipart.sh
> index 85cbf67..e3270a7 100755
> --- a/test/T190-multipart.sh
> +++ b/test/T190-multipart.sh
> @@ -104,6 +104,30 @@ Content-Transfer-Encoding: base64
>  7w0K
>  --==-=-=--
>  EOF
> +
> +cat <<EOF > content_types
> +From: Todd <todd@electricoding.com>
> +To: todd@electricoding.com
> +Subject: odd content types
> +Date: Fri, 05 Jan 2001 15:42:57 +0000
> +User-Agent: Notmuch/0.5 (http://notmuchmail.org) Emacs/23.3.1 (i486-pc-linux-gnu)
> +Message-ID: <87liy5ap01.fsf@yoom.home.cworth.org>
> +MIME-Version: 1.0
> +Content-Type: multipart/alternative; boundary="==-=-=="
> +
> +--==-=-==
> +Content-Type: application/unique_identifier
> +
> +<p>This is an embedded message, with a multipart/alternative part.</p>
> +
> +--==-=-==
> +Content-Type: text/some_other_identifier
> +
> +This is an embedded message, with a multipart/alternative part.
> +
> +--==-=-==--
> +EOF
> +cat content_types >> ${MAIL_DIR}/odd_content_type
>  notmuch new > /dev/null
>
>  test_begin_subtest "--format=text --part=0, full message"
> @@ -727,4 +751,12 @@ test_begin_subtest "html parts included"
>  notmuch show --format=json --include-html id:htmlmessage > OUTPUT
>  test_expect_equal_json "$(cat OUTPUT)" "$(cat EXPECTED.withhtml)"
>
> +test_begin_subtest "indexes content-type"
> +output=$(notmuch search contenttype:application/unique_identifier | notmuch_search_sanitize)
> +test_expect_equal "$output" "thread:XXX   2001-01-05 [1/1] Todd; odd content types (inbox unread)"
> +
> +output=$(notmuch search contenttype:text/some_other_identifier | notmuch_search_sanitize)
> +test_expect_equal "$output" "thread:XXX   2001-01-05 [1/1] Todd; odd content types (inbox unread)"
> +
> +
>  test_done
> --
> 1.9.1
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH] Index Content-Type of attachments with a contenttype prefix

[Todd]

>>>>> "DB" == David Bremner <david@tethera.net> writes:

    DB> Todd <todd@electricoding.com> writes:
    >> I wanted to tag messages with calendar invitations, but couldn't as
    >> the information wasn't indexed.
    >> 
    >> This patch allows for queries for like:
    >> 
    >> Find calendar invites
    >> - contenttype:text/calendar or contenttype:applicaton/ics
    >> 
    >> Find any image attachments
    >> - contenttype:image
    >> 
    >> Find all patches
    >> - contenttype:text/x-patch

    DB> The main issue,  that I won't really address in this message (because I'm
    DB> hoping Austin finds time to comment) is upgrading the database. The short
    DB> version is that a new "database feature" needs to be created.

    I just discovered and starting using notmuch this week, so I'm not
    entirely familiar with the project yet.  I had suspected there might
    be more work :)  I'll look into the "database feature".

    My other thought was to just index the content-type with the
    attachment prefix.  This would have made the first search that I
    attempted (without reading the docs/source) work
    (e.g. attachment:text/calendar or attachment:application/ics).

    Is this preferred to adding a new search term?


    >> --- a/NEWS
    >> +++ b/NEWS
    >> @@ -15,6 +15,12 @@ keyboard shortcuts to saved searches.
    >> Command-Line Interface
    >> ----------------------

    DB> Minor point, 0.19 was released, you should start a new NEWS section for
    DB> 0.20 with date UNRELEASED

    I'll make that change.

    >> --- a/completion/notmuch-completion.bash
    >> +++ b/completion/notmuch-completion.bash

    DB> If you can without too much suffering, it would be nice to update the zsh completion at the same
    DB> time.

    I don't use zsh, but I'll look into what's needed.

    >> +The **contenttype:** prefix can be used to search for specific
    >> +content-types of attachments to email messages (as specified by the
    >> +sender).
    >> +

    DB> I'm not 100% sure, but I did wonder if the docs should mention MIME
    DB> somewhere, for people searching.

    Agreed.
    
    >> { "attachment",		"XATTACHMENT" },
    >> +    { "contenttype",		"XCONTENTTYPE"},
    >> { "subject",		"XSUBJECT"},

    DB> I didn't work through all the details, but I did wonder if it was in
    DB> some sense redundant to be indexing contenttype and also the existing
    DB> attachement and encrypted pseudotags.  I guess this might be one of
    DB> those cases where we are stuck with the extra indexing for now, until we
    DB> sort out some query parsing issues.

    DB> d

    

    - Todd

Re: [PATCH] Index Content-Type of attachments with a contenttype prefix

[Todd]

>>>>> "Jani" == Jani Nikula <jani@nikula.org> writes:

    Jani> On Sat, 10 Jan 2015, Todd <todd@electricoding.com> wrote:
    >> I wanted to tag messages with calendar invitations, but couldn't as
    >> the information wasn't indexed.
    >> 
    >> This patch allows for queries for like:
    >> 
    >> Find calendar invites
    >> - contenttype:text/calendar or contenttype:applicaton/ics
    >> 
    >> Find any image attachments
    >> - contenttype:image
    >> 
    >> Find all patches
    >> - contenttype:text/x-patch
    >> 
    >> 
    >> - Todd
    >> 
    >> ---
    >> NEWS                               |  6 ++++++
    >> completion/notmuch-completion.bash |  2 +-
    >> doc/man7/notmuch-search-terms.rst  |  6 ++++++
    >> emacs/notmuch.el                   |  2 +-
    >> lib/database.cc                    |  1 +
    >> lib/index.cc                       |  5 +++++
    >> test/T190-multipart.sh             | 32 ++++++++++++++++++++++++++++++++

    Jani> IMO these could be split into several patches.

    No problem, I'll split them up the next time I post.

    >> 7 files changed, 52 insertions(+), 2 deletions(-)
    >> 
    >> diff --git a/NEWS b/NEWS
    >> index 44e8d05..5f4622c 100644
    >> --- a/NEWS
    >> +++ b/NEWS
    >> @@ -15,6 +15,12 @@ keyboard shortcuts to saved searches.
    >> Command-Line Interface
    >> ----------------------
    >> 
    >> +There is a new `contenttype:` search prefix
    >> +
    >> +  The new `contenttype:` search prefix allows searching for the
    >> +  content-type of attachments, which is now indexed by `notmuch
    >> +  insert`. See the `notmuch-search-terms` manual page for details.
    >> +

    Jani> Admittedly I did not have the time to dig into details, but I think
    Jani> "attachment" is misleading, as it's really all mime parts, right?

    Jani> Will this also index the Content-Type: header of the message itself,
    Jani> regardless of whether it has mime structure or not? Maybe it
    Jani> should?

    Yes, all mime-parts. It does not index the Content-Type of the
    message itself.  That probably wouldn't be difficult to add if it's
    a desired feature, but if there are plans for indexing other message
    headers it may fit better there.

    I also wasn't too happy with a "contenttype" keyword and debated
    just indexing the information under "attachment" along with the
    filename.

    >> Stopped `notmuch dump` failing if someone writes to the database
    >> 
    >> The dump command now takes the write lock when running. This
    >> diff --git a/completion/notmuch-completion.bash b/completion/notmuch-completion.bash
    >> index d58dc8b..05b5969 100644
    >> --- a/completion/notmuch-completion.bash
    >> +++ b/completion/notmuch-completion.bash
    >> @@ -61,7 +61,7 @@ _notmuch_search_terms()
    >> sed "s|^$path/||" | grep -v "\(^\|/\)\(cur\|new\|tmp\)$" ) )
    >> ;;
    >> *)
    >> -	    local search_terms="from: to: subject: attachment: tag: id: thread: folder: path: date:"
    >> +	    local search_terms="from: to: subject: attachment: contenttype: tag: id: thread: folder: path: date:"
    >> compopt -o nospace
    >> COMPREPLY=( $(compgen -W "${search_terms}" -- ${cur}) )
    >> ;;
    >> diff --git a/doc/man7/notmuch-search-terms.rst b/doc/man7/notmuch-search-terms.rst
    >> index 1acdaa0..d126ce6 100644
    >> --- a/doc/man7/notmuch-search-terms.rst
    >> +++ b/doc/man7/notmuch-search-terms.rst
    >> @@ -40,6 +40,8 @@ indicate user-supplied values):
    >> 
    >> -  attachment:<word>
    >> 
    >> +-  contenttype:<word>
    >> +
    >> -  tag:<tag> (or is:<tag>)
    >> 
    >> -  id:<message-id>
    >> @@ -66,6 +68,10 @@ by including quotation marks around the phrase, immediately following
    >> The **attachment:** prefix can be used to search for specific filenames
    >> (or extensions) of attachments to email messages.
    >> 
    >> +The **contenttype:** prefix can be used to search for specific
    >> +content-types of attachments to email messages (as specified by the
    >> +sender).
    >> +
    >> For **tag:** and **is:** valid tag values include **inbox** and
    >> **unread** by default for new messages added by **notmuch new** as well
    >> as any other tag values added manually with **notmuch tag**.
    >> diff --git a/emacs/notmuch.el b/emacs/notmuch.el
    >> index 218486a..702700c 100644
    >> --- a/emacs/notmuch.el
    >> +++ b/emacs/notmuch.el
    >> @@ -858,7 +858,7 @@ PROMPT is the string to prompt with."
    >> (lexical-let
    >> ((completions
    >> (append (list "folder:" "path:" "thread:" "id:" "date:" "from:" "to:"
    >> -		      "subject:" "attachment:")
    >> +		      "subject:" "attachment:" "contenttype:")
    >> (mapcar (lambda (tag)
    >> (concat "tag:" (notmuch-escape-boolean-term tag)))
    >> (process-lines notmuch-command "search" "--output=tags" "*")))))
    >> diff --git a/lib/database.cc b/lib/database.cc
    >> index 3601f9d..a7a64c9 100644
    >> --- a/lib/database.cc
    >> +++ b/lib/database.cc
    >> @@ -254,6 +254,7 @@ static prefix_t PROBABILISTIC_PREFIX[]= {
    >> { "from",			"XFROM" },
    >> { "to",			"XTO" },
    >> { "attachment",		"XATTACHMENT" },
    >> +    { "contenttype",		"XCONTENTTYPE"},
    >> { "subject",		"XSUBJECT"},

    Jani> Is the use of probabilistic prefix intentional? I think it's probably
    Jani> the right thing to do, but just checking.

    I'm not familiar with Xapian and just followed the precedence of
    attachment.  

    Jani> BR,
    Jani> Jani.

    - Todd

[PATCH v2 0/5] Index the content-type of MIME parts

[Todd]

This set of patches allows for queries for like:

Find calendar invites
- mimetype:text/calendar or mimetype:application/ics

Find any image attachments
- mimetype:image

Find all patches
- mimetype:text/x-patch

It is a rework of a previous set of patches that now adds a new
database feature, only available in newly created databases, which
indicates that all of the mime parts are indexed.  Indexing of new
messages on an existing database should work fine as well, but the
feature won't be enabled via upgrade.

Code reviews welcome, I'm not familiar with this code base.

Todd (5):
  Add a failing unit test for indexed mime types
  Add the NOTMUCH_FEATURE_INDEXED_MIMETYPES database feature
  Add indexing for the mimetype term
  Update completions for Emacs and bash
  Update documentation

 NEWS                               | 16 ++++++++++++++++
 completion/notmuch-completion.bash |  2 +-
 doc/man7/notmuch-search-terms.rst  |  6 ++++++
 emacs/notmuch.el                   |  2 +-
 lib/database-private.h             | 15 ++++++++++++---
 lib/database.cc                    | 11 +++++++++--
 lib/index.cc                       |  6 ++++++
 test/T190-multipart.sh             | 32 ++++++++++++++++++++++++++++++++
 8 files changed, 83 insertions(+), 7 deletions(-)

--
1.9.1

[PATCH v2 1/5] Add a failing unit test for indexed mime types

[Todd]

---
 test/T190-multipart.sh | 32 ++++++++++++++++++++++++++++++++
 1 file changed, 32 insertions(+)

diff --git a/test/T190-multipart.sh b/test/T190-multipart.sh
index 85cbf67..26f7c7a 100755
--- a/test/T190-multipart.sh
+++ b/test/T190-multipart.sh
@@ -104,6 +104,30 @@ Content-Transfer-Encoding: base64
 7w0K
 --==-=-=--
 EOF
+
+cat <<EOF > content_types
+From: Todd <todd@example.com>
+To: todd@example.com
+Subject: odd content types
+Date: Mon, 12 Jan 2014 18:12:32 +0000
+User-Agent: Notmuch/0.5 (http://notmuchmail.org) Emacs/23.3.1 (i486-pc-linux-gnu)
+Message-ID: <KfjfO2WJBw2hrV2p0gjT@example.com>
+MIME-Version: 1.0
+Content-Type: multipart/alternative; boundary="==-=-=="
+
+--==-=-==
+Content-Type: application/unique_identifier
+
+<p>This is an embedded message, with a multipart/alternative part.</p>
+
+--==-=-==
+Content-Type: text/some_other_identifier
+
+This is an embedded message, with a multipart/alternative part.
+
+--==-=-==--
+EOF
+cat content_types >> ${MAIL_DIR}/odd_content_type
 notmuch new > /dev/null
 
 test_begin_subtest "--format=text --part=0, full message"
@@ -727,4 +751,12 @@ test_begin_subtest "html parts included"
 notmuch show --format=json --include-html id:htmlmessage > OUTPUT
 test_expect_equal_json "$(cat OUTPUT)" "$(cat EXPECTED.withhtml)"
 
+test_begin_subtest "indexes mime-type"
+output=$(notmuch search mimetype:application/unique_identifier | notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2014-01-12 [1/1] Todd; odd content types (inbox unread)"
+
+output=$(notmuch search mimetype:text/some_other_identifier | notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2014-01-12 [1/1] Todd; odd content types (inbox unread)"
+
+
 test_done
-- 
1.9.1

[PATCH v2 2/5] Add the NOTMUCH_FEATURE_INDEXED_MIMETYPES database feature

[Todd]

---
 lib/database-private.h | 15 ++++++++++++---
 lib/database.cc        | 10 ++++++++--
 2 files changed, 20 insertions(+), 5 deletions(-)

diff --git a/lib/database-private.h b/lib/database-private.h
index 15e03cc..6d6fa2c 100644
--- a/lib/database-private.h
+++ b/lib/database-private.h
@@ -92,6 +92,14 @@ enum _notmuch_features {
      *
      * Introduced: version 3. */
     NOTMUCH_FEATURE_GHOSTS = 1 << 4,
+
+
+    /* If set, then the database was created after the introduction of
+     * indexed mime types. If unset, then the database may contain a
+     * mixture of messages with indexed and non-indexed mime types.
+     *
+     * Introduced: version 3. */
+    NOTMUCH_FEATURE_INDEXED_MIMETYPES = 1 << 5,
 };
 
 /* In C++, a named enum is its own type, so define bitwise operators
@@ -161,9 +169,10 @@ struct _notmuch_database {
 
 /* Current database features.  If any of these are missing from a
  * database, request an upgrade.
- * NOTMUCH_FEATURE_FROM_SUBJECT_ID_VALUES is not included because
- * upgrade doesn't currently introduce the feature (though brand new
- * databases will have it). */
+ * NOTMUCH_FEATURE_FROM_SUBJECT_ID_VALUES and
+ * NOTMUCH_FEATURE_INDEXED_MIMETYPES are not included because upgrade
+ * doesn't currently introduce the features (though brand new databases
+ * will have it). */
 #define NOTMUCH_FEATURES_CURRENT \
     (NOTMUCH_FEATURE_FILE_TERMS | NOTMUCH_FEATURE_DIRECTORY_DOCS | \
      NOTMUCH_FEATURE_BOOL_FOLDER | NOTMUCH_FEATURE_GHOSTS)
diff --git a/lib/database.cc b/lib/database.cc
index 3601f9d..2de60f8 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -304,6 +304,11 @@ static const struct {
       "exact folder:/path: search", "rw" },
     { NOTMUCH_FEATURE_GHOSTS,
       "mail documents for missing messages", "w"},
+    /* Knowledge of the index mime-types are not required for reading
+     * a database because a reader will just be unable to query
+     * them. */
+    { NOTMUCH_FEATURE_INDEXED_MIMETYPES,
+      "mime-types in database", "w"},
 };
 
 const char *
@@ -646,9 +651,10 @@ notmuch_database_create (const char *path, notmuch_database_t **database)
     if (status)
 	goto DONE;
 
-    /* Upgrade doesn't add this feature to existing databases, but new
-     * databases have it. */
+    /* Upgrade doesn't add these feature to existing databases, but
+     * new databases have them. */
     notmuch->features |= NOTMUCH_FEATURE_FROM_SUBJECT_ID_VALUES;
+    notmuch->features |= NOTMUCH_FEATURE_INDEXED_MIMETYPES;
 
     status = notmuch_database_upgrade (notmuch, NULL, NULL);
     if (status) {
-- 
1.9.1

[PATCH v2 3/5] Add indexing for the mimetype term

[Todd]

---
 lib/database.cc | 1 +
 lib/index.cc    | 6 ++++++
 2 files changed, 7 insertions(+)

diff --git a/lib/database.cc b/lib/database.cc
index 2de60f8..7c7a267 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -254,6 +254,7 @@ static prefix_t PROBABILISTIC_PREFIX[]= {
     { "from",			"XFROM" },
     { "to",			"XTO" },
     { "attachment",		"XATTACHMENT" },
+    { "mimetype",		"XMIMETYPE"},
     { "subject",		"XSUBJECT"},
 };
 
diff --git a/lib/index.cc b/lib/index.cc
index 1a2e63d..2e27558 100644
--- a/lib/index.cc
+++ b/lib/index.cc
@@ -318,6 +318,12 @@ _index_mime_part (notmuch_message_t *message,
 	return;
     }
 
+
+    GMimeContentType*  content_type = g_mime_object_get_content_type(part);
+    if (content_type) {
+	_notmuch_message_gen_terms (message, "mimetype", g_mime_content_type_to_string(content_type));
+    }
+
     if (GMIME_IS_MULTIPART (part)) {
 	GMimeMultipart *multipart = GMIME_MULTIPART (part);
 	int i;
-- 
1.9.1

[PATCH v2 4/5] Update completions for Emacs and bash

[Todd]

This adds completions for Emacs and bash, ZSH does not appear to have
completions for search terms.
---
 completion/notmuch-completion.bash | 2 +-
 emacs/notmuch.el                   | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/completion/notmuch-completion.bash b/completion/notmuch-completion.bash
index d58dc8b..05b5969 100644
--- a/completion/notmuch-completion.bash
+++ b/completion/notmuch-completion.bash
@@ -61,7 +61,7 @@ _notmuch_search_terms()
 		sed "s|^$path/||" | grep -v "\(^\|/\)\(cur\|new\|tmp\)$" ) )
 	    ;;
 	*)
-	    local search_terms="from: to: subject: attachment: tag: id: thread: folder: path: date:"
+	    local search_terms="from: to: subject: attachment: contenttype: tag: id: thread: folder: path: date:"
 	    compopt -o nospace
 	    COMPREPLY=( $(compgen -W "${search_terms}" -- ${cur}) )
 	    ;;
diff --git a/emacs/notmuch.el b/emacs/notmuch.el
index 218486a..ab00454 100644
--- a/emacs/notmuch.el
+++ b/emacs/notmuch.el
@@ -858,7 +858,7 @@ PROMPT is the string to prompt with."
   (lexical-let
       ((completions
 	(append (list "folder:" "path:" "thread:" "id:" "date:" "from:" "to:"
-		      "subject:" "attachment:")
+		      "subject:" "attachment:" "mimetype:")
 		(mapcar (lambda (tag)
 			  (concat "tag:" (notmuch-escape-boolean-term tag)))
 			(process-lines notmuch-command "search" "--output=tags" "*")))))
-- 
1.9.1

[PATCH v2 5/5] Update documentation

[Todd]

---
 NEWS                              | 16 ++++++++++++++++
 doc/man7/notmuch-search-terms.rst |  6 ++++++
 2 files changed, 22 insertions(+)

diff --git a/NEWS b/NEWS
index 44e8d05..0df7977 100644
--- a/NEWS
+++ b/NEWS
@@ -1,3 +1,19 @@
+Notmuch 0.20 (TBD)
+=========================
+
+Overview
+--------
+
+Command-Line Interface
+----------------------
+
+There is a new `mimetype:` search prefix
+
+  The new `mimetype:` search prefix allows searching for the
+  content-type of attachments, which is now indexed by `notmuch
+  insert`. See the `notmuch-search-terms` manual page for details.
+
+
 Notmuch 0.19 (2014-11-14)
 =========================
 
diff --git a/doc/man7/notmuch-search-terms.rst b/doc/man7/notmuch-search-terms.rst
index 1acdaa0..8d1acfc 100644
--- a/doc/man7/notmuch-search-terms.rst
+++ b/doc/man7/notmuch-search-terms.rst
@@ -40,6 +40,8 @@ indicate user-supplied values):
 
 -  attachment:<word>
 
+-  mimetype:<word>
+
 -  tag:<tag> (or is:<tag>)
 
 -  id:<message-id>
@@ -66,6 +68,10 @@ by including quotation marks around the phrase, immediately following
 The **attachment:** prefix can be used to search for specific filenames
 (or extensions) of attachments to email messages.
 
+The **mimetype:** prefix can be used to search for the specific
+content-types of mime parts within email messages (as specified by the
+sender).
+
 For **tag:** and **is:** valid tag values include **inbox** and
 **unread** by default for new messages added by **notmuch new** as well
 as any other tag values added manually with **notmuch tag**.
-- 
1.9.1

Re: [PATCH v2 1/5] Add a failing unit test for indexed mime types

[Jani Nikula]

On Wed, 14 Jan 2015, Todd <todd@electricoding.com> wrote:
> ---
>  test/T190-multipart.sh | 32 ++++++++++++++++++++++++++++++++
>  1 file changed, 32 insertions(+)
>
> diff --git a/test/T190-multipart.sh b/test/T190-multipart.sh
> index 85cbf67..26f7c7a 100755
> --- a/test/T190-multipart.sh
> +++ b/test/T190-multipart.sh
> @@ -104,6 +104,30 @@ Content-Transfer-Encoding: base64
>  7w0K
>  --==-=-=--
>  EOF
> +
> +cat <<EOF > content_types
> +From: Todd <todd@example.com>
> +To: todd@example.com
> +Subject: odd content types
> +Date: Mon, 12 Jan 2014 18:12:32 +0000
> +User-Agent: Notmuch/0.5 (http://notmuchmail.org) Emacs/23.3.1 (i486-pc-linux-gnu)
> +Message-ID: <KfjfO2WJBw2hrV2p0gjT@example.com>
> +MIME-Version: 1.0
> +Content-Type: multipart/alternative; boundary="==-=-=="
> +
> +--==-=-==
> +Content-Type: application/unique_identifier
> +
> +<p>This is an embedded message, with a multipart/alternative part.</p>
> +
> +--==-=-==
> +Content-Type: text/some_other_identifier
> +
> +This is an embedded message, with a multipart/alternative part.
> +
> +--==-=-==--
> +EOF
> +cat content_types >> ${MAIL_DIR}/odd_content_type
>  notmuch new > /dev/null
>  
>  test_begin_subtest "--format=text --part=0, full message"
> @@ -727,4 +751,12 @@ test_begin_subtest "html parts included"
>  notmuch show --format=json --include-html id:htmlmessage > OUTPUT
>  test_expect_equal_json "$(cat OUTPUT)" "$(cat EXPECTED.withhtml)"
>  
> +test_begin_subtest "indexes mime-type"

test_subtest_known_broken

> +output=$(notmuch search mimetype:application/unique_identifier | notmuch_search_sanitize)
> +test_expect_equal "$output" "thread:XXX   2014-01-12 [1/1] Todd; odd content types (inbox unread)"
> +

test_begin_subtest "indexes mime-type #2"
test_subtest_known_broken

Then you'd remove test_subtest_known_broken when you add the feature.

> +output=$(notmuch search mimetype:text/some_other_identifier | notmuch_search_sanitize)
> +test_expect_equal "$output" "thread:XXX   2014-01-12 [1/1] Todd; odd content types (inbox unread)"

I think I'd like to see a subtest for searching
mimetype:multipart/alternative (which should *not* find the message,
IIUC).

BR,
Jani.

> +
> +
>  test_done
> -- 
> 1.9.1
>
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH v2 3/5] Add indexing for the mimetype term

[Jani Nikula]

Please add a commit message.

On Wed, 14 Jan 2015, Todd <todd@electricoding.com> wrote:
> ---
>  lib/database.cc | 1 +
>  lib/index.cc    | 6 ++++++
>  2 files changed, 7 insertions(+)
>
> diff --git a/lib/database.cc b/lib/database.cc
> index 2de60f8..7c7a267 100644
> --- a/lib/database.cc
> +++ b/lib/database.cc
> @@ -254,6 +254,7 @@ static prefix_t PROBABILISTIC_PREFIX[]= {
>      { "from",			"XFROM" },
>      { "to",			"XTO" },
>      { "attachment",		"XATTACHMENT" },
> +    { "mimetype",		"XMIMETYPE"},
>      { "subject",		"XSUBJECT"},
>  };
>  
> diff --git a/lib/index.cc b/lib/index.cc
> index 1a2e63d..2e27558 100644
> --- a/lib/index.cc
> +++ b/lib/index.cc
> @@ -318,6 +318,12 @@ _index_mime_part (notmuch_message_t *message,
>  	return;
>      }
>  
> +

Superfluous newline.

> +    GMimeContentType*  content_type = g_mime_object_get_content_type(part);

Please replace "*  " with " *".

> +    if (content_type) {
> +	_notmuch_message_gen_terms (message, "mimetype", g_mime_content_type_to_string(content_type));
> +    }

Check g_mime_content_type_to_string return value, and g_free the result
afterwards.

BR,
Jani.

> +
>      if (GMIME_IS_MULTIPART (part)) {
>  	GMimeMultipart *multipart = GMIME_MULTIPART (part);
>  	int i;
> -- 
> 1.9.1
>
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH v2 4/5] Update completions for Emacs and bash

[Jani Nikula]

On Wed, 14 Jan 2015, Todd <todd@electricoding.com> wrote:
> This adds completions for Emacs and bash, ZSH does not appear to have
> completions for search terms.
> ---
>  completion/notmuch-completion.bash | 2 +-
>  emacs/notmuch.el                   | 2 +-
>  2 files changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/completion/notmuch-completion.bash b/completion/notmuch-completion.bash
> index d58dc8b..05b5969 100644
> --- a/completion/notmuch-completion.bash
> +++ b/completion/notmuch-completion.bash
> @@ -61,7 +61,7 @@ _notmuch_search_terms()
>  		sed "s|^$path/||" | grep -v "\(^\|/\)\(cur\|new\|tmp\)$" ) )
>  	    ;;
>  	*)
> -	    local search_terms="from: to: subject: attachment: tag: id: thread: folder: path: date:"
> +	    local search_terms="from: to: subject: attachment: contenttype: tag: id: thread: folder: path: date:"

Should be mimetype.

BR,
Jani.

>  	    compopt -o nospace
>  	    COMPREPLY=( $(compgen -W "${search_terms}" -- ${cur}) )
>  	    ;;
> diff --git a/emacs/notmuch.el b/emacs/notmuch.el
> index 218486a..ab00454 100644
> --- a/emacs/notmuch.el
> +++ b/emacs/notmuch.el
> @@ -858,7 +858,7 @@ PROMPT is the string to prompt with."
>    (lexical-let
>        ((completions
>  	(append (list "folder:" "path:" "thread:" "id:" "date:" "from:" "to:"
> -		      "subject:" "attachment:")
> +		      "subject:" "attachment:" "mimetype:")
>  		(mapcar (lambda (tag)
>  			  (concat "tag:" (notmuch-escape-boolean-term tag)))
>  			(process-lines notmuch-command "search" "--output=tags" "*")))))
> -- 
> 1.9.1
>
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH v2 1/5] Add a failing unit test for indexed mime types

[Jani Nikula]

On Thu, 15 Jan 2015, Jani Nikula <jani@nikula.org> wrote:
> On Wed, 14 Jan 2015, Todd <todd@electricoding.com> wrote:
>> ---
>>  test/T190-multipart.sh | 32 ++++++++++++++++++++++++++++++++
>>  1 file changed, 32 insertions(+)
>>
>> diff --git a/test/T190-multipart.sh b/test/T190-multipart.sh
>> index 85cbf67..26f7c7a 100755
>> --- a/test/T190-multipart.sh
>> +++ b/test/T190-multipart.sh
>> @@ -104,6 +104,30 @@ Content-Transfer-Encoding: base64
>>  7w0K
>>  --==-=-=--
>>  EOF
>> +
>> +cat <<EOF > content_types
>> +From: Todd <todd@example.com>
>> +To: todd@example.com
>> +Subject: odd content types
>> +Date: Mon, 12 Jan 2014 18:12:32 +0000
>> +User-Agent: Notmuch/0.5 (http://notmuchmail.org) Emacs/23.3.1 (i486-pc-linux-gnu)
>> +Message-ID: <KfjfO2WJBw2hrV2p0gjT@example.com>
>> +MIME-Version: 1.0
>> +Content-Type: multipart/alternative; boundary="==-=-=="
>> +
>> +--==-=-==
>> +Content-Type: application/unique_identifier
>> +
>> +<p>This is an embedded message, with a multipart/alternative part.</p>
>> +
>> +--==-=-==
>> +Content-Type: text/some_other_identifier
>> +
>> +This is an embedded message, with a multipart/alternative part.
>> +
>> +--==-=-==--
>> +EOF
>> +cat content_types >> ${MAIL_DIR}/odd_content_type
>>  notmuch new > /dev/null
>>  
>>  test_begin_subtest "--format=text --part=0, full message"
>> @@ -727,4 +751,12 @@ test_begin_subtest "html parts included"
>>  notmuch show --format=json --include-html id:htmlmessage > OUTPUT
>>  test_expect_equal_json "$(cat OUTPUT)" "$(cat EXPECTED.withhtml)"
>>  
>> +test_begin_subtest "indexes mime-type"
>
> test_subtest_known_broken
>
>> +output=$(notmuch search mimetype:application/unique_identifier | notmuch_search_sanitize)
>> +test_expect_equal "$output" "thread:XXX   2014-01-12 [1/1] Todd; odd content types (inbox unread)"
>> +
>
> test_begin_subtest "indexes mime-type #2"
> test_subtest_known_broken
>
> Then you'd remove test_subtest_known_broken when you add the feature.
>
>> +output=$(notmuch search mimetype:text/some_other_identifier | notmuch_search_sanitize)
>> +test_expect_equal "$output" "thread:XXX   2014-01-12 [1/1] Todd; odd content types (inbox unread)"
>
> I think I'd like to see a subtest for searching
> mimetype:multipart/alternative (which should *not* find the message,
> IIUC).

Looking at the code in patch 3, I think it will index that. Please add
the test so we know. ;)

>
> BR,
> Jani.
>
>> +
>> +
>>  test_done
>> -- 
>> 1.9.1
>>
>> _______________________________________________
>> notmuch mailing list
>> notmuch@notmuchmail.org
>> http://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH v2 2/5] Add the NOTMUCH_FEATURE_INDEXED_MIMETYPES database feature

[Jani Nikula]

Austin, would you mind having a look at this one please?

Thanks,
Jani.

On Wed, 14 Jan 2015, Todd <todd@electricoding.com> wrote:
> ---
>  lib/database-private.h | 15 ++++++++++++---
>  lib/database.cc        | 10 ++++++++--
>  2 files changed, 20 insertions(+), 5 deletions(-)
>
> diff --git a/lib/database-private.h b/lib/database-private.h
> index 15e03cc..6d6fa2c 100644
> --- a/lib/database-private.h
> +++ b/lib/database-private.h
> @@ -92,6 +92,14 @@ enum _notmuch_features {
>       *
>       * Introduced: version 3. */
>      NOTMUCH_FEATURE_GHOSTS = 1 << 4,
> +
> +
> +    /* If set, then the database was created after the introduction of
> +     * indexed mime types. If unset, then the database may contain a
> +     * mixture of messages with indexed and non-indexed mime types.
> +     *
> +     * Introduced: version 3. */
> +    NOTMUCH_FEATURE_INDEXED_MIMETYPES = 1 << 5,
>  };
>  
>  /* In C++, a named enum is its own type, so define bitwise operators
> @@ -161,9 +169,10 @@ struct _notmuch_database {
>  
>  /* Current database features.  If any of these are missing from a
>   * database, request an upgrade.
> - * NOTMUCH_FEATURE_FROM_SUBJECT_ID_VALUES is not included because
> - * upgrade doesn't currently introduce the feature (though brand new
> - * databases will have it). */
> + * NOTMUCH_FEATURE_FROM_SUBJECT_ID_VALUES and
> + * NOTMUCH_FEATURE_INDEXED_MIMETYPES are not included because upgrade
> + * doesn't currently introduce the features (though brand new databases
> + * will have it). */
>  #define NOTMUCH_FEATURES_CURRENT \
>      (NOTMUCH_FEATURE_FILE_TERMS | NOTMUCH_FEATURE_DIRECTORY_DOCS | \
>       NOTMUCH_FEATURE_BOOL_FOLDER | NOTMUCH_FEATURE_GHOSTS)
> diff --git a/lib/database.cc b/lib/database.cc
> index 3601f9d..2de60f8 100644
> --- a/lib/database.cc
> +++ b/lib/database.cc
> @@ -304,6 +304,11 @@ static const struct {
>        "exact folder:/path: search", "rw" },
>      { NOTMUCH_FEATURE_GHOSTS,
>        "mail documents for missing messages", "w"},
> +    /* Knowledge of the index mime-types are not required for reading
> +     * a database because a reader will just be unable to query
> +     * them. */
> +    { NOTMUCH_FEATURE_INDEXED_MIMETYPES,
> +      "mime-types in database", "w"},
>  };
>  
>  const char *
> @@ -646,9 +651,10 @@ notmuch_database_create (const char *path, notmuch_database_t **database)
>      if (status)
>  	goto DONE;
>  
> -    /* Upgrade doesn't add this feature to existing databases, but new
> -     * databases have it. */
> +    /* Upgrade doesn't add these feature to existing databases, but
> +     * new databases have them. */
>      notmuch->features |= NOTMUCH_FEATURE_FROM_SUBJECT_ID_VALUES;
> +    notmuch->features |= NOTMUCH_FEATURE_INDEXED_MIMETYPES;
>  
>      status = notmuch_database_upgrade (notmuch, NULL, NULL);
>      if (status) {
> -- 
> 1.9.1
>
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH v2 2/5] Add the NOTMUCH_FEATURE_INDEXED_MIMETYPES database feature

[Austin Clements]

Just one nit. Otherwise this patch LGTM.

On January 15, 2015 12:20:08 PM EST, Jani Nikula <jani@nikula.org> wrote:
>
>Austin, would you mind having a look at this one please?
>
>Thanks,
>Jani.
>
>On Wed, 14 Jan 2015, Todd <todd@electricoding.com> wrote:
>> ---
>>  lib/database-private.h | 15 ++++++++++++---
>>  lib/database.cc        | 10 ++++++++--
>>  2 files changed, 20 insertions(+), 5 deletions(-)
>>
>> diff --git a/lib/database-private.h b/lib/database-private.h
>> index 15e03cc..6d6fa2c 100644
>> --- a/lib/database-private.h
>> +++ b/lib/database-private.h
>> @@ -92,6 +92,14 @@ enum _notmuch_features {
>>       *
>>       * Introduced: version 3. */
>>      NOTMUCH_FEATURE_GHOSTS = 1 << 4,
>> +
>> +
>> +    /* If set, then the database was created after the introduction
>of
>> +     * indexed mime types. If unset, then the database may contain a
>> +     * mixture of messages with indexed and non-indexed mime types.
>> +     *
>> +     * Introduced: version 3. */
>> +    NOTMUCH_FEATURE_INDEXED_MIMETYPES = 1 << 5,
>>  };
>>  
>>  /* In C++, a named enum is its own type, so define bitwise operators
>> @@ -161,9 +169,10 @@ struct _notmuch_database {
>>  
>>  /* Current database features.  If any of these are missing from a
>>   * database, request an upgrade.
>> - * NOTMUCH_FEATURE_FROM_SUBJECT_ID_VALUES is not included because
>> - * upgrade doesn't currently introduce the feature (though brand new
>> - * databases will have it). */
>> + * NOTMUCH_FEATURE_FROM_SUBJECT_ID_VALUES and
>> + * NOTMUCH_FEATURE_INDEXED_MIMETYPES are not included because
>upgrade
>> + * doesn't currently introduce the features (though brand new
>databases
>> + * will have it). */
>>  #define NOTMUCH_FEATURES_CURRENT \
>>      (NOTMUCH_FEATURE_FILE_TERMS | NOTMUCH_FEATURE_DIRECTORY_DOCS | \
>>       NOTMUCH_FEATURE_BOOL_FOLDER | NOTMUCH_FEATURE_GHOSTS)
>> diff --git a/lib/database.cc b/lib/database.cc
>> index 3601f9d..2de60f8 100644
>> --- a/lib/database.cc
>> +++ b/lib/database.cc
>> @@ -304,6 +304,11 @@ static const struct {
>>        "exact folder:/path: search", "rw" },
>>      { NOTMUCH_FEATURE_GHOSTS,
>>        "mail documents for missing messages", "w"},
>> +    /* Knowledge of the index mime-types are not required for
>reading
>> +     * a database because a reader will just be unable to query
>> +     * them. */
>> +    { NOTMUCH_FEATURE_INDEXED_MIMETYPES,
>> +      "mime-types in database", "w"},

I would label this "indexed MIME types" to be closer to the enum and because "MIME" is an acronym and hence should be capitalized.

>>  };
>>  
>>  const char *
>> @@ -646,9 +651,10 @@ notmuch_database_create (const char *path,
>notmuch_database_t **database)
>>      if (status)
>>  	goto DONE;
>>  
>> -    /* Upgrade doesn't add this feature to existing databases, but
>new
>> -     * databases have it. */
>> +    /* Upgrade doesn't add these feature to existing databases, but
>> +     * new databases have them. */
>>      notmuch->features |= NOTMUCH_FEATURE_FROM_SUBJECT_ID_VALUES;
>> +    notmuch->features |= NOTMUCH_FEATURE_INDEXED_MIMETYPES;
>>  
>>      status = notmuch_database_upgrade (notmuch, NULL, NULL);
>>      if (status) {
>> -- 
>> 1.9.1
>>
>> _______________________________________________
>> notmuch mailing list
>> notmuch@notmuchmail.org
>> http://notmuchmail.org/mailman/listinfo/notmuch

[PATCH v3 1/5] Add failing unit tests for indexed mime types

[Todd]

---
 test/T190-multipart.sh | 44 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 44 insertions(+)

diff --git a/test/T190-multipart.sh b/test/T190-multipart.sh
index 85cbf67..57f1b61 100755
--- a/test/T190-multipart.sh
+++ b/test/T190-multipart.sh
@@ -104,6 +104,30 @@ Content-Transfer-Encoding: base64
 7w0K
 --==-=-=--
 EOF
+
+cat <<EOF > content_types
+From: Todd <todd@example.com>
+To: todd@example.com
+Subject: odd content types
+Date: Mon, 12 Jan 2014 18:12:32 +0000
+User-Agent: Notmuch/0.5 (http://notmuchmail.org) Emacs/23.3.1 (i486-pc-linux-gnu)
+Message-ID: <KfjfO2WJBw2hrV2p0gjT@example.com>
+MIME-Version: 1.0
+Content-Type: multipart/alternative; boundary="==-=-=="
+
+--==-=-==
+Content-Type: application/unique_identifier
+
+<p>This is an embedded message, with a multipart/alternative part.</p>
+
+--==-=-==
+Content-Type: text/some_other_identifier
+
+This is an embedded message, with a multipart/alternative part.
+
+--==-=-==--
+EOF
+cat content_types >> ${MAIL_DIR}/odd_content_type
 notmuch new > /dev/null
 
 test_begin_subtest "--format=text --part=0, full message"
@@ -727,4 +751,24 @@ test_begin_subtest "html parts included"
 notmuch show --format=json --include-html id:htmlmessage > OUTPUT
 test_expect_equal_json "$(cat OUTPUT)" "$(cat EXPECTED.withhtml)"
 
+test_begin_subtest "indexes mime-type #1"
+test_subtest_known_broken
+output=$(notmuch search mimetype:application/unique_identifier | notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2014-01-12 [1/1] Todd; odd content types (inbox unread)"
+
+test_begin_subtest "indexes mime-type #2"
+test_subtest_known_broken
+output=$(notmuch search mimetype:text/some_other_identifier | notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2014-01-12 [1/1] Todd; odd content types (inbox unread)"
+
+test_begin_subtest "indexes mime-type #3"
+test_subtest_known_broken
+output=$(notmuch search from:todd and mimetype:multipart/alternative | notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2014-01-12 [1/1] Todd; odd content types (inbox unread)"
+
+test_begin_subtest "search for non-existent mime-type"
+test_subtest_known_broken
+output=$(notmuch search mimetype:non-existent/mime-type | notmuch_search_sanitize)
+test_expect_equal "$output" ""
+
 test_done
-- 
1.9.1

[PATCH v3 2/5] Add the NOTMUCH_FEATURE_INDEXED_MIMETYPES database feature

[Todd]

---
 lib/database-private.h | 15 ++++++++++++---
 lib/database.cc        | 10 ++++++++--
 2 files changed, 20 insertions(+), 5 deletions(-)

diff --git a/lib/database-private.h b/lib/database-private.h
index 15e03cc..6d6fa2c 100644
--- a/lib/database-private.h
+++ b/lib/database-private.h
@@ -92,6 +92,14 @@ enum _notmuch_features {
      *
      * Introduced: version 3. */
     NOTMUCH_FEATURE_GHOSTS = 1 << 4,
+
+
+    /* If set, then the database was created after the introduction of
+     * indexed mime types. If unset, then the database may contain a
+     * mixture of messages with indexed and non-indexed mime types.
+     *
+     * Introduced: version 3. */
+    NOTMUCH_FEATURE_INDEXED_MIMETYPES = 1 << 5,
 };
 
 /* In C++, a named enum is its own type, so define bitwise operators
@@ -161,9 +169,10 @@ struct _notmuch_database {
 
 /* Current database features.  If any of these are missing from a
  * database, request an upgrade.
- * NOTMUCH_FEATURE_FROM_SUBJECT_ID_VALUES is not included because
- * upgrade doesn't currently introduce the feature (though brand new
- * databases will have it). */
+ * NOTMUCH_FEATURE_FROM_SUBJECT_ID_VALUES and
+ * NOTMUCH_FEATURE_INDEXED_MIMETYPES are not included because upgrade
+ * doesn't currently introduce the features (though brand new databases
+ * will have it). */
 #define NOTMUCH_FEATURES_CURRENT \
     (NOTMUCH_FEATURE_FILE_TERMS | NOTMUCH_FEATURE_DIRECTORY_DOCS | \
      NOTMUCH_FEATURE_BOOL_FOLDER | NOTMUCH_FEATURE_GHOSTS)
diff --git a/lib/database.cc b/lib/database.cc
index 3601f9d..0d2c417 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -304,6 +304,11 @@ static const struct {
       "exact folder:/path: search", "rw" },
     { NOTMUCH_FEATURE_GHOSTS,
       "mail documents for missing messages", "w"},
+    /* Knowledge of the index mime-types are not required for reading
+     * a database because a reader will just be unable to query
+     * them. */
+    { NOTMUCH_FEATURE_INDEXED_MIMETYPES,
+      "indexed MIME types", "w"},
 };
 
 const char *
@@ -646,9 +651,10 @@ notmuch_database_create (const char *path, notmuch_database_t **database)
     if (status)
 	goto DONE;
 
-    /* Upgrade doesn't add this feature to existing databases, but new
-     * databases have it. */
+    /* Upgrade doesn't add these feature to existing databases, but
+     * new databases have them. */
     notmuch->features |= NOTMUCH_FEATURE_FROM_SUBJECT_ID_VALUES;
+    notmuch->features |= NOTMUCH_FEATURE_INDEXED_MIMETYPES;
 
     status = notmuch_database_upgrade (notmuch, NULL, NULL);
     if (status) {
-- 
1.9.1

[PATCH v3 3/5] Add indexing for the mimetype term

[Todd]

Adds the indexing and removes the broken test flag
---
 lib/database.cc        |  1 +
 lib/index.cc           | 10 ++++++++++
 test/T190-multipart.sh |  4 ----
 3 files changed, 11 insertions(+), 4 deletions(-)

diff --git a/lib/database.cc b/lib/database.cc
index 0d2c417..3974e2e 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -254,6 +254,7 @@ static prefix_t PROBABILISTIC_PREFIX[]= {
     { "from",			"XFROM" },
     { "to",			"XTO" },
     { "attachment",		"XATTACHMENT" },
+    { "mimetype",		"XMIMETYPE"},
     { "subject",		"XSUBJECT"},
 };
 
diff --git a/lib/index.cc b/lib/index.cc
index 1a2e63d..c88ed8d 100644
--- a/lib/index.cc
+++ b/lib/index.cc
@@ -318,6 +318,16 @@ _index_mime_part (notmuch_message_t *message,
 	return;
     }
 
+    GMimeContentType *content_type = g_mime_object_get_content_type(part);
+    if (content_type) {
+	char *mime_string = g_mime_content_type_to_string(content_type);
+	if (mime_string)
+	{
+	    _notmuch_message_gen_terms (message, "mimetype", mime_string);
+	    g_free(mime_string);
+	}
+    }
+
     if (GMIME_IS_MULTIPART (part)) {
 	GMimeMultipart *multipart = GMIME_MULTIPART (part);
 	int i;
diff --git a/test/T190-multipart.sh b/test/T190-multipart.sh
index 57f1b61..de7d361 100755
--- a/test/T190-multipart.sh
+++ b/test/T190-multipart.sh
@@ -752,22 +752,18 @@ notmuch show --format=json --include-html id:htmlmessage > OUTPUT
 test_expect_equal_json "$(cat OUTPUT)" "$(cat EXPECTED.withhtml)"
 
 test_begin_subtest "indexes mime-type #1"
-test_subtest_known_broken
 output=$(notmuch search mimetype:application/unique_identifier | notmuch_search_sanitize)
 test_expect_equal "$output" "thread:XXX   2014-01-12 [1/1] Todd; odd content types (inbox unread)"
 
 test_begin_subtest "indexes mime-type #2"
-test_subtest_known_broken
 output=$(notmuch search mimetype:text/some_other_identifier | notmuch_search_sanitize)
 test_expect_equal "$output" "thread:XXX   2014-01-12 [1/1] Todd; odd content types (inbox unread)"
 
 test_begin_subtest "indexes mime-type #3"
-test_subtest_known_broken
 output=$(notmuch search from:todd and mimetype:multipart/alternative | notmuch_search_sanitize)
 test_expect_equal "$output" "thread:XXX   2014-01-12 [1/1] Todd; odd content types (inbox unread)"
 
 test_begin_subtest "search for non-existent mime-type"
-test_subtest_known_broken
 output=$(notmuch search mimetype:non-existent/mime-type | notmuch_search_sanitize)
 test_expect_equal "$output" ""
 
-- 
1.9.1

[PATCH v3 4/5] Update completions for Emacs and bash

[Todd]

This adds completions for Emacs and bash, ZSH does not appear to have
completions for search terms.
---
 completion/notmuch-completion.bash | 2 +-
 emacs/notmuch.el                   | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/completion/notmuch-completion.bash b/completion/notmuch-completion.bash
index d58dc8b..39320f8 100644
--- a/completion/notmuch-completion.bash
+++ b/completion/notmuch-completion.bash
@@ -61,7 +61,7 @@ _notmuch_search_terms()
 		sed "s|^$path/||" | grep -v "\(^\|/\)\(cur\|new\|tmp\)$" ) )
 	    ;;
 	*)
-	    local search_terms="from: to: subject: attachment: tag: id: thread: folder: path: date:"
+	    local search_terms="from: to: subject: attachment: mimetype: tag: id: thread: folder: path: date:"
 	    compopt -o nospace
 	    COMPREPLY=( $(compgen -W "${search_terms}" -- ${cur}) )
 	    ;;
diff --git a/emacs/notmuch.el b/emacs/notmuch.el
index 218486a..ab00454 100644
--- a/emacs/notmuch.el
+++ b/emacs/notmuch.el
@@ -858,7 +858,7 @@ PROMPT is the string to prompt with."
   (lexical-let
       ((completions
 	(append (list "folder:" "path:" "thread:" "id:" "date:" "from:" "to:"
-		      "subject:" "attachment:")
+		      "subject:" "attachment:" "mimetype:")
 		(mapcar (lambda (tag)
 			  (concat "tag:" (notmuch-escape-boolean-term tag)))
 			(process-lines notmuch-command "search" "--output=tags" "*")))))
-- 
1.9.1

[PATCH v3 5/5] Update documentation

[Todd]

---
 NEWS                              | 11 +++++++++++
 doc/man7/notmuch-search-terms.rst |  6 ++++++
 2 files changed, 17 insertions(+)

diff --git a/NEWS b/NEWS
index abb5ad7..baa4519 100644
--- a/NEWS
+++ b/NEWS
@@ -1,4 +1,5 @@
 Notmuch 0.20 (UNRELEASED)
+=========================
 
 Contrib
 -------
@@ -7,6 +8,16 @@ Contrib
 previously provided by `notmuch-deliver` should now be provided by
 `notmuch insert`, provided by the main notmuch binary.
 
+Command-Line Interface
+----------------------
+
+There is a new `mimetype:` search prefix
+
+  The new `mimetype:` search prefix allows searching for the
+  content-type of attachments, which is now indexed by `notmuch
+  insert`. See the `notmuch-search-terms` manual page for details.
+
+
 Notmuch 0.19 (2014-11-14)
 =========================
 
diff --git a/doc/man7/notmuch-search-terms.rst b/doc/man7/notmuch-search-terms.rst
index 1acdaa0..8d1acfc 100644
--- a/doc/man7/notmuch-search-terms.rst
+++ b/doc/man7/notmuch-search-terms.rst
@@ -40,6 +40,8 @@ indicate user-supplied values):
 
 -  attachment:<word>
 
+-  mimetype:<word>
+
 -  tag:<tag> (or is:<tag>)
 
 -  id:<message-id>
@@ -66,6 +68,10 @@ by including quotation marks around the phrase, immediately following
 The **attachment:** prefix can be used to search for specific filenames
 (or extensions) of attachments to email messages.
 
+The **mimetype:** prefix can be used to search for the specific
+content-types of mime parts within email messages (as specified by the
+sender).
+
 For **tag:** and **is:** valid tag values include **inbox** and
 **unread** by default for new messages added by **notmuch new** as well
 as any other tag values added manually with **notmuch tag**.
-- 
1.9.1

Re: [PATCH v3 1/5] Add failing unit tests for indexed mime types

[David Bremner]

Todd <todd@electricoding.com> writes:
> +test_begin_subtest "search for non-existent mime-type"
> +test_subtest_known_broken
> +output=$(notmuch search mimetype:non-existent/mime-type | notmuch_search_sanitize)
> +test_expect_equal "$output" ""
> +

This one is not in fact broken, since the search silently returns
nothing in either case.  Not sure if others have stricter ideas, but for
me it would be ok to remote this one test_subtest_known_broken and note
the situation in the commit message.

Re: [PATCH v3 3/5] Add indexing for the mimetype term

[David Bremner]

Todd <todd@electricoding.com> writes:

> Adds the indexing and removes the broken test flag
> ---
>  lib/database.cc        |  1 +
>  lib/index.cc           | 10 ++++++++++
>  test/T190-multipart.sh |  4 ----
>  3 files changed, 11 insertions(+), 4 deletions(-)
>
> diff --git a/lib/database.cc b/lib/database.cc
> index 0d2c417..3974e2e 100644
> --- a/lib/database.cc
> +++ b/lib/database.cc
> @@ -254,6 +254,7 @@ static prefix_t PROBABILISTIC_PREFIX[]= {
>      { "from",			"XFROM" },
>      { "to",			"XTO" },
>      { "attachment",		"XATTACHMENT" },
> +    { "mimetype",		"XMIMETYPE"},
>      { "subject",		"XSUBJECT"},
>  };

I think the commit message should articulate why we are indexing this as
a probabilistic prefix, rather than as a boolean prefix. In particular,
this gives people a last chance to complain.

The reference I know is http://xapian.org/docs/queryparser.html

If I understand correctly (it would be great if you could test this
Todd) , with a probabilistic prefix,

   mimetime:pdf

will match

application/pdf
image/pdf
application/x-pdf
application/x-ext-pdf

but not

application/x-bzpdf
application/x-gzpdf
application/x-xzpdf

On the whole, this is probably more beneficial than bad.  The downside
of probabilistic prefixes/fields is that they are not "anchored", so
there is no easy way to distinguish

      application/pdf

from

      pdf
      application/x-pdf

I guess in a perfect world this would also be explained in
notmuch-search-terms(7), but that's pretty much orthogonal to this
series.

d

Re: [PATCH v3 5/5] Update documentation

[David Bremner]

Todd <todd@electricoding.com> writes:

> +  The new `mimetype:` search prefix allows searching for the
> +  content-type of attachments, which is now indexed by `notmuch
> +  insert`. See the `notmuch-search-terms` manual page for detai

it strikes me as odd to single out notmuch-insert here, since mimetypes
are also indexed by notmuch-new (on new messages). 

>  
> +-  mimetype:<word>
> +

I'm curious what other people think, but my personal preference would be
for the mimetype: docs to more closely match the subject: docs, since
that's how they work.

Re: [PATCH v3 3/5] Add indexing for the mimetype term

[Todd]

[-- Attachment #1: Type: text/plain, Size: 3041 bytes --]


>>>>> "DB" == David Bremner <david@tethera.net> writes:

    DB> Todd <todd@electricoding.com> writes:
    >> Adds the indexing and removes the broken test flag
    >> ---
    >> lib/database.cc        |  1 +
    >> lib/index.cc           | 10 ++++++++++
    >> test/T190-multipart.sh |  4 ----
    >> 3 files changed, 11 insertions(+), 4 deletions(-)
    >>
    >> diff --git a/lib/database.cc b/lib/database.cc
    >> index 0d2c417..3974e2e 100644
    >> --- a/lib/database.cc
    >> +++ b/lib/database.cc
    >> @@ -254,6 +254,7 @@ static prefix_t PROBABILISTIC_PREFIX[]= {
    >> { "from",			"XFROM" },
    >> { "to",			"XTO" },
    >> { "attachment",		"XATTACHMENT" },
    >> +    { "mimetype",		"XMIMETYPE"},
    >> { "subject",		"XSUBJECT"},
    >> };

    DB> I think the commit message should articulate why we are indexing this as
    DB> a probabilistic prefix, rather than as a boolean prefix. In particular,
    DB> this gives people a last chance to complain.

    DB> The reference I know is http://xapian.org/docs/queryparser.html

    DB> If I understand correctly (it would be great if you could test this
    DB> Todd) , with a probabilistic prefix,

    DB>    mimetime:pdf

    DB> will match

    DB> application/pdf
    DB> image/pdf
    DB> application/x-pdf
    DB> application/x-ext-pdf

    DB> but not

    DB> application/x-bzpdf
    DB> application/x-gzpdf
    DB> application/x-xzpdf

    I just tested, and it does work this way with your examples.  I
    *believe* from reading the docs, that xapian is treating the full
    MIME-type queries as phrase searches anyway due to the embedded
    slashes.

    From http://xapian.org/docs/queryparser.html:

         A phrase surrounded with double quotes ("") matches documents
         containing that exact phrase. Hyphenated words are also treated
         as phrases, as are cases such as filenames and email addresses
         (e.g. /etc/passwd or president@whitehouse.gov).

    I think that we'll get good behavior from the types of queries that
    will typically be performed due to this automatic phrasing.



    DB> On the whole, this is probably more beneficial than bad.  The downside
    DB> of probabilistic prefixes/fields is that they are not "anchored", so
    DB> there is no easy way to distinguish

    DB>       application/pdf

    DB> from

    DB>       pdf
    DB>       application/x-pdf

    DB> I guess in a perfect world this would also be explained in
    DB> notmuch-search-terms(7), but that's pretty much orthogonal to this
    DB> series.

    If separate messages with application/pdf and application/x-pdf are
    indexed, then:
    
    mimetype:application/x-pdf finds only the application/x-pdf
    mimetype:application/pdf finds only the application/pdf
    mimetype:pdf finds both of the messages

    I am fairly sure that this behaviour is a result of the automatic
    phrasing mentioned above.

    - Todd
    
    DB> d

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]

[PATCH v4 0/5] Index the content-type of MIME parts

[Todd]

I think I've finished incorporating the feedback.  The final
notmuch-search-terms.rst could use more details, but it should
probably occur after the recent patch that was posted documenting the
probablistic indexing/searching has been committed.

Todd (5):
  test: Add failing unit tests for indexed mime types
  Add the NOTMUCH_FEATURE_INDEXED_MIMETYPES database feature
  Add indexing for the mimetype term
  Update completions for Emacs and bash
  Update documentation

 NEWS                               | 11 +++++++++++
 completion/notmuch-completion.bash |  2 +-
 doc/man7/notmuch-search-terms.rst  |  6 ++++++
 emacs/notmuch.el                   |  2 +-
 lib/database-private.h             | 15 ++++++++++++---
 lib/database.cc                    | 11 +++++++++--
 lib/index.cc                       | 10 ++++++++++
 test/T190-multipart.sh             | 36 ++++++++++++++++++++++++++++++++++++
 8 files changed, 86 insertions(+), 7 deletions(-)

--
1.9.1

[PATCH v4 1/5] test: Add failing unit tests for indexed mime types

[Todd]

Adds three failing unit tests for searching of mime-types.

An attempt was made at adding a negative test (i.e. searching for a
non-existent mime-type and ensuring it didn't return a message), but
that test would always pass making it pointless.
---
 test/T190-multipart.sh | 39 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 39 insertions(+)

diff --git a/test/T190-multipart.sh b/test/T190-multipart.sh
index 85cbf67..a97dc28 100755
--- a/test/T190-multipart.sh
+++ b/test/T190-multipart.sh
@@ -104,6 +104,30 @@ Content-Transfer-Encoding: base64
 7w0K
 --==-=-=--
 EOF
+
+cat <<EOF > content_types
+From: Todd <todd@example.com>
+To: todd@example.com
+Subject: odd content types
+Date: Mon, 12 Jan 2014 18:12:32 +0000
+User-Agent: Notmuch/0.5 (http://notmuchmail.org) Emacs/23.3.1 (i486-pc-linux-gnu)
+Message-ID: <KfjfO2WJBw2hrV2p0gjT@example.com>
+MIME-Version: 1.0
+Content-Type: multipart/alternative; boundary="==-=-=="
+
+--==-=-==
+Content-Type: application/unique_identifier
+
+<p>This is an embedded message, with a multipart/alternative part.</p>
+
+--==-=-==
+Content-Type: text/some_other_identifier
+
+This is an embedded message, with a multipart/alternative part.
+
+--==-=-==--
+EOF
+cat content_types >> ${MAIL_DIR}/odd_content_type
 notmuch new > /dev/null
 
 test_begin_subtest "--format=text --part=0, full message"
@@ -727,4 +751,19 @@ test_begin_subtest "html parts included"
 notmuch show --format=json --include-html id:htmlmessage > OUTPUT
 test_expect_equal_json "$(cat OUTPUT)" "$(cat EXPECTED.withhtml)"
 
+test_begin_subtest "indexes mime-type #1"
+test_subtest_known_broken
+output=$(notmuch search mimetype:application/unique_identifier | notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2014-01-12 [1/1] Todd; odd content types (inbox unread)"
+
+test_begin_subtest "indexes mime-type #2"
+test_subtest_known_broken
+output=$(notmuch search mimetype:text/some_other_identifier | notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2014-01-12 [1/1] Todd; odd content types (inbox unread)"
+
+test_begin_subtest "indexes mime-type #3"
+test_subtest_known_broken
+output=$(notmuch search from:todd and mimetype:multipart/alternative | notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2014-01-12 [1/1] Todd; odd content types (inbox unread)"
+
 test_done
-- 
1.9.1

[PATCH v4 2/5] Add the NOTMUCH_FEATURE_INDEXED_MIMETYPES database feature

[Todd]

This feature will exist in all newly created databases, but there is
no upgrade provided for it.  If this flag exists, it indicates that
the database was created after the indexed MIME-types feature was
added.
---
 lib/database-private.h | 15 ++++++++++++---
 lib/database.cc        | 10 ++++++++--
 2 files changed, 20 insertions(+), 5 deletions(-)

diff --git a/lib/database-private.h b/lib/database-private.h
index 15e03cc..6d6fa2c 100644
--- a/lib/database-private.h
+++ b/lib/database-private.h
@@ -92,6 +92,14 @@ enum _notmuch_features {
      *
      * Introduced: version 3. */
     NOTMUCH_FEATURE_GHOSTS = 1 << 4,
+
+
+    /* If set, then the database was created after the introduction of
+     * indexed mime types. If unset, then the database may contain a
+     * mixture of messages with indexed and non-indexed mime types.
+     *
+     * Introduced: version 3. */
+    NOTMUCH_FEATURE_INDEXED_MIMETYPES = 1 << 5,
 };
 
 /* In C++, a named enum is its own type, so define bitwise operators
@@ -161,9 +169,10 @@ struct _notmuch_database {
 
 /* Current database features.  If any of these are missing from a
  * database, request an upgrade.
- * NOTMUCH_FEATURE_FROM_SUBJECT_ID_VALUES is not included because
- * upgrade doesn't currently introduce the feature (though brand new
- * databases will have it). */
+ * NOTMUCH_FEATURE_FROM_SUBJECT_ID_VALUES and
+ * NOTMUCH_FEATURE_INDEXED_MIMETYPES are not included because upgrade
+ * doesn't currently introduce the features (though brand new databases
+ * will have it). */
 #define NOTMUCH_FEATURES_CURRENT \
     (NOTMUCH_FEATURE_FILE_TERMS | NOTMUCH_FEATURE_DIRECTORY_DOCS | \
      NOTMUCH_FEATURE_BOOL_FOLDER | NOTMUCH_FEATURE_GHOSTS)
diff --git a/lib/database.cc b/lib/database.cc
index 3601f9d..0d2c417 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -304,6 +304,11 @@ static const struct {
       "exact folder:/path: search", "rw" },
     { NOTMUCH_FEATURE_GHOSTS,
       "mail documents for missing messages", "w"},
+    /* Knowledge of the index mime-types are not required for reading
+     * a database because a reader will just be unable to query
+     * them. */
+    { NOTMUCH_FEATURE_INDEXED_MIMETYPES,
+      "indexed MIME types", "w"},
 };
 
 const char *
@@ -646,9 +651,10 @@ notmuch_database_create (const char *path, notmuch_database_t **database)
     if (status)
 	goto DONE;
 
-    /* Upgrade doesn't add this feature to existing databases, but new
-     * databases have it. */
+    /* Upgrade doesn't add these feature to existing databases, but
+     * new databases have them. */
     notmuch->features |= NOTMUCH_FEATURE_FROM_SUBJECT_ID_VALUES;
+    notmuch->features |= NOTMUCH_FEATURE_INDEXED_MIMETYPES;
 
     status = notmuch_database_upgrade (notmuch, NULL, NULL);
     if (status) {
-- 
1.9.1

[PATCH v4 3/5] Add indexing for the mimetype term

[Todd]

This adds the indexing support for the "mimetype:" term and removes
the broken test flag.  The indexing is probablistic in Xapian terms,
which gives a better experience to end users.  Standard content-types
of the form "foo/bar" are automatically interpreted as phrases in
Xapian due to the embedded slash.

Assume, separate messages with application/pdf and application/x-pdf
are indexed, then:

- mimetype:application/x-pdf will find only the application/x-pdf
- mimetype:application/pdf will find only the application/pdf
- mimetype:pdf will find both of the messages
---
 lib/database.cc        |  1 +
 lib/index.cc           | 10 ++++++++++
 test/T190-multipart.sh |  3 ---
 3 files changed, 11 insertions(+), 3 deletions(-)

diff --git a/lib/database.cc b/lib/database.cc
index 0d2c417..3974e2e 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -254,6 +254,7 @@ static prefix_t PROBABILISTIC_PREFIX[]= {
     { "from",			"XFROM" },
     { "to",			"XTO" },
     { "attachment",		"XATTACHMENT" },
+    { "mimetype",		"XMIMETYPE"},
     { "subject",		"XSUBJECT"},
 };
 
diff --git a/lib/index.cc b/lib/index.cc
index 1a2e63d..c88ed8d 100644
--- a/lib/index.cc
+++ b/lib/index.cc
@@ -318,6 +318,16 @@ _index_mime_part (notmuch_message_t *message,
 	return;
     }
 
+    GMimeContentType *content_type = g_mime_object_get_content_type(part);
+    if (content_type) {
+	char *mime_string = g_mime_content_type_to_string(content_type);
+	if (mime_string)
+	{
+	    _notmuch_message_gen_terms (message, "mimetype", mime_string);
+	    g_free(mime_string);
+	}
+    }
+
     if (GMIME_IS_MULTIPART (part)) {
 	GMimeMultipart *multipart = GMIME_MULTIPART (part);
 	int i;
diff --git a/test/T190-multipart.sh b/test/T190-multipart.sh
index a97dc28..ad8d29e 100755
--- a/test/T190-multipart.sh
+++ b/test/T190-multipart.sh
@@ -752,17 +752,14 @@ notmuch show --format=json --include-html id:htmlmessage > OUTPUT
 test_expect_equal_json "$(cat OUTPUT)" "$(cat EXPECTED.withhtml)"
 
 test_begin_subtest "indexes mime-type #1"
-test_subtest_known_broken
 output=$(notmuch search mimetype:application/unique_identifier | notmuch_search_sanitize)
 test_expect_equal "$output" "thread:XXX   2014-01-12 [1/1] Todd; odd content types (inbox unread)"
 
 test_begin_subtest "indexes mime-type #2"
-test_subtest_known_broken
 output=$(notmuch search mimetype:text/some_other_identifier | notmuch_search_sanitize)
 test_expect_equal "$output" "thread:XXX   2014-01-12 [1/1] Todd; odd content types (inbox unread)"
 
 test_begin_subtest "indexes mime-type #3"
-test_subtest_known_broken
 output=$(notmuch search from:todd and mimetype:multipart/alternative | notmuch_search_sanitize)
 test_expect_equal "$output" "thread:XXX   2014-01-12 [1/1] Todd; odd content types (inbox unread)"
 
-- 
1.9.1

[PATCH v4 4/5] Update completions for Emacs and bash

[Todd]

This adds completions for both Emacs and bash. ZSH does not appear to
have completions for search terms.
---
 completion/notmuch-completion.bash | 2 +-
 emacs/notmuch.el                   | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/completion/notmuch-completion.bash b/completion/notmuch-completion.bash
index d58dc8b..39320f8 100644
--- a/completion/notmuch-completion.bash
+++ b/completion/notmuch-completion.bash
@@ -61,7 +61,7 @@ _notmuch_search_terms()
 		sed "s|^$path/||" | grep -v "\(^\|/\)\(cur\|new\|tmp\)$" ) )
 	    ;;
 	*)
-	    local search_terms="from: to: subject: attachment: tag: id: thread: folder: path: date:"
+	    local search_terms="from: to: subject: attachment: mimetype: tag: id: thread: folder: path: date:"
 	    compopt -o nospace
 	    COMPREPLY=( $(compgen -W "${search_terms}" -- ${cur}) )
 	    ;;
diff --git a/emacs/notmuch.el b/emacs/notmuch.el
index 218486a..ab00454 100644
--- a/emacs/notmuch.el
+++ b/emacs/notmuch.el
@@ -858,7 +858,7 @@ PROMPT is the string to prompt with."
   (lexical-let
       ((completions
 	(append (list "folder:" "path:" "thread:" "id:" "date:" "from:" "to:"
-		      "subject:" "attachment:")
+		      "subject:" "attachment:" "mimetype:")
 		(mapcar (lambda (tag)
 			  (concat "tag:" (notmuch-escape-boolean-term tag)))
 			(process-lines notmuch-command "search" "--output=tags" "*")))))
-- 
1.9.1

[PATCH v4 5/5] Update documentation

[Todd]

Adds new entry to the NEWS file, and updates the search terms section
of the man page.  The search terms section needs to be updated again
once the new section in the documentation covering probablistic terms
has been committed.
---
 NEWS                              | 11 +++++++++++
 doc/man7/notmuch-search-terms.rst |  6 ++++++
 2 files changed, 17 insertions(+)

diff --git a/NEWS b/NEWS
index 53e06aa..a4f2a3f 100644
--- a/NEWS
+++ b/NEWS
@@ -1,4 +1,5 @@
 Notmuch 0.20 (UNRELEASED)
+=========================
 
 Documentation
 -------------
@@ -13,6 +14,16 @@ Contrib
 previously provided by `notmuch-deliver` should now be provided by
 `notmuch insert`, provided by the main notmuch binary.
 
+Command-Line Interface
+----------------------
+
+There is a new `mimetype:` search prefix
+
+  The new `mimetype:` search prefix allows searching for the
+  content-type of attachments, which is now indexed by. See the
+  `notmuch-search-terms` manual page for details.
+
+
 Notmuch 0.19 (2014-11-14)
 =========================
 
diff --git a/doc/man7/notmuch-search-terms.rst b/doc/man7/notmuch-search-terms.rst
index 1acdaa0..315e849 100644
--- a/doc/man7/notmuch-search-terms.rst
+++ b/doc/man7/notmuch-search-terms.rst
@@ -40,6 +40,8 @@ indicate user-supplied values):
 
 -  attachment:<word>
 
+-  mimetype:<word>
+
 -  tag:<tag> (or is:<tag>)
 
 -  id:<message-id>
@@ -66,6 +68,10 @@ by including quotation marks around the phrase, immediately following
 The **attachment:** prefix can be used to search for specific filenames
 (or extensions) of attachments to email messages.
 
+The **mimetype:** prefix will be used to match text from the
+content-types of MIME parts within email messages (as specified by the
+sender).
+
 For **tag:** and **is:** valid tag values include **inbox** and
 **unread** by default for new messages added by **notmuch new** as well
 as any other tag values added manually with **notmuch tag**.
-- 
1.9.1

Re: [PATCH v4 5/5] Update documentation

[Jani Nikula]

On Fri, 23 Jan 2015, Todd <todd@electricoding.com> wrote:
> Adds new entry to the NEWS file, and updates the search terms section
> of the man page.  The search terms section needs to be updated again
> once the new section in the documentation covering probablistic terms
> has been committed.
> ---
>  NEWS                              | 11 +++++++++++
>  doc/man7/notmuch-search-terms.rst |  6 ++++++
>  2 files changed, 17 insertions(+)
>
> diff --git a/NEWS b/NEWS
> index 53e06aa..a4f2a3f 100644
> --- a/NEWS
> +++ b/NEWS
> @@ -1,4 +1,5 @@
>  Notmuch 0.20 (UNRELEASED)
> +=========================
>  
>  Documentation
>  -------------
> @@ -13,6 +14,16 @@ Contrib
>  previously provided by `notmuch-deliver` should now be provided by
>  `notmuch insert`, provided by the main notmuch binary.
>  
> +Command-Line Interface
> +----------------------
> +
> +There is a new `mimetype:` search prefix
> +
> +  The new `mimetype:` search prefix allows searching for the
> +  content-type of attachments, which is now indexed by. See the

"indexed by." ?

Otherwise the series looks good to me, tests pass, and I rebuilt my own
database with it and gave it a spin. Seems to work, and I like it.


BR,
Jani.



> +  `notmuch-search-terms` manual page for details.
> +
> +
>  Notmuch 0.19 (2014-11-14)
>  =========================
>  
> diff --git a/doc/man7/notmuch-search-terms.rst b/doc/man7/notmuch-search-terms.rst
> index 1acdaa0..315e849 100644
> --- a/doc/man7/notmuch-search-terms.rst
> +++ b/doc/man7/notmuch-search-terms.rst
> @@ -40,6 +40,8 @@ indicate user-supplied values):
>  
>  -  attachment:<word>
>  
> +-  mimetype:<word>
> +
>  -  tag:<tag> (or is:<tag>)
>  
>  -  id:<message-id>
> @@ -66,6 +68,10 @@ by including quotation marks around the phrase, immediately following
>  The **attachment:** prefix can be used to search for specific filenames
>  (or extensions) of attachments to email messages.
>  
> +The **mimetype:** prefix will be used to match text from the
> +content-types of MIME parts within email messages (as specified by the
> +sender).
> +
>  For **tag:** and **is:** valid tag values include **inbox** and
>  **unread** by default for new messages added by **notmuch new** as well
>  as any other tag values added manually with **notmuch tag**.
> -- 
> 1.9.1
>
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH v4 0/5] Index the content-type of MIME parts

[David Bremner]

Todd <todd@electricoding.com> writes:

> I think I've finished incorporating the feedback.  The final
> notmuch-search-terms.rst could use more details, but it should
> probably occur after the recent patch that was posted documenting the
> probablistic indexing/searching has been committed.

Pushed this version, with amended NEWS, to master.

d