Multiple header sources for an index.header query prefix?

Multiple header sources for an index.header query prefix?

[Keith Amidon]

I've been a very happy user of notmuch for a very long time. Thank you 
for such a great tool!

Recently I've been working on implementing a new workflow for handling 
email which uses automated tagging much more heavily. In doing so, I 
want to use some custom header indexes to construct some of the searches 
and it seems like it would be very convenient to be able to index 
multiple custom headers under the same query prefix. For example, I'd 
like to be able to index both X-spam and X-Spam-Result under the query 
prefix "Spam:".

I tried just adding two custom header lines to the config file with the 
same query prefix, like:

[index]
header.Spam=X-Spam
header.Spam=X-Spam-Result

It appeared that after reindexing, notmuch just used the final line with 
the same query prefix. In addition, the python bindings config parser 
fails on such a configuration with an error that the header value is 
already in use. Checking the code, it this behavior seems consistent.

I started thinking about how I would add this feature and was thinking 
it might be better to support in the configuration by using semi-colon 
separated values like some of the other config options. For example:

[index]
header.Spam=X-Spam;X-Spam-Result

Does this seem a reasonable way to go? Is this an extension of the 
custom header indexing behavior that you would be interested in? 
Finally, any suggestions for approaching an implementation? In 
particular, I believe there were some recent changes to also store the 
configuration in the database. Any thoughts on how this suggestion would 
interact with that?

Thanks for your help!

Regards, Keith

Re: Multiple header sources for an index.header query prefix?

[David Bremner]

> I started thinking about how I would add this feature and was thinking 
> it might be better to support in the configuration by using semi-colon 
> separated values like some of the other config options. For example:
>
> [index]
> header.Spam=X-Spam;X-Spam-Result
>
> Does this seem a reasonable way to go? Is this an extension of the 
> custom header indexing behavior that you would be interested in?

It sounds reasonable.

> Finally, any suggestions for approaching an implementation? In 
> particular, I believe there were some recent changes to also store the 
> configuration in the database. Any thoughts on how this suggestion would 
> interact with that?

An implementation would need to update
_notmuch_message_index_user_headers to iterate (using
notmuch_config_get_values_string) over the desired headers. Basically
the call to `_notmuch_message_gen_terms` needs to be repeated once for
each header, using the same value for `prefix_name`.

And of course the tests and notmuch-search-terms(7) would need to be
updated.

d

Re: Multiple header sources for an index.header query prefix?

[Keith Amidon]

On 8/5/21 3:44 AM, David Bremner wrote
>> I started thinking about how I would add this feature and was thinking
>> it might be better to support in the configuration by using semi-colon
>> separated values like some of the other config options. For example:
>>
>> [index]
>> header.Spam=X-Spam;X-Spam-Result
>>
>> Does this seem a reasonable way to go? Is this an extension of the
>> custom header indexing behavior that you would be interested in?
> It sounds reasonable.
>
>> Finally, any suggestions for approaching an implementation? In
>> particular, I believe there were some recent changes to also store the
>> configuration in the database. Any thoughts on how this suggestion would
>> interact with that?
> An implementation would need to update
> _notmuch_message_index_user_headers to iterate (using
> notmuch_config_get_values_string) over the desired headers. Basically
> the call to `_notmuch_message_gen_terms` needs to be repeated once for
> each header, using the same value for `prefix_name`.
>
> And of course the tests and notmuch-search-terms(7) would need to be
> updated.

Thanks for the encouragement and suggestions. The time it has taken me 
to respond to this thread is a fairly accurate indication of the amount 
of time I have to work on this right now, but it is something that I'd 
definitely would find useful so I'll plug away on it when I can make 
time and update the list when I hopefully get something working.

Thanks again, Keith

Re: Multiple header sources for an index.header query prefix?

[David Bremner]

Keith Amidon <camalot@picnicpark.org> writes:

>>> I started thinking about how I would add this feature and was thinking
>>> it might be better to support in the configuration by using semi-colon
>>> separated values like some of the other config options. For example:
>>>
>>> [index]
>>> header.Spam=X-Spam;X-Spam-Result
> Thanks for the encouragement and suggestions. The time it has taken me 
> to respond to this thread is a fairly accurate indication of the amount 
> of time I have to work on this right now, but it is something that I'd 
> definitely would find useful so I'll plug away on it when I can make 
> time and update the list when I hopefully get something working.
>

I should repeat my advertising for the (in-progress) s-expression query
parser [1], which would allow defining macros to group various user
defined headers. So assuming headers XSpam and XSpamResult are defined,

$ notmuch config set squery.Spam '(macro (term) (or (XSpam term) (XSpamResult term)))'
$ notmuch search --query=sexp '(Spam "foo bar baz")'

If that seems workable for you, you could just wait ;).

d

[1] id:20210812170728.1348333-2-david@tethera.net

Re: Multiple header sources for an index.header query prefix?

[Keith Amidon]

On 8/15/21 12:27 PM, David Bremner wrote:
> I should repeat my advertising for the (in-progress) s-expression query
> parser [1], which would allow defining macros to group various user
> defined headers. So assuming headers XSpam and XSpamResult are defined,
>
> $ notmuch config set squery.Spam '(macro (term) (or (XSpam term) (XSpamResult term)))'
> $ notmuch search --query=sexp '(Spam "foo bar baz")'
>
> If that seems workable for you, you could just wait ;).

Neat! I'll check it out. Thanks for the pointer.  --- Keith\r