[PATCH] Display extra headers for emacs-mua - db config option

[PATCH] Display extra headers for emacs-mua - db config option

[Johan Parin]

Modify format_headers_sprinter so that it returns some additional headers in the
sexp, instead of a small fixed set of headers.

The extra header list is configured by the database config option
`show.extra_headers'.

This is required in order for the elisp variable
`notmuch-message-headers' to work.

See this bug report:

  https://notmuchmail.org/pipermail/notmuch/2017/026069.html
---
 doc/man1/notmuch-config.rst |  6 ++++++
 notmuch-config.c            |  7 ++++---
 notmuch-show.c              | 41 ++++++++++++++++++++++++++++++++++++-
 3 files changed, 50 insertions(+), 4 deletions(-)

diff --git a/doc/man1/notmuch-config.rst b/doc/man1/notmuch-config.rst
index 28487079..0eb59883 100644
--- a/doc/man1/notmuch-config.rst
+++ b/doc/man1/notmuch-config.rst
@@ -204,6 +204,12 @@ The available configuration items are described below.
     supported. See **notmuch-search-terms(7)** for a list of existing
     prefixes, and an explanation of probabilistic prefixes.
 
+**show.extra_headers**
+    A list of extra headers that will be output by **notmuch show**
+    with ``--format=sexp``, if present in the message.
+
+    Default: empty list.
+
 **built_with.<name>**
     Compile time feature <name>. Current possibilities include
     "compact" (see **notmuch-compact(1)**) and "field_processor" (see
diff --git a/notmuch-config.c b/notmuch-config.c
index 1b079e85..6554ad9b 100644
--- a/notmuch-config.c
+++ b/notmuch-config.c
@@ -841,9 +841,10 @@ typedef struct config_key {
 
 static struct config_key
     config_key_table[] = {
-    { "index.decrypt",   true,   false,  NULL },
-    { "index.header.",   true,   true,   validate_field_name },
-    { "query.",          true,   true,   NULL },
+    { "index.decrypt",      true,   false,  NULL },
+    { "index.header.",      true,   true,   validate_field_name },
+    { "query.",             true,   true,   NULL },
+    { "show.extra_headers", true,   false,  NULL }
 };
 
 static config_key_info_t *
diff --git a/notmuch-show.c b/notmuch-show.c
index 21792a57..4c77468f 100644
--- a/notmuch-show.c
+++ b/notmuch-show.c
@@ -18,11 +18,16 @@
  * Author: Carl Worth <cworth@cworth.org>
  */
 
+#include <string.h>
+
 #include "notmuch-client.h"
 #include "gmime-filter-reply.h"
 #include "sprinter.h"
 #include "zlib-extra.h"
 
+static notmuch_database_t *notmuch = NULL;
+
+
 static const char *
 _get_tags_as_string (const void *ctx, notmuch_message_t *message)
 {
@@ -195,6 +200,38 @@ _is_from_line (const char *line)
 	return 0;
 }
 
+/* Output extra headers if configured with the `show.extra_headers'
+ * database configuration option
+ */
+void
+format_extra_headers_sprinter (sprinter_t *sp, GMimeMessage *message)
+{
+    GMimeHeaderList *header_list;
+    GMimeHeader *header;
+    char *extra_headers, *tofree, *header_name;
+
+    if (notmuch == NULL)
+	return;
+
+    if (notmuch_database_get_config (notmuch, "show.extra_headers",
+				     &extra_headers) != NOTMUCH_STATUS_SUCCESS)
+	return;
+
+    header_list  = g_mime_object_get_header_list (GMIME_OBJECT(message));
+
+    tofree = extra_headers;
+    while ( (header_name = strsep(&extra_headers, ";")) != NULL) {
+
+	header = g_mime_header_list_get_header (header_list, header_name);
+	if (header == NULL)
+	    continue;
+
+	sp->map_key (sp, g_mime_header_get_name(header));
+	sp->string (sp, g_mime_header_get_value(header));
+    }
+    free (tofree);
+}
+
 void
 format_headers_sprinter (sprinter_t *sp, GMimeMessage *message,
 			 bool reply, const _notmuch_message_crypto_t *msg_crypto)
@@ -253,6 +290,9 @@ format_headers_sprinter (sprinter_t *sp, GMimeMessage *message,
     } else {
 	sp->map_key (sp, "Date");
 	sp->string (sp, g_mime_message_get_date_string (sp, message));
+
+	/* Output extra headers the user has configured in the database, if any */
+	format_extra_headers_sprinter (sp, message);
     }
 
     sp->end (sp);
@@ -1152,7 +1192,6 @@ static const notmuch_show_format_t *formatters[] = {
 int
 notmuch_show_command (notmuch_config_t *config, int argc, char *argv[])
 {
-    notmuch_database_t *notmuch;
     notmuch_query_t *query;
     char *query_string;
     int opt_index, ret;
-- 
2.21.0 (Apple Git-122)

Re: [PATCH] Display extra headers for emacs-mua - db config option

[Tomi Ollila]

On Sat, Nov 16 2019, Johan Parin wrote:

> Modify format_headers_sprinter so that it returns some additional headers
> in the 
> sexp, instead of a small fixed set of headers.
>
> The extra header list is configured by the database config option
> `show.extra_headers'.
>
> This is required in order for the elisp variable
> `notmuch-message-headers' to work.

I did not look the patch, but if there is going to be configuration option
(either in config file, or in database), this way is (IMO) the most 
sensible alternative (add to the default options) -- no "full list" or 
"suppress header" options.

Tomi

>
> See this bug report:
>
>   https://notmuchmail.org/pipermail/notmuch/2017/026069.html
> ---
>  doc/man1/notmuch-config.rst |  6 ++++++
>  notmuch-config.c            |  7 ++++---
>  notmuch-show.c              | 41 ++++++++++++++++++++++++++++++++++++-
>  3 files changed, 50 insertions(+), 4 deletions(-)
>
> diff --git a/doc/man1/notmuch-config.rst b/doc/man1/notmuch-config.rst
> index 28487079..0eb59883 100644
> --- a/doc/man1/notmuch-config.rst
> +++ b/doc/man1/notmuch-config.rst
> @@ -204,6 +204,12 @@ The available configuration items are described below.
>      supported. See **notmuch-search-terms(7)** for a list of existing
>      prefixes, and an explanation of probabilistic prefixes.
>  
> +**show.extra_headers**
> +    A list of extra headers that will be output by **notmuch show**
> +    with ``--format=sexp``, if present in the message.
> +
> +    Default: empty list.
> +
>  **built_with.<name>**
>      Compile time feature <name>. Current possibilities include
>      "compact" (see **notmuch-compact(1)**) and "field_processor" (see
> diff --git a/notmuch-config.c b/notmuch-config.c
> index 1b079e85..6554ad9b 100644
> --- a/notmuch-config.c
> +++ b/notmuch-config.c
> @@ -841,9 +841,10 @@ typedef struct config_key {
>  
>  static struct config_key
>      config_key_table[] = {
> -    { "index.decrypt",   true,   false,  NULL },
> -    { "index.header.",   true,   true,   validate_field_name },
> -    { "query.",          true,   true,   NULL },
> +    { "index.decrypt",      true,   false,  NULL },
> +    { "index.header.",      true,   true,   validate_field_name },
> +    { "query.",             true,   true,   NULL },
> +    { "show.extra_headers", true,   false,  NULL }
>  };
>  
>  static config_key_info_t *
> diff --git a/notmuch-show.c b/notmuch-show.c
> index 21792a57..4c77468f 100644
> --- a/notmuch-show.c
> +++ b/notmuch-show.c
> @@ -18,11 +18,16 @@
>   * Author: Carl Worth <cworth@cworth.org>
>   */
>  
> +#include <string.h>
> +
>  #include "notmuch-client.h"
>  #include "gmime-filter-reply.h"
>  #include "sprinter.h"
>  #include "zlib-extra.h"
>  
> +static notmuch_database_t *notmuch = NULL;
> +
> +
>  static const char *
>  _get_tags_as_string (const void *ctx, notmuch_message_t *message)
>  {
> @@ -195,6 +200,38 @@ _is_from_line (const char *line)
>  	return 0;
>  }
>  
> +/* Output extra headers if configured with the `show.extra_headers'
> + * database configuration option
> + */
> +void
> +format_extra_headers_sprinter (sprinter_t *sp, GMimeMessage *message)
> +{
> +    GMimeHeaderList *header_list;
> +    GMimeHeader *header;
> +    char *extra_headers, *tofree, *header_name;
> +
> +    if (notmuch == NULL)
> +	return;
> +
> +    if (notmuch_database_get_config (notmuch, "show.extra_headers",
> +				     &extra_headers) != NOTMUCH_STATUS_SUCCESS)
> +	return;
> +
> +    header_list  = g_mime_object_get_header_list (GMIME_OBJECT(message));
> +
> +    tofree = extra_headers;
> +    while ( (header_name = strsep(&extra_headers, ";")) != NULL) {
> +
> +	header = g_mime_header_list_get_header (header_list, header_name);
> +	if (header == NULL)
> +	    continue;
> +
> +	sp->map_key (sp, g_mime_header_get_name(header));
> +	sp->string (sp, g_mime_header_get_value(header));
> +    }
> +    free (tofree);
> +}
> +
>  void
>  format_headers_sprinter (sprinter_t *sp, GMimeMessage *message,
>  			 bool reply, const _notmuch_message_crypto_t *msg_crypto)
> @@ -253,6 +290,9 @@ format_headers_sprinter (sprinter_t *sp, GMimeMessage *message,
>      } else {
>  	sp->map_key (sp, "Date");
>  	sp->string (sp, g_mime_message_get_date_string (sp, message));
> +
> +	/* Output extra headers the user has configured in the database, if any */
> +	format_extra_headers_sprinter (sp, message);
>      }
>  
>      sp->end (sp);
> @@ -1152,7 +1192,6 @@ static const notmuch_show_format_t *formatters[] = {
>  int
>  notmuch_show_command (notmuch_config_t *config, int argc, char *argv[])
>  {
> -    notmuch_database_t *notmuch;
>      notmuch_query_t *query;
>      char *query_string;
>      int opt_index, ret;
> -- 
> 2.21.0 (Apple Git-122)
>
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> https://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH] Display extra headers for emacs-mua - db config option

[Johan Parin]

This is another version of my attempt to get configurability on extra
headers to be displayed in the emacs MUA. It is a modification of a
previous patch where the extra headers returned by notmuch-show were
hard coded. In this version, the extra headers is configured by a
database config option `show.extra_headers'. As I see it the advantage
with this approach is:

- It gives full flexibility to choose any extra header to be
  displayed. The required steps are two (1) configure
  `show.extra_headers' in the db and (2) set `notmuch-message-headers'
  as desired.
- There is absolutely no performance concern.

The change is also very simple, low impact.

There is no API change (if that is considered a maintenance cost).

If a decision would later be taken to have format_headers_sprinter
return all headers, then users having adopted the configuration option
would still have the same functionality, albeit with an unused db config
entry.

I think this patch is complete, I also added an update to the config man
page. I guess this patch does not require updating the schemata (?)


/Johan

Re: [PATCH] Display extra headers for emacs-mua - db config option

[Daniel Kahn Gillmor]

On Sat 2019-11-16 17:27:23 +0100, Johan Parin wrote:
> Modify format_headers_sprinter so that it returns some additional headers in the
> sexp, instead of a small fixed set of headers.
>
> The extra header list is configured by the database config option
> `show.extra_headers'.
>
> This is required in order for the elisp variable
> `notmuch-message-headers' to work.

Thanks for this work, Johan, and for your persistence on this
functionality.

> diff --git a/notmuch-show.c b/notmuch-show.c
> index 21792a57..4c77468f 100644
> --- a/notmuch-show.c
> +++ b/notmuch-show.c
> @@ -18,11 +18,16 @@
>   * Author: Carl Worth <cworth@cworth.org>
>   */
>  
> +#include <string.h>
> +
>  #include "notmuch-client.h"
>  #include "gmime-filter-reply.h"
>  #include "sprinter.h"
>  #include "zlib-extra.h"
>  
> +static notmuch_database_t *notmuch = NULL;
> +
> +
>  static const char *
>  _get_tags_as_string (const void *ctx, notmuch_message_t *message)
>  {
> @@ -195,6 +200,38 @@ _is_from_line (const char *line)
>  	return 0;
>  }
>  
> +/* Output extra headers if configured with the `show.extra_headers'
> + * database configuration option
> + */
> +void
> +format_extra_headers_sprinter (sprinter_t *sp, GMimeMessage *message)
> +{
> +    GMimeHeaderList *header_list;
> +    GMimeHeader *header;
> +    char *extra_headers, *tofree, *header_name;
> +
> +    if (notmuch == NULL)
> +	return;
> +
> +    if (notmuch_database_get_config (notmuch, "show.extra_headers",
> +				     &extra_headers) != NOTMUCH_STATUS_SUCCESS)
> +	return;
> +
> +    header_list  = g_mime_object_get_header_list (GMIME_OBJECT(message));
> +
> +    tofree = extra_headers;
> +    while ( (header_name = strsep(&extra_headers, ";")) != NULL) {
> +
> +	header = g_mime_header_list_get_header (header_list, header_name);
> +	if (header == NULL)
> +	    continue;
> +
> +	sp->map_key (sp, g_mime_header_get_name(header));
> +	sp->string (sp, g_mime_header_get_value(header));
> +    }
> +    free (tofree);
> +}
> +
>  void
>  format_headers_sprinter (sprinter_t *sp, GMimeMessage *message,
>  			 bool reply, const _notmuch_message_crypto_t *msg_crypto)
> @@ -253,6 +290,9 @@ format_headers_sprinter (sprinter_t *sp, GMimeMessage *message,
>      } else {
>  	sp->map_key (sp, "Date");
>  	sp->string (sp, g_mime_message_get_date_string (sp, message));
> +
> +	/* Output extra headers the user has configured in the database, if any */
> +	format_extra_headers_sprinter (sp, message);
>      }
>  
>      sp->end (sp);
> @@ -1152,7 +1192,6 @@ static const notmuch_show_format_t *formatters[] = {
>  int
>  notmuch_show_command (notmuch_config_t *config, int argc, char *argv[])
>  {
> -    notmuch_database_t *notmuch;
>      notmuch_query_t *query;
>      char *query_string;
>      int opt_index, ret;

I'm a little weirded out by the move to a static notmuch_database_t
*notmuch object.  Are we doing this because we don't want to pass around
the database to internal functions?  I know that the scope of
nomtuch-show.c is basically "global scope", but i worry that it makes
the code more difficult to read and maintain.

It's also not a common idiom in the rest of the codebase (at least not
one that i've seen).

Is it that much worse to pass around the notmuch_database_t *?

     --dkg

Re: [PATCH] Display extra headers for emacs-mua - db config option

[David Bremner]

Daniel Kahn Gillmor <dkg@fifthhorseman.net> writes:

>
> I'm a little weirded out by the move to a static notmuch_database_t
> *notmuch object.  Are we doing this because we don't want to pass around
> the database to internal functions?  I know that the scope of
> nomtuch-show.c is basically "global scope", but i worry that it makes
> the code more difficult to read and maintain.

I had a similar reaction.

>
> It's also not a common idiom in the rest of the codebase (at least not
> one that i've seen).
>
> Is it that much worse to pass around the notmuch_database_t *?

There are some call chains 3 or 4 deep that would need to be modified. I
_think_ that there is always a notmuch_config_t available, so one option
would be to stash the headers or the database there. We do have the
convention other places (mainly in  lib/) of objects having a pointer to
their "parent" database.

Re: [PATCH] Display extra headers for emacs-mua - db config option

[Johan Parin]

Daniel Kahn Gillmor <dkg@fifthhorseman.net> writes:

>
> I'm a little weirded out by the move to a static notmuch_database_t
> *notmuch object.  Are we doing this because we don't want to pass
> around the database to internal functions?

Yes

> I know that the scope of nomtuch-show.c is basically "global scope",
> but i worry that it makes the code more difficult to read and
> maintain.
>

I agree it's not so nice with globals in general.

>
> Is it that much worse to pass around the notmuch_database_t *?
>

It has a lot of code impact, it really propagates to a lot of
places. For instance it also impacts the json and text api, because
those need to have the same signatures as the json api. Also the
database needs to be opened in places where it's currently not opened,
such as in notmuch_search_command().

I have started on such a patch and can certainly complete that if this
is the direction we want to move. I can warn you that there is a lot of
code changes (although they are cosmetic).


/Johan

Re: [PATCH] Display extra headers for emacs-mua - db config option

[Johan Parin]

[-- Attachment #1: Type: text/plain, Size: 553 bytes --]


Johan Parin <johanparin@gmail.com> writes:

> Daniel Kahn Gillmor <dkg@fifthhorseman.net> writes:
>>
>> Is it that much worse to pass around the notmuch_database_t *?
>>
>
> It has a lot of code impact, it really propagates to a lot of
> places. For instance it also impacts the json and text api, because
> those need to have the same signatures as the json api. Also the
> database needs to be opened in places where it's currently not opened,
> such as in notmuch_search_command().
>

Here is a taste (not fully tested, but seems to work).

/Johan


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Display-extra-headers-for-emacs-mua-db-config-option.patch --]
[-- Type: text/x-patch, Size: 23077 bytes --]

From 706a0f784a101b05ac82a462ff6c19c0cef550af Mon Sep 17 00:00:00 2001
From: Johan Parin <johan.parin@gmail.com>
Date: Sun, 17 Nov 2019 13:15:39 +0100
Subject: [PATCH] Display extra headers for emacs-mua - db config option

Modify format_headers_sprinter so that it returns some additional headers in the
sexp, instead of a small fixed set of headers.

The extra header list is configured by the database config option
`show.extra_headers'.

This is required in order for the elisp variable
`notmuch-message-headers' to work.

See this bug report:

  https://notmuchmail.org/pipermail/notmuch/2017/026069.html

This version avoids the file global notmuch database variable in
notmuch-show.c by passing around the database pointer in a lot of
function calls.
---
 doc/man1/notmuch-config.rst |   6 ++
 notmuch-client.h            |  12 ++--
 notmuch-config.c            |   7 ++-
 notmuch-reply.c             |  13 ++--
 notmuch-search.c            |  29 ++++++---
 notmuch-show.c              | 114 ++++++++++++++++++++++++++++--------
 sprinter-json.c             |   3 +-
 sprinter-sexp.c             |   3 +-
 sprinter-text.c             |  10 +++-
 sprinter.h                  |  12 ++--
 10 files changed, 155 insertions(+), 54 deletions(-)

diff --git a/doc/man1/notmuch-config.rst b/doc/man1/notmuch-config.rst
index 28487079..0eb59883 100644
--- a/doc/man1/notmuch-config.rst
+++ b/doc/man1/notmuch-config.rst
@@ -204,6 +204,12 @@ The available configuration items are described below.
     supported. See **notmuch-search-terms(7)** for a list of existing
     prefixes, and an explanation of probabilistic prefixes.
 
+**show.extra_headers**
+    A list of extra headers that will be output by **notmuch show**
+    with ``--format=sexp``, if present in the message.
+
+    Default: empty list.
+
 **built_with.<name>**
     Compile time feature <name>. Current possibilities include
     "compact" (see **notmuch-compact(1)**) and "field_processor" (see
diff --git a/notmuch-client.h b/notmuch-client.h
index 74690054..d4402231 100644
--- a/notmuch-client.h
+++ b/notmuch-client.h
@@ -64,8 +64,10 @@ struct sprinter;
 struct notmuch_show_params;
 
 typedef struct notmuch_show_format {
-    struct sprinter *(*new_sprinter)(const void *ctx, FILE *stream);
-    notmuch_status_t (*part)(const void *ctx, struct sprinter *sprinter,
+    struct sprinter *(*new_sprinter)(notmuch_database_t *notmuch,
+				     const void *ctx, FILE *stream);
+    notmuch_status_t (*part)(notmuch_database_t *notmuch,
+			     const void *ctx, struct sprinter *sprinter,
 			     struct mime_node *node, int indent,
 			     const struct notmuch_show_params *params);
 } notmuch_show_format_t;
@@ -227,12 +229,14 @@ notmuch_status_t
 show_one_part (const char *filename, int part);
 
 void
-format_part_sprinter (const void *ctx, struct sprinter *sp, mime_node_t *node,
+format_part_sprinter (notmuch_database_t *notmuch,
+		      const void *ctx, struct sprinter *sp, mime_node_t *node,
 		      bool output_body,
 		      bool include_html);
 
 void
-format_headers_sprinter (struct sprinter *sp, GMimeMessage *message,
+format_headers_sprinter (notmuch_database_t *notmuch,
+			 struct sprinter *sp, GMimeMessage *message,
 			 bool reply, const _notmuch_message_crypto_t *msg_crypto);
 
 typedef enum {
diff --git a/notmuch-config.c b/notmuch-config.c
index 1b079e85..6554ad9b 100644
--- a/notmuch-config.c
+++ b/notmuch-config.c
@@ -841,9 +841,10 @@ typedef struct config_key {
 
 static struct config_key
     config_key_table[] = {
-    { "index.decrypt",   true,   false,  NULL },
-    { "index.header.",   true,   true,   validate_field_name },
-    { "query.",          true,   true,   NULL },
+    { "index.decrypt",      true,   false,  NULL },
+    { "index.header.",      true,   true,   validate_field_name },
+    { "query.",             true,   true,   NULL },
+    { "show.extra_headers", true,   false,  NULL }
 };
 
 static config_key_info_t *
diff --git a/notmuch-reply.c b/notmuch-reply.c
index 2c30f6f9..5d468c88 100644
--- a/notmuch-reply.c
+++ b/notmuch-reply.c
@@ -614,7 +614,8 @@ enum {
 };
 
 static int
-do_reply (notmuch_config_t *config,
+do_reply (notmuch_database_t *notmuch,
+	  notmuch_config_t *config,
 	  notmuch_query_t *query,
 	  notmuch_show_params_t *params,
 	  int format,
@@ -640,9 +641,9 @@ do_reply (notmuch_config_t *config,
 	}
 
 	if (format == FORMAT_JSON)
-	    sp = sprinter_json_create (config, stdout);
+	    sp = sprinter_json_create (notmuch, config, stdout);
 	else
-	    sp = sprinter_sexp_create (config, stdout);
+	    sp = sprinter_sexp_create (notmuch, config, stdout);
     }
 
     status = notmuch_query_search_messages (query, &messages);
@@ -670,11 +671,11 @@ do_reply (notmuch_config_t *config,
 	    sp->map_key (sp, "reply-headers");
 	    /* FIXME: send msg_crypto here to avoid killing the
 	     * subject line on reply to encrypted messages! */
-	    format_headers_sprinter (sp, reply, true, NULL);
+	    format_headers_sprinter (notmuch, sp, reply, true, NULL);
 
 	    /* Start the original */
 	    sp->map_key (sp, "original");
-	    format_part_sprinter (config, sp, node, true, false);
+	    format_part_sprinter (notmuch, config, sp, node, true, false);
 
 	    /* End */
 	    sp->end (sp);
@@ -765,7 +766,7 @@ notmuch_reply_command (notmuch_config_t *config, int argc, char *argv[])
 	return EXIT_FAILURE;
     }
 
-    if (do_reply (config, query, &params, format, reply_all) != 0)
+    if (do_reply (notmuch, config, query, &params, format, reply_all) != 0)
 	return EXIT_FAILURE;
 
     _notmuch_crypto_cleanup (&params.crypto);
diff --git a/notmuch-search.c b/notmuch-search.c
index fd0b58c5..4a025530 100644
--- a/notmuch-search.c
+++ b/notmuch-search.c
@@ -673,7 +673,8 @@ do_search_tags (const search_context_t *ctx)
 }
 
 static int
-_notmuch_search_prepare (search_context_t *ctx, notmuch_config_t *config, int argc, char *argv[])
+_notmuch_search_prepare (notmuch_database_t *notmuch, search_context_t *ctx,
+			 notmuch_config_t *config, int argc, char *argv[])
 {
     char *query_str;
     unsigned int i;
@@ -681,20 +682,20 @@ _notmuch_search_prepare (search_context_t *ctx, notmuch_config_t *config, int ar
 
     switch (ctx->format_sel) {
     case NOTMUCH_FORMAT_TEXT:
-	ctx->format = sprinter_text_create (config, stdout);
+	ctx->format = sprinter_text_create (notmuch, config, stdout);
 	break;
     case NOTMUCH_FORMAT_TEXT0:
 	if (ctx->output == OUTPUT_SUMMARY) {
 	    fprintf (stderr, "Error: --format=text0 is not compatible with --output=summary.\n");
 	    return EXIT_FAILURE;
 	}
-	ctx->format = sprinter_text0_create (config, stdout);
+	ctx->format = sprinter_text0_create (notmuch, config, stdout);
 	break;
     case NOTMUCH_FORMAT_JSON:
-	ctx->format = sprinter_json_create (config, stdout);
+	ctx->format = sprinter_json_create (notmuch, config, stdout);
 	break;
     case NOTMUCH_FORMAT_SEXP:
-	ctx->format = sprinter_sexp_create (config, stdout);
+	ctx->format = sprinter_sexp_create (notmuch, config, stdout);
 	break;
     default:
 	/* this should never happen */
@@ -803,6 +804,7 @@ static const notmuch_opt_desc_t common_options[] = {
 int
 notmuch_search_command (notmuch_config_t *config, int argc, char *argv[])
 {
+    notmuch_database_t *notmuch;
     search_context_t *ctx = &search_context;
     int opt_index, ret;
 
@@ -841,10 +843,16 @@ notmuch_search_command (notmuch_config_t *config, int argc, char *argv[])
 	return EXIT_FAILURE;
     }
 
-    if (_notmuch_search_prepare (ctx, config,
+    if (notmuch_database_open (notmuch_config_get_database_path (config),
+			       NOTMUCH_DATABASE_MODE_READ_ONLY, &notmuch))
+	return EXIT_FAILURE;
+
+    if (_notmuch_search_prepare (notmuch, ctx, config,
 				 argc - opt_index, argv + opt_index))
 	return EXIT_FAILURE;
 
+    notmuch_database_destroy (notmuch);
+
     switch (ctx->output) {
     case OUTPUT_SUMMARY:
     case OUTPUT_THREADS:
@@ -869,6 +877,7 @@ notmuch_search_command (notmuch_config_t *config, int argc, char *argv[])
 int
 notmuch_address_command (notmuch_config_t *config, int argc, char *argv[])
 {
+    notmuch_database_t *notmuch;
     search_context_t *ctx = &search_context;
     int opt_index, ret;
 
@@ -907,10 +916,16 @@ notmuch_address_command (notmuch_config_t *config, int argc, char *argv[])
 	return EXIT_FAILURE;
     }
 
-    if (_notmuch_search_prepare (ctx, config,
+    if (notmuch_database_open (notmuch_config_get_database_path (config),
+			       NOTMUCH_DATABASE_MODE_READ_ONLY, &notmuch))
+	return EXIT_FAILURE;
+
+    if (_notmuch_search_prepare (notmuch, ctx, config,
 				 argc - opt_index, argv + opt_index))
 	return EXIT_FAILURE;
 
+    notmuch_database_destroy (notmuch);
+
     ctx->addresses = g_hash_table_new_full (strcase_hash, strcase_equal,
 					    _talloc_free_for_g_hash,
 					    _list_free_for_g_hash);
diff --git a/notmuch-show.c b/notmuch-show.c
index 21792a57..94d91aa6 100644
--- a/notmuch-show.c
+++ b/notmuch-show.c
@@ -18,11 +18,16 @@
  * Author: Carl Worth <cworth@cworth.org>
  */
 
+#include <string.h>
+
 #include "notmuch-client.h"
 #include "gmime-filter-reply.h"
 #include "sprinter.h"
 #include "zlib-extra.h"
 
+static notmuch_database_t *notmuch = NULL;
+
+
 static const char *
 _get_tags_as_string (const void *ctx, notmuch_message_t *message)
 {
@@ -195,8 +200,42 @@ _is_from_line (const char *line)
 	return 0;
 }
 
+/* Output extra headers if configured with the `show.extra_headers'
+ * database configuration option
+ */
 void
-format_headers_sprinter (sprinter_t *sp, GMimeMessage *message,
+format_extra_headers_sprinter (notmuch_database_t *notmuch,
+			       sprinter_t *sp, GMimeMessage *message)
+{
+    GMimeHeaderList *header_list;
+    GMimeHeader *header;
+    char *extra_headers, *tofree, *header_name;
+
+    if (notmuch == NULL)
+	return;
+
+    if (notmuch_database_get_config (notmuch, "show.extra_headers",
+				     &extra_headers) != NOTMUCH_STATUS_SUCCESS)
+	return;
+
+    header_list  = g_mime_object_get_header_list (GMIME_OBJECT(message));
+
+    tofree = extra_headers;
+    while ( (header_name = strsep(&extra_headers, ";")) != NULL) {
+
+	header = g_mime_header_list_get_header (header_list, header_name);
+	if (header == NULL)
+	    continue;
+
+	sp->map_key (sp, g_mime_header_get_name(header));
+	sp->string (sp, g_mime_header_get_value(header));
+    }
+    free (tofree);
+}
+
+void
+format_headers_sprinter (notmuch_database_t *notmuch,
+			 sprinter_t *sp, GMimeMessage *message,
 			 bool reply, const _notmuch_message_crypto_t *msg_crypto)
 {
     /* Any changes to the JSON or S-Expression format should be
@@ -253,6 +292,9 @@ format_headers_sprinter (sprinter_t *sp, GMimeMessage *message,
     } else {
 	sp->map_key (sp, "Date");
 	sp->string (sp, g_mime_message_get_date_string (sp, message));
+
+	/* Output extra headers the user has configured in the database, if any */
+	format_extra_headers_sprinter (notmuch, sp, message);
     }
 
     sp->end (sp);
@@ -486,7 +528,8 @@ format_part_sigstatus_sprinter (sprinter_t *sp, GMimeSignatureList *siglist)
 }
 
 static notmuch_status_t
-format_part_text (const void *ctx, sprinter_t *sp, mime_node_t *node,
+format_part_text (notmuch_database_t *notmuch,
+		  const void *ctx, sprinter_t *sp, mime_node_t *node,
 		  int indent, const notmuch_show_params_t *params)
 {
     /* The disposition and content-type metadata are associated with
@@ -499,6 +542,8 @@ format_part_text (const void *ctx, sprinter_t *sp, mime_node_t *node,
     const char *part_type;
     int i;
 
+    (void) notmuch;
+
     if (node->envelope_file) {
 	notmuch_message_t *message = node->envelope_file;
 
@@ -576,7 +621,8 @@ format_part_text (const void *ctx, sprinter_t *sp, mime_node_t *node,
     }
 
     for (i = 0; i < node->nchildren; i++)
-	format_part_text (ctx, sp, mime_node_child (node, i), indent, params);
+	format_part_text (notmuch, ctx, sp, mime_node_child (node, i),
+			  indent, params);
 
     if (GMIME_IS_MESSAGE (node->part))
 	g_mime_stream_printf (stream, "\fbody}\n");
@@ -610,7 +656,8 @@ format_omitted_part_meta_sprinter (sprinter_t *sp, GMimeObject *meta, GMimePart
 }
 
 void
-format_part_sprinter (const void *ctx, sprinter_t *sp, mime_node_t *node,
+format_part_sprinter (notmuch_database_t *notmuch,
+		      const void *ctx, sprinter_t *sp, mime_node_t *node,
 		      bool output_body,
 		      bool include_html)
 {
@@ -625,7 +672,8 @@ format_part_sprinter (const void *ctx, sprinter_t *sp, mime_node_t *node,
 	if (output_body) {
 	    sp->map_key (sp, "body");
 	    sp->begin_list (sp);
-	    format_part_sprinter (ctx, sp, mime_node_child (node, 0), true, include_html);
+	    format_part_sprinter (notmuch, ctx, sp, mime_node_child (node, 0),
+				  true, include_html);
 	    sp->end (sp);
 	}
 
@@ -679,7 +727,8 @@ format_part_sprinter (const void *ctx, sprinter_t *sp, mime_node_t *node,
 	}
 
 	sp->map_key (sp, "headers");
-	format_headers_sprinter (sp, GMIME_MESSAGE (node->part), false, msg_crypto);
+	format_headers_sprinter (notmuch, sp, GMIME_MESSAGE (node->part),
+				 false, msg_crypto);
 
 	sp->end (sp);
 	return;
@@ -771,7 +820,8 @@ format_part_sprinter (const void *ctx, sprinter_t *sp, mime_node_t *node,
 	sp->begin_map (sp);
 
 	sp->map_key (sp, "headers");
-	format_headers_sprinter (sp, GMIME_MESSAGE (node->part), false, NULL);
+	format_headers_sprinter (notmuch, sp, GMIME_MESSAGE (node->part),
+				 false, NULL);
 
 	sp->map_key (sp, "body");
 	sp->begin_list (sp);
@@ -779,7 +829,8 @@ format_part_sprinter (const void *ctx, sprinter_t *sp, mime_node_t *node,
     }
 
     for (i = 0; i < node->nchildren; i++)
-	format_part_sprinter (ctx, sp, mime_node_child (node, i), true, include_html);
+	format_part_sprinter (notmuch, ctx, sp, mime_node_child (node, i),
+			      true, include_html);
 
     /* Close content structures */
     for (i = 0; i < nclose; i++)
@@ -789,11 +840,12 @@ format_part_sprinter (const void *ctx, sprinter_t *sp, mime_node_t *node,
 }
 
 static notmuch_status_t
-format_part_sprinter_entry (const void *ctx, sprinter_t *sp,
+format_part_sprinter_entry (notmuch_database_t *notmuch,
+			    const void *ctx, sprinter_t *sp,
 			    mime_node_t *node, unused (int indent),
 			    const notmuch_show_params_t *params)
 {
-    format_part_sprinter (ctx, sp, node, params->output_body, params->include_html);
+    format_part_sprinter (notmuch, ctx, sp, node, params->output_body, params->include_html);
 
     return NOTMUCH_STATUS_SUCCESS;
 }
@@ -804,7 +856,8 @@ format_part_sprinter_entry (const void *ctx, sprinter_t *sp,
  * http://qmail.org/qmail-manual-html/man5/mbox.html
  */
 static notmuch_status_t
-format_part_mbox (const void *ctx, unused (sprinter_t *sp), mime_node_t *node,
+format_part_mbox (notmuch_database_t *notmuch,
+		  const void *ctx, unused (sprinter_t *sp), mime_node_t *node,
 		  unused (int indent),
 		  unused (const notmuch_show_params_t *params))
 {
@@ -822,6 +875,8 @@ format_part_mbox (const void *ctx, unused (sprinter_t *sp), mime_node_t *node,
     ssize_t line_size;
     ssize_t line_len;
 
+    (void) notmuch;
+
     if (! message)
 	INTERNAL_ERROR ("format_part_mbox requires a root part");
 
@@ -856,10 +911,13 @@ format_part_mbox (const void *ctx, unused (sprinter_t *sp), mime_node_t *node,
 }
 
 static notmuch_status_t
-format_part_raw (unused (const void *ctx), unused (sprinter_t *sp),
+format_part_raw (notmuch_database_t *notmuch,
+		 unused (const void *ctx), unused (sprinter_t *sp),
 		 mime_node_t *node, unused (int indent),
 		 const notmuch_show_params_t *params)
 {
+    (void) notmuch;
+
     if (node->envelope_file) {
 	/* Special case the entire message to avoid MIME parsing. */
 	const char *filename;
@@ -930,7 +988,8 @@ format_part_raw (unused (const void *ctx), unused (sprinter_t *sp),
 }
 
 static notmuch_status_t
-show_message (void *ctx,
+show_message (notmuch_database_t *notmuch,
+	      void *ctx,
 	      const notmuch_show_format_t *format,
 	      sprinter_t *sp,
 	      notmuch_message_t *message,
@@ -951,7 +1010,7 @@ show_message (void *ctx,
 	goto DONE;
     part = mime_node_seek_dfs (root, (params->part < 0 ? 0 : params->part));
     if (part)
-	status = format->part (local, sp, part, indent, params);
+	status = format->part (notmuch, local, sp, part, indent, params);
     if (params->crypto.decrypt == NOTMUCH_DECRYPT_TRUE && session_key_count_error == NOTMUCH_STATUS_SUCCESS) {
 	unsigned int new_session_keys = 0;
 	if (notmuch_message_count_properties (message, "session-key", &new_session_keys) == NOTMUCH_STATUS_SUCCESS &&
@@ -971,7 +1030,8 @@ show_message (void *ctx,
 }
 
 static notmuch_status_t
-show_messages (void *ctx,
+show_messages (notmuch_database_t *notmuch,
+	       void *ctx,
 	       const notmuch_show_format_t *format,
 	       sprinter_t *sp,
 	       notmuch_messages_t *messages,
@@ -999,7 +1059,8 @@ show_messages (void *ctx,
 	next_indent = indent;
 
 	if ((match && (! excluded || ! params->omit_excluded)) || params->entire_thread) {
-	    status = show_message (ctx, format, sp, message, indent, params);
+	    status = show_message (notmuch, ctx, format, sp, message, indent,
+				   params);
 	    if (status && ! res)
 		res = status;
 	    next_indent = indent + 1;
@@ -1007,7 +1068,8 @@ show_messages (void *ctx,
 	    sp->null (sp);
 	}
 
-	status = show_messages (ctx,
+	status = show_messages (notmuch,
+				ctx,
 				format, sp,
 				notmuch_message_get_replies (message),
 				next_indent,
@@ -1027,7 +1089,8 @@ show_messages (void *ctx,
 
 /* Formatted output of single message */
 static int
-do_show_single (void *ctx,
+do_show_single (notmuch_database_t *notmuch,
+		void *ctx,
 		notmuch_query_t *query,
 		const notmuch_show_format_t *format,
 		sprinter_t *sp,
@@ -1060,13 +1123,14 @@ do_show_single (void *ctx,
 
     notmuch_message_set_flag (message, NOTMUCH_MESSAGE_FLAG_MATCH, 1);
 
-    return show_message (ctx, format, sp, message, 0, params)
+    return show_message (notmuch, ctx, format, sp, message, 0, params)
 	   != NOTMUCH_STATUS_SUCCESS;
 }
 
 /* Formatted output of threads */
 static int
-do_show (void *ctx,
+do_show (notmuch_database_t *notmuch,
+	 void *ctx,
 	 notmuch_query_t *query,
 	 const notmuch_show_format_t *format,
 	 sprinter_t *sp,
@@ -1094,7 +1158,7 @@ do_show (void *ctx,
 	    INTERNAL_ERROR ("Thread %s has no toplevel messages.\n",
 			    notmuch_thread_get_thread_id (thread));
 
-	status = show_messages (ctx, format, sp, messages, 0, params);
+	status = show_messages (notmuch, ctx, format, sp, messages, 0, params);
 	if (status && ! res)
 	    res = status;
 
@@ -1152,7 +1216,6 @@ static const notmuch_show_format_t *formatters[] = {
 int
 notmuch_show_command (notmuch_config_t *config, int argc, char *argv[])
 {
-    notmuch_database_t *notmuch;
     notmuch_query_t *query;
     char *query_string;
     int opt_index, ret;
@@ -1284,13 +1347,14 @@ notmuch_show_command (notmuch_config_t *config, int argc, char *argv[])
 
     /* Create structure printer. */
     formatter = formatters[format];
-    sprinter = formatter->new_sprinter (config, stdout);
+    sprinter = formatter->new_sprinter (notmuch, config, stdout);
 
     params.out_stream = g_mime_stream_stdout_new ();
 
     /* If a single message is requested we do not use search_excludes. */
     if (single_message) {
-	ret = do_show_single (config, query, formatter, sprinter, &params);
+	ret = do_show_single (notmuch, config, query, formatter, sprinter,
+			      &params);
     } else {
 	/* We always apply set the exclude flag. The
 	 * exclude=true|false option controls whether or not we return
@@ -1317,7 +1381,7 @@ notmuch_show_command (notmuch_config_t *config, int argc, char *argv[])
 	    params.omit_excluded = false;
 	}
 
-	ret = do_show (config, query, formatter, sprinter, &params);
+	ret = do_show (notmuch, config, query, formatter, sprinter, &params);
     }
 
   DONE:
diff --git a/sprinter-json.c b/sprinter-json.c
index c6ec8577..8f718e6c 100644
--- a/sprinter-json.c
+++ b/sprinter-json.c
@@ -171,7 +171,8 @@ json_separator (struct sprinter *sp)
 }
 
 struct sprinter *
-sprinter_json_create (const void *ctx, FILE *stream)
+sprinter_json_create (notmuch_database_t *notmuch, const void *ctx,
+		      FILE *stream)
 {
     static const struct sprinter_json template = {
 	.vtable = {
diff --git a/sprinter-sexp.c b/sprinter-sexp.c
index 6891ea42..2e6b1174 100644
--- a/sprinter-sexp.c
+++ b/sprinter-sexp.c
@@ -206,7 +206,8 @@ sexp_separator (struct sprinter *sp)
 }
 
 struct sprinter *
-sprinter_sexp_create (const void *ctx, FILE *stream)
+sprinter_sexp_create (notmuch_database_t *notmuch, const void *ctx,
+		      FILE *stream)
 {
     static const struct sprinter_sexp template = {
 	.vtable = {
diff --git a/sprinter-text.c b/sprinter-text.c
index 648b54b1..201bbe12 100644
--- a/sprinter-text.c
+++ b/sprinter-text.c
@@ -113,7 +113,8 @@ text_map_key (unused (struct sprinter *sp), unused (const char *key))
 }
 
 struct sprinter *
-sprinter_text_create (const void *ctx, FILE *stream)
+sprinter_text_create (notmuch_database_t *notmuch, const void *ctx,
+		      FILE *stream)
 {
     static const struct sprinter_text template = {
 	.vtable = {
@@ -133,6 +134,8 @@ sprinter_text_create (const void *ctx, FILE *stream)
     };
     struct sprinter_text *res;
 
+    (void) notmuch;
+
     res = talloc (ctx, struct sprinter_text);
     if (! res)
 	return NULL;
@@ -143,11 +146,12 @@ sprinter_text_create (const void *ctx, FILE *stream)
 }
 
 struct sprinter *
-sprinter_text0_create (const void *ctx, FILE *stream)
+sprinter_text0_create (notmuch_database_t *notmuch, const void *ctx,
+		       FILE *stream)
 {
     struct sprinter *sp;
 
-    sp = sprinter_text_create (ctx, stream);
+    sp = sprinter_text_create (notmuch, ctx, stream);
     if (! sp)
 	return NULL;
 
diff --git a/sprinter.h b/sprinter.h
index 182b1a8b..20f32e1c 100644
--- a/sprinter.h
+++ b/sprinter.h
@@ -65,20 +65,24 @@ typedef struct sprinter {
 /* Create a new unstructured printer that emits the default text format
  * for "notmuch search". */
 struct sprinter *
-sprinter_text_create (const void *ctx, FILE *stream);
+sprinter_text_create (notmuch_database_t *notmuch, const void *ctx,
+		      FILE *stream);
 
 /* Create a new unstructured printer that emits the text format for
  * "notmuch search", with each field separated by a null character
  * instead of the newline character. */
 struct sprinter *
-sprinter_text0_create (const void *ctx, FILE *stream);
+sprinter_text0_create (notmuch_database_t *notmuch, const void *ctx,
+		       FILE *stream);
 
 /* Create a new structure printer that emits JSON. */
 struct sprinter *
-sprinter_json_create (const void *ctx, FILE *stream);
+sprinter_json_create (notmuch_database_t *notmuch, const void *ctx,
+		      FILE *stream);
 
 /* Create a new structure printer that emits S-Expressions. */
 struct sprinter *
-sprinter_sexp_create (const void *ctx, FILE *stream);
+sprinter_sexp_create (notmuch_database_t *notmuch, const void *ctx,
+		      FILE *stream);
 
 #endif // NOTMUCH_SPRINTER_H
-- 
2.21.0 (Apple Git-122)

Re: [PATCH] Display extra headers for emacs-mua - db config option

[Daniel Kahn Gillmor]

[-- Attachment #1: Type: text/plain, Size: 1407 bytes --]

On Thu 2019-11-21 22:56:04 +0100, Johan Parin wrote:
> Here is a taste (not fully tested, but seems to work).

Oof, i see what you mean :(

I haven't tried to implement this a different way, so i don't know
whether there isn't a shorter cut to what we want, but yow it is a lot.

Are we doing something more deeply, architecturally wrong that saddles
us these two rather displeasing choices?

> diff --git a/doc/man1/notmuch-config.rst b/doc/man1/notmuch-config.rst
> index 28487079..0eb59883 100644
> --- a/doc/man1/notmuch-config.rst
> +++ b/doc/man1/notmuch-config.rst
> @@ -204,6 +204,12 @@ The available configuration items are described below.
>      supported. See **notmuch-search-terms(7)** for a list of existing
>      prefixes, and an explanation of probabilistic prefixes.
>  
> +**show.extra_headers**
> +    A list of extra headers that will be output by **notmuch show**
> +    with ``--format=sexp``, if present in the message.
> +
> +    Default: empty list.
> +
>  **built_with.<name>**
>      Compile time feature <name>. Current possibilities include
>      "compact" (see **notmuch-compact(1)**) and "field_processor" (see

Given the implementation choices, i think you want **[STORED IN
DATABASE]** here (see the definitions for query.<name> and
index.header.<prefix> and index.decrypt).

Thanks for taking the time to sort this out!

       --dkg

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: [PATCH] Display extra headers for emacs-mua - db config option

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 2412 bytes --]

On Thu, Nov 21 2019, Johan Parin wrote:
> Johan Parin <johanparin@gmail.com> writes:
>> Daniel Kahn Gillmor <dkg@fifthhorseman.net> writes:
>>>
>>> Is it that much worse to pass around the notmuch_database_t *?

My opinion is that we should not use the static notmuch_database_t
pointer. I think any desire to reach for that is, at best, papering over
some other problem that would be better fixed itself.

>> It has a lot of code impact, it really propagates to a lot of
>> places.
...
> Here is a taste (not fully tested, but seems to work).
...
>  static int
> -do_reply (notmuch_config_t *config,
> +do_reply (notmuch_database_t *notmuch,
> +	  notmuch_config_t *config,

This passing around of the pair of notmuch_database_t* and a
notmuch_config_t* all over the place looks like an anti-pattern to me,
(and something that was just being hidden by the static
notmuch_database_t* that would be better to be fixed properly).

One option for reducing that pair to a single pointer would be to move
the configuration into the database, (as Daniel has been arguing for a
while).

I've argued against that in the past, (and obviously, I came up with the
current approach originally). But the historical situation has already
led to some unpleasant mixing of configuration state, (where some state
is in the configuration file and other state is in the database and it's
hard to predict which is where and which things can be controlled in the
configuration file).

And I think a patch like the above with the "notmuch, config" all over
the place would be another unpleasant result of the historical
convention.

So, I wouldn't be opposed to having the configuration move into the
database entirely. I would definitely like to see a tool that allows for
a dump/restore operation of the configuration state, (so users can still
edit configuration parameters with a text editor and with
fully-documented parameters). Bonus points if the users can also capture
their own commentary/explanation of values they are setting (as they can
do with the current configuration file).

And if things do move, existing configuration files should be updated
with a comment describing what that new dump/restore mechanism is and
that the current configuration file will no longer be used, (though, at
the transition point, obviously configuration parameters should be
copied from the configuration file into the database).

-Carl

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: [PATCH] Display extra headers for emacs-mua - db config option

[David Bremner]

Johan Parin <johanparin@gmail.com> writes:

> +
> +    if (notmuch_database_get_config (notmuch, "show.extra_headers",
> +				     &extra_headers) != NOTMUCH_STATUS_SUCCESS)
> +	return;

Apologies for being late to the discussion of where to store the
configuration. So far we have only stored configuration in the database
where it affected the behaviour of the library API. I know some people
(e.g. dkg) have suggested it would be better to store all of the
configuration in the database for consistency, while others are
disgruntled that some of the configuration is not editable with text
editor. The approach here would represent a change, which I'm not
necessarily against, but I think we need to think it through.

> +
> +    header_list  = g_mime_object_get_header_list (GMIME_OBJECT(message));
> +
> +    tofree = extra_headers;

It's somewhat a matter of taste, but I would rather introduce a variable
for the pointer to be mutated by strsep, and call  free (extra_headers) below

Re: [PATCH] Display extra headers for emacs-mua - db config option

[Daniel Kahn Gillmor]

[-- Attachment #1: Type: text/plain, Size: 1216 bytes --]

On Thu 2019-11-21 08:27:04 -0400, David Bremner wrote:
> Apologies for being late to the discussion of where to store the
> configuration. So far we have only stored configuration in the database
> where it affected the behaviour of the library API.

While i'm being ambitious, i'd like also to eventually consider moving
more of the existing CLI functionality into being accessible from the
library.  I think this would help downstream MUAs that use notmuch who
currently can't (or don't want to, for whatever reason) take advantage
of the existing CLI, but can use the library.

If we stuff more config in the config file, then the behavior of any
future move to the library will have to grapple with the config move
(currently the library never reads the config file, it just opens the
database).

> I know some people (e.g. dkg) have suggested it would be better to
> store all of the configuration in the database for consistency, while
> others are disgruntled that some of the configuration is not editable
> with text editor.

It would still editable with a text editor -- you just need to edit the
output of "notmuch dump --include config" and feed the result back into
"notmuch restore" :)

         --dkg

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: [PATCH] Display extra headers for emacs-mua - db config option

[Tomi Ollila]

On Fri, Nov 22 2019, Daniel Kahn Gillmor wrote:

> On Thu 2019-11-21 08:27:04 -0400, David Bremner wrote:
>> Apologies for being late to the discussion of where to store the
>> configuration. So far we have only stored configuration in the database
>> where it affected the behaviour of the library API.
>
> While i'm being ambitious, i'd like also to eventually consider moving
> more of the existing CLI functionality into being accessible from the
> library.  I think this would help downstream MUAs that use notmuch who
> currently can't (or don't want to, for whatever reason) take advantage
> of the existing CLI, but can use the library.
>
> If we stuff more config in the config file, then the behavior of any
> future move to the library will have to grapple with the config move
> (currently the library never reads the config file, it just opens the
> database).

How can library open the database if it doesn't read the config file
-- the config file defines where database is located =D

>> I know some people (e.g. dkg) have suggested it would be better to
>> store all of the configuration in the database for consistency, while
>> others are disgruntled that some of the configuration is not editable
>> with text editor.
>
> It would still editable with a text editor -- you just need to edit the
> output of "notmuch dump --include config" and feed the result back into
> "notmuch restore" :)

If the library can somehow guess the location of the database without
reading the config file ;D I think dumping -- editing -- restoring
the database is tolerable solution -- provided it is clearly documented
for people like me who wants to edit configuration files w/ text editor(*),
for forgets these instructions easily. If, in the future, we still have
configuration file(+), notmuch could also write such instructions to
the config file.

>
>          --dkg

Tomi

(*) like someone may know I'm not too fond of editing configuration files
    when there are alternatives available (like in this case) -- however
    is someone else(tm) takes the time to create this solution and it uses
    configuration files, I'll use it...

(+) if we dropped the configuration file, then notmuch, and library could
    open database from ~/mail/notmuch/ by default, or from location pointed
    by e.g. NOTMUCH_DATABASE_DIR -- as an additional benefit(?) notmuch
    would pollute user's $HOME by one file less -- the `.notmuch-config`.

moving the config into the database [was: Re: [PATCH] Display extra headers for emacs-mua - db config option]

[Daniel Kahn Gillmor]

[-- Attachment #1: Type: text/plain, Size: 1843 bytes --]

On Thu 2019-11-21 23:38:06 +0200, Tomi Ollila wrote:
> How can library open the database if it doesn't read the config file
> -- the config file defines where database is located =D

The *only* thing i think worth keeping in the config file is ultimately
the location of the database, though i would eventually prefer that we
introduce a NOTMUCH_DATABASE env var, and use it if we can't find a
config.

This is the moral equivalent of the NOTMUCH_CONFIG env var, as far as
i'm concerned, but more rationalized.

and *eventually eventually* drop the config file entirely, yes.

> If the library can somehow guess the location of the database without
> reading the config file ;D I think dumping -- editing -- restoring
> the database is tolerable solution -- provided it is clearly documented
> for people like me who wants to edit configuration files w/ text
> editor(*),

Better than documenting, i'd be happy if we would add a "notmuch config
edit" subcommand, which handles the above sequence for you, invoking
$EDITOR at the appropriate time.

The only caveat i see there is if the end user wants to inject comments
in the config file, which would then be stripped out in between these
invocations.  perhaps someone who finds these comments in config files
super important could propose a way to stash them in the db and recover
them during "notmuch config edit" as well :)

> (+) if we dropped the configuration file, then notmuch, and library could
>     open database from ~/mail/notmuch/ by default, or from location pointed
>     by e.g. NOTMUCH_DATABASE_DIR -- as an additional benefit(?) notmuch
>     would pollute user's $HOME by one file less -- the `.notmuch-config`.

You named the env var differently than i did, but i'd be happy to go
with your name, since it seems we arrived at the same design
independently :)

     --dkg

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: moving the config into the database [was: Re: [PATCH] Display extra headers for emacs-mua - db config option]

[Jorge P. de Morais Neto]

Em [2019-11-22 sex 10:43:40+0800], Daniel Kahn Gillmor escreveu:

> Better than documenting, i'd be happy if we would add a "notmuch config
> edit" subcommand, which handles the above sequence for you, invoking
> $EDITOR at the appropriate time.
>
> The only caveat i see there is if the end user wants to inject comments
> in the config file, which would then be stripped out in between these
> invocations.  perhaps someone who finds these comments in config files
> super important could propose a way to stash them in the db and recover
> them during "notmuch config edit" as well :)

I sync my two Notmuch configuration files -- personal notebook and
workplace desktop -- over Unison via a USB flash drive.  This way, when
I want to reconfigure Notmuch in one machine, I have the other machine's
configuration as reference.  However, I do not sync the entire Notmuch
databases because, since they are big and change frequently, I suppose
they would wear the USB flash drive and also increase the
synchronization time.

Thus for my use case it would be useful if "notmuch config edit" could
automatically keep a perennial text file reflecting the configuration in
the database.  That text file would then be synchronized by Unison.

Regards
-- 
- <https://jorgemorais.gitlab.io/justice-for-rms/>
- I am Brazilian.  I hope my English is correct and I welcome feedback.
- Please adopt free formats like PDF, ODF, Org, LaTeX, Opus, WebM and 7z.
- Free/libre software for Replicant, LineageOS and Android: https://f-droid.org
- [[https://www.gnu.org/philosophy/free-sw.html][What is free software?]]

Re: moving the config into the database [was: Re: [PATCH] Display extra headers for emacs-mua - db config option]

[Jameson Graef Rollins]

On Sun, Dec 08 2019, Jorge P. de Morais Neto <jorge+list@disroot.org> wrote:
> Em [2019-11-22 sex 10:43:40+0800], Daniel Kahn Gillmor escreveu:
>
>> Better than documenting, i'd be happy if we would add a "notmuch config
>> edit" subcommand, which handles the above sequence for you, invoking
>> $EDITOR at the appropriate time.
>>
>> The only caveat i see there is if the end user wants to inject comments
>> in the config file, which would then be stripped out in between these
>> invocations.  perhaps someone who finds these comments in config files
>> super important could propose a way to stash them in the db and recover
>> them during "notmuch config edit" as well :)
>
> I sync my two Notmuch configuration files -- personal notebook and
> workplace desktop -- over Unison via a USB flash drive.  This way, when
> I want to reconfigure Notmuch in one machine, I have the other machine's
> configuration as reference.  However, I do not sync the entire Notmuch
> databases because, since they are big and change frequently, I suppose
> they would wear the USB flash drive and also increase the
> synchronization time.
>
> Thus for my use case it would be useful if "notmuch config edit" could
> automatically keep a perennial text file reflecting the configuration in
> the database.  That text file would then be synchronized by Unison.

You can already use 'notmuch config list' to dump every configuration
item to stdout.  Would that be sufficient for personal synchronization
purposes.

jamie.

Re: moving the config into the database [was: Re: [PATCH] Display extra headers for emacs-mua - db config option]

[Jorge P. de Morais Neto]

Em [2019-12-08 dom 09:12:55-0800], Jameson Graef Rollins escreveu:

> You can already use 'notmuch config list' to dump every configuration
> item to stdout.  Would that be sufficient for personal synchronization
> purposes.

But then I would have to remember invoking
~notmuch config list > ~/notmuch-config~
every time I changed Notmuch configuration.  It would be more convenient
and less error-prone for Notmuch to keep a perennial text file
reflecting the configuration in the database.  Oh, I that file should be
updated not only by ~notmuch edit config~, but also by ~notmuch config
set~.

Regards
-- 
- <https://jorgemorais.gitlab.io/justice-for-rms/>
- I am Brazilian.  I hope my English is correct and I welcome feedback.
- Please adopt free formats like PDF, ODF, Org, LaTeX, Opus, WebM and 7z.
- Free/libre software for Replicant, LineageOS and Android: https://f-droid.org
- [[https://www.gnu.org/philosophy/free-sw.html][What is free software?]]

Re: moving the config into the database [was: Re: [PATCH] Display extra headers for emacs-mua - db config option]

[Daniel Kahn Gillmor]

[-- Attachment #1: Type: text/plain, Size: 1188 bytes --]

On Sun 2019-12-08 15:19:38 -0300, Jorge P. de Morais Neto wrote:
> Em [2019-12-08 dom 09:12:55-0800], Jameson Graef Rollins escreveu:
>
>> You can already use 'notmuch config list' to dump every configuration
>> item to stdout.  Would that be sufficient for personal synchronization
>> purposes.
>
> But then I would have to remember invoking
> ~notmuch config list > ~/notmuch-config~
> every time I changed Notmuch configuration.  It would be more convenient
> and less error-prone for Notmuch to keep a perennial text file
> reflecting the configuration in the database.  Oh, I that file should be
> updated not only by ~notmuch edit config~, but also by ~notmuch config
> set~.

One problem with such a "perennial text file" is that people will want
to edit it.  When they do, what should happen?

So the proposal to maintain such a file seems like we would be adding an
interface to notmuch that will not be used by many people, and will
eventually hurt (or at least confuse) users in surprising ways.

I'd rather keep notmuch itself simpler, and expect users who are syncing
to sync things like the output of "notmuch dump" (which already includes
all db-stored config).

    --dkg

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: moving the config into the database [was: Re: [PATCH] Display extra headers for emacs-mua - db config option]

[Jorge P. de Morais Neto]

Hello.

Em [2019-12-09 seg 13:31:22-0500], Daniel Kahn Gillmor escreveu:

> One problem with such a "perennial text file" is that people will want
> to edit it.  When they do, what should happen?

Well, the file could be named "notmuch-config-view" and the contents
could include comments explaining it is just a view.  Besides, the
generation of such a file could be opt-in.

I hope you don't mind my insistence (it is my personality) and, of
course, there is still the problem of slightly increasing Notmuch's
complexity to provide a small benefit for relatively few people.  It is
(obviously) your call, since you understand much more about the Notmuch
codebase (and free/libre software development in general) than me.
Besides, it is not me who am going to actually implement this (I am
unfortunately unable to contribute C code to Notmuch at this moment).

Regards
-- 
- <https://jorgemorais.gitlab.io/justice-for-rms/>
- I am Brazilian.  I hope my English is correct and I welcome feedback.
- Please adopt free formats like PDF, ODF, Org, LaTeX, Opus, WebM and 7z.
- Free/libre software for Replicant, LineageOS and Android: https://f-droid.org
- [[https://www.gnu.org/philosophy/free-sw.html][What is free software?]]

Re: moving the config into the database [was: Re: [PATCH] Display extra headers for emacs-mua - db config option]

[Daniel Kahn Gillmor]

[-- Attachment #1: Type: text/plain, Size: 436 bytes --]

On Tue 2019-12-10 09:46:14 -0300, Jorge P. de Morais Neto wrote:
> I hope you don't mind my insistence

Insistence is fine -- it sounds like you have a real need for your
backup/sync strategy, and if we can figure out a way to support it, that
would be good.  perhaps we should talk about different ways to get what
you're looking for, though.

Perhaps you would be satisfied with some sort of "post-config-update" hook?

        --dkg

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: moving the config into the database [was: Re: [PATCH] Display extra headers for emacs-mua - db config option]

[Jorge P. de Morais Neto]

Hello.

Em [2019-12-10 ter 12:01:06-0500], Daniel Kahn Gillmor escreveu:

> Perhaps you would be satisfied with some sort of "post-config-update" hook?

Yes, I think that would fulfill my needs.  And it would satisfy not only
people who use Unison, but also people who use some "realtime" (don't
know if that is the right word) synchronization tool like Netxcloud.

Regards
-- 
- <https://jorgemorais.gitlab.io/justice-for-rms/>
- I am Brazilian.  I hope my English is correct and I welcome feedback.
- Please adopt free formats like PDF, ODF, Org, LaTeX, Opus, WebM and 7z.
- Free/libre software for Replicant, LineageOS and Android: https://f-droid.org
- [[https://www.gnu.org/philosophy/free-sw.html][What is free software?]]

Re: moving the config into the database [was: Re: [PATCH] Display extra headers for emacs-mua - db config option]

[Jameson Graef Rollins]

On Sun, Dec 08 2019, Jorge P. de Morais Neto <jorge+list@disroot.org> wrote:
> Em [2019-12-08 dom 09:12:55-0800], Jameson Graef Rollins escreveu:
>
>> You can already use 'notmuch config list' to dump every configuration
>> item to stdout.  Would that be sufficient for personal synchronization
>> purposes.
>
> But then I would have to remember invoking
> ~notmuch config list > ~/notmuch-config~
> every time I changed Notmuch configuration.

Rather than remember, why not just have your synchronization script
retrieve the config at sync time using notmuch config/dump.  That way
you don't have to remember to do anything, and you can be agnostic about
how the configuration info is stored on disk.

Fwiw I support dkg's idea to store the config in the database itself.  I
think that's a clean simplification that makes a lot of sense.

jamie.

Re: moving the config into the database [was: Re: [PATCH] Display extra headers for emacs-mua - db config option]

[David Edmondson]

On Tuesday, 2019-12-10 at 08:11:56 -08, Jameson Graef Rollins wrote:

> Fwiw I support dkg's idea to store the config in the database itself.  I
> think that's a clean simplification that makes a lot of sense.

I'm not inherently against this, but I'm unsure exactly how it would
work.

For example, I set new.tags and new.ignore in my current
.notmuch-config. If they move into the database, how will I set them
before I have a database?

It seems that only “notmuch new” is currently prepared to create a new
database, but it will also import all of my email before I have a chance
to set the above parameters.

It seems that the obvious answer is “notmuch config” will create a
database if necessary. Is that the plan?

dme.
-- 
They like the smell of it in Hollywood.

Re: moving the config into the database [was: Re: [PATCH] Display extra headers for emacs-mua - db config option]

[David Bremner]

David Edmondson <dme@dme.org> writes:

> It seems that only “notmuch new” is currently prepared to create a new
> database, but it will also import all of my email before I have a chance
> to set the above parameters.
>
> It seems that the obvious answer is “notmuch config” will create a
> database if necessary. Is that the plan?

Wouldn't it make sense to have "notmuch setup" do this?

d

Re: moving the config into the database [was: Re: [PATCH] Display extra headers for emacs-mua - db config option]

[David Edmondson]

On Wednesday, 2019-12-11 at 10:00:07 -04, David Bremner wrote:

> David Edmondson <dme@dme.org> writes:
>
>> It seems that only “notmuch new” is currently prepared to create a new
>> database, but it will also import all of my email before I have a chance
>> to set the above parameters.
>>
>> It seems that the obvious answer is “notmuch config” will create a
>> database if necessary. Is that the plan?
>
> Wouldn't it make sense to have "notmuch setup" do this?

Oh, yes, of course. I had forgotten that existed :-)

dme.
-- 
Tell her I'll be waiting, in the usual place.

Re: moving the config into the database [was: Re: [PATCH] Display extra headers for emacs-mua - db config option]

[Daniel Kahn Gillmor]

[-- Attachment #1: Type: text/plain, Size: 639 bytes --]

On Wed 2019-12-11 10:00:07 -0400, David Bremner wrote:
> David Edmondson <dme@dme.org> writes:
>
>> It seems that only “notmuch new” is currently prepared to create a new
>> database, but it will also import all of my email before I have a chance
>> to set the above parameters.
>>
>> It seems that the obvious answer is “notmuch config” will create a
>> database if necessary. Is that the plan?
>
> Wouldn't it make sense to have "notmuch setup" do this?

It has always puzzled me that "notmuch setup" did not actually set up
the database in the first place.  So yes, i think this is the right
answer.

    --dkg

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]