unofficial mirror of notmuch@notmuchmail.org
 help / color / mirror / code / Atom feed
* [PATCH/RFC 0/3] Maildir custom flags and notmuch tags
@ 2015-11-26  2:16 Igor Almeida
  2015-11-26  2:16 ` [PATCH/RFC 1/3] David Bremner's patch for custom maildir flags Igor Almeida
                   ` (3 more replies)
  0 siblings, 4 replies; 12+ messages in thread
From: Igor Almeida @ 2015-11-26  2:16 UTC (permalink / raw)
  To: notmuch

Building on Bremner's patch at
<1411805835-3563-1-git-send-email-david@tethera.net>, this updates
'notmuch new' to tag messages based on lower-case letters in the file
name.
I'm testing this with an IMAP server, offlineimap and notmuch,
and seems to work well enough for my unidirectional use case.
Eventually I'd like to implement the reverse path, i.e. tagging in
notmuch and seeing IMAP keywords show up in the server, but that's for
later.

Comments are very welcome.

Igor Almeida (3):
  David Bremner's patch for custom maildir flags
  notmuch new: tag messages based on maildir custom flags
  notmuch new: sync maildir custom flag user configuration

 lib/database.cc  | 61 +++++++++++++++++++++++++++++++++++++++
 lib/message.cc   | 88 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 lib/notmuch.h    | 18 ++++++++++++
 notmuch-client.h |  4 +++
 notmuch-config.c | 24 +++++++++++++++-
 notmuch-new.c    | 49 +++++++++++++++++++++++++++++++
 6 files changed, 243 insertions(+), 1 deletion(-)

-- 
2.5.3

^ permalink raw reply	[flat|nested] 12+ messages in thread

* [PATCH/RFC 1/3] David Bremner's patch for custom maildir flags
  2015-11-26  2:16 [PATCH/RFC 0/3] Maildir custom flags and notmuch tags Igor Almeida
@ 2015-11-26  2:16 ` Igor Almeida
  2015-11-26  2:16 ` [PATCH/RFC 2/3] notmuch new: tag messages based on maildir custom flags Igor Almeida
                   ` (2 subsequent siblings)
  3 siblings, 0 replies; 12+ messages in thread
From: Igor Almeida @ 2015-11-26  2:16 UTC (permalink / raw)
  To: notmuch

---
 lib/database.cc | 61 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 lib/notmuch.h   | 18 +++++++++++++++++
 2 files changed, 79 insertions(+)

diff --git a/lib/database.cc b/lib/database.cc
index 3b342f1..592cbcc 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -1329,6 +1329,67 @@ notmuch_database_get_version (notmuch_database_t *notmuch)
     return version;
 }
 
+notmuch_status_t
+notmuch_database_get_maildir_keyword (notmuch_database_t *notmuch,
+				      int index, const char **tag)
+{
+    string tag_string;
+    const char *key;
+    const char *str;
+
+    if (!notmuch || !tag)
+	return NOTMUCH_STATUS_NULL_POINTER;
+
+    if (index < 0 || index > ('z' - 'a'))
+	return NOTMUCH_STATUS_UNSUPPORTED_OPERATION;
+
+    key = talloc_asprintf(notmuch, "maildir_keyword_%c", 'a' + index);
+    if (!key)
+	return NOTMUCH_STATUS_OUT_OF_MEMORY;
+
+    *tag = NULL;
+    tag_string = notmuch->xapian_db->get_metadata (key);
+    if (tag_string.empty ())
+	return NOTMUCH_STATUS_SUCCESS;
+
+    str = tag_string.c_str ();
+    if (str == NULL || *str == '\0')
+	return NOTMUCH_STATUS_SUCCESS;
+
+    *tag = str;
+
+    return NOTMUCH_STATUS_SUCCESS;
+}
+
+notmuch_status_t
+notmuch_database_set_maildir_keyword (notmuch_database_t *notmuch,
+				      int index, const char *tag)
+{
+    string tag_string;
+    const char *key;
+    notmuch_status_t ret;
+    Xapian::WritableDatabase *db;
+
+    if (!notmuch || !tag)
+	return NOTMUCH_STATUS_NULL_POINTER;
+
+    ret = _notmuch_database_ensure_writable (notmuch);
+    if (ret)
+	return ret;
+
+    if (index < 0 || index > ('z' - 'a'))
+	return NOTMUCH_STATUS_UNSUPPORTED_OPERATION;
+
+    key = talloc_asprintf(notmuch, "maildir_keyword_%c", 'a' + index);
+    if (!key)
+	return NOTMUCH_STATUS_OUT_OF_MEMORY;
+
+    db = static_cast <Xapian::WritableDatabase *> (notmuch->xapian_db);
+    db->set_metadata (key, tag);
+
+    return NOTMUCH_STATUS_SUCCESS;
+}
+
 notmuch_bool_t
 notmuch_database_needs_upgrade (notmuch_database_t *notmuch)
 {
diff --git a/lib/notmuch.h b/lib/notmuch.h
index 310a8b8..779a5ea 100644
--- a/lib/notmuch.h
+++ b/lib/notmuch.h
@@ -674,6 +674,24 @@ notmuch_tags_t *
 notmuch_database_get_all_tags (notmuch_database_t *db);
 
 /**
+ * Return the tag corresponding to a maildir keyword.
+ *
+ */
+notmuch_status_t
+notmuch_database_get_maildir_keyword(notmuch_database_t *db,
+				     int index, const char **tag);
+
+/**
+ * Set the tag corresponding to a maildir keyword.
+ *
+ * Note that no messages have their tags modified by this call.
+ */
+
+notmuch_status_t
+notmuch_database_set_maildir_keyword(notmuch_database_t *db,
+				     int index, const char *tag);
+
+/**
  * Create a new query for 'database'.
  *
  * Here, 'database' should be an open database, (see
-- 
2.5.3

^ permalink raw reply related	[flat|nested] 12+ messages in thread

* [PATCH/RFC 2/3] notmuch new: tag messages based on maildir custom flags
  2015-11-26  2:16 [PATCH/RFC 0/3] Maildir custom flags and notmuch tags Igor Almeida
  2015-11-26  2:16 ` [PATCH/RFC 1/3] David Bremner's patch for custom maildir flags Igor Almeida
@ 2015-11-26  2:16 ` Igor Almeida
  2015-11-26  2:16 ` [PATCH/RFC 3/3] notmuch new: sync maildir custom flag user configuration Igor Almeida
  2016-01-10 13:59 ` [PATCH/RFC 0/3] Maildir custom flags and notmuch tags David Bremner
  3 siblings, 0 replies; 12+ messages in thread
From: Igor Almeida @ 2015-11-26  2:16 UTC (permalink / raw)
  To: notmuch

Iterate through the flags in the message's filename, creating tags and
tagging the message as they appear.
This uses Bremner/Dovecot's convention of maildir_keyword_xxx, where xxx a char
between 'a' and 'z'.

Since more than one file may point to the same message in the database
(think renames), the tags related to maildir custom flags are always
removed and later re-added to maintain consistency.

Signed-off-by: Igor Almeida <igor.contato@gmail.com>
---
 lib/message.cc | 88 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 88 insertions(+)

diff --git a/lib/message.cc b/lib/message.cc
index 26b5e76..8a89dee 100644
--- a/lib/message.cc
+++ b/lib/message.cc
@@ -1301,6 +1301,13 @@ notmuch_message_maildir_flags_to_tags (notmuch_message_t *message)
     unsigned i;
     int seen_maildir_info = 0;
 
+    /* For custom maildir flags */
+    char *c; /* to iterate over combined_flags */
+    const char *custom_flag_dbase_name;
+    int index;
+    notmuch_bool_t cancel_custom_flags = FALSE;
+    notmuch_bool_t must_free = FALSE;
+
     for (filenames = notmuch_message_get_filenames (message);
 	 notmuch_filenames_valid (filenames);
 	 notmuch_filenames_move_to_next (filenames))
@@ -1349,6 +1356,87 @@ notmuch_message_maildir_flags_to_tags (notmuch_message_t *message)
 	if (status)
 	    return status;
     }
+
+    /* Now let's find custom maildir flags */
+
+    status = notmuch_message_freeze (message);
+    if (status)
+	goto finish;
+
+    cancel_custom_flags = FALSE;
+
+    /* First we remove the tags for custom maildir flags, then
+     * we add only the ones in combined_flags
+     */
+    char letter;
+    for (letter = 'a'; letter <= 'z'; letter++) {
+	index = letter - 'a';
+	status = notmuch_database_get_maildir_keyword (
+	    message->notmuch, index, &custom_flag_dbase_name);
+	if (status) {
+	    /* TODO probably OOM, what now? */
+	}
+
+	if (custom_flag_dbase_name == NULL) {
+	    /* We don't have a custom flag for this letter, try the next
+	     * one now
+	     */
+	    continue;
+	}
+
+	status = notmuch_message_remove_tag (message, custom_flag_dbase_name);
+	if (status) {
+	    /* TODO tag too long? */
+	}
+    }
+
+    /* Go through combined_flags and tag the message accordingly */
+
+    for (c = combined_flags; *c; c++) {
+	if (*c >= 'a' && *c <= 'z') {
+	    index = *c - 'a';
+
+	    status = notmuch_database_get_maildir_keyword(
+		message->notmuch, index, &custom_flag_dbase_name);
+	    if (status) {
+		cancel_custom_flags = TRUE;
+		break;
+	    } else {
+		if (custom_flag_dbase_name == NULL) {
+		    /* Custom flag does not yet exist */
+		    custom_flag_dbase_name = talloc_asprintf (message,
+			"maildir_keyword_%c", 'a' + index);
+		    /* Add to the database */
+		    notmuch_database_set_maildir_keyword (
+			message->notmuch, index, custom_flag_dbase_name);
+
+		    must_free = TRUE;
+		}
+
+		/* Tag the message */
+		status = notmuch_message_add_tag (message,
+		    custom_flag_dbase_name);
+
+		if (must_free) {
+		    talloc_free((void*)custom_flag_dbase_name);
+		    must_free = FALSE;
+		}
+
+		if (status) {
+		    cancel_custom_flags = TRUE;
+		    break;
+		}
+	    }
+	}
+    }
+
+    if (cancel_custom_flags) {
+	/* TODO rollback the add_tag's */
+    }
+
+    status = notmuch_message_thaw (message);
+
+finish:
     status = notmuch_message_thaw (message);
 
     talloc_free (combined_flags);
-- 
2.5.3

^ permalink raw reply related	[flat|nested] 12+ messages in thread

* [PATCH/RFC 3/3] notmuch new: sync maildir custom flag user configuration
  2015-11-26  2:16 [PATCH/RFC 0/3] Maildir custom flags and notmuch tags Igor Almeida
  2015-11-26  2:16 ` [PATCH/RFC 1/3] David Bremner's patch for custom maildir flags Igor Almeida
  2015-11-26  2:16 ` [PATCH/RFC 2/3] notmuch new: tag messages based on maildir custom flags Igor Almeida
@ 2015-11-26  2:16 ` Igor Almeida
  2016-01-10 13:59 ` [PATCH/RFC 0/3] Maildir custom flags and notmuch tags David Bremner
  3 siblings, 0 replies; 12+ messages in thread
From: Igor Almeida @ 2015-11-26  2:16 UTC (permalink / raw)
  To: notmuch

This allows the user to specify in the config file how maildir custom
flags translate to notmuch tags (this mapping is stored in the db metadata).

Command 'notmuch new' will pick up changes in the config and update the
database metadata, but will not retag the messages.

Example config:
	[maildir]
	synchronize_flags=true
	customflag_j = todo
	customflag_e = important

Signed-off-by: Igor Almeida <igor.contato@gmail.com>
---
 notmuch-client.h |  4 ++++
 notmuch-config.c | 24 +++++++++++++++++++++++-
 notmuch-new.c    | 49 +++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 76 insertions(+), 1 deletion(-)

diff --git a/notmuch-client.h b/notmuch-client.h
index 3bd2903..9f5bb8f 100644
--- a/notmuch-client.h
+++ b/notmuch-client.h
@@ -324,6 +324,10 @@ void
 notmuch_config_set_maildir_synchronize_flags (notmuch_config_t *config,
 					      notmuch_bool_t synchronize_flags);
 
+const char *
+notmuch_config_get_maildir_customflag_map (notmuch_config_t *config,
+					   const char maildir_custom_flag);
+
 const char **
 notmuch_config_get_search_exclude_tags (notmuch_config_t *config, size_t *length);
 
diff --git a/notmuch-config.c b/notmuch-config.c
index d252bb2..f2a25b1 100644
--- a/notmuch-config.c
+++ b/notmuch-config.c
@@ -72,7 +72,7 @@ static const char user_config_comment[] =
 static const char maildir_config_comment[] =
     " Maildir compatibility configuration\n"
     "\n"
-    " The following option is supported here:\n"
+    " The following options are supported here:\n"
     "\n"
     "\tsynchronize_flags      Valid values are true and false.\n"
     "\n"
@@ -87,6 +87,11 @@ static const char maildir_config_comment[] =
     "\t\tR	replied\n"
     "\t\tS	unread (added when 'S' flag is not present)\n"
     "\n"
+    "\tIf true, then the following option is also considered, and messages\n"
+    "\twill be tagged based on the lower-case letters in the maildir flags:\n"
+    "\n"
+    "\tcustomflag_[a-z]       Valid values are tag names.\n"
+    "\n"
     "\tThe \"notmuch new\" command will notice flag changes in filenames\n"
     "\tand update tags, while the \"notmuch tag\" and \"notmuch restore\"\n"
     "\tcommands will notice tag changes and update flags in filenames\n";
@@ -127,6 +132,8 @@ struct _notmuch_config {
     notmuch_bool_t maildir_synchronize_flags;
     const char **search_exclude_tags;
     size_t search_exclude_tags_length;
+
+    char *customflag_map[26 /* 'a' to 'z' */];
 };
 
 static int
@@ -931,3 +938,18 @@ notmuch_config_set_maildir_synchronize_flags (notmuch_config_t *config,
 			    "maildir", "synchronize_flags", synchronize_flags);
     config->maildir_synchronize_flags = synchronize_flags;
 }
+
+const char *
+notmuch_config_get_maildir_customflag_map (notmuch_config_t *config,
+					   const char maildir_customflag)
+{
+    char key[] = "customflag_a";
+    key[sizeof(key) - 2] = maildir_customflag;
+
+    int index = maildir_customflag - 'a';
+
+    return _config_get (
+	config,
+	&config->customflag_map[index],
+	"maildir", key);
+}
diff --git a/notmuch-new.c b/notmuch-new.c
index d45d0af..d4bb6c7 100644
--- a/notmuch-new.c
+++ b/notmuch-new.c
@@ -923,6 +923,50 @@ print_results (const add_files_state_t *state)
     printf ("\n");
 }
 
+static void
+_sync_config_and_database_customflag_mapping (notmuch_config_t *config,
+					      notmuch_database_t *notmuch)
+{
+    char c;
+    const char *config_customflag;
+    const char *db_customflag;
+
+    notmuch_status_t status;
+    int index;
+
+    /* Compare the config mapping and the db mapping, warn the user and
+     * overwrite the db if they differ
+     */
+    for (c = 'a'; c <= 'z'; c++) {
+	index = c - 'a';
+	config_customflag = notmuch_config_get_maildir_customflag_map (config, c);
+
+	if (config_customflag != NULL) {
+	    status = notmuch_database_get_maildir_keyword (
+		notmuch, index, &db_customflag);
+	    if (status) {
+		/* TODO what now? break and let 'notmuch new' fail later? */
+	    } else {
+		if (db_customflag != NULL &&
+			strcmp(config_customflag, db_customflag)) {
+		    printf ("Your configuration for maildir custom flags has "
+			    "changed.\n"
+			    "I will overwrite the mapping in the db metadata, "
+			    "but you will have to retag the messages "
+			    "yourself, with something like this:\n"
+			    "\tnotmuch tag +%s tag:%s\n"
+			    "\tnotmuch tag -%s tag:%s\n",
+				config_customflag, db_customflag,
+				db_customflag, config_customflag);
+		}
+
+		notmuch_database_set_maildir_keyword (
+		    notmuch, index, config_customflag);
+	    }
+	}
+    }
+}
+
 int
 notmuch_new_command (notmuch_config_t *config, int argc, char *argv[])
 {
@@ -1068,6 +1112,11 @@ notmuch_new_command (notmuch_config_t *config, int argc, char *argv[])
     if (notmuch == NULL)
 	return EXIT_FAILURE;
 
+    /* Make sure our database's maildir customflag mapping matches the config,
+     * warn the user if not
+     */
+    _sync_config_and_database_customflag_mapping (config, notmuch);
+
     /* Set up our handler for SIGINT. We do this after having
      * potentially done a database upgrade we this interrupt handler
      * won't support. */
-- 
2.5.3

^ permalink raw reply related	[flat|nested] 12+ messages in thread

* Re: [PATCH/RFC 0/3] Maildir custom flags and notmuch tags
  2015-11-26  2:16 [PATCH/RFC 0/3] Maildir custom flags and notmuch tags Igor Almeida
                   ` (2 preceding siblings ...)
  2015-11-26  2:16 ` [PATCH/RFC 3/3] notmuch new: sync maildir custom flag user configuration Igor Almeida
@ 2016-01-10 13:59 ` David Bremner
  2016-02-02 16:52   ` Jan Pobrislo
  3 siblings, 1 reply; 12+ messages in thread
From: David Bremner @ 2016-01-10 13:59 UTC (permalink / raw)
  To: Igor Almeida, notmuch

Igor Almeida <igor.contato@gmail.com> writes:

> Building on Bremner's patch at
> <1411805835-3563-1-git-send-email-david@tethera.net>, this updates
> 'notmuch new' to tag messages based on lower-case letters in the file
> name.
> I'm testing this with an IMAP server, offlineimap and notmuch,
> and seems to work well enough for my unidirectional use case.
> Eventually I'd like to implement the reverse path, i.e. tagging in
> notmuch and seeing IMAP keywords show up in the server, but that's for
> later.
>
> Comments are very welcome.
>

Sorry about the long silence. I _have_ been thinking about and working
on this. This message summarizes my current thinking/plan.

I'm not very enthusiastic about the about the several places this can
get out of sync

1) config info in .notmuch-config can be out of sync with the "real
config" in the database.  This part is only until the next run of
notmuch new, so it's not so bad.

2) When the interpretation of maildir flags changes, there doesn't seem
to be a sensible way of retagging messages.

Some others have objected to supporting a nonstandard extension used (I
think) only by dovecot. For me this is not so bad, as long as the code
required specifically for this is not too large / intrusive.

So here are the pieces I think we need. 

a) some code in notmuch-new to sync flags a-z to a fixed set of tags
  [1]. This soundspretty much the functionality you have now, although I
  haven't really reviewed that part of the series.

b) some code to update metadata directly from the CLI (notmuch
   config/notmuch tag), probably using something like [2] as a back
   end. This metadata maps the fixed tags like $maildir_flag_k to and
   from some friendly format.

c) The use of xapian field processors to translate e.g. a query
   "tag:my-sensible-thing" into "tag:$maildir_flag_k". Optionally do the
   reverse when returning lists of tags. FieldProcessors are only
   available in Xapian 1.3.x, but apparently 1.3 series is nearing a
   stable release, so it's a sane time to optionally depend on it.
   I guess as a short term hack, this could be done by shell wrappers
   (not something we'd ship, but it might make it usable for you).          

d) code to dump and restore the metadata as part of the normal dump
   files. This is at least in progress [2].

In terms of how these changes relate to other goals

b,d) have strong connects with allowing database level config. Among
     other reasons this will help make bindings users less second class
     citizens
      

  c) has overlap with allowing date fields that are not ranges (among
     other things). At least all the machinery to configure xapian 1.2
     versus 1.3, and the understanding how to use field processors, is
     reusable.

I'd understand perfectly if you prefer to rebase your simpler solution
on top of master, but if you want to keep working on this, then I guess
(a), (b), and (c) are all things you would work on.

Probably (b) needs the library api from [2], which now that I start
typing this epic long message, I could split off from the dump/restore
stuff. In particular managing the config metadata only needs about half
of the LOC; the iterator is only needed for dump, I think.

David

[1]: eventually it might make sense to split the tags that are somehow
automatic from those that represent user choice. We already have this
issue with tag:encrypted and friends.

[2]: id:1452394301-29499-1-git-send-email-david@tethera.net

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [PATCH/RFC 0/3] Maildir custom flags and notmuch tags
  2016-01-10 13:59 ` [PATCH/RFC 0/3] Maildir custom flags and notmuch tags David Bremner
@ 2016-02-02 16:52   ` Jan Pobrislo
  2016-02-03 12:03     ` David Bremner
  0 siblings, 1 reply; 12+ messages in thread
From: Jan Pobrislo @ 2016-02-02 16:52 UTC (permalink / raw)
  To: notmuch

Hello. I've been thinking about this feature for quite some while.
Having tags stored and synchronized together with mail would certainly
make life easier for a lot of notmuch user. And the upside is that you
can already do that with Dovecot's dsync tool.

But from what I've seen in the patches (as far as I understood them) it
doesn't really work anything like keyword storage in Dovecot - aside
from using maildir flags a-z.

I've been missing comprehensive source of information on various things
maildir-related, so I've written this: https://notmuchmail.org/software/
It includes links to specifications of the two keyword storage formats
for maildir: Dovecot's and Courier's. There's a lot more that can be
added on that page, all the MUAs for starters.

I think that using fixed mapping for flag meaning is a good POC step,
but that won't work without explicit support from the synchronizer to
map specific keywords to always same tags. I don't really know what
offlineimap does to synchronize keywords, if anything. Dsync already
does what it does - which is obviously to use the full dovecot format.

Having the mapping in the maildir rather than database can work well
because you have one unambiguous format that both synchronizer and
notmuch can use without much hassle. And it will scale up to 26
different tags per maildir, after that it will unfortunately stop
working without resorting to dovecot-specific formats, but I think it
still covers 98% of use-cases or so.

On the other hand, you can hack up a script that renames mail
post-dsync to conform to expected static mapping. The change detection
with just one auxiliary file is simple too, as opposed to the
subdirectory format Courier uses.

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [PATCH/RFC 0/3] Maildir custom flags and notmuch tags
  2016-02-02 16:52   ` Jan Pobrislo
@ 2016-02-03 12:03     ` David Bremner
  2016-02-03 14:32       ` Jan Pobrislo
  0 siblings, 1 reply; 12+ messages in thread
From: David Bremner @ 2016-02-03 12:03 UTC (permalink / raw)
  To: Jan Pobrislo, notmuch

Jan Pobrislo <ccx@webprojekty.cz> writes:
>
> I think that using fixed mapping for flag meaning is a good POC step,
> but that won't work without explicit support from the synchronizer to
> map specific keywords to always same tags. I don't really know what
> offlineimap does to synchronize keywords, if anything. Dsync already
> does what it does - which is obviously to use the full dovecot format.
>

I see, you're talking about this "dovecot-keywords" file I guess

  http://wiki2.dovecot.org/MailboxFormat/Maildir

Some questions that spring to mind:

- This is clearly dovecot specific; I wonder what fraction of
  our users would benefit. I suppose that's a question about any scheme
  involving maildir-flags a-z; at least those can be synchronized ootb
  by several tools.

- Notmuch new currently only indexes one copy of a message, so two files
  in different maildirs (i.e. a list and inbox) would be pretty much a
  crapshoot which tags get applied. We intend to change this behaviour
  eventually, but no one is working on it currently.

- even if/when this behaviour changes, there is still the problem of
  reconciling different tag mappings from several maildirs.

On the other hand, maybe not much change to the notmuch core would be
needed to at least experiment with this, using e.g. hooks to
notmuch-insert and notmuch-new.

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [PATCH/RFC 0/3] Maildir custom flags and notmuch tags
  2016-02-03 12:03     ` David Bremner
@ 2016-02-03 14:32       ` Jan Pobrislo
  2016-02-03 15:38         ` David Bremner
  0 siblings, 1 reply; 12+ messages in thread
From: Jan Pobrislo @ 2016-02-03 14:32 UTC (permalink / raw)
  To: David Bremner, notmuch

On Wed, 03 Feb 2016 08:03:08 -0400
David Bremner <david@tethera.net> wrote:

> I see, you're talking about this "dovecot-keywords" file I guess
> 
>   http://wiki2.dovecot.org/MailboxFormat/Maildir

Indeed.

> Some questions that spring to mind:
> 
> - This is clearly dovecot specific; I wonder what fraction of
>   our users would benefit. I suppose that's a question about any
> scheme involving maildir-flags a-z; at least those can be
> synchronized ootb by several tools.

Every such maildir extension out there that I know of was invented for
some specific application. Out of the two documented formats there are,
the dovecot-keywords file is:

1) more limited (26 tags maximum)
2) simpler to implement, especially wrt. detecting changes
3) usable out of the box with some tools, as you noted

Of course, one could go invent some another format, but in the end it'd
be application-specific too, and I don't see it succeeding without help
of mail-synchronizer authors.

> - Notmuch new currently only indexes one copy of a message, so two
> files in different maildirs (i.e. a list and inbox) would be pretty
> much a crapshoot which tags get applied. We intend to change this
> behaviour eventually, but no one is working on it currently.
> 
> - even if/when this behaviour changes, there is still the problem of
>   reconciling different tag mappings from several maildirs.

This is nothing new though, current synchronize_tags has the very same
problem. I'm not sure how much of a problem it is in practice, but it
probably should be addressed in some way.

Here possibly the best course of action would be to leave it open-ended
and provide user-definable conflict resolution hook, as I can't really
think of "one size fits all" solution.

> On the other hand, maybe not much change to the notmuch core would be
> needed to at least experiment with this, using e.g. hooks to
> notmuch-insert and notmuch-new.

I was pondering this before and would find it rather neat, but it is
bit more complicated than it might first seem. The hook needs to be
able to add arbitrary files to be watched for changes and deduce from
that which files need their tags re-read.

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [PATCH/RFC 0/3] Maildir custom flags and notmuch tags
  2016-02-03 14:32       ` Jan Pobrislo
@ 2016-02-03 15:38         ` David Bremner
  2016-02-03 15:39           ` David Bremner
                             ` (2 more replies)
  0 siblings, 3 replies; 12+ messages in thread
From: David Bremner @ 2016-02-03 15:38 UTC (permalink / raw)
  To: Jan Pobrislo, notmuch

Jan Pobrislo <ccx@webprojekty.cz> writes:

>
> Every such maildir extension out there that I know of was invented for
> some specific application. Out of the two documented formats there are,
> the dovecot-keywords file is:
>
> 1) more limited (26 tags maximum)
> 2) simpler to implement, especially wrt. detecting changes
> 3) usable out of the box with some tools, as you noted

Actually, I just mean the extra maildir flags. I guess only the dovecot
sync tool syncs dovecot-keywords. And I don't think "notmuch+dsync
users" is a very big group, fwiw.

>> On the other hand, maybe not much change to the notmuch core would be
>> needed to at least experiment with this, using e.g. hooks to
>> notmuch-insert and notmuch-new.
>
> I was pondering this before and would find it rather neat, but it is
> bit more complicated than it might first seem. The hook needs to be
> able to add arbitrary files to be watched for changes and deduce from
> that which files need their tags re-read.

perhaps lastmod: queries can help here. that's more or less what they
added for.

d

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [PATCH/RFC 0/3] Maildir custom flags and notmuch tags
  2016-02-03 15:38         ` David Bremner
@ 2016-02-03 15:39           ` David Bremner
  2016-02-03 16:00           ` Jan Pobrislo
       [not found]           ` <20160203165650.4ae528c1@dorje.v103.te2000>
  2 siblings, 0 replies; 12+ messages in thread
From: David Bremner @ 2016-02-03 15:39 UTC (permalink / raw)
  To: Jan Pobrislo, notmuch

David Bremner <david@tethera.net> writes:

> Jan Pobrislo <ccx@webprojekty.cz> writes:
>
>>
>> Every such maildir extension out there that I know of was invented for
>> some specific application. Out of the two documented formats there are,
>> the dovecot-keywords file is:
>>
>> 1) more limited (26 tags maximum)
>> 2) simpler to implement, especially wrt. detecting changes
>> 3) usable out of the box with some tools, as you noted
>
> Actually, I just mean the extra maildir flags. I guess only the dovecot
> sync tool syncs dovecot-keywords. And I don't think "notmuch+dsync
> users" is a very big group, fwiw.

Straight rsync would also transfer those files, of course.

d

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [PATCH/RFC 0/3] Maildir custom flags and notmuch tags
  2016-02-03 15:38         ` David Bremner
  2016-02-03 15:39           ` David Bremner
@ 2016-02-03 16:00           ` Jan Pobrislo
       [not found]           ` <20160203165650.4ae528c1@dorje.v103.te2000>
  2 siblings, 0 replies; 12+ messages in thread
From: Jan Pobrislo @ 2016-02-03 16:00 UTC (permalink / raw)
  To: notmuch

On Wed, 03 Feb 2016 11:38:30 -0400
David Bremner <david@tethera.net> wrote:
> > I was pondering this before and would find it rather neat, but it is
> > bit more complicated than it might first seem. The hook needs to be
> > able to add arbitrary files to be watched for changes and deduce
> > from that which files need their tags re-read.  
> 
> perhaps lastmod: queries can help here. that's more or less what they
> added for.
> 
> d
> 

That only helps with applying tags from notmuch to maildir, not
vice-versa, which is the main thing I was talking about.

But it got me thinking about potential notmuch<->dovecot-keywords
synchronizer running as a separate pass. Although it would have to keep
it's own database of mtimes, tags, etc. is still pretty feasible to
make without changes to notmuch itself.

---
David: Sorry about the duplicate mail. Having to use stupid MUA ATM.

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [PATCH/RFC 0/3] Maildir custom flags and notmuch tags
       [not found]           ` <20160203165650.4ae528c1@dorje.v103.te2000>
@ 2016-02-03 16:04             ` David Bremner
  0 siblings, 0 replies; 12+ messages in thread
From: David Bremner @ 2016-02-03 16:04 UTC (permalink / raw)
  To: Jan Pobrislo; +Cc: notmuch

Jan Pobrislo <ccx@webprojekty.cz> writes:

> On Wed, 03 Feb 2016 11:38:30 -0400
> David Bremner <david@tethera.net> wrote:
>> > I was pondering this before and would find it rather neat, but it is
>> > bit more complicated than it might first seem. The hook needs to be
>> > able to add arbitrary files to be watched for changes and deduce
>> > from that which files need their tags re-read.  
>> 
>> perhaps lastmod: queries can help here. that's more or less what they
>> added for.
>> 
>> d
>> 
>
> That only helps with applying tags from notmuch to maildir, not
> vice-versa, which is the main thing I was talking about.

For changes to dovecot-keywords files, you're on your own. But changes
to the actual maildir flags are file renames, which should trigger an
increase in lastmod counter; if not I'd consider that a bug.

d

^ permalink raw reply	[flat|nested] 12+ messages in thread

end of thread, other threads:[~2016-02-03 16:04 UTC | newest]

Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2015-11-26  2:16 [PATCH/RFC 0/3] Maildir custom flags and notmuch tags Igor Almeida
2015-11-26  2:16 ` [PATCH/RFC 1/3] David Bremner's patch for custom maildir flags Igor Almeida
2015-11-26  2:16 ` [PATCH/RFC 2/3] notmuch new: tag messages based on maildir custom flags Igor Almeida
2015-11-26  2:16 ` [PATCH/RFC 3/3] notmuch new: sync maildir custom flag user configuration Igor Almeida
2016-01-10 13:59 ` [PATCH/RFC 0/3] Maildir custom flags and notmuch tags David Bremner
2016-02-02 16:52   ` Jan Pobrislo
2016-02-03 12:03     ` David Bremner
2016-02-03 14:32       ` Jan Pobrislo
2016-02-03 15:38         ` David Bremner
2016-02-03 15:39           ` David Bremner
2016-02-03 16:00           ` Jan Pobrislo
     [not found]           ` <20160203165650.4ae528c1@dorje.v103.te2000>
2016-02-03 16:04             ` David Bremner

Code repositories for project(s) associated with this public inbox

	https://yhetil.org/notmuch.git/

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).