unofficial mirror of notmuch@notmuchmail.org
 help / color / mirror / code / Atom feed
* [PATCH] introduce new.rename_tags for renamed (moved) messages
@ 2018-07-06 14:04 Michael J Gruber
  2018-09-08  0:47 ` David Bremner
  2018-09-21  6:48 ` [PATCH] " Gaute Hope
  0 siblings, 2 replies; 10+ messages in thread
From: Michael J Gruber @ 2018-07-06 14:04 UTC (permalink / raw)
  To: notmuch

IMAP clients (such as webmail) use folders to mark messages as junk
etc., some even to mark messages as trash ("move to trash"). Such a
change is reported by notmuch as a rename; the message is not tagged
with new.tags since it is not new, so that there is no way to act upon a
rename.

Introduce new.rename_tags (default: not set) which are added by `notmuch
new` to renamed messages. This allows to act upon renames, e.g. to keep
the IMAP folder structure in sync with tags with a tool like `afew` or
homecooked scripts simply by filtering for this tag in the same ways as
one would filter for new messages using new.tags.

Signed-off-by: Michael J Gruber <git@grubix.eu>
---
 NEWS                        | 11 +++++++++++
 doc/man1/notmuch-config.rst |  6 ++++++
 notmuch-client.h            |  8 ++++++++
 notmuch-config.c            | 26 ++++++++++++++++++++++++++
 notmuch-new.c               | 23 +++++++++++++++++++++++
 5 files changed, 74 insertions(+)

diff --git a/NEWS b/NEWS
index 240d594b..e3b75e74 100644
--- a/NEWS
+++ b/NEWS
@@ -1,3 +1,14 @@
+Notmuch 0.28 (UNRELEASED)
+=========================
+
+New command-line features
+-------------------------
+
+User-configurable tags for renamed messages
+
+  A new "new.rename_tags" option is available in the configuration file to
+  determine which tags are applied to renamed (moved) messages.
+
 Notmuch 0.27 (2018-06-13)
 =========================
 
diff --git a/doc/man1/notmuch-config.rst b/doc/man1/notmuch-config.rst
index 89909808..9e4198a1 100644
--- a/doc/man1/notmuch-config.rst
+++ b/doc/man1/notmuch-config.rst
@@ -77,6 +77,12 @@ The available configuration items are described below.
 
     Default: ``unread;inbox``.
 
+**new.rename_tags**
+    A list of tags that will be added to all messages which
+    **notmuch new** identifies as renamed (moved).
+
+    Default: not set.
+
 **new.ignore**
     A list to specify files and directories that will not be searched
     for messages by **notmuch new**. Each entry in the list is either:
diff --git a/notmuch-client.h b/notmuch-client.h
index 6c84ecc0..5e1e6b66 100644
--- a/notmuch-client.h
+++ b/notmuch-client.h
@@ -316,6 +316,14 @@ notmuch_config_set_new_tags (notmuch_config_t *config,
 			     const char *new_tags[],
 			     size_t length);
 
+const char **
+notmuch_config_get_rename_tags (notmuch_config_t *config,
+			     size_t *length);
+void
+notmuch_config_set_rename_tags (notmuch_config_t *config,
+			     const char *rename_tags[],
+			     size_t length);
+
 const char **
 notmuch_config_get_new_ignore (notmuch_config_t *config,
 			       size_t *length);
diff --git a/notmuch-config.c b/notmuch-config.c
index e1b16609..02f7d247 100644
--- a/notmuch-config.c
+++ b/notmuch-config.c
@@ -132,6 +132,8 @@ struct _notmuch_config {
     size_t user_other_email_length;
     const char **new_tags;
     size_t new_tags_length;
+    const char **rename_tags;
+    size_t rename_tags_length;
     const char **new_ignore;
     size_t new_ignore_length;
     bool maildir_synchronize_flags;
@@ -712,6 +714,14 @@ notmuch_config_get_new_tags (notmuch_config_t *config,   size_t *length)
 			     &(config->new_tags_length), length);
 }
 
+const char **
+notmuch_config_get_rename_tags (notmuch_config_t *config,   size_t *length)
+{
+    return _config_get_list (config, "new", "rename_tags",
+			     &(config->rename_tags),
+			     &(config->rename_tags_length), length);
+}
+
 const char **
 notmuch_config_get_new_ignore (notmuch_config_t *config, size_t *length)
 {
@@ -738,6 +748,15 @@ notmuch_config_set_new_tags (notmuch_config_t *config,
 		     &(config->new_tags));
 }
 
+void
+notmuch_config_set_rename_tags (notmuch_config_t *config,
+				     const char *list[],
+				     size_t length)
+{
+    _config_set_list (config, "new", "rename_tags", list, length,
+		     &(config->rename_tags));
+}
+
 void
 notmuch_config_set_new_ignore (notmuch_config_t *config,
 			       const char *list[],
@@ -867,6 +886,13 @@ notmuch_config_command_get (notmuch_config_t *config, char *item)
 	tags = notmuch_config_get_new_tags (config, &length);
 	for (i = 0; i < length; i++)
 	    printf ("%s\n", tags[i]);
+    } else if (strcmp(item, "new.rename_tags") == 0) {
+	const char **tags;
+	size_t i, length;
+
+	tags = notmuch_config_get_rename_tags (config, &length);
+	for (i = 0; i < length; i++)
+	    printf ("%s\n", tags[i]);
     } else if (STRNCMP_LITERAL (item, BUILT_WITH_PREFIX) == 0) {
 	printf ("%s\n",
 		notmuch_built_with (item + strlen (BUILT_WITH_PREFIX)) ? "true" : "false");
diff --git a/notmuch-new.c b/notmuch-new.c
index 6a54a1a1..e6d3dc82 100644
--- a/notmuch-new.c
+++ b/notmuch-new.c
@@ -50,6 +50,8 @@ typedef struct {
     bool full_scan;
     const char **new_tags;
     size_t new_tags_length;
+    const char **rename_tags;
+    size_t rename_tags_length;
     const char **ignore_verbatim;
     size_t ignore_verbatim_length;
     regex_t *ignore_regex;
@@ -948,9 +950,18 @@ remove_filename (notmuch_database_t *notmuch,
 
     status = notmuch_database_remove_message (notmuch, path);
     if (status == NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID) {
+        const char **tag;
 	add_files_state->renamed_messages++;
+	notmuch_message_freeze (message);
+
+	for (tag = add_files_state->rename_tags; tag != NULL && *tag != NULL; tag++) {
+	    notmuch_message_add_tag (message, *tag);
+	}
+
+
 	if (add_files_state->synchronize_flags == true)
 	    notmuch_message_maildir_flags_to_tags (message);
+	notmuch_message_thaw (message);
 	status = NOTMUCH_STATUS_SUCCESS;
     } else if (status == NOTMUCH_STATUS_SUCCESS) {
 	add_files_state->removed_messages++;
@@ -1095,6 +1106,7 @@ notmuch_new_command (notmuch_config_t *config, int argc, char *argv[])
 	add_files_state.verbosity = VERBOSITY_VERBOSE;
 
     add_files_state.new_tags = notmuch_config_get_new_tags (config, &add_files_state.new_tags_length);
+    add_files_state.rename_tags = notmuch_config_get_rename_tags (config, &add_files_state.rename_tags_length);
     add_files_state.synchronize_flags = notmuch_config_get_maildir_synchronize_flags (config);
     db_path = notmuch_config_get_database_path (config);
     add_files_state.db_path = db_path;
@@ -1113,6 +1125,17 @@ notmuch_new_command (notmuch_config_t *config, int argc, char *argv[])
 	}
     }
 
+    for (i = 0; i < add_files_state.rename_tags_length; i++) {
+	const char *error_msg;
+
+	error_msg = illegal_tag (add_files_state.rename_tags[i], false);
+	if (error_msg) {
+	    fprintf (stderr, "Error: tag '%s' in rename.tags: %s\n",
+		     add_files_state.rename_tags[i], error_msg);
+	    return EXIT_FAILURE;
+	}
+    }
+
     if (hooks) {
 	ret = notmuch_run_hook (db_path, "pre-new");
 	if (ret)
-- 
2.18.0.226.g7b49cad896

^ permalink raw reply related	[flat|nested] 10+ messages in thread

* Re: [PATCH] introduce new.rename_tags for renamed (moved) messages
  2018-07-06 14:04 [PATCH] introduce new.rename_tags for renamed (moved) messages Michael J Gruber
@ 2018-09-08  0:47 ` David Bremner
  2018-09-18 15:32   ` [PATCH v2] " Michael J Gruber
  2018-09-21  6:48 ` [PATCH] " Gaute Hope
  1 sibling, 1 reply; 10+ messages in thread
From: David Bremner @ 2018-09-08  0:47 UTC (permalink / raw)
  To: Michael J Gruber, notmuch

Michael J Gruber <git@grubix.eu> writes:

>
> Introduce new.rename_tags (default: not set) which are added by `notmuch
> new` to renamed messages. This allows to act upon renames, e.g. to keep
> the IMAP folder structure in sync with tags with a tool like `afew` or
> homecooked scripts simply by filtering for this tag in the same ways as
> one would filter for new messages using new.tags.

The idea seems OK to me. I was hoping for some more feedback from
others, but here we are.

> +        const char **tag;
>  	add_files_state->renamed_messages++;
> +	notmuch_message_freeze (message);
> +
> +	for (tag = add_files_state->rename_tags; tag != NULL && *tag != NULL; tag++) {
> +	    notmuch_message_add_tag (message, *tag);
> +	}
> +
> +

extra blank line

>  	if (add_files_state->synchronize_flags == true)
>  	    notmuch_message_maildir_flags_to_tags (message);
> +	notmuch_message_thaw (message);

Did you have a specific reason for putting the _thaw after the existing
maildir_flags_to_tags? It's probably not important, but if there's no
good reason it seems more natural before.

As a new feature this needs some tests before it can be merged.  You can
start by looking at the tests for new.tags for inspiration I guess
(T050-new.sh and T340-maildir-sync.sh).  One case that occured to me is
that when duplicate files (with the same message-id) exist, deleting one
of them is detected as a rename.

^ permalink raw reply	[flat|nested] 10+ messages in thread

* [PATCH v2] introduce new.rename_tags for renamed (moved) messages
  2018-09-08  0:47 ` David Bremner
@ 2018-09-18 15:32   ` Michael J Gruber
  2019-04-02 12:40     ` [PATCH] performance-tests: tests for renamed/copied files in notmuch new David Bremner
  2019-04-03  1:28     ` [PATCH v2] introduce new.rename_tags for renamed (moved) messages David Bremner
  0 siblings, 2 replies; 10+ messages in thread
From: Michael J Gruber @ 2018-09-18 15:32 UTC (permalink / raw)
  To: notmuch

IMAP clients (such as webmail) use folders to mark messages as junk
etc., some even to mark messages as trash ("move to trash"). Such a
change is reported by notmuch as a rename; the message is not tagged
with new.tags since it is not new, so that there is no way to act upon a
rename.

Introduce new.rename_tags (default: not set) which are added by `notmuch
new` to renamed messages. This allows to act upon renames, e.g. to keep
the IMAP folder structure in sync with tags with a tool like `afew` or
homecooked scripts simply by filtering for this tag in the same ways as
one would filter for new messages using new.tags.

Signed-off-by: Michael J Gruber <git@grubix.eu>
---
Changed since v1:
- acted upon review comments (blank line, _thaw position)
- added 3 tests (mv, cp, cp-rm)
- treat copies as renames, too

The reasoning behind the latter is: If you use a mapping between folders
and tags, then a copy to an additional location should alert the
"mapper" to update that mapping; that's what the rename tag is for.
Maybe it should be named "renew" after all? But it's just the
folder/label name that is/needs to be renewed, nothing else about the
message.

Interdiff against v1:
  diff --git a/notmuch-new.c b/notmuch-new.c
  index e6d3dc82..e893fa21 100644
  --- a/notmuch-new.c
  +++ b/notmuch-new.c
  @@ -401,6 +401,13 @@ add_file (notmuch_database_t *notmuch, const char *filename,
   	break;
       /* Non-fatal issues (go on to next file). */
       case NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID:
  +	notmuch_message_freeze (message);
  +
  +	for (tag = state->rename_tags; tag != NULL && *tag != NULL; tag++) {
  +	    notmuch_message_add_tag (message, *tag);
  +	}
  +
  +	notmuch_message_thaw (message);
   	if (state->synchronize_flags)
   	    notmuch_message_maildir_flags_to_tags (message);
   	break;
  @@ -958,10 +965,9 @@ remove_filename (notmuch_database_t *notmuch,
   	    notmuch_message_add_tag (message, *tag);
   	}
   
  -
  +	notmuch_message_thaw (message);
   	if (add_files_state->synchronize_flags == true)
   	    notmuch_message_maildir_flags_to_tags (message);
  -	notmuch_message_thaw (message);
   	status = NOTMUCH_STATUS_SUCCESS;
       } else if (status == NOTMUCH_STATUS_SUCCESS) {
   	add_files_state->removed_messages++;
  diff --git a/test/T340-maildir-sync.sh b/test/T340-maildir-sync.sh
  index 7fece5f2..44f32ad2 100755
  --- a/test/T340-maildir-sync.sh
  +++ b/test/T340-maildir-sync.sh
  @@ -196,6 +196,36 @@ notmuch search 'subject:"File in new"' | notmuch_search_sanitize > output
   test_expect_equal "$(< output)" \
   "thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; File in new/ (test unread)"
   
  +test_begin_subtest "Renamed files get default renamed tags"
  +OLDCONFIG=$(notmuch config get new.rename_tags)
  +notmuch config set new.rename_tags "renamed"
  +mv $MAIL_DIR/new/file-in-new $MAIL_DIR/new/file-in-new-renamed
  +notmuch new
  +notmuch config set new.rename_tags $OLDCONFIG
  +notmuch search 'subject:"File in new"' | notmuch_search_sanitize > output
  +test_expect_equal "$(< output)" \
  +"thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; File in new/ (renamed test unread)"
  +
  +test_begin_subtest "Copied files do get new default renamed tags"
  +OLDCONFIG=$(notmuch config get new.rename_tags)
  +notmuch config set new.rename_tags "copied"
  +cp $MAIL_DIR/new/file-in-new-renamed $MAIL_DIR/new/file-in-new-copied
  +notmuch new
  +notmuch config set new.rename_tags $OLDCONFIG
  +notmuch search 'subject:"File in new"' | notmuch_search_sanitize > output
  +test_expect_equal "$(< output)" \
  +"thread:XXX   2001-01-05 [1/1(2)] Notmuch Test Suite; File in new/ (copied renamed test unread)"
  +
  +test_begin_subtest "Renamed files (cp+rm) get default renamed tags"
  +OLDCONFIG=$(notmuch config get new.rename_tags)
  +notmuch config set new.rename_tags "cprm"
  +rm $MAIL_DIR/new/file-in-new-renamed
  +notmuch new
  +notmuch config set new.rename_tags $OLDCONFIG
  +notmuch search 'subject:"File in new"' | notmuch_search_sanitize > output
  +test_expect_equal "$(< output)" \
  +"thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; File in new/ (copied cprm renamed test unread)"
  +
   for tag in draft flagged passed replied; do
       test_begin_subtest "$tag is valid in new.tags"
       OLDCONFIG=$(notmuch config get new.tags)

 NEWS                        | 11 +++++++++++
 doc/man1/notmuch-config.rst |  6 ++++++
 notmuch-client.h            |  8 ++++++++
 notmuch-config.c            | 26 ++++++++++++++++++++++++++
 notmuch-new.c               | 29 +++++++++++++++++++++++++++++
 test/T340-maildir-sync.sh   | 30 ++++++++++++++++++++++++++++++
 6 files changed, 110 insertions(+)

diff --git a/NEWS b/NEWS
index 240d594b..e3b75e74 100644
--- a/NEWS
+++ b/NEWS
@@ -1,3 +1,14 @@
+Notmuch 0.28 (UNRELEASED)
+=========================
+
+New command-line features
+-------------------------
+
+User-configurable tags for renamed messages
+
+  A new "new.rename_tags" option is available in the configuration file to
+  determine which tags are applied to renamed (moved) messages.
+
 Notmuch 0.27 (2018-06-13)
 =========================
 
diff --git a/doc/man1/notmuch-config.rst b/doc/man1/notmuch-config.rst
index 89909808..9e4198a1 100644
--- a/doc/man1/notmuch-config.rst
+++ b/doc/man1/notmuch-config.rst
@@ -77,6 +77,12 @@ The available configuration items are described below.
 
     Default: ``unread;inbox``.
 
+**new.rename_tags**
+    A list of tags that will be added to all messages which
+    **notmuch new** identifies as renamed (moved).
+
+    Default: not set.
+
 **new.ignore**
     A list to specify files and directories that will not be searched
     for messages by **notmuch new**. Each entry in the list is either:
diff --git a/notmuch-client.h b/notmuch-client.h
index 6c84ecc0..5e1e6b66 100644
--- a/notmuch-client.h
+++ b/notmuch-client.h
@@ -316,6 +316,14 @@ notmuch_config_set_new_tags (notmuch_config_t *config,
 			     const char *new_tags[],
 			     size_t length);
 
+const char **
+notmuch_config_get_rename_tags (notmuch_config_t *config,
+			     size_t *length);
+void
+notmuch_config_set_rename_tags (notmuch_config_t *config,
+			     const char *rename_tags[],
+			     size_t length);
+
 const char **
 notmuch_config_get_new_ignore (notmuch_config_t *config,
 			       size_t *length);
diff --git a/notmuch-config.c b/notmuch-config.c
index e1b16609..02f7d247 100644
--- a/notmuch-config.c
+++ b/notmuch-config.c
@@ -132,6 +132,8 @@ struct _notmuch_config {
     size_t user_other_email_length;
     const char **new_tags;
     size_t new_tags_length;
+    const char **rename_tags;
+    size_t rename_tags_length;
     const char **new_ignore;
     size_t new_ignore_length;
     bool maildir_synchronize_flags;
@@ -712,6 +714,14 @@ notmuch_config_get_new_tags (notmuch_config_t *config,   size_t *length)
 			     &(config->new_tags_length), length);
 }
 
+const char **
+notmuch_config_get_rename_tags (notmuch_config_t *config,   size_t *length)
+{
+    return _config_get_list (config, "new", "rename_tags",
+			     &(config->rename_tags),
+			     &(config->rename_tags_length), length);
+}
+
 const char **
 notmuch_config_get_new_ignore (notmuch_config_t *config, size_t *length)
 {
@@ -738,6 +748,15 @@ notmuch_config_set_new_tags (notmuch_config_t *config,
 		     &(config->new_tags));
 }
 
+void
+notmuch_config_set_rename_tags (notmuch_config_t *config,
+				     const char *list[],
+				     size_t length)
+{
+    _config_set_list (config, "new", "rename_tags", list, length,
+		     &(config->rename_tags));
+}
+
 void
 notmuch_config_set_new_ignore (notmuch_config_t *config,
 			       const char *list[],
@@ -867,6 +886,13 @@ notmuch_config_command_get (notmuch_config_t *config, char *item)
 	tags = notmuch_config_get_new_tags (config, &length);
 	for (i = 0; i < length; i++)
 	    printf ("%s\n", tags[i]);
+    } else if (strcmp(item, "new.rename_tags") == 0) {
+	const char **tags;
+	size_t i, length;
+
+	tags = notmuch_config_get_rename_tags (config, &length);
+	for (i = 0; i < length; i++)
+	    printf ("%s\n", tags[i]);
     } else if (STRNCMP_LITERAL (item, BUILT_WITH_PREFIX) == 0) {
 	printf ("%s\n",
 		notmuch_built_with (item + strlen (BUILT_WITH_PREFIX)) ? "true" : "false");
diff --git a/notmuch-new.c b/notmuch-new.c
index 6a54a1a1..e893fa21 100644
--- a/notmuch-new.c
+++ b/notmuch-new.c
@@ -50,6 +50,8 @@ typedef struct {
     bool full_scan;
     const char **new_tags;
     size_t new_tags_length;
+    const char **rename_tags;
+    size_t rename_tags_length;
     const char **ignore_verbatim;
     size_t ignore_verbatim_length;
     regex_t *ignore_regex;
@@ -399,6 +401,13 @@ add_file (notmuch_database_t *notmuch, const char *filename,
 	break;
     /* Non-fatal issues (go on to next file). */
     case NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID:
+	notmuch_message_freeze (message);
+
+	for (tag = state->rename_tags; tag != NULL && *tag != NULL; tag++) {
+	    notmuch_message_add_tag (message, *tag);
+	}
+
+	notmuch_message_thaw (message);
 	if (state->synchronize_flags)
 	    notmuch_message_maildir_flags_to_tags (message);
 	break;
@@ -948,7 +957,15 @@ remove_filename (notmuch_database_t *notmuch,
 
     status = notmuch_database_remove_message (notmuch, path);
     if (status == NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID) {
+        const char **tag;
 	add_files_state->renamed_messages++;
+	notmuch_message_freeze (message);
+
+	for (tag = add_files_state->rename_tags; tag != NULL && *tag != NULL; tag++) {
+	    notmuch_message_add_tag (message, *tag);
+	}
+
+	notmuch_message_thaw (message);
 	if (add_files_state->synchronize_flags == true)
 	    notmuch_message_maildir_flags_to_tags (message);
 	status = NOTMUCH_STATUS_SUCCESS;
@@ -1095,6 +1112,7 @@ notmuch_new_command (notmuch_config_t *config, int argc, char *argv[])
 	add_files_state.verbosity = VERBOSITY_VERBOSE;
 
     add_files_state.new_tags = notmuch_config_get_new_tags (config, &add_files_state.new_tags_length);
+    add_files_state.rename_tags = notmuch_config_get_rename_tags (config, &add_files_state.rename_tags_length);
     add_files_state.synchronize_flags = notmuch_config_get_maildir_synchronize_flags (config);
     db_path = notmuch_config_get_database_path (config);
     add_files_state.db_path = db_path;
@@ -1113,6 +1131,17 @@ notmuch_new_command (notmuch_config_t *config, int argc, char *argv[])
 	}
     }
 
+    for (i = 0; i < add_files_state.rename_tags_length; i++) {
+	const char *error_msg;
+
+	error_msg = illegal_tag (add_files_state.rename_tags[i], false);
+	if (error_msg) {
+	    fprintf (stderr, "Error: tag '%s' in rename.tags: %s\n",
+		     add_files_state.rename_tags[i], error_msg);
+	    return EXIT_FAILURE;
+	}
+    }
+
     if (hooks) {
 	ret = notmuch_run_hook (db_path, "pre-new");
 	if (ret)
diff --git a/test/T340-maildir-sync.sh b/test/T340-maildir-sync.sh
index 7fece5f2..44f32ad2 100755
--- a/test/T340-maildir-sync.sh
+++ b/test/T340-maildir-sync.sh
@@ -196,6 +196,36 @@ notmuch search 'subject:"File in new"' | notmuch_search_sanitize > output
 test_expect_equal "$(< output)" \
 "thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; File in new/ (test unread)"
 
+test_begin_subtest "Renamed files get default renamed tags"
+OLDCONFIG=$(notmuch config get new.rename_tags)
+notmuch config set new.rename_tags "renamed"
+mv $MAIL_DIR/new/file-in-new $MAIL_DIR/new/file-in-new-renamed
+notmuch new
+notmuch config set new.rename_tags $OLDCONFIG
+notmuch search 'subject:"File in new"' | notmuch_search_sanitize > output
+test_expect_equal "$(< output)" \
+"thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; File in new/ (renamed test unread)"
+
+test_begin_subtest "Copied files do get new default renamed tags"
+OLDCONFIG=$(notmuch config get new.rename_tags)
+notmuch config set new.rename_tags "copied"
+cp $MAIL_DIR/new/file-in-new-renamed $MAIL_DIR/new/file-in-new-copied
+notmuch new
+notmuch config set new.rename_tags $OLDCONFIG
+notmuch search 'subject:"File in new"' | notmuch_search_sanitize > output
+test_expect_equal "$(< output)" \
+"thread:XXX   2001-01-05 [1/1(2)] Notmuch Test Suite; File in new/ (copied renamed test unread)"
+
+test_begin_subtest "Renamed files (cp+rm) get default renamed tags"
+OLDCONFIG=$(notmuch config get new.rename_tags)
+notmuch config set new.rename_tags "cprm"
+rm $MAIL_DIR/new/file-in-new-renamed
+notmuch new
+notmuch config set new.rename_tags $OLDCONFIG
+notmuch search 'subject:"File in new"' | notmuch_search_sanitize > output
+test_expect_equal "$(< output)" \
+"thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; File in new/ (copied cprm renamed test unread)"
+
 for tag in draft flagged passed replied; do
     test_begin_subtest "$tag is valid in new.tags"
     OLDCONFIG=$(notmuch config get new.tags)
-- 
2.19.0.612.g94276ab026

^ permalink raw reply related	[flat|nested] 10+ messages in thread

* Re: [PATCH] introduce new.rename_tags for renamed (moved) messages
  2018-07-06 14:04 [PATCH] introduce new.rename_tags for renamed (moved) messages Michael J Gruber
  2018-09-08  0:47 ` David Bremner
@ 2018-09-21  6:48 ` Gaute Hope
       [not found]   ` <CAA19uiRXVEp8kPdxUw4yDvVLmG0DSXpxaKKn-ZC0uFC97uJi_Q@mail.gmail.com>
  1 sibling, 1 reply; 10+ messages in thread
From: Gaute Hope @ 2018-09-21  6:48 UTC (permalink / raw)
  To: Michael J Gruber, notmuch

[-- Attachment #1: Type: text/plain, Size: 1227 bytes --]

Michael J Gruber writes on July 6, 2018 16:04:
> IMAP clients (such as webmail) use folders to mark messages as junk
> etc., some even to mark messages as trash ("move to trash"). Such a
> change is reported by notmuch as a rename; the message is not tagged
> with new.tags since it is not new, so that there is no way to act upon a
> rename.
>
> Introduce new.rename_tags (default: not set) which are added by `notmuch
> new` to renamed messages. This allows to act upon renames, e.g. to keep
> the IMAP folder structure in sync with tags with a tool like `afew` or
> homecooked scripts simply by filtering for this tag in the same ways as
> one would filter for new messages using new.tags.

Hi,

think this would be very useful. I suggested something similar way back[0]
(no doubt bit-rotted by now), and seem to remember there were some
issues with cases where a rename would not be detected. Might be worth
checking out !

[0]  id:1396800683-9164-1-git-send-email-eg@gaute.vetsj.com

BTW: My use-case was solved by using `lastmod:` queries in keywsync [1]
(later obsoleted by gmailieer[2]).

[1] https://github.com/gauteh/abunchoftags
[2] https://github.com/gauteh/gmailieer

Regards, Gaute


[-- Attachment #2: Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [PATCH] introduce new.rename_tags for renamed (moved) messages
       [not found]   ` <CAA19uiRXVEp8kPdxUw4yDvVLmG0DSXpxaKKn-ZC0uFC97uJi_Q@mail.gmail.com>
@ 2018-09-21  8:34     ` Gaute Hope
  0 siblings, 0 replies; 10+ messages in thread
From: Gaute Hope @ 2018-09-21  8:34 UTC (permalink / raw)
  To: Michael J Gruber; +Cc: notmuch

[-- Attachment #1: Type: text/plain, Size: 697 bytes --]

Michael J Gruber writes on September 21, 2018 10:22:
> Lastmod and (re)new tags serve different purposes, I would thank. The
> latter is a flag that can be cleared to signal that a certain job has
> been done (by whomever). Whereas, if one actor acts upon a lastmod
> change by modifying the db that will change the lastmod and (possibly)
> trigger another actor to act upon that. Different scenarios which look
> the same if you have one "actor" only.

Agreed, very useful if all scenarios can be reliably caught! Maybe it 
was the "message changed without rename" that was problematic, since 
offlineimap would change the X-Keywords header without renaming the 
file.


- gaute


[-- Attachment #2: Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 10+ messages in thread

* [PATCH] performance-tests: tests for renamed/copied files in notmuch new
  2018-09-18 15:32   ` [PATCH v2] " Michael J Gruber
@ 2019-04-02 12:40     ` David Bremner
  2019-04-02 12:45       ` David Bremner
  2019-04-03 20:05       ` Tomi Ollila
  2019-04-03  1:28     ` [PATCH v2] introduce new.rename_tags for renamed (moved) messages David Bremner
  1 sibling, 2 replies; 10+ messages in thread
From: David Bremner @ 2019-04-02 12:40 UTC (permalink / raw)
  To: Michael J Gruber, notmuch

Several people have observed that this is surprisingly slow, and we
have a proposal to add tagging into this code path, so we want to make
sure it doesn't imply too much of a performance hit.
---
 performance-test/T00-new.sh | 30 ++++++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)

I added these tests to help evaluate Michael's propesed patch. I'll send the results in a seperate email.

diff --git a/performance-test/T00-new.sh b/performance-test/T00-new.sh
index 68750129..cec28d58 100755
--- a/performance-test/T00-new.sh
+++ b/performance-test/T00-new.sh
@@ -12,4 +12,34 @@ for i in $(seq 2 6); do
     time_run "notmuch new #$i" 'notmuch new'
 done
 
+manifest=$(mktemp manifestXXXXXX)
+
+count=0
+total=0
+while read -r name ; do
+    if [ $((total % 4 )) -eq 0 ]; then
+        echo $name >> $manifest
+        count=$((count + 1))
+    fi
+    total=$((total + 1))
+done < <(find mail -type f ! -path 'mail/.notmuch/*' )
+
+while read -r name ; do
+    mv $name ${name}.renamed
+done <  $manifest
+
+time_run "new ($count mv)" 'notmuch new'
+
+while read -r name ; do
+    mv ${name}.renamed $name
+done <  $manifest
+
+time_run "new ($count mv back)" 'notmuch new'
+
+while read -r name ; do
+    cp ${name} $name.copy
+done <  $manifest
+
+time_run "new ($count cp)" 'notmuch new'
+
 time_done
-- 
2.20.1

^ permalink raw reply related	[flat|nested] 10+ messages in thread

* Re: [PATCH] performance-tests: tests for renamed/copied files in notmuch new
  2019-04-02 12:40     ` [PATCH] performance-tests: tests for renamed/copied files in notmuch new David Bremner
@ 2019-04-02 12:45       ` David Bremner
  2019-04-03 20:05       ` Tomi Ollila
  1 sibling, 0 replies; 10+ messages in thread
From: David Bremner @ 2019-04-02 12:45 UTC (permalink / raw)
  To: Michael J Gruber, notmuch

[-- Attachment #1: Type: text/plain, Size: 772 bytes --]

David Bremner <david@tethera.net> writes:

> Several people have observed that this is surprisingly slow, and we
> have a proposal to add tagging into this code path, so we want to make
> sure it doesn't imply too much of a performance hit.

On my SSD / 8th gen i7 / 32G RAM based debian workstation it seems OK,
with a max slowdown of 1.4% (mean of 5 runs). I'd like to see similar
figures for spinning rust and older CPUs. This is with Xapian 1.4.11;
ideally so would other comparisons.

|         | Before |  After | slowdown % |
| initial | 517.69 | 522.13 |        0.9 |
| mv      | 313.16 | 317.57 |        1.4 |
| mv back | 315.21 | 316.73 |        0.5 |
| cp      | 170.36 | 171.08 |        0.4 |
#+TBLFM: $4=100*($3/$2-1);f1

Raw data is attached for the curious

[-- Attachment #2: trials.txt --]
[-- Type: text/plain, Size: 7146 bytes --]

with patch

T00-new.sh: Testing notmuch new                         [0.4 large]
                        Wall(s) Usr(s)  Sys(s)  Res(K)  In/Out(512B)
  Initial notmuch new   521.81  405.90  111.16  273992  0/15876448
  notmuch new #2        0.02    0.01    0.00    12632   0/144
  notmuch new #3        0.00    0.00    0.00    8872    0/8
  notmuch new #4        0.00    0.00    0.00    8900    0/8
  notmuch new #5        0.00    0.00    0.00    9004    0/8
  notmuch new #6        0.00    0.00    0.00    8728    0/8
  new (52374 mv)        316.30  219.89  95.42   146136  0/4611440
  new (52374 mv back)   312.67  217.23  94.55   147476  0/4986256
  new (52374 cp)        170.76  125.47  43.85   114360  0/4099808

T00-new.sh: Testing notmuch new                         [0.4 large]
                        Wall(s) Usr(s)  Sys(s)  Res(K)  In/Out(512B)
  Initial notmuch new   523.08  405.39  112.02  274152  0/15871216
  notmuch new #2        0.02    0.00    0.01    12372   0/144
  notmuch new #3        0.02    0.01    0.00    8984    0/8
  notmuch new #4        0.01    0.00    0.00    8820    0/8
  notmuch new #5        0.00    0.00    0.00    8776    0/8
  notmuch new #6        0.00    0.00    0.00    8776    0/8
  new (52374 mv)        317.27  221.69  94.55   146424  0/4864352
  new (52374 mv back)   318.88  222.51  95.31   146764  0/5063728
  new (52374 cp)        172.09  126.18  44.75   114548  0/4073552

T00-new.sh: Testing notmuch new                         [0.4 large]
                        Wall(s) Usr(s)  Sys(s)  Res(K)  In/Out(512B)
  Initial notmuch new   523.58  407.64  112.15  274100  0/16211056
  notmuch new #2        0.02    0.01    0.00    12644   0/144
  notmuch new #3        0.00    0.00    0.00    8828    0/8
  notmuch new #4        0.00    0.00    0.00    8884    0/8
  notmuch new #5        0.00    0.00    0.00    8916    0/8
  notmuch new #6        0.00    0.00    0.00    8896    0/8
  new (52374 mv)        318.53  222.16  95.60   146368  0/5056016
  new (52374 mv back)   315.98  219.54  95.64   146796  0/4766496
  new (52374 cp)        170.79  124.49  44.63   114560  0/4075616

T00-new.sh: Testing notmuch new                         [0.4 large]
                        Wall(s) Usr(s)  Sys(s)  Res(K)  In/Out(512B)
  Initial notmuch new   519.77  403.60  111.33  274212  0/15964240
  notmuch new #2        0.02    0.00    0.01    12456   0/144
  notmuch new #3        0.00    0.00    0.00    9080    0/8
  notmuch new #4        0.00    0.00    0.00    8884    0/8
  notmuch new #5        0.00    0.00    0.00    9004    0/8
  notmuch new #6        0.00    0.00    0.00    8840    0/8
  new (52374 mv)        318.98  222.26  95.94   146332  0/4854864
  new (52374 mv back)   319.39  221.87  96.65   146944  0/4969296
  new (52374 cp)        170.72  124.50  44.63   114224  0/3817216

T00-new.sh: Testing notmuch new                         [0.4 large]
                        Wall(s) Usr(s)  Sys(s)  Res(K)  In/Out(512B)
  Initial notmuch new   522.43  406.40  111.85  273936  0/16188560
  notmuch new #2        0.88    0.01    0.01    12512   0/144
  notmuch new #3        0.03    0.01    0.01    8756    0/8
  notmuch new #4        0.00    0.00    0.00    8852    0/8
  notmuch new #5        0.00    0.00    0.00    8832    0/8
  notmuch new #6        0.00    0.00    0.00    8864    0/8
  new (52374 mv)        316.75  219.96  95.79   146496  0/4866064
  new (52374 mv back)   316.73  220.40  95.33   146760  0/5034928
  new (52374 cp)        171.03  125.37  44.34   114384  0/4096496

----------------------------------------------------------------------

without patch:

T00-new.sh: Testing notmuch new                         [0.4 large]
                        Wall(s) Usr(s)  Sys(s)  Res(K)  In/Out(512B)
  Initial notmuch new   517.73  403.06  110.61  284648  0/14723776
  notmuch new #2        0.01    0.00    0.00    8760    0/8
  notmuch new #3        0.01    0.00    0.01    8928    0/8
  notmuch new #4        0.01    0.01    0.00    8812    0/8
  notmuch new #5        0.01    0.00    0.01    8928    0/8
  notmuch new #6        0.00    0.00    0.00    8832    0/8
  new (52374 mv)        314.48  218.67  94.55   146560  0/4774288
  new (52374 mv back)   319.43  222.46  96.09   146780  0/4731808
  new (52374 cp)        169.92  125.07  43.83   114476  0/3946896

T00-new.sh: Testing notmuch new                         [0.4 large]
                        Wall(s) Usr(s)  Sys(s)  Res(K)  In/Out(512B)
  Initial notmuch new   516.03  401.91  109.94  284668  0/14725200
  notmuch new #2        0.88    0.01    0.01    12420   0/144
  notmuch new #3        0.01    0.00    0.00    8888    0/8
  notmuch new #4        0.00    0.00    0.00    8760    0/8
  notmuch new #5        0.00    0.00    0.00    8848    0/8
  notmuch new #6        0.00    0.00    0.00    8864    0/8
  new (52374 mv)        311.78  217.74  93.29   146456  8/4774576
  new (52374 mv back)   312.88  219.64  92.07   147064  0/4785104
  new (52374 cp)        170.95  125.16  44.57   114384  0/3688016

T00-new.sh: Testing notmuch new                         [0.4 large]
                        Wall(s) Usr(s)  Sys(s)  Res(K)  In/Out(512B)
  Initial notmuch new   517.58  404.56  110.22  284608  0/15282160
  notmuch new #2        0.02    0.00    0.01    12664   0/144
  notmuch new #3        0.00    0.00    0.00    8956    0/8
  notmuch new #4        0.00    0.00    0.00    8804    0/8
  notmuch new #5        0.00    0.00    0.00    8892    0/8
  notmuch new #6        0.00    0.00    0.00    8844    0/8
  new (52374 mv)        313.10  217.43  94.61   146108  0/4980176
  new (52374 mv back)   315.64  219.17  95.49   147176  0/4943408
  new (52374 cp)        169.27  124.51  43.71   114500  0/3946928

T00-new.sh: Testing notmuch new                         [0.4 large]
                        Wall(s) Usr(s)  Sys(s)  Res(K)  In/Out(512B)
  Initial notmuch new   518.28  402.97  111.69  284836  0/14991824
  notmuch new #2        0.00    0.00    0.00    8804    0/8
  notmuch new #3        0.00    0.00    0.00    8764    0/8
  notmuch new #4        0.00    0.00    0.00    8792    0/8
  notmuch new #5        0.00    0.00    0.00    9068    0/8
  notmuch new #6        0.00    0.00    0.00    8820    0/8
  new (52374 mv)        314.22  218.72  94.45   146276  0/4761712
  new (52374 mv back)   314.39  220.18  93.40   147288  0/4832464
  new (52374 cp)        170.49  125.64  43.84   114380  0/3948064

T00-new.sh: Testing notmuch new                         [0.4 large]
                        Wall(s) Usr(s)  Sys(s)  Res(K)  In/Out(512B)
  Initial notmuch new   518.81  405.03  110.53  284696  0/14964736
  notmuch new #2        0.94    0.00    0.00    8808    0/144
  notmuch new #3        0.01    0.00    0.00    9008    0/8
  notmuch new #4        0.00    0.00    0.00    8768    0/8
  notmuch new #5        0.00    0.00    0.00    8916    0/8
  notmuch new #6        0.00    0.00    0.00    8776    0/8
  new (52374 mv)        312.24  217.13  93.96   146492  0/4772608
  new (52374 mv back)   313.70  218.13  93.98   147172  0/4753408
  new (52374 cp)        171.15  125.01  44.42   114472  0/3970096


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: trials.org --]
[-- Type: text/x-org, Size: 3915 bytes --]

with patch

| Initial notmuch new | 519.77 | 403.60 | 111.33 | 274212 | 0/15964240 |
| Initial notmuch new | 521.81 | 405.90 | 111.16 | 273992 | 0/15876448 |
| Initial notmuch new | 522.43 | 406.40 | 111.85 | 273936 | 0/16188560 |
| Initial notmuch new | 523.08 | 405.39 | 112.02 | 274152 | 0/15871216 |
| Initial notmuch new | 523.58 | 407.64 | 112.15 | 274100 | 0/16211056 |
|                     | 522.13 |        |        |        |            |
#+TBLFM: @6$2=vmean(@1..@5);f2

| new (52374 cp) | 170.72 | 124.50 | 44.63 | 114224 | 0/3817216 |
| new (52374 cp) | 170.76 | 125.47 | 43.85 | 114360 | 0/4099808 |
| new (52374 cp) | 170.79 | 124.49 | 44.63 | 114560 | 0/4075616 |
| new (52374 cp) | 171.03 | 125.37 | 44.34 | 114384 | 0/4096496 |
| new (52374 cp) | 172.09 | 126.18 | 44.75 | 114548 | 0/4073552 |
|                | 171.08 |        |       |        |           |
#+TBLFM: @6$2=vmean(@1..@5);f2

| new (52374 mv) | 316.30 | 219.89 | 95.42 | 146136 | 0/4611440 |
| new (52374 mv) | 316.75 | 219.96 | 95.79 | 146496 | 0/4866064 |
| new (52374 mv) | 317.27 | 221.69 | 94.55 | 146424 | 0/4864352 |
| new (52374 mv) | 318.53 | 222.16 | 95.60 | 146368 | 0/5056016 |
| new (52374 mv) | 318.98 | 222.26 | 95.94 | 146332 | 0/4854864 |
|                | 317.57 |        |       |        |           |
#+TBLFM: @6$2=vmean(@1..@5);f2

| new (52374 mv back) | 312.67 | 217.23 | 94.55 | 147476 | 0/4986256 |
| new (52374 mv back) | 315.98 | 219.54 | 95.64 | 146796 | 0/4766496 |
| new (52374 mv back) | 316.73 | 220.40 | 95.33 | 146760 | 0/5034928 |
| new (52374 mv back) | 318.88 | 222.51 | 95.31 | 146764 | 0/5063728 |
| new (52374 mv back) | 319.39 | 221.87 | 96.65 | 146944 | 0/4969296 |
|                     | 316.73 |        |       |        |           |
#+TBLFM: @6$2=vmean(@1..@5);f2


----------------------------------------------------------------------

without patch:

| Initial notmuch new | 516.03 | 401.91 | 109.94 | 284668 | 0/14725200 |
| Initial notmuch new | 517.58 | 404.56 | 110.22 | 284608 | 0/15282160 |
| Initial notmuch new | 517.73 | 403.06 | 110.61 | 284648 | 0/14723776 |
| Initial notmuch new | 518.28 | 402.97 | 111.69 | 284836 | 0/14991824 |
| Initial notmuch new | 518.81 | 405.03 | 110.53 | 284696 | 0/14964736 |
|                     | 517.69 |        |        |        |            |
#+TBLFM: @6$2=vmean(@1..@5);f2

| new (52374 cp) | 169.27 | 124.51 | 43.71 | 114500 | 0/3946928 |
| new (52374 cp) | 169.92 | 125.07 | 43.83 | 114476 | 0/3946896 |
| new (52374 cp) | 170.49 | 125.64 | 43.84 | 114380 | 0/3948064 |
| new (52374 cp) | 170.95 | 125.16 | 44.57 | 114384 | 0/3688016 |
| new (52374 cp) | 171.15 | 125.01 | 44.42 | 114472 | 0/3970096 |
|                | 170.36 |        |       |        |           |
#+TBLFM: @6$2=vmean(@1..@5);f2


| new (52374 mv) | 311.78 | 217.74 | 93.29 | 146456 | 8/4774576 |
| new (52374 mv) | 312.24 | 217.13 | 93.96 | 146492 | 0/4772608 |
| new (52374 mv) | 313.10 | 217.43 | 94.61 | 146108 | 0/4980176 |
| new (52374 mv) | 314.22 | 218.72 | 94.45 | 146276 | 0/4761712 |
| new (52374 mv) | 314.48 | 218.67 | 94.55 | 146560 | 0/4774288 |
|                | 313.16 |        |       |        |           |
#+TBLFM: @6$2=vmean(@1..@5);f2

| new (52374 mv back) | 312.88 | 219.64 | 92.07 | 147064 | 0/4785104 |
| new (52374 mv back) | 313.70 | 218.13 | 93.98 | 147172 | 0/4753408 |
| new (52374 mv back) | 314.39 | 220.18 | 93.40 | 147288 | 0/4832464 |
| new (52374 mv back) | 315.64 | 219.17 | 95.49 | 147176 | 0/4943408 |
| new (52374 mv back) | 319.43 | 222.46 | 96.09 | 146780 | 0/4731808 |
|                     | 315.21 |        |       |        |           |
#+TBLFM: @6$2=vmean(@1..@5);f2


|         | Before |  After | slowdown % |
| initial | 517.69 | 522.13 |        0.9 |
| mv      | 313.16 | 317.57 |        1.4 |
| mv back | 315.21 | 316.73 |        0.5 |
| cp      | 170.36 | 171.08 |        0.4 |
#+TBLFM: $4=100*($3/$2-1);f1

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [PATCH v2] introduce new.rename_tags for renamed (moved) messages
  2018-09-18 15:32   ` [PATCH v2] " Michael J Gruber
  2019-04-02 12:40     ` [PATCH] performance-tests: tests for renamed/copied files in notmuch new David Bremner
@ 2019-04-03  1:28     ` David Bremner
  1 sibling, 0 replies; 10+ messages in thread
From: David Bremner @ 2019-04-03  1:28 UTC (permalink / raw)
  To: Michael J Gruber, notmuch

Michael J Gruber <git@grubix.eu> writes:

> Changed since v1:
> - acted upon review comments (blank line, _thaw position)
> - added 3 tests (mv, cp, cp-rm)
> - treat copies as renames, too

Apologies for taking so long to get back to this. As a general comment,
does this same tagging "hook" make sense for notmuch-insert as well?
That's not meant to delay this series, just food for thought.

> The reasoning behind the latter is: If you use a mapping between folders
> and tags, then a copy to an additional location should alert the
> "mapper" to update that mapping; that's what the rename tag is for.
> Maybe it should be named "renew" after all? But it's just the
> folder/label name that is/needs to be renewed, nothing else about the
> message.

It _is_ confusing to use rename to refer to copies as well as actual
renames. I don't find "renew" better though. I wonder about something
like "new.new_path_tags". That's a bit weird with the repeated "new", I
grant you.   Maybe "new.path_change_tags"

> +**new.rename_tags**
> +    A list of tags that will be added to all messages which
> +    **notmuch new** identifies as renamed (moved).
> +
> +    Default: not set.
> +

Even if the name stays the same, you'll need to update the blurb to
mention copies.

> +void
> +notmuch_config_set_rename_tags (notmuch_config_t *config,
> +			     const char *rename_tags[],
> +			     size_t length);
> +

If this is only used in notmuch-config.c, I don't think it needs to
be exported (at least until they are needed). Of course it's also worth
asking if we want to call them notmuch-setup.c; I suspect not doing so
is OK, but I haven't checked in detail.


> diff --git a/test/T340-maildir-sync.sh b/test/T340-maildir-sync.sh

These tests don't really have to do with maildir syncing (in the sense
that notmuch uses the word), that's about syncing maildir flags to
notmuch tags. I think they'd be better in T050-new.sh.

> index 7fece5f2..44f32ad2 100755
> --- a/test/T340-maildir-sync.sh
> +++ b/test/T340-maildir-sync.sh
> @@ -196,6 +196,36 @@ notmuch search 'subject:"File in new"' | notmuch_search_sanitize > output
>  test_expect_equal "$(< output)" \
>  "thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; File in new/ (test unread)"
>  
> +test_begin_subtest "Renamed files get default renamed tags"
> +OLDCONFIG=$(notmuch config get new.rename_tags)
> +notmuch config set new.rename_tags "renamed"
> +mv $MAIL_DIR/new/file-in-new $MAIL_DIR/new/file-in-new-renamed
> +notmuch new
> +notmuch config set new.rename_tags $OLDCONFIG
> +notmuch search 'subject:"File in new"' | notmuch_search_sanitize > output
> +test_expect_equal "$(< output)" \
> +"thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; File in new/ (renamed test unread)"

Most of the newer tests use the pattern

cat << EOF > EXPECTED
thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; File in new/ (renamed test unread)
EOF
test_expect_equal_file EXPECTED output

we also mainly use OUTPUT for the file, and $output as a variable
containing output.

Unfortunately the searches would need to be updated as well to work in
T050-new. 

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [PATCH] performance-tests: tests for renamed/copied files in notmuch new
  2019-04-02 12:40     ` [PATCH] performance-tests: tests for renamed/copied files in notmuch new David Bremner
  2019-04-02 12:45       ` David Bremner
@ 2019-04-03 20:05       ` Tomi Ollila
  2019-04-05 16:59         ` David Bremner
  1 sibling, 1 reply; 10+ messages in thread
From: Tomi Ollila @ 2019-04-03 20:05 UTC (permalink / raw)
  To: David Bremner, Michael J Gruber, notmuch

On Tue, Apr 02 2019, David Bremner wrote:

> Several people have observed that this is surprisingly slow, and we
> have a proposal to add tagging into this code path, so we want to make
> sure it doesn't imply too much of a performance hit.
> ---
>  performance-test/T00-new.sh | 30 ++++++++++++++++++++++++++++++
>  1 file changed, 30 insertions(+)
>
> I added these tests to help evaluate Michael's propesed patch. I'll send the results in a seperate email.
>
> diff --git a/performance-test/T00-new.sh b/performance-test/T00-new.sh
> index 68750129..cec28d58 100755
> --- a/performance-test/T00-new.sh
> +++ b/performance-test/T00-new.sh
> @@ -12,4 +12,34 @@ for i in $(seq 2 6); do
>      time_run "notmuch new #$i" 'notmuch new'
>  done
>  
> +manifest=$(mktemp manifestXXXXXX)
> +
> +count=0
> +total=0
> +while read -r name ; do
> +    if [ $((total % 4 )) -eq 0 ]; then
> +        echo $name >> $manifest
> +        count=$((count + 1))
> +    fi
> +    total=$((total + 1))
> +done < <(find mail -type f ! -path 'mail/.notmuch/*' )

// this comment was written last in this email, just for fun >;) //

find mail -type f ! -path 'mail/.notmuch/*' | sed -n '1~4 p' > $manifest
count=`wc $manifest`

(I'd be interested which one of the above were faster -- my suggestion
does quite a many more forks and execve's but abowe read loop 200 000 
read(2)'s and [lf]seek(2)s (and then 50 000 opens). 
well, probably no-one would notice difference...)

> +
> +while read -r name ; do
> +    mv $name ${name}.renamed
> +done <  $manifest

--------'12' -- 2 spaces above (and below...)

luckily bash read builtin does not read input byte at a time
(IIRC it read 128 bytes, then scanned for newline and then seeked
 -- in this case it can, since file was redirected -- fd is seekable)

50 000 mv(1) executions definitely take time.

perl -nle 'rename $_, "$_.renamed"' $manifest  would be significantly faster 

> +
> +time_run "new ($count mv)" 'notmuch new'
> +
> +while read -r name ; do
> +    mv ${name}.renamed $name
> +done <  $manifest
> +
> +time_run "new ($count mv back)" 'notmuch new'
> +
> +while read -r name ; do
> +    cp ${name} $name.copy
> +done <  $manifest

perl -nle 'link $_, "$_.copy"' $manifest  ?

> +
> +time_run "new ($count cp)" 'notmuch new'
> +
>  time_done
> -- 
> 2.20.1
>
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> https://notmuchmail.org/mailman/listinfo/notmuch

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [PATCH] performance-tests: tests for renamed/copied files in notmuch new
  2019-04-03 20:05       ` Tomi Ollila
@ 2019-04-05 16:59         ` David Bremner
  0 siblings, 0 replies; 10+ messages in thread
From: David Bremner @ 2019-04-05 16:59 UTC (permalink / raw)
  To: Tomi Ollila, Michael J Gruber, notmuch

Tomi Ollila <tomi.ollila@iki.fi> writes:

>> +done < <(find mail -type f ! -path 'mail/.notmuch/*' )
>
> // this comment was written last in this email, just for fun >;) //
>
> find mail -type f ! -path 'mail/.notmuch/*' | sed -n '1~4 p' > $manifest
> count=`wc $manifest`
>
> (I'd be interested which one of the above were faster -- my suggestion
> does quite a many more forks and execve's but abowe read loop 200 000 
> read(2)'s and [lf]seek(2)s (and then 50 000 opens). 
> well, probably no-one would notice difference...)

Your version is _much_ faster so I used it, along with the other
suggestions in the version I just pushed to master.

d

^ permalink raw reply	[flat|nested] 10+ messages in thread

end of thread, other threads:[~2019-04-05 16:59 UTC | newest]

Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-07-06 14:04 [PATCH] introduce new.rename_tags for renamed (moved) messages Michael J Gruber
2018-09-08  0:47 ` David Bremner
2018-09-18 15:32   ` [PATCH v2] " Michael J Gruber
2019-04-02 12:40     ` [PATCH] performance-tests: tests for renamed/copied files in notmuch new David Bremner
2019-04-02 12:45       ` David Bremner
2019-04-03 20:05       ` Tomi Ollila
2019-04-05 16:59         ` David Bremner
2019-04-03  1:28     ` [PATCH v2] introduce new.rename_tags for renamed (moved) messages David Bremner
2018-09-21  6:48 ` [PATCH] " Gaute Hope
     [not found]   ` <CAA19uiRXVEp8kPdxUw4yDvVLmG0DSXpxaKKn-ZC0uFC97uJi_Q@mail.gmail.com>
2018-09-21  8:34     ` Gaute Hope

Code repositories for project(s) associated with this public inbox

	https://yhetil.org/notmuch.git/

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).