[PATCH v2] emacs: call "notmuch tag" only once when archiving a thread

[PATCH v2] emacs: call "notmuch tag" only once when archiving a thread

[Jani Nikula]

Optimize thread archiving by combining all the -inbox tagging
operations to a single "notmuch tag" call. Also skip redisplay of tag
changes in current buffer, as it is immediately killed by the
archiving functions.

For threads in the order of tens or a hundred inbox tagged messages,
this gives a noticeable speedup. On two different machines, archiving
a thread of about 50 inbox tagged messages goes down from 10+ seconds
to about 0.5 seconds.

The bottleneck is not within emacs; the same behaviour can be observed
in the cli. This patch is a quick fix to thread archiving, but it
seems clear that generally the thread tagging functions should be
refactored to do tagging in one go. This approach would have the added
benefit of being more reliable: any of the individual tagging
operations might face a locked database, leading to partial results.

This introduces a limitation to the number of messages that can be
archived at the same time (through ARG_MAX limiting the command
line). While at least on Linux this seems more like a theoretical
limitation than a real one, it could be avoided by archiving at most a
few hundred messages at a time.

Signed-off-by: Jani Nikula <jani@nikula.org>

---

v1 is at id:"1325615346-8302-1-git-send-email-jani@nikula.org".

Although this saves me several minutes a day, I don't have the time
for further improvements. I'm just too slow writing elisp...
---
 emacs/notmuch-show.el |   19 +++++++++++++++++--
 1 files changed, 17 insertions(+), 2 deletions(-)

diff --git a/emacs/notmuch-show.el b/emacs/notmuch-show.el
index 7469e2e..a0b8eb3 100644
--- a/emacs/notmuch-show.el
+++ b/emacs/notmuch-show.el
@@ -1614,6 +1614,21 @@ added."
       (if show-next
 	  (notmuch-search-show-thread)))))
 
+(defun notmuch-show-archive-thread-quick ()
+  "Remove \"inbox\" tag from the current set of messages.
+
+Note: This function does not call `notmuch-show-set-tags' on the
+messages to redisplay the changed tags. This is meant to be
+called by functions that archive the messages and kill the buffer
+afterwards."
+  (goto-char (point-min))
+  (let (message-ids)
+    (loop do
+	  (add-to-list 'message-ids (notmuch-show-get-message-id))
+	  until (not (notmuch-show-goto-message-next)))
+    (when message-ids
+      (notmuch-tag (mapconcat 'identity message-ids " OR ") "-inbox"))))
+
 (defun notmuch-show-archive-thread (&optional unarchive)
   "Archive each message in thread.
 
@@ -1637,13 +1652,13 @@ buffer."
 (defun notmuch-show-archive-thread-then-next ()
   "Archive each message in thread, then show next thread from search."
   (interactive)
-  (notmuch-show-archive-thread)
+  (notmuch-show-archive-thread-quick)
   (notmuch-show-next-thread t))
 
 (defun notmuch-show-archive-thread-then-exit ()
   "Archive each message in thread, then exit back to search results."
   (interactive)
-  (notmuch-show-archive-thread)
+  (notmuch-show-archive-thread-quick)
   (notmuch-show-next-thread))
 
 (defun notmuch-show-archive-message (&optional unarchive)
-- 
1.7.5.4

[PATCH v3] emacs: Call "notmuch tag" once when applying tag changes to a thread.

[David Edmondson]

Optimize thread tagging by combining all the tagging operations to a
single "notmuch tag" call.

For threads in the order of tens or a hundred inbox tagged messages,
this gives a noticeable speedup. On two different machines, archiving
a thread of about 50 inbox tagged messages goes down from 10+ seconds
to about 0.5 seconds.

The bottleneck is not within emacs; the same behaviour can be observed
in the CLI. This approach has the added benefit of being more
reliable: any of the individual tagging operations might face a locked
database, leading to partial results.

This introduces a limitation to the number of messages that can be
archived at the same time (through ARG_MAX limiting the command
line). While at least on Linux this seems more like a theoretical
limitation than a real one, it could be avoided by archiving at most a
few hundred messages at a time.

Based on code from Jani Nikula <jani@nikula.org>.
---

Batch all thread tagging rather than archive being a special case.

 emacs/notmuch-show.el |   19 ++++++++++---------
 1 files changed, 10 insertions(+), 9 deletions(-)

diff --git a/emacs/notmuch-show.el b/emacs/notmuch-show.el
index 7469e2e..bbba482 100644
--- a/emacs/notmuch-show.el
+++ b/emacs/notmuch-show.el
@@ -1582,16 +1582,17 @@ argument, hide all of the messages."
   (backward-button 1))
 
 (defun notmuch-show-tag-thread-internal (tag &optional remove)
-  "Add tag to the current set of messages.
+  "Add TAG to the current set of messages.
 
-If the remove switch is given, tags will be removed instead of
-added."
-  (goto-char (point-min))
-  (let ((tag-function (if remove
-			  'notmuch-show-remove-tag
-			'notmuch-show-add-tag)))
-    (loop do (funcall tag-function tag)
-	  until (not (notmuch-show-goto-message-next)))))
+If REMOVE is non-nil, TAG will be removed rather than added."
+  (save-excursion
+    (goto-char (point-min))
+    (let ((message-ids
+	  (loop collect (notmuch-show-get-message-id)
+		until (not (notmuch-show-goto-message-next)))))
+      (if message-ids
+	  (notmuch-tag (mapconcat #'identity message-ids " OR ")
+		       (concat (if remove "-" "+") tag))))))
 
 (defun notmuch-show-add-tag-thread (tag)
   "Add tag to all messages in the current thread."
-- 
1.7.8.3

Re: [PATCH v3] emacs: Call "notmuch tag" once when applying tag changes to a thread.

[Tomi Ollila]

On Wed,  8 Feb 2012 08:36:09 +0000, David Edmondson <dme@dme.org> wrote:
> Optimize thread tagging by combining all the tagging operations to a
> single "notmuch tag" call.
> 
> For threads in the order of tens or a hundred inbox tagged messages,
> this gives a noticeable speedup. On two different machines, archiving
> a thread of about 50 inbox tagged messages goes down from 10+ seconds
> to about 0.5 seconds.
> 
> The bottleneck is not within emacs; the same behaviour can be observed
> in the CLI. This approach has the added benefit of being more
> reliable: any of the individual tagging operations might face a locked
> database, leading to partial results.
> 
> This introduces a limitation to the number of messages that can be
> archived at the same time (through ARG_MAX limiting the command
> line). While at least on Linux this seems more like a theoretical
> limitation than a real one, it could be avoided by archiving at most a
> few hundred messages at a time.

I did a simple test program:

--8<----8<----8<----8<----8<----8<----8<----8<----8<----8<----8<--
#!/usr/bin/perl

use strict;
use warnings;

die "Usage: $0 <arglen> <# of args>\n" unless @ARGV == 2;

my $arg = 'x' x $ARGV[0];
my @args = ( $arg ) x $ARGV[1];

print "One arg: '$arg'\n";
print 'Number of args: ', scalar @args, "\n";

exec '/bin/true', @args;
--8<----8<----8<----8<----8<----8<----8<----8<----8<----8<----8<--

This program executes /bin/true with given number of args all args
of some length: i.e.

./test_cmdlimit 100 10000 

makes 10000 100 character arguments (total of million characters)
and executes /bin/true with those ten thousand 100-char args.

In one machine where  getconf ARG_MAX  returns 131072
(Debian Lenny ia32)

./test_cmdlimit 200 10000   succeeds
but
./test_cmdlimit 20 100000   gives
Can't exec "/bin/true": Argument list too long at ./test_cmdlimit.pl line 14.

Hmm, actually there:
./test_cmdlimit 19 100000   succeeds and
./test_cmdlimit.pl 209 10000 fails

More with args which lengths are 1 and 2 chars:

./test_cmdlimit.pl 1 1046163 succeeds
./test_cmdlimit.pl 1 1046164 fails

./test_cmdlimit.pl 2 697442 succeeds
./test_cmdlimit.pl 2 697443 fails

1046163 was close to 1048576 (1024 * 1024) --
697442 * 2 is 1394884...

From these I can make an educated guess that when message id:s
are typically between 30 to 70 characters something like
20 000 messages are safe to be tagged at once in this
test system (./test_cmdlimit.pl 50 40000 succeeds).

> 
> Based on code from Jani Nikula <jani@nikula.org>.

Tomi

[PATCH v4] emacs: Call "notmuch tag" once when applying tag changes to a thread.

[David Edmondson]

Optimize thread tagging by combining all the tagging operations to a
single "notmuch tag" call.

For threads in the order of tens or a hundred inbox tagged messages,
this gives a noticeable speedup. On two different machines, archiving
a thread of about 50 inbox tagged messages goes down from 10+ seconds
to about 0.5 seconds.

The bottleneck is not within emacs; the same behaviour can be observed
in the CLI. This approach has the added benefit of being more
reliable: any of the individual tagging operations might face a locked
database, leading to partial results.

This introduces a limitation to the number of messages that can be
archived at the same time (through ARG_MAX limiting the command
line). While at least on Linux this seems more like a theoretical
limitation than a real one, it could be avoided by archiving at most a
few hundred messages at a time.

Based on code from Jani Nikula <jani@nikula.org>.
---

v4:
- Rebased after Dmitry's tagging changes (yay!).

 emacs/notmuch-show.el |   18 +++++++++++-------
 1 files changed, 11 insertions(+), 7 deletions(-)

diff --git a/emacs/notmuch-show.el b/emacs/notmuch-show.el
index faa9f9b..9294bc8 100644
--- a/emacs/notmuch-show.el
+++ b/emacs/notmuch-show.el
@@ -1599,14 +1599,18 @@ argument, hide all of the messages."
   (backward-button 1))
 
 (defun notmuch-show-tag-thread-internal (tag &optional remove)
-  "Add tag to the current set of messages.
+  "Add TAG to the current set of messages.
 
-If the remove switch is given, tags will be removed instead of
-added."
-  (goto-char (point-min))
-  (let ((op (if remove "-" "+")))
-    (loop do (notmuch-show-tag-message (concat op tag))
-	  until (not (notmuch-show-goto-message-next)))))
+If REMOVE is non-nil, TAG will be removed rather than added."
+  (save-excursion
+    (goto-char (point-min))
+    (let ((tag-op (concat (if remove "-" "+") tag))
+	  (message-ids
+	   (loop collect (notmuch-show-get-message-id)
+		 until (not (notmuch-show-goto-message-next)))))
+      (if message-ids
+	  (notmuch-tag (mapconcat #'identity message-ids " OR ")
+		       tag-op)))))
 
 (defun notmuch-show-add-tag-thread (tag)
   "Add tag to all messages in the current thread."
-- 
1.7.8.3

Re: [PATCH v4] emacs: Call "notmuch tag" once when applying tag changes to a thread.

[Dmitry Kurochkin]

On Wed,  8 Feb 2012 16:43:15 +0000, David Edmondson <dme@dme.org> wrote:
> Optimize thread tagging by combining all the tagging operations to a
> single "notmuch tag" call.
> 
> For threads in the order of tens or a hundred inbox tagged messages,
> this gives a noticeable speedup. On two different machines, archiving
> a thread of about 50 inbox tagged messages goes down from 10+ seconds
> to about 0.5 seconds.
> 
> The bottleneck is not within emacs; the same behaviour can be observed
> in the CLI. This approach has the added benefit of being more
> reliable: any of the individual tagging operations might face a locked
> database, leading to partial results.
> 
> This introduces a limitation to the number of messages that can be
> archived at the same time (through ARG_MAX limiting the command
> line). While at least on Linux this seems more like a theoretical
> limitation than a real one, it could be avoided by archiving at most a
> few hundred messages at a time.
> 
> Based on code from Jani Nikula <jani@nikula.org>.
> ---
> 
> v4:
> - Rebased after Dmitry's tagging changes (yay!).
> 

Now, that my tagging changes are pushed, the same effect can be (and
should be) achieved by some code cleanup and function reuse.

This patch is obsoleted by [1].

Regards,
  Dmitry

[1] id:"1328719731-13402-1-git-send-email-dmitry.kurochkin@gmail.com"

>  emacs/notmuch-show.el |   18 +++++++++++-------
>  1 files changed, 11 insertions(+), 7 deletions(-)
> 
> diff --git a/emacs/notmuch-show.el b/emacs/notmuch-show.el
> index faa9f9b..9294bc8 100644
> --- a/emacs/notmuch-show.el
> +++ b/emacs/notmuch-show.el
> @@ -1599,14 +1599,18 @@ argument, hide all of the messages."
>    (backward-button 1))
>  
>  (defun notmuch-show-tag-thread-internal (tag &optional remove)
> -  "Add tag to the current set of messages.
> +  "Add TAG to the current set of messages.
>  
> -If the remove switch is given, tags will be removed instead of
> -added."
> -  (goto-char (point-min))
> -  (let ((op (if remove "-" "+")))
> -    (loop do (notmuch-show-tag-message (concat op tag))
> -	  until (not (notmuch-show-goto-message-next)))))
> +If REMOVE is non-nil, TAG will be removed rather than added."
> +  (save-excursion
> +    (goto-char (point-min))
> +    (let ((tag-op (concat (if remove "-" "+") tag))
> +	  (message-ids
> +	   (loop collect (notmuch-show-get-message-id)
> +		 until (not (notmuch-show-goto-message-next)))))
> +      (if message-ids
> +	  (notmuch-tag (mapconcat #'identity message-ids " OR ")
> +		       tag-op)))))
>  
>  (defun notmuch-show-add-tag-thread (tag)
>    "Add tag to all messages in the current thread."
> -- 
> 1.7.8.3
> 
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch