provide various lazy-show options

provide various lazy-show options

[David Bremner]

It is a bit of an unfortunate profusion of options, but it isn't
really clear to me yet if there is one best heuristic for all
use-cases. The overall observation is that for most threads (even up
to several hundred messages), the time to return the query from the
notmuch CLI is much smaller than that to render it in emacs. This
series expands upon some existing ideas of not rendering large text/*
attachments to "lazily" render message bodies for messages satisfying
certain criteria. In one of the introduced test cases this yields
about a 10x speedup. Of course the comparison is not completly fair as
the user probably has to do more work in opening messages to read
them.

I can imagine a couple of different UI improvements. One would be just
choosing some better default values for the parameters discussed in
this series. Another would be to provide a nice ui for temporarily
changing them and re-rendering (perhaps analogous to the limit
commands already present).

[PATCH 1/5] perf-test/emacs: test showing longish threads

[David Bremner]

This particular thread takes about 100 times longer to display in
emacs than on the command line.

The parameter notmuch-show-max-text-part-size exists, but is currently
ineffective for this task because it only hides attachments, not
part 0.

In future commits, the parameter notmuch-show-{depth,height}-limit
will trigger a similar kind of hiding for bodies as
n-s-m-text-part-size already does for attachments.
---
 performance-test/T06-emacs.sh | 63 +++++++++++++++++++++++++++++++++++
 1 file changed, 63 insertions(+)

diff --git a/performance-test/T06-emacs.sh b/performance-test/T06-emacs.sh
index 66f0be58..c92bbd66 100755
--- a/performance-test/T06-emacs.sh
+++ b/performance-test/T06-emacs.sh
@@ -18,4 +18,67 @@ time_emacs "tag messages" \
    (notmuch-tag msg (list \"+test\"))
    (notmuch-tag msg (list \"-test\"))))"
 
+time_emacs "show warmup" \
+	   '(notmuch-show "thread:{id:tip-4f8219875a0dad2cfad9e93a3fafcd9626db98d2@git.kernel.org}")'
+
+time_emacs "show thread #1" \
+	   '(notmuch-show "thread:{id:tip-4f8219875a0dad2cfad9e93a3fafcd9626db98d2@git.kernel.org}")'
+
+time_emacs "depth bound #1" \
+	   '(let ((notmuch-show-depth-limit 0))
+		(notmuch-show "thread:{id:tip-4f8219875a0dad2cfad9e93a3fafcd9626db98d2@git.kernel.org}"))'
+
+time_emacs "height bound #1" \
+	   '(let ((notmuch-show-height-limit -1))
+		(notmuch-show "thread:{id:tip-4f8219875a0dad2cfad9e93a3fafcd9626db98d2@git.kernel.org}"))'
+
+time_emacs "size bound #1" \
+	   '(let ((notmuch-show-max-text-part-size 1))
+		(notmuch-show "thread:{id:tip-4f8219875a0dad2cfad9e93a3fafcd9626db98d2@git.kernel.org}"))'
+
+time_emacs "show thread #2" \
+	   '(notmuch-show "thread:{id:20101208005731.943729010@clark.site}")'
+
+time_emacs "depth bound #2" \
+	   '(let ((notmuch-show-depth-limit 0))
+		(notmuch-show "thread:{id:20101208005731.943729010@clark.site}"))'
+
+time_emacs "height bound #2" \
+	   '(let ((notmuch-show-height-limit -1))
+		(notmuch-show "thread:{id:20101208005731.943729010@clark.site}"))'
+
+time_emacs "size bound #2" \
+	   '(let ((notmuch-show-max-text-part-size 1))
+		(notmuch-show "thread:{id:20101208005731.943729010@clark.site}"))'
+
+time_emacs "show thread #3" \
+	   '(notmuch-show "thread:{id:20120109014938.GE20796@mit.edu}")'
+
+time_emacs "depth bound #3" \
+	   '(let ((notmuch-show-depth-limit 0))
+		(notmuch-show "thread:{id:20120109014938.GE20796@mit.edu}"))'
+
+time_emacs "height bound #3" \
+	   '(let ((notmuch-show-height-limit -1))
+		(notmuch-show "thread:{id:20120109014938.GE20796@mit.edu}"))'
+
+time_emacs "size bound #3" \
+	   '(let ((notmuch-show-max-text-part-size 1))
+		(notmuch-show "thread:{id:20120109014938.GE20796@mit.edu}"))'
+
+time_emacs "show thread #4" \
+	   '(notmuch-show "thread:{id:1280704593.25620.48.camel@mulgrave.site}")'
+
+time_emacs "depth bound #4" \
+	   '(let ((notmuch-show-depth-limit 0))
+		(notmuch-show "thread:{id:1280704593.25620.48.camel@mulgrave.site}"))'
+
+time_emacs "height bound #4" \
+	   '(let ((notmuch-show-height-limit -1))
+		(notmuch-show "thread:{id:1280704593.25620.48.camel@mulgrave.site}"))'
+
+time_emacs "size bound #4" \
+	   '(let ((notmuch-show-max-text-part-size 1))
+		(notmuch-show "thread:{id:1280704593.25620.48.camel@mulgrave.site}"))'
+
 time_done
-- 
2.35.2

[PATCH 2/5] emacs/show: optionally insert bodies of "deep messages" lazily

[David Bremner]

This leverages the machinery already there to insert buttons for
attachments.

A potential use-case is browsing the top layers of the tree to decide
which of the lower subtrees to read.
---
 doc/notmuch-emacs.rst                         |   3 +
 emacs/notmuch-show.el                         |  21 +++-
 test/T450-emacs-show.sh                       |  18 +++
 .../notmuch-show-depth                        |  44 +++++++
 .../notmuch-show-depth-1                      | 119 ++++++++++++++++++
 5 files changed, 203 insertions(+), 2 deletions(-)
 create mode 100644 test/emacs-show.expected-output/notmuch-show-depth
 create mode 100644 test/emacs-show.expected-output/notmuch-show-depth-1

diff --git a/doc/notmuch-emacs.rst b/doc/notmuch-emacs.rst
index 970cd7b7..c2d16c5f 100644
--- a/doc/notmuch-emacs.rst
+++ b/doc/notmuch-emacs.rst
@@ -277,6 +277,9 @@ Display of messages can be controlled by the following variables
 :index:`notmuch-show-header-line`
        |docstring::notmuch-show-header-line|
 
+:index:`notmuch-show-depth-limit`
+       |docstring::notmuch-show-depth-limit|
+
 :index:`notmuch-multipart/alternative-discouraged`
 
    Which mime types to hide by default for multipart messages.
diff --git a/emacs/notmuch-show.el b/emacs/notmuch-show.el
index aded1ee7..bac259f2 100644
--- a/emacs/notmuch-show.el
+++ b/emacs/notmuch-show.el
@@ -111,6 +111,19 @@ displayed."
 		 (function :tag "Function"))
   :group 'notmuch-show)
 
+(defcustom notmuch-show-depth-limit nil
+  "Depth beyond which message bodies are displayed lazily.
+
+If bound to an integer, any message with tree depth greater than
+this will have its body display lazily, initially
+inserting only a button.
+
+If this variable is set to nil (the default) no such lazy
+insertion is done."
+    :type '(choice (const :tag "No limit" nil)
+                   (number :tag "Limit" 10))
+    :group 'notmuch-show)
+
 (defcustom notmuch-show-relative-dates t
   "Display relative dates in the message summary line."
   :type 'boolean
@@ -1031,17 +1044,21 @@ is t, hide the part initially and show the button."
 		    (> notmuch-show-max-text-part-size 0)
 		    (> (length (plist-get part :content))
 		       notmuch-show-max-text-part-size)))
+	 (deep (and notmuch-show-depth-limit
+		    (> depth notmuch-show-depth-limit)))
 	 (beg (point))
 	 ;; This default header-p function omits the part button for
 	 ;; the first (or only) part if this is text/plain.
-	 (button (and (funcall notmuch-show-insert-header-p-function part hide)
+	 (button (and (or deep
+			  (funcall notmuch-show-insert-header-p-function part hide))
 		      (notmuch-show-insert-part-header
 		       nth mime-type
 		       (and content-type (downcase content-type))
 		       (plist-get part :filename))))
-	 ;; Hide the part initially if HIDE is t, or if it is too long
+	 ;; Hide the part initially if HIDE is t, or if it is too long/deep
 	 ;; and we have a button to allow toggling.
 	 (show-part (not (or (equal hide t)
+			     (and deep button)
 			     (and long button))))
 	 (content-beg (point)))
     ;; Store the computed mime-type for later use (e.g. by attachment handlers).
diff --git a/test/T450-emacs-show.sh b/test/T450-emacs-show.sh
index 0c1f6204..0749dc9d 100755
--- a/test/T450-emacs-show.sh
+++ b/test/T450-emacs-show.sh
@@ -91,6 +91,24 @@ test_emacs '(let ((notmuch-show-only-matching-messages t))
 	(test-visible-output))'
 test_expect_equal_file $EXPECTED/notmuch-show-elide-non-matching-messages-on OUTPUT
 
+test_begin_subtest "Hide bodies of messages by depth"
+test_emacs '(let ((notmuch-show-depth-limit -1))
+	(notmuch-search "thread:{id:87ocn0qh6d.fsf@yoom.home.cworth.org}")
+	(notmuch-test-wait)
+	(notmuch-search-show-thread)
+	(notmuch-test-wait)
+	(test-visible-output))'
+test_expect_equal_file $EXPECTED/notmuch-show-depth OUTPUT
+
+test_begin_subtest "Hide bodies of messages (depth > 1)"
+test_emacs '(let ((notmuch-show-depth-limit 1))
+	(notmuch-search "thread:{id:87ocn0qh6d.fsf@yoom.home.cworth.org}")
+	(notmuch-test-wait)
+	(notmuch-search-show-thread)
+	(notmuch-test-wait)
+	(test-visible-output))'
+test_expect_equal_file $EXPECTED/notmuch-show-depth-1 OUTPUT
+
 test_begin_subtest "notmuch-show: elide non-matching messages (w/ notmuch-show-toggle-elide-non-matching)"
 test_emacs '(let ((notmuch-show-only-matching-messages nil))
 	(notmuch-search "from:lars@seas.harvard.edu and subject:\"Maildir storage\"")
diff --git a/test/emacs-show.expected-output/notmuch-show-depth b/test/emacs-show.expected-output/notmuch-show-depth
new file mode 100644
index 00000000..8299519c
--- /dev/null
+++ b/test/emacs-show.expected-output/notmuch-show-depth
@@ -0,0 +1,44 @@
+Lars Kellogg-Stedman <lars@seas.harvard.edu> (2009-11-17) (inbox signed)
+Subject: [notmuch] Working with Maildir storage?
+To: notmuch@notmuchmail.org
+Date: Tue, 17 Nov 2009 14:00:54 -0500
+
+[ multipart/mixed (hidden) ]
+ Mikhail Gusarov <dottedmag@dottedmag.net> (2009-11-17) (inbox signed unread)
+ Subject: Re: [notmuch] Working with Maildir storage?
+ To: notmuch@notmuchmail.org
+ Date: Wed, 18 Nov 2009 01:02:38 +0600
+
+ [ multipart/mixed (hidden) ]
+  Lars Kellogg-Stedman <lars@seas.harvard.edu> (2009-11-17) (inbox signed)
+  Subject: Re: [notmuch] Working with Maildir storage?
+  To: Mikhail Gusarov <dottedmag@dottedmag.net>
+  Cc: notmuch@notmuchmail.org
+  Date: Tue, 17 Nov 2009 15:33:01 -0500
+
+  [ multipart/mixed (hidden) ]
+   Mikhail Gusarov <dottedmag@dottedmag.net> (2009-11-17) (inbox unread)
+   Subject: [notmuch] Working with Maildir storage?
+   To: notmuch@notmuchmail.org
+   Date: Wed, 18 Nov 2009 02:50:48 +0600
+
+   [ text/plain (hidden) ]
+   Keith Packard <keithp@keithp.com> (2009-11-17) (inbox unread)
+   Subject: [notmuch] Working with Maildir storage?
+   To: notmuch@notmuchmail.org
+   Date: Tue, 17 Nov 2009 13:24:13 -0800
+
+   [ text/plain (hidden) ]
+    Lars Kellogg-Stedman <lars@seas.harvard.edu> (2009-11-18) (inbox signed unread)
+    Subject: Re: [notmuch] Working with Maildir storage?
+    To: Keith Packard <keithp@keithp.com>
+    Cc: notmuch@notmuchmail.org
+    Date: Tue, 17 Nov 2009 19:50:40 -0500
+
+    [ multipart/mixed (hidden) ]
+ Carl Worth <cworth@cworth.org> (2009-11-18) (inbox unread)
+ Subject: [notmuch] Working with Maildir storage?
+ To: notmuch@notmuchmail.org
+ Date: Wed, 18 Nov 2009 02:08:10 -0800
+
+ [ text/plain (hidden) ]
diff --git a/test/emacs-show.expected-output/notmuch-show-depth-1 b/test/emacs-show.expected-output/notmuch-show-depth-1
new file mode 100644
index 00000000..e7c376bb
--- /dev/null
+++ b/test/emacs-show.expected-output/notmuch-show-depth-1
@@ -0,0 +1,119 @@
+Lars Kellogg-Stedman <lars@seas.harvard.edu> (2009-11-17) (inbox signed)
+Subject: [notmuch] Working with Maildir storage?
+To: notmuch@notmuchmail.org
+Date: Tue, 17 Nov 2009 14:00:54 -0500
+
+[ multipart/mixed ]
+[ multipart/signed ]
+[ Unknown key ID 0xD74695063141ACD8 or unsupported algorithm ]
+[ text/plain ]
+I saw the LWN article and decided to take a look at notmuch.  I'm
+currently using mutt and mairix to index and read a collection of
+Maildir mail folders (around 40,000 messages total).
+
+notmuch indexed the messages without complaint, but my attempt at
+searching bombed out. Running, for example:
+
+  notmuch search storage
+
+Resulted in 4604 lines of errors along the lines of:
+
+  Error opening
+  /home/lars/Mail/read-messages.2008/cur/1246413773.24928_27334.hostname,U=3026:2,S:
+  Too many open files
+
+I'm curious if this is expected behavior (i.e., notmuch does not work
+with Maildir) or if something else is going on.
+
+Cheers,
+
+[ 4-line signature. Click/Enter to show. ]
+[ application/pgp-signature ]
+[ text/plain ]
+[ 4-line signature. Click/Enter to show. ]
+ Mikhail Gusarov <dottedmag@dottedmag.net> (2009-11-17) (inbox signed unread)
+ Subject: Re: [notmuch] Working with Maildir storage?
+ To: notmuch@notmuchmail.org
+ Date: Wed, 18 Nov 2009 01:02:38 +0600
+
+ [ multipart/mixed ]
+ [ multipart/signed ]
+ [ Unknown key ID 0x9D20F6503E338888 or unsupported algorithm ]
+ [ text/plain ]
+
+ Twas brillig at 14:00:54 17.11.2009 UTC-05 when lars@seas.harvard.edu did
+ gyre and gimble:
+
+  LK> Resulted in 4604 lines of errors along the lines of:
+
+  LK>   Error opening
+  LK>  
+ /home/lars/Mail/read-messages.2008/cur/1246413773.24928_27334.hostname,U=3026:2,S:
+  LK>   Too many open files
+
+ See the patch just posted here.
+
+ [ 2-line signature. Click/Enter to show. ]
+ [ application/pgp-signature ]
+ [ text/plain ]
+ [ 4-line signature. Click/Enter to show. ]
+  Lars Kellogg-Stedman <lars@seas.harvard.edu> (2009-11-17) (inbox signed)
+  Subject: Re: [notmuch] Working with Maildir storage?
+  To: Mikhail Gusarov <dottedmag@dottedmag.net>
+  Cc: notmuch@notmuchmail.org
+  Date: Tue, 17 Nov 2009 15:33:01 -0500
+
+  [ multipart/mixed (hidden) ]
+   Mikhail Gusarov <dottedmag@dottedmag.net> (2009-11-17) (inbox unread)
+   Subject: [notmuch] Working with Maildir storage?
+   To: notmuch@notmuchmail.org
+   Date: Wed, 18 Nov 2009 02:50:48 +0600
+
+   [ text/plain (hidden) ]
+   Keith Packard <keithp@keithp.com> (2009-11-17) (inbox unread)
+   Subject: [notmuch] Working with Maildir storage?
+   To: notmuch@notmuchmail.org
+   Date: Tue, 17 Nov 2009 13:24:13 -0800
+
+   [ text/plain (hidden) ]
+    Lars Kellogg-Stedman <lars@seas.harvard.edu> (2009-11-18) (inbox signed unread)
+    Subject: Re: [notmuch] Working with Maildir storage?
+    To: Keith Packard <keithp@keithp.com>
+    Cc: notmuch@notmuchmail.org
+    Date: Tue, 17 Nov 2009 19:50:40 -0500
+
+    [ multipart/mixed (hidden) ]
+ Carl Worth <cworth@cworth.org> (2009-11-18) (inbox unread)
+ Subject: [notmuch] Working with Maildir storage?
+ To: notmuch@notmuchmail.org
+ Date: Wed, 18 Nov 2009 02:08:10 -0800
+
+ On Tue, 17 Nov 2009 14:00:54 -0500, Lars Kellogg-Stedman <lars at
+ seas.harvard.edu> wrote:
+ > I saw the LWN article and decided to take a look at notmuch.  I'm
+ > currently using mutt and mairix to index and read a collection of
+ > Maildir mail folders (around 40,000 messages total).
+
+ Welcome, Lars!
+
+ I hadn't even seen that Keith's blog post had been picked up by lwn.net.
+ That's very interesting. So, thanks for coming and trying out notmuch.
+
+ >   Error opening
+ > /home/lars/Mail/read-messages.2008/cur/1246413773.24928_27334.hostname,U=3026:2,S:
+ >   Too many open files
+
+ Sadly, the lwn article coincided with me having just introduced this
+ bug, and then getting on a Trans-Atlantic flight. So I fixed the bug
+ fairly quickly, but there was quite a bit of latency before I could push
+ the fix out. It should be fixed now.
+
+ > I'm curious if this is expected behavior (i.e., notmuch does not work
+ > with Maildir) or if something else is going on.
+
+ Notmuch works just fine with maildir---it's one of the things that it
+ likes the best.
+
+ Happy hacking,
+
+ -Carl
-- 
2.35.2

[PATCH 3/5] emacs/show: obey notmuch-show-max-text-part-size for part 0

[David Bremner]

This parameter was originally introduced to hide large attachements
that happened to be text/plain. From a performance point of view,
there is no reason not to also hide large message bodies.
---
 doc/notmuch-emacs.rst                         |  3 +
 emacs/notmuch-show.el                         |  2 +-
 test/T450-emacs-show.sh                       | 18 ++++
 .../notmuch-show-size                         | 64 +++++++++++++
 .../notmuch-show-size-450                     | 89 +++++++++++++++++++
 5 files changed, 175 insertions(+), 1 deletion(-)
 create mode 100644 test/emacs-show.expected-output/notmuch-show-size
 create mode 100644 test/emacs-show.expected-output/notmuch-show-size-450

diff --git a/doc/notmuch-emacs.rst b/doc/notmuch-emacs.rst
index c2d16c5f..d6fa37e0 100644
--- a/doc/notmuch-emacs.rst
+++ b/doc/notmuch-emacs.rst
@@ -280,6 +280,9 @@ Display of messages can be controlled by the following variables
 :index:`notmuch-show-depth-limit`
        |docstring::notmuch-show-depth-limit|
 
+:index:`notmuch-show-max-text-part-size`
+       |docstring::notmuch-show-max-text-part-size|
+
 :index:`notmuch-multipart/alternative-discouraged`
 
    Which mime types to hide by default for multipart messages.
diff --git a/emacs/notmuch-show.el b/emacs/notmuch-show.el
index bac259f2..a25ea061 100644
--- a/emacs/notmuch-show.el
+++ b/emacs/notmuch-show.el
@@ -1049,7 +1049,7 @@ is t, hide the part initially and show the button."
 	 (beg (point))
 	 ;; This default header-p function omits the part button for
 	 ;; the first (or only) part if this is text/plain.
-	 (button (and (or deep
+	 (button (and (or deep long
 			  (funcall notmuch-show-insert-header-p-function part hide))
 		      (notmuch-show-insert-part-header
 		       nth mime-type
diff --git a/test/T450-emacs-show.sh b/test/T450-emacs-show.sh
index 0749dc9d..e9c59cd3 100755
--- a/test/T450-emacs-show.sh
+++ b/test/T450-emacs-show.sh
@@ -109,6 +109,24 @@ test_emacs '(let ((notmuch-show-depth-limit 1))
 	(test-visible-output))'
 test_expect_equal_file $EXPECTED/notmuch-show-depth-1 OUTPUT
 
+test_begin_subtest "Hide bodies of messages by size"
+test_emacs '(let ((notmuch-show-max-text-part-size 1))
+	(notmuch-search "thread:{id:87ocn0qh6d.fsf@yoom.home.cworth.org}")
+	(notmuch-test-wait)
+	(notmuch-search-show-thread)
+	(notmuch-test-wait)
+	(test-visible-output))'
+test_expect_equal_file $EXPECTED/notmuch-show-size OUTPUT
+
+test_begin_subtest "Hide bodies of messages by size > 450"
+test_emacs '(let ((notmuch-show-max-text-part-size 450))
+	(notmuch-search "thread:{id:87ocn0qh6d.fsf@yoom.home.cworth.org}")
+	(notmuch-test-wait)
+	(notmuch-search-show-thread)
+	(notmuch-test-wait)
+	(test-visible-output))'
+test_expect_equal_file $EXPECTED/notmuch-show-size-450 OUTPUT
+
 test_begin_subtest "notmuch-show: elide non-matching messages (w/ notmuch-show-toggle-elide-non-matching)"
 test_emacs '(let ((notmuch-show-only-matching-messages nil))
 	(notmuch-search "from:lars@seas.harvard.edu and subject:\"Maildir storage\"")
diff --git a/test/emacs-show.expected-output/notmuch-show-size b/test/emacs-show.expected-output/notmuch-show-size
new file mode 100644
index 00000000..cdde467e
--- /dev/null
+++ b/test/emacs-show.expected-output/notmuch-show-size
@@ -0,0 +1,64 @@
+Lars Kellogg-Stedman <lars@seas.harvard.edu> (2009-11-17) (inbox signed)
+Subject: [notmuch] Working with Maildir storage?
+To: notmuch@notmuchmail.org
+Date: Tue, 17 Nov 2009 14:00:54 -0500
+
+[ multipart/mixed ]
+[ multipart/signed ]
+[ Unknown key ID 0xD74695063141ACD8 or unsupported algorithm ]
+[ text/plain (hidden) ]
+[ application/pgp-signature ]
+[ text/plain (hidden) ]
+ Mikhail Gusarov <dottedmag@dottedmag.net> (2009-11-17) (inbox signed unread)
+ Subject: Re: [notmuch] Working with Maildir storage?
+ To: notmuch@notmuchmail.org
+ Date: Wed, 18 Nov 2009 01:02:38 +0600
+
+ [ multipart/mixed ]
+ [ multipart/signed ]
+ [ Unknown key ID 0x9D20F6503E338888 or unsupported algorithm ]
+ [ text/plain (hidden) ]
+ [ application/pgp-signature ]
+ [ text/plain (hidden) ]
+  Lars Kellogg-Stedman <lars@seas.harvard.edu> (2009-11-17) (inbox signed)
+  Subject: Re: [notmuch] Working with Maildir storage?
+  To: Mikhail Gusarov <dottedmag@dottedmag.net>
+  Cc: notmuch@notmuchmail.org
+  Date: Tue, 17 Nov 2009 15:33:01 -0500
+
+  [ multipart/mixed ]
+  [ multipart/signed ]
+  [ Unknown key ID 0xD74695063141ACD8 or unsupported algorithm ]
+  [ text/plain (hidden) ]
+  [ application/pgp-signature ]
+  [ text/plain (hidden) ]
+   Mikhail Gusarov <dottedmag@dottedmag.net> (2009-11-17) (inbox unread)
+   Subject: [notmuch] Working with Maildir storage?
+   To: notmuch@notmuchmail.org
+   Date: Wed, 18 Nov 2009 02:50:48 +0600
+
+   [ text/plain (hidden) ]
+   Keith Packard <keithp@keithp.com> (2009-11-17) (inbox unread)
+   Subject: [notmuch] Working with Maildir storage?
+   To: notmuch@notmuchmail.org
+   Date: Tue, 17 Nov 2009 13:24:13 -0800
+
+   [ text/plain (hidden) ]
+    Lars Kellogg-Stedman <lars@seas.harvard.edu> (2009-11-18) (inbox signed unread)
+    Subject: Re: [notmuch] Working with Maildir storage?
+    To: Keith Packard <keithp@keithp.com>
+    Cc: notmuch@notmuchmail.org
+    Date: Tue, 17 Nov 2009 19:50:40 -0500
+
+    [ multipart/mixed ]
+    [ multipart/signed ]
+    [ Unknown key ID 0xD74695063141ACD8 or unsupported algorithm ]
+    [ text/plain (hidden) ]
+    [ application/pgp-signature ]
+    [ text/plain (hidden) ]
+ Carl Worth <cworth@cworth.org> (2009-11-18) (inbox unread)
+ Subject: [notmuch] Working with Maildir storage?
+ To: notmuch@notmuchmail.org
+ Date: Wed, 18 Nov 2009 02:08:10 -0800
+
+ [ text/plain (hidden) ]
diff --git a/test/emacs-show.expected-output/notmuch-show-size-450 b/test/emacs-show.expected-output/notmuch-show-size-450
new file mode 100644
index 00000000..ec34612e
--- /dev/null
+++ b/test/emacs-show.expected-output/notmuch-show-size-450
@@ -0,0 +1,89 @@
+Lars Kellogg-Stedman <lars@seas.harvard.edu> (2009-11-17) (inbox signed)
+Subject: [notmuch] Working with Maildir storage?
+To: notmuch@notmuchmail.org
+Date: Tue, 17 Nov 2009 14:00:54 -0500
+
+[ multipart/mixed ]
+[ multipart/signed ]
+[ Unknown key ID 0xD74695063141ACD8 or unsupported algorithm ]
+[ text/plain (hidden) ]
+[ application/pgp-signature ]
+[ text/plain ]
+[ 4-line signature. Click/Enter to show. ]
+ Mikhail Gusarov <dottedmag@dottedmag.net> (2009-11-17) (inbox signed unread)
+ Subject: Re: [notmuch] Working with Maildir storage?
+ To: notmuch@notmuchmail.org
+ Date: Wed, 18 Nov 2009 01:02:38 +0600
+
+ [ multipart/mixed ]
+ [ multipart/signed ]
+ [ Unknown key ID 0x9D20F6503E338888 or unsupported algorithm ]
+ [ text/plain ]
+
+ Twas brillig at 14:00:54 17.11.2009 UTC-05 when lars@seas.harvard.edu did
+ gyre and gimble:
+
+  LK> Resulted in 4604 lines of errors along the lines of:
+
+  LK>   Error opening
+  LK>  
+ /home/lars/Mail/read-messages.2008/cur/1246413773.24928_27334.hostname,U=3026:2,S:
+  LK>   Too many open files
+
+ See the patch just posted here.
+
+ [ 2-line signature. Click/Enter to show. ]
+ [ application/pgp-signature ]
+ [ text/plain ]
+ [ 4-line signature. Click/Enter to show. ]
+  Lars Kellogg-Stedman <lars@seas.harvard.edu> (2009-11-17) (inbox signed)
+  Subject: Re: [notmuch] Working with Maildir storage?
+  To: Mikhail Gusarov <dottedmag@dottedmag.net>
+  Cc: notmuch@notmuchmail.org
+  Date: Tue, 17 Nov 2009 15:33:01 -0500
+
+  [ multipart/mixed ]
+  [ multipart/signed ]
+  [ Unknown key ID 0xD74695063141ACD8 or unsupported algorithm ]
+  [ text/plain (hidden) ]
+  [ application/pgp-signature ]
+  [ text/plain ]
+  [ 4-line signature. Click/Enter to show. ]
+   Mikhail Gusarov <dottedmag@dottedmag.net> (2009-11-17) (inbox unread)
+   Subject: [notmuch] Working with Maildir storage?
+   To: notmuch@notmuchmail.org
+   Date: Wed, 18 Nov 2009 02:50:48 +0600
+
+   [ text/plain (hidden) ]
+   Keith Packard <keithp@keithp.com> (2009-11-17) (inbox unread)
+   Subject: [notmuch] Working with Maildir storage?
+   To: notmuch@notmuchmail.org
+   Date: Tue, 17 Nov 2009 13:24:13 -0800
+
+   [ text/plain (hidden) ]
+    Lars Kellogg-Stedman <lars@seas.harvard.edu> (2009-11-18) (inbox signed unread)
+    Subject: Re: [notmuch] Working with Maildir storage?
+    To: Keith Packard <keithp@keithp.com>
+    Cc: notmuch@notmuchmail.org
+    Date: Tue, 17 Nov 2009 19:50:40 -0500
+
+    [ multipart/mixed ]
+    [ multipart/signed ]
+    [ Unknown key ID 0xD74695063141ACD8 or unsupported algorithm ]
+    [ text/plain ]
+    > I've also pushed a slightly more complicated (and complete) fix to my
+    > private notmuch repository
+
+    The version of lib/messages.cc in your repo doesn't build because it's
+    missing "#include <stdint.h>" (for the uint32_t on line 466).
+
+    [ 4-line signature. Click/Enter to show. ]
+    [ application/pgp-signature ]
+    [ text/plain ]
+    [ 4-line signature. Click/Enter to show. ]
+ Carl Worth <cworth@cworth.org> (2009-11-18) (inbox unread)
+ Subject: [notmuch] Working with Maildir storage?
+ To: notmuch@notmuchmail.org
+ Date: Wed, 18 Nov 2009 02:08:10 -0800
+
+ [ text/plain (hidden) ]
-- 
2.35.2

[PATCH 4/5] emacs/show: introduce notmuch-show-height-limit

[David Bremner]

Analogous to notmuch-show-depth-limit, it requires a bit more
calculation, but that cost seems dominated by the rendering cost.
---
 doc/notmuch-emacs.rst                         |  3 +
 emacs/notmuch-show.el                         | 32 +++++-
 test/T450-emacs-show.sh                       | 20 ++++
 .../notmuch-show-height-0                     | 97 +++++++++++++++++++
 4 files changed, 151 insertions(+), 1 deletion(-)
 create mode 100644 test/emacs-show.expected-output/notmuch-show-height-0

diff --git a/doc/notmuch-emacs.rst b/doc/notmuch-emacs.rst
index d6fa37e0..be7e324d 100644
--- a/doc/notmuch-emacs.rst
+++ b/doc/notmuch-emacs.rst
@@ -280,6 +280,9 @@ Display of messages can be controlled by the following variables
 :index:`notmuch-show-depth-limit`
        |docstring::notmuch-show-depth-limit|
 
+:index:`notmuch-show-height-limit`
+       |docstring::notmuch-show-height-limit|
+
 :index:`notmuch-show-max-text-part-size`
        |docstring::notmuch-show-max-text-part-size|
 
diff --git a/emacs/notmuch-show.el b/emacs/notmuch-show.el
index a25ea061..7d6c9c1f 100644
--- a/emacs/notmuch-show.el
+++ b/emacs/notmuch-show.el
@@ -124,6 +124,19 @@ insertion is done."
                    (number :tag "Limit" 10))
     :group 'notmuch-show)
 
+(defcustom notmuch-show-height-limit nil
+  "Height (from leaves) beyond which message bodies are displayed lazily.
+
+If bound to an integer, any message with height in the message
+tree greater than this will have its body displayed lazily,
+initially only a button.
+
+If this variable is set to nil (the default) no such lazy
+display is done."
+    :type '(choice (const :tag "No limit" nil)
+                   (number :tag "Limit" 10))
+    :group 'notmuch-show)
+
 (defcustom notmuch-show-relative-dates t
   "Display relative dates in the message summary line."
   :type 'boolean
@@ -506,6 +519,18 @@ Return unchanged ADDRESS if parsing fails."
       ;; Otherwise format the name and address together.
       (concat p-name " <" p-address ">"))))
 
+(defun notmuch-show--mark-height (tree)
+  "Calculate and cache height (distance from deepest descendent)"
+  (let* ((msg (car tree))
+	 (children (cadr tree))
+	 (cached-height (plist-get msg :height)))
+    (or cached-height
+	(let ((height
+	       (if (null children) 0
+		 (1+ (apply #'max (mapcar #'notmuch-show--mark-height children))))))
+	  (plist-put msg :height height)
+	  height))))
+
 (defun notmuch-show-insert-headerline (headers date tags depth)
   "Insert a notmuch style headerline based on HEADERS for a
 message at DEPTH in the current thread."
@@ -1040,16 +1065,19 @@ is t, hide the part initially and show the button."
   (let* ((content-type (plist-get part :content-type))
 	 (mime-type (notmuch-show-mime-type part))
 	 (nth (plist-get part :id))
+	 (height (plist-get msg :height))
 	 (long (and (notmuch-match-content-type mime-type "text/*")
 		    (> notmuch-show-max-text-part-size 0)
 		    (> (length (plist-get part :content))
 		       notmuch-show-max-text-part-size)))
 	 (deep (and notmuch-show-depth-limit
 		    (> depth notmuch-show-depth-limit)))
+	 (high (and notmuch-show-height-limit
+		    (> height notmuch-show-height-limit)))
 	 (beg (point))
 	 ;; This default header-p function omits the part button for
 	 ;; the first (or only) part if this is text/plain.
-	 (button (and (or deep long
+	 (button (and (or deep long high
 			  (funcall notmuch-show-insert-header-p-function part hide))
 		      (notmuch-show-insert-part-header
 		       nth mime-type
@@ -1059,6 +1087,7 @@ is t, hide the part initially and show the button."
 	 ;; and we have a button to allow toggling.
 	 (show-part (not (or (equal hide t)
 			     (and deep button)
+			     (and high button)
 			     (and long button))))
 	 (content-beg (point)))
     ;; Store the computed mime-type for later use (e.g. by attachment handlers).
@@ -1202,6 +1231,7 @@ is t, hide the part initially and show the button."
 	(replies (cadr tree)))
     ;; We test whether there is a message or just some replies.
     (when msg
+      (notmuch-show--mark-height tree)
       (notmuch-show-insert-msg msg depth))
     (notmuch-show-insert-thread replies (1+ depth))))
 
diff --git a/test/T450-emacs-show.sh b/test/T450-emacs-show.sh
index e9c59cd3..64f174cf 100755
--- a/test/T450-emacs-show.sh
+++ b/test/T450-emacs-show.sh
@@ -100,6 +100,26 @@ test_emacs '(let ((notmuch-show-depth-limit -1))
 	(test-visible-output))'
 test_expect_equal_file $EXPECTED/notmuch-show-depth OUTPUT
 
+
+test_begin_subtest "Hide bodies of messages by height"
+test_emacs '(let ((notmuch-show-height-limit -1))
+	(notmuch-search "thread:{id:87ocn0qh6d.fsf@yoom.home.cworth.org}")
+	(notmuch-test-wait)
+	(notmuch-search-show-thread)
+	(notmuch-test-wait)
+	(test-visible-output))'
+# folding all messages by height or depth should look the same
+test_expect_equal_file $EXPECTED/notmuch-show-depth OUTPUT
+
+test_begin_subtest "Hide bodies of messages; show only leaves."
+test_emacs '(let ((notmuch-show-height-limit 0))
+	(notmuch-search "thread:{id:87ocn0qh6d.fsf@yoom.home.cworth.org}")
+	(notmuch-test-wait)
+	(notmuch-search-show-thread)
+	(notmuch-test-wait)
+	(test-visible-output))'
+test_expect_equal_file $EXPECTED/notmuch-show-height-0 OUTPUT
+
 test_begin_subtest "Hide bodies of messages (depth > 1)"
 test_emacs '(let ((notmuch-show-depth-limit 1))
 	(notmuch-search "thread:{id:87ocn0qh6d.fsf@yoom.home.cworth.org}")
diff --git a/test/emacs-show.expected-output/notmuch-show-height-0 b/test/emacs-show.expected-output/notmuch-show-height-0
new file mode 100644
index 00000000..d646353e
--- /dev/null
+++ b/test/emacs-show.expected-output/notmuch-show-height-0
@@ -0,0 +1,97 @@
+Lars Kellogg-Stedman <lars@seas.harvard.edu> (2009-11-17) (inbox signed)
+Subject: [notmuch] Working with Maildir storage?
+To: notmuch@notmuchmail.org
+Date: Tue, 17 Nov 2009 14:00:54 -0500
+
+[ multipart/mixed (hidden) ]
+ Mikhail Gusarov <dottedmag@dottedmag.net> (2009-11-17) (inbox signed unread)
+ Subject: Re: [notmuch] Working with Maildir storage?
+ To: notmuch@notmuchmail.org
+ Date: Wed, 18 Nov 2009 01:02:38 +0600
+
+ [ multipart/mixed (hidden) ]
+  Lars Kellogg-Stedman <lars@seas.harvard.edu> (2009-11-17) (inbox signed)
+  Subject: Re: [notmuch] Working with Maildir storage?
+  To: Mikhail Gusarov <dottedmag@dottedmag.net>
+  Cc: notmuch@notmuchmail.org
+  Date: Tue, 17 Nov 2009 15:33:01 -0500
+
+  [ multipart/mixed (hidden) ]
+   Mikhail Gusarov <dottedmag@dottedmag.net> (2009-11-17) (inbox unread)
+   Subject: [notmuch] Working with Maildir storage?
+   To: notmuch@notmuchmail.org
+   Date: Wed, 18 Nov 2009 02:50:48 +0600
+
+   Twas brillig at 15:33:01 17.11.2009 UTC-05 when lars at seas.harvard.edu
+   did gyre and gimble:
+
+    LK> Is the list archived anywhere?  The obvious archives
+    LK> (http://notmuchmail.org/pipermail/notmuch/) aren't available, and I
+    LK> think I subscribed too late to get the patch (I only just saw the
+    LK> discussion about it).
+
+    LK> It doesn't look like the patch is in git yet.
+
+   Just has been pushed
+
+   [ 10-line signature. Click/Enter to show. ]
+   Keith Packard <keithp@keithp.com> (2009-11-17) (inbox unread)
+   Subject: [notmuch] Working with Maildir storage?
+   To: notmuch@notmuchmail.org
+   Date: Tue, 17 Nov 2009 13:24:13 -0800
+
+   [ text/plain (hidden) ]
+    Lars Kellogg-Stedman <lars@seas.harvard.edu> (2009-11-18) (inbox signed unread)
+    Subject: Re: [notmuch] Working with Maildir storage?
+    To: Keith Packard <keithp@keithp.com>
+    Cc: notmuch@notmuchmail.org
+    Date: Tue, 17 Nov 2009 19:50:40 -0500
+
+    [ multipart/mixed ]
+    [ multipart/signed ]
+    [ Unknown key ID 0xD74695063141ACD8 or unsupported algorithm ]
+    [ text/plain ]
+    > I've also pushed a slightly more complicated (and complete) fix to my
+    > private notmuch repository
+
+    The version of lib/messages.cc in your repo doesn't build because it's
+    missing "#include <stdint.h>" (for the uint32_t on line 466).
+
+    [ 4-line signature. Click/Enter to show. ]
+    [ application/pgp-signature ]
+    [ text/plain ]
+    [ 4-line signature. Click/Enter to show. ]
+ Carl Worth <cworth@cworth.org> (2009-11-18) (inbox unread)
+ Subject: [notmuch] Working with Maildir storage?
+ To: notmuch@notmuchmail.org
+ Date: Wed, 18 Nov 2009 02:08:10 -0800
+
+ On Tue, 17 Nov 2009 14:00:54 -0500, Lars Kellogg-Stedman <lars at
+ seas.harvard.edu> wrote:
+ > I saw the LWN article and decided to take a look at notmuch.  I'm
+ > currently using mutt and mairix to index and read a collection of
+ > Maildir mail folders (around 40,000 messages total).
+
+ Welcome, Lars!
+
+ I hadn't even seen that Keith's blog post had been picked up by lwn.net.
+ That's very interesting. So, thanks for coming and trying out notmuch.
+
+ >   Error opening
+ > /home/lars/Mail/read-messages.2008/cur/1246413773.24928_27334.hostname,U=3026:2,S:
+ >   Too many open files
+
+ Sadly, the lwn article coincided with me having just introduced this
+ bug, and then getting on a Trans-Atlantic flight. So I fixed the bug
+ fairly quickly, but there was quite a bit of latency before I could push
+ the fix out. It should be fixed now.
+
+ > I'm curious if this is expected behavior (i.e., notmuch does not work
+ > with Maildir) or if something else is going on.
+
+ Notmuch works just fine with maildir---it's one of the things that it
+ likes the best.
+
+ Happy hacking,
+
+ -Carl
-- 
2.35.2

[PATCH 5/5] doc/emacs: add discussion about notmuch-show-perfomance.

[David Bremner]

Most of the information is already in the docstrings, but hopefully
by grouping it is more discoverable.
---
 doc/notmuch-emacs.rst | 41 +++++++++++++++++++++++++++++++----------
 1 file changed, 31 insertions(+), 10 deletions(-)

diff --git a/doc/notmuch-emacs.rst b/doc/notmuch-emacs.rst
index be7e324d..f1f7d6a9 100644
--- a/doc/notmuch-emacs.rst
+++ b/doc/notmuch-emacs.rst
@@ -266,7 +266,7 @@ pressing RET after positioning the cursor on a hidden part.
 ``?``
     Display full set of key bindings
 
-Display of messages can be controlled by the following variables
+Display of messages can be controlled by the following variables; see also :ref:`show-large`.
 
 :index:`notmuch-message-headers`
        |docstring::notmuch-message-headers|
@@ -277,15 +277,6 @@ Display of messages can be controlled by the following variables
 :index:`notmuch-show-header-line`
        |docstring::notmuch-show-header-line|
 
-:index:`notmuch-show-depth-limit`
-       |docstring::notmuch-show-depth-limit|
-
-:index:`notmuch-show-height-limit`
-       |docstring::notmuch-show-height-limit|
-
-:index:`notmuch-show-max-text-part-size`
-       |docstring::notmuch-show-max-text-part-size|
-
 :index:`notmuch-multipart/alternative-discouraged`
 
    Which mime types to hide by default for multipart messages.
@@ -307,6 +298,36 @@ Display of messages can be controlled by the following variables
            (t
             (list "text/html" "multipart/related")))))
 
+.. _show-large:
+
+Dealing with large messages and threads
+---------------------------------------
+
+If you are finding :ref:`notmuch-show` is annoyingly slow displaying
+large messages, you can customize
+:index:`notmuch-show-max-text-part-size`.  If you want to speed up the
+display of large threads (with or without large messages), there are
+several options.  First, you can display the same query in one of the
+other modes. :ref:`notmuch-unthreaded` is the most robust for
+extremely large queries, but :ref:`notmuch-tree` is also be faster
+than :ref:`notmuch-show` in general, since it only renders a single
+message a time. If you prefer to stay with the rendered thread
+("conversation") view of :ref:`notmuch-show`, you can customize the
+variables :index:`notmuch-show-depth-limit`,
+:index:`notmuch-show-height-limit` and
+:index:`notmuch-show-max-text-part-size` to limit the amount of
+rendering done initially. Note that these limits are implicitly
+*AND*-ed together, and combinations might have surprising effects.
+
+:index:`notmuch-show-depth-limit`
+       |docstring::notmuch-show-depth-limit|
+
+:index:`notmuch-show-height-limit`
+       |docstring::notmuch-show-height-limit|
+
+:index:`notmuch-show-max-text-part-size`
+       |docstring::notmuch-show-max-text-part-size|
+
 .. _show-copy:
 
 Copy to kill-ring
-- 
2.35.2

Re: provide various lazy-show options

[David Bremner]

David Bremner <david@tethera.net> writes:

> It is a bit of an unfortunate profusion of options, but it isn't
> really clear to me yet if there is one best heuristic for all
> use-cases. The overall observation is that for most threads (even up
> to several hundred messages), the time to return the query from the
> notmuch CLI is much smaller than that to render it in emacs. This
> series expands upon some existing ideas of not rendering large text/*
> attachments to "lazily" render message bodies for messages satisfying
> certain criteria. In one of the introduced test cases this yields
> about a 10x speedup. Of course the comparison is not completly fair as
> the user probably has to do more work in opening messages to read
> them.

I have applied a slightly tweaked series to master

- one or two indentation fixes in the elisp
- fixed the subject on last patch
- changed AND to OR in the docs. The wording is still a bit imprecise,
  doc patches welcome!

d