v2 lazy body insertion

v2 lazy body insertion

[David Bremner]

This obsoletes the series at [1]. Compared to that series, this does
not introduce an extra layer of buttons, but just toggles more parts
closed.  It still seems to be a similar (maybe even better, which is
suprising) performance improvement.  While doing this revision I
realized that notmuch-show-max-text-part-size can play a similar
role. There are some details to check there, in particular the setting
of 0 to disable is unfortunate.

In some sense the heuristic of only (fully) displaying shallow
messages is backwards. Most likely the thing user wants to read are
things near the leaves of the discussion tree. That will require
either extending the notmuch-show CLI output (boo!) or a second
recursive traversal of every message in emacs. I'll have to see if the
latter is a performance problem.

This revision also adds a few more performance tests, to avoid
overfitting to the display of one thread.

[1]: id:20220605170851.1595387-1-david@tethera.net

[RFC PATCH v2 1/2] perf-test/emacs: test showing longish threads

[David Bremner]

This particular thread takes about 100 times longer to display in
emacs than on the command line.

In a future commit, the parameter notmuch-show-depth-limit will
trigger lazy display of message bodies, which should improve
performance (at the expense of extra clicks to view a message).
---
 performance-test/T06-emacs.sh | 31 +++++++++++++++++++++++++++++++
 1 file changed, 31 insertions(+)

diff --git a/performance-test/T06-emacs.sh b/performance-test/T06-emacs.sh
index 66f0be58..09131b7a 100755
--- a/performance-test/T06-emacs.sh
+++ b/performance-test/T06-emacs.sh
@@ -18,4 +18,35 @@ time_emacs "tag messages" \
    (notmuch-tag msg (list \"+test\"))
    (notmuch-tag msg (list \"-test\"))))"
 
+time_emacs "show warmup" \
+	   '(notmuch-show "thread:{id:tip-4f8219875a0dad2cfad9e93a3fafcd9626db98d2@git.kernel.org}")'
+
+time_emacs "show thread #1" \
+	   '(notmuch-show "thread:{id:tip-4f8219875a0dad2cfad9e93a3fafcd9626db98d2@git.kernel.org}")'
+
+time_emacs "lazy show #1" \
+	   '(let ((notmuch-show-depth-limit 0))
+		(notmuch-show "thread:{id:tip-4f8219875a0dad2cfad9e93a3fafcd9626db98d2@git.kernel.org}"))'
+
+time_emacs "show thread #2" \
+	   '(notmuch-show "thread:{id:20101208005731.943729010@clark.site}")'
+
+time_emacs "lazy show #2" \
+	   '(let ((notmuch-show-depth-limit 0))
+		(notmuch-show "thread:{id:20101208005731.943729010@clark.site}"))'
+
+time_emacs "show thread #3" \
+	   '(notmuch-show "thread:{id:20120109014938.GE20796@mit.edu}")'
+
+time_emacs "lazy show #3" \
+	   '(let ((notmuch-show-depth-limit 0))
+		(notmuch-show "thread:{id:20120109014938.GE20796@mit.edu}"))'
+
+time_emacs "show thread #4" \
+	   '(notmuch-show "thread:{id:1280704593.25620.48.camel@mulgrave.site}")'
+
+time_emacs "lazy show #4" \
+	   '(let ((notmuch-show-depth-limit 0))
+		(notmuch-show "thread:{id:1280704593.25620.48.camel@mulgrave.site}"))'
+
 time_done
-- 
2.35.2

[RFC PATCH v2 2/2] emacs/show: optionally insert bodies lazily

[David Bremner]

This leverages the machinery already there to insert buttons for attachments.
---
 doc/notmuch-emacs.rst                         |   3 +
 emacs/notmuch-show.el                         |  23 +++-
 test/T450-emacs-show.sh                       |  18 +++
 .../notmuch-show-depth                        |  44 +++++++
 .../notmuch-show-depth-1                      | 119 ++++++++++++++++++
 5 files changed, 204 insertions(+), 3 deletions(-)
 create mode 100644 test/emacs-show.expected-output/notmuch-show-depth
 create mode 100644 test/emacs-show.expected-output/notmuch-show-depth-1

diff --git a/doc/notmuch-emacs.rst b/doc/notmuch-emacs.rst
index 970cd7b7..c2d16c5f 100644
--- a/doc/notmuch-emacs.rst
+++ b/doc/notmuch-emacs.rst
@@ -277,6 +277,9 @@ Display of messages can be controlled by the following variables
 :index:`notmuch-show-header-line`
        |docstring::notmuch-show-header-line|
 
+:index:`notmuch-show-depth-limit`
+       |docstring::notmuch-show-depth-limit|
+
 :index:`notmuch-multipart/alternative-discouraged`
 
    Which mime types to hide by default for multipart messages.
diff --git a/emacs/notmuch-show.el b/emacs/notmuch-show.el
index aded1ee7..8a50fd54 100644
--- a/emacs/notmuch-show.el
+++ b/emacs/notmuch-show.el
@@ -111,6 +111,19 @@ displayed."
 		 (function :tag "Function"))
   :group 'notmuch-show)
 
+(defcustom notmuch-show-depth-limit nil
+  "Depth beyond which message bodies are inserted lazily.
+
+If bound to an integer, any message with tree depth greater than
+this insert will have its body inserted lazily, initially
+inserting only a button to do the actual insertion.
+
+If this variable is set to nil (the default) no such lazy
+insertion is done."
+    :type '(choice (const :tag "No limit" nil)
+                   (number :tag "Limit" 10))
+    :group 'notmuch-show)
+
 (defcustom notmuch-show-relative-dates t
   "Display relative dates in the message summary line."
   :type 'boolean
@@ -1031,18 +1044,22 @@ is t, hide the part initially and show the button."
 		    (> notmuch-show-max-text-part-size 0)
 		    (> (length (plist-get part :content))
 		       notmuch-show-max-text-part-size)))
+	 (deep (and notmuch-show-depth-limit
+		    (> depth notmuch-show-depth-limit)))
 	 (beg (point))
 	 ;; This default header-p function omits the part button for
 	 ;; the first (or only) part if this is text/plain.
-	 (button (and (funcall notmuch-show-insert-header-p-function part hide)
+	 (button (and (or deep
+			  (funcall notmuch-show-insert-header-p-function part hide))
 		      (notmuch-show-insert-part-header
 		       nth mime-type
 		       (and content-type (downcase content-type))
 		       (plist-get part :filename))))
-	 ;; Hide the part initially if HIDE is t, or if it is too long
+	 ;; Hide the part initially if HIDE is t, or if it is too long/deep
 	 ;; and we have a button to allow toggling.
 	 (show-part (not (or (equal hide t)
-			     (and long button))))
+			     (and long button)
+			     (and deep button))))
 	 (content-beg (point)))
     ;; Store the computed mime-type for later use (e.g. by attachment handlers).
     (plist-put part :computed-type mime-type)
diff --git a/test/T450-emacs-show.sh b/test/T450-emacs-show.sh
index 0c1f6204..da9851b3 100755
--- a/test/T450-emacs-show.sh
+++ b/test/T450-emacs-show.sh
@@ -91,6 +91,24 @@ test_emacs '(let ((notmuch-show-only-matching-messages t))
 	(test-visible-output))'
 test_expect_equal_file $EXPECTED/notmuch-show-elide-non-matching-messages-on OUTPUT
 
+test_begin_subtest "Hide bodies of messages"
+test_emacs '(let ((notmuch-show-depth-limit -1))
+	(notmuch-search "thread:{id:87ocn0qh6d.fsf@yoom.home.cworth.org}")
+	(notmuch-test-wait)
+	(notmuch-search-show-thread)
+	(notmuch-test-wait)
+	(test-visible-output))'
+test_expect_equal_file $EXPECTED/notmuch-show-depth OUTPUT
+
+test_begin_subtest "Hide bodies of messages (depth > 1)"
+test_emacs '(let ((notmuch-show-depth-limit 1))
+	(notmuch-search "thread:{id:87ocn0qh6d.fsf@yoom.home.cworth.org}")
+	(notmuch-test-wait)
+	(notmuch-search-show-thread)
+	(notmuch-test-wait)
+	(test-visible-output))'
+test_expect_equal_file $EXPECTED/notmuch-show-depth-1 OUTPUT
+
 test_begin_subtest "notmuch-show: elide non-matching messages (w/ notmuch-show-toggle-elide-non-matching)"
 test_emacs '(let ((notmuch-show-only-matching-messages nil))
 	(notmuch-search "from:lars@seas.harvard.edu and subject:\"Maildir storage\"")
diff --git a/test/emacs-show.expected-output/notmuch-show-depth b/test/emacs-show.expected-output/notmuch-show-depth
new file mode 100644
index 00000000..8299519c
--- /dev/null
+++ b/test/emacs-show.expected-output/notmuch-show-depth
@@ -0,0 +1,44 @@
+Lars Kellogg-Stedman <lars@seas.harvard.edu> (2009-11-17) (inbox signed)
+Subject: [notmuch] Working with Maildir storage?
+To: notmuch@notmuchmail.org
+Date: Tue, 17 Nov 2009 14:00:54 -0500
+
+[ multipart/mixed (hidden) ]
+ Mikhail Gusarov <dottedmag@dottedmag.net> (2009-11-17) (inbox signed unread)
+ Subject: Re: [notmuch] Working with Maildir storage?
+ To: notmuch@notmuchmail.org
+ Date: Wed, 18 Nov 2009 01:02:38 +0600
+
+ [ multipart/mixed (hidden) ]
+  Lars Kellogg-Stedman <lars@seas.harvard.edu> (2009-11-17) (inbox signed)
+  Subject: Re: [notmuch] Working with Maildir storage?
+  To: Mikhail Gusarov <dottedmag@dottedmag.net>
+  Cc: notmuch@notmuchmail.org
+  Date: Tue, 17 Nov 2009 15:33:01 -0500
+
+  [ multipart/mixed (hidden) ]
+   Mikhail Gusarov <dottedmag@dottedmag.net> (2009-11-17) (inbox unread)
+   Subject: [notmuch] Working with Maildir storage?
+   To: notmuch@notmuchmail.org
+   Date: Wed, 18 Nov 2009 02:50:48 +0600
+
+   [ text/plain (hidden) ]
+   Keith Packard <keithp@keithp.com> (2009-11-17) (inbox unread)
+   Subject: [notmuch] Working with Maildir storage?
+   To: notmuch@notmuchmail.org
+   Date: Tue, 17 Nov 2009 13:24:13 -0800
+
+   [ text/plain (hidden) ]
+    Lars Kellogg-Stedman <lars@seas.harvard.edu> (2009-11-18) (inbox signed unread)
+    Subject: Re: [notmuch] Working with Maildir storage?
+    To: Keith Packard <keithp@keithp.com>
+    Cc: notmuch@notmuchmail.org
+    Date: Tue, 17 Nov 2009 19:50:40 -0500
+
+    [ multipart/mixed (hidden) ]
+ Carl Worth <cworth@cworth.org> (2009-11-18) (inbox unread)
+ Subject: [notmuch] Working with Maildir storage?
+ To: notmuch@notmuchmail.org
+ Date: Wed, 18 Nov 2009 02:08:10 -0800
+
+ [ text/plain (hidden) ]
diff --git a/test/emacs-show.expected-output/notmuch-show-depth-1 b/test/emacs-show.expected-output/notmuch-show-depth-1
new file mode 100644
index 00000000..e7c376bb
--- /dev/null
+++ b/test/emacs-show.expected-output/notmuch-show-depth-1
@@ -0,0 +1,119 @@
+Lars Kellogg-Stedman <lars@seas.harvard.edu> (2009-11-17) (inbox signed)
+Subject: [notmuch] Working with Maildir storage?
+To: notmuch@notmuchmail.org
+Date: Tue, 17 Nov 2009 14:00:54 -0500
+
+[ multipart/mixed ]
+[ multipart/signed ]
+[ Unknown key ID 0xD74695063141ACD8 or unsupported algorithm ]
+[ text/plain ]
+I saw the LWN article and decided to take a look at notmuch.  I'm
+currently using mutt and mairix to index and read a collection of
+Maildir mail folders (around 40,000 messages total).
+
+notmuch indexed the messages without complaint, but my attempt at
+searching bombed out. Running, for example:
+
+  notmuch search storage
+
+Resulted in 4604 lines of errors along the lines of:
+
+  Error opening
+  /home/lars/Mail/read-messages.2008/cur/1246413773.24928_27334.hostname,U=3026:2,S:
+  Too many open files
+
+I'm curious if this is expected behavior (i.e., notmuch does not work
+with Maildir) or if something else is going on.
+
+Cheers,
+
+[ 4-line signature. Click/Enter to show. ]
+[ application/pgp-signature ]
+[ text/plain ]
+[ 4-line signature. Click/Enter to show. ]
+ Mikhail Gusarov <dottedmag@dottedmag.net> (2009-11-17) (inbox signed unread)
+ Subject: Re: [notmuch] Working with Maildir storage?
+ To: notmuch@notmuchmail.org
+ Date: Wed, 18 Nov 2009 01:02:38 +0600
+
+ [ multipart/mixed ]
+ [ multipart/signed ]
+ [ Unknown key ID 0x9D20F6503E338888 or unsupported algorithm ]
+ [ text/plain ]
+
+ Twas brillig at 14:00:54 17.11.2009 UTC-05 when lars@seas.harvard.edu did
+ gyre and gimble:
+
+  LK> Resulted in 4604 lines of errors along the lines of:
+
+  LK>   Error opening
+  LK>  
+ /home/lars/Mail/read-messages.2008/cur/1246413773.24928_27334.hostname,U=3026:2,S:
+  LK>   Too many open files
+
+ See the patch just posted here.
+
+ [ 2-line signature. Click/Enter to show. ]
+ [ application/pgp-signature ]
+ [ text/plain ]
+ [ 4-line signature. Click/Enter to show. ]
+  Lars Kellogg-Stedman <lars@seas.harvard.edu> (2009-11-17) (inbox signed)
+  Subject: Re: [notmuch] Working with Maildir storage?
+  To: Mikhail Gusarov <dottedmag@dottedmag.net>
+  Cc: notmuch@notmuchmail.org
+  Date: Tue, 17 Nov 2009 15:33:01 -0500
+
+  [ multipart/mixed (hidden) ]
+   Mikhail Gusarov <dottedmag@dottedmag.net> (2009-11-17) (inbox unread)
+   Subject: [notmuch] Working with Maildir storage?
+   To: notmuch@notmuchmail.org
+   Date: Wed, 18 Nov 2009 02:50:48 +0600
+
+   [ text/plain (hidden) ]
+   Keith Packard <keithp@keithp.com> (2009-11-17) (inbox unread)
+   Subject: [notmuch] Working with Maildir storage?
+   To: notmuch@notmuchmail.org
+   Date: Tue, 17 Nov 2009 13:24:13 -0800
+
+   [ text/plain (hidden) ]
+    Lars Kellogg-Stedman <lars@seas.harvard.edu> (2009-11-18) (inbox signed unread)
+    Subject: Re: [notmuch] Working with Maildir storage?
+    To: Keith Packard <keithp@keithp.com>
+    Cc: notmuch@notmuchmail.org
+    Date: Tue, 17 Nov 2009 19:50:40 -0500
+
+    [ multipart/mixed (hidden) ]
+ Carl Worth <cworth@cworth.org> (2009-11-18) (inbox unread)
+ Subject: [notmuch] Working with Maildir storage?
+ To: notmuch@notmuchmail.org
+ Date: Wed, 18 Nov 2009 02:08:10 -0800
+
+ On Tue, 17 Nov 2009 14:00:54 -0500, Lars Kellogg-Stedman <lars at
+ seas.harvard.edu> wrote:
+ > I saw the LWN article and decided to take a look at notmuch.  I'm
+ > currently using mutt and mairix to index and read a collection of
+ > Maildir mail folders (around 40,000 messages total).
+
+ Welcome, Lars!
+
+ I hadn't even seen that Keith's blog post had been picked up by lwn.net.
+ That's very interesting. So, thanks for coming and trying out notmuch.
+
+ >   Error opening
+ > /home/lars/Mail/read-messages.2008/cur/1246413773.24928_27334.hostname,U=3026:2,S:
+ >   Too many open files
+
+ Sadly, the lwn article coincided with me having just introduced this
+ bug, and then getting on a Trans-Atlantic flight. So I fixed the bug
+ fairly quickly, but there was quite a bit of latency before I could push
+ the fix out. It should be fixed now.
+
+ > I'm curious if this is expected behavior (i.e., notmuch does not work
+ > with Maildir) or if something else is going on.
+
+ Notmuch works just fine with maildir---it's one of the things that it
+ likes the best.
+
+ Happy hacking,
+
+ -Carl
-- 
2.35.2