[emacs] optimizing notmuch-search to only parse displayed messages

[emacs] optimizing notmuch-search to only parse displayed messages

[Tristan Cacqueray]

[-- Attachment #1: Type: text/plain, Size: 4772 bytes --]

Hi,

I have been testing emacs as a mail client recently, so first, thank you for
all the work on notmuch and the emacs mode, it's really neat :)

My main issue is that, while notmuch-search is super fast to show the first
messages, it seems like it is processing the whole query output in the
background, which is resulting in a long cpu load for large search...

Reading through the notmuch.el code, it seems like the search process output
is ingested by a scratch buffer, and iiuc we can't control how fast the stdout
gets processed. I tried adding some delay on the process-filter but that
doesn't seems to work.

I'm pretty much a lisp beginer, thus I don't know how doable this is, but
can we replace that scratch buffer or the make-process usage by something
that enables stdout's reading to be controled based on the window status
(e.g. only read more when the window reaches the end of the buffer) ?

As an alternative solution, I've looked into sending STOP and CONT signal to
the process using the patch below. It's a bit buggy when changing windows,
and it assumes the active window is the one being scrolled, but it does
make the notmuch-search window reads and parses the message when needed...

What would be the best way to improve notmuch-search efficiency?
(beside adding search limit to the queries...)

Cheers,
-Tristan


[PATCH] wip: stop notmuch-search process until window reach end of buffer

This change updates the process-filter function to send a SIG_STOP
signal to the notmuch-search process when the window is close to
the end of the buffer. Then a scroll-functions is used to send the
SIG_CONT when the window reaches the end of the buffer.
---
 emacs/notmuch.el | 43 ++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 42 insertions(+), 1 deletion(-)

diff --git a/emacs/notmuch.el b/emacs/notmuch.el
index 804e78ab..e42396e3 100644
--- a/emacs/notmuch.el
+++ b/emacs/notmuch.el
@@ -864,8 +864,29 @@ sets the :orig-tag property."
       (setq notmuch-search-target-thread "found")
       (goto-char pos))))

+;; notmuch-windows alist (window-name . process)
+(setq notmuch-windows nil)
+(defun notmuch-scroller (window window-start)
+  ;; (message "DEBUG: scroller position %d (%s)" (- (point-max) (window-end) 8192) (selected-window))
+  (if (<= (- (point-max) (window-end) 8192) 0)
+      ;; Window is near the end of the buffer
+      (progn
+        (let ((proc (alist-get (selected-window) notmuch-windows)))
+          (if (not (equal proc nil))
+              ;; Window is in notmuch-windows list
+              (progn
+                (message "DEBUG: Sending SIG_CONT signal to %d (%s)" (process-id proc) (selected-window))
+                ;; Resume the process
+                (signal-process proc 18)
+                ;; Remove it from the notmuch-windows list
+                (setq notmuch-windows (delq (assoc (selected-window) notmuch-windows) notmuch-windows))
+                ))))))
+;; Install the hook
+(add-hook 'window-scroll-functions 'notmuch-scroller)
+
 (defun notmuch-search-process-filter (proc string)
   "Process and filter the output of \"notmuch search\""
+  ;; (message "DEBUG: notmuch-search-process-filter [%d]" (length string))
   (let ((results-buf (process-buffer proc))
        (parse-buf (process-get proc 'parse-buf))
        (inhibit-read-only t)
@@ -877,7 +898,27 @@ sets the :orig-tag property."
          (goto-char (point-max))
          (insert string))
        (notmuch-sexp-parse-partial-list 'notmuch-search-append-result
-                                        results-buf)))))
+                                        results-buf))))
+
+  ;; (message "DEBUG: parser position %d" (- (window-end) (point-max) -16384))
+  (if (<= (- (window-end) (point-max) -16384) 0)
+      ;; Buffer is past the end of the window
+      (let ((old-proc (alist-get (selected-window) notmuch-windows)))
+        ;; Remove previous proc window association
+        (if (and old-proc (not (equal old-proc proc)))
+            (progn
+              (message "DEBUG: Removing stalled association")
+              (setq notmuch-windows (delq (assoc (selected-window) notmuch-windows) notmuch-windows))
+              ))
+        ;; Stop proc and associate to selected window
+        (if (not old-proc)
+            (progn
+              (add-to-list 'notmuch-windows `(,(selected-window) . ,proc))
+              (message "DEBUG: Sending SIG_STOP signal to %d (%s))" (process-id proc) (selected-window))
+              (signal-process proc 19)
+              ))))
+  )
+

 (defun notmuch-search-tag-all (tag-changes)
   "Add/remove tags from all messages in current search buffer.
--
2.19.2


[-- Attachment #2: Type: application/pgp-signature, Size: 488 bytes --]

Re: [emacs] optimizing notmuch-search to only parse displayed messages

[Tristan Cacqueray]

[-- Attachment #1.1: Type: text/plain, Size: 2053 bytes --]


On Mon, Dec 17, 2018 at 07:59 Tristan Cacqueray wrote:
> Hi,
>
> I have been testing emacs as a mail client recently, so first, thank you for
> all the work on notmuch and the emacs mode, it's really neat :)
>
> My main issue is that, while notmuch-search is super fast to show the first
> messages, it seems like it is processing the whole query output in the
> background, which is resulting in a long cpu load for large search...
>
> Reading through the notmuch.el code, it seems like the search process output
> is ingested by a scratch buffer, and iiuc we can't control how fast the stdout
> gets processed. I tried adding some delay on the process-filter but that
> doesn't seems to work.
>
> I'm pretty much a lisp beginer, thus I don't know how doable this is, but
> can we replace that scratch buffer or the make-process usage by something
> that enables stdout's reading to be controled based on the window status
> (e.g. only read more when the window reaches the end of the buffer) ?
>
> As an alternative solution, I've looked into sending STOP and CONT signal to
> the process using the patch below. It's a bit buggy when changing windows,
> and it assumes the active window is the one being scrolled, but it does
> make the notmuch-search window reads and parses the message when needed...
>
> What would be the best way to improve notmuch-search efficiency?
> (beside adding search limit to the queries...)
>

Hi,

Here is a quick follow-up from the irc discussions.

The SIGSTOP trick doesn't work well with ssh wrapper, because the signal
doesn't propagate to the remote process with the standard ssh client.
Moreover, resuming a search after a db update may not work too. However
this works well for me, and below is an improved implementation as other
may find it useful.

Alternative solutions are:
- the query limit patch[0], but extending reload the whole search,
- use pagination, but this needs a get_msgs() patch.

Thank you all for the suggestions.
-Tristan

[0]: https://notmuchmail.org/pipermail/notmuch/2011/006297.html



[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1.2: 0001-wip-add-notmuch-progressive-search-custom.patch --]
[-- Type: text/x-diff, Size: 8148 bytes --]

From d0738252bb62929581ea863d40af0fb6c1827543 Mon Sep 17 00:00:00 2001
From: Tristan Cacqueray <tdecacqu@redhat.com>
Date: Mon, 17 Dec 2018 05:01:17 +0000
Subject: [PATCH] wip: add notmuch-progressive-search custom

This change updates the process-filter function to send a SIGSTOP
signal to the notmuch-search process when the window is close to
the end of the buffer. Then a scroll-functions is used to send a
SIGCONT signal when the window reaches the end of the buffer.

This change also adds a notmuch-flush-buffer function to read the
whole search process outputs.

This change also ignore the SIGCHLD handler for the notmuch-search
process to properly reap gpg child process.
---
 emacs/notmuch.el | 121 +++++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 118 insertions(+), 3 deletions(-)

diff --git a/emacs/notmuch.el b/emacs/notmuch.el
index 804e78ab..fd2bc7ef 100644
--- a/emacs/notmuch.el
+++ b/emacs/notmuch.el
@@ -864,6 +864,99 @@ sets the :orig-tag property."
       (setq notmuch-search-target-thread "found")
       (goto-char pos))))
 
+(defcustom notmuch-progressive-search nil
+  "Enable progressive search.
+When set to non nil, notmuch-search process is paused until viewport
+reach the end of the buffer. Use the notmuch-flush-buffer to force
+read all the messages."
+  :type 'boolean
+  :group 'notmuch-search)
+
+(defvar notmuch--paused-procs nil
+  "The list of notmuch search buffer that are paused.")
+(defvar notmuch--procs-to-flush nil
+  "The list of notmuch search buffer the user requested to be flushed.")
+(defvar notmuch--search-gc-timer nil
+  "The buffer process garbage collector timer.")
+
+(defconst notmuch--search-stop-limit -16384
+  "The number of points between buffer and window to stop the search process.")
+(defconst notmuch--search-cont-limit 8192
+  "The number of points between window and buffer to cont the search process.")
+(defconst notmuch--search-process-ttl 3600
+  "The number of seconds before a paused search process is killed.")
+
+(defun notmuch--terminate (proc)
+  "Send sigterm to process group"
+  (message "DEBUG(term): Sending SIGTERM to %d (%s)" (process-id proc) (current-buffer))
+  (signal-process proc 15)
+  (notmuch--paused-procs-remove proc t)
+  (delete-process proc))
+
+(defun notmuch--paused-procs-gc ()
+  "Cleanup old paused processes and the one associated with killed buffers."
+  (message "DEBUG(gc): called at %s" (current-time-string))
+  (dolist (proc notmuch--paused-procs)
+    (let ((buffer (process-buffer proc)))
+      (message "DEBUG(gc): processing %S associated with %S (starttime %S)"
+               proc buffer (car (cdr (alist-get 'etime (process-attributes (process-id proc))))))
+      (unless (buffer-live-p buffer)
+        (message "DEBUG(gc): Deleting because buffer is killed")
+        (notmuch--terminate proc))
+      (if (member (process-status proc) '(run stop))
+          (unless (<= (car (cdr (alist-get 'etime (process-attributes (process-id proc)))))
+                      notmuch--search-process-ttl)
+            (message "DEBUG(gc): Deleting because process is too old")
+            (notmuch--terminate proc))
+        (message "DEBUG(gc): Deleting because process is dead")
+        (notmuch--paused-procs-remove proc nil))))
+  (dolist (proc notmuch--procs-to-flush)
+    (unless (process-live-p proc)
+        (setq notmuch--procs-to-flush (delete proc notmuch--procs-to-flush))))
+  (when (and (timerp notmuch--search-gc-timer)
+             (not notmuch--paused-procs)
+             (not notmuch--procs-to-flush))
+    (message "DEBUG(gc): removing the timer")
+    ;; Remove un-needed timer
+    (cancel-timer notmuch--search-gc-timer)))
+
+(defun notmuch--paused-procs-remove (proc resume)
+  "Remove PROC from the paused list and send SIGCONT when resume is t"
+  (when resume
+    ;; Send SIGCONT signal
+    (message "DEBUG(r): Sending SIGCONT to %d (%s)" (process-id proc) (current-buffer))
+    (signal-process proc 18))
+  (setq notmuch--paused-procs (delete proc notmuch--paused-procs))
+  (message "DEBUG(r) paused-procs are %S" notmuch--paused-procs)
+  (unless notmuch--paused-procs
+    ;; Remove un-needed scroll hook
+    (remove-hook 'window-scroll-functions 'notmuch--scroller)))
+
+(defun notmuch-flush-buffer ()
+  "Manually flush the search process stdout associated with the current buffer."
+  (interactive)
+  (let ((proc (get-buffer-process (current-buffer))))
+    (if (not (member proc notmuch--paused-procs))
+        (error "Notmuch search process is not paused")
+      (message "DEBUG(fb): Adding %S to the flush list for buffer %S" proc (current-buffer))
+      (add-to-list 'notmuch--procs-to-flush proc)
+      (notmuch--paused-procs-remove proc t)))
+  (message "DEBUG(fb): procs-to-flush are %S" notmuch--procs-to-flush))
+
+(defun notmuch--scroller (window window-start)
+  "Resume search process when WINDOW reaches the end of the buffer.
+
+This is added to the 'window-scroll-functions' when a search process is stopped.
+WINDOW-START unused."
+  ;; (message "DEBUG(s): scroller position %d (%s)"
+  ;;   (- (point-max) (window-end) notmuch--search-cont-limit) (current-buffer))
+  (when (<= (- (point-max) (window-end) notmuch--search-cont-limit) 0)
+    ;; Window is near the end of the buffer
+    (let ((proc (get-buffer-process (current-buffer))))
+      (when (and proc (member proc notmuch--paused-procs))
+        ;; Buffer is a stopped notmuch buffer
+        (notmuch--paused-procs-remove proc t)))))
+
 (defun notmuch-search-process-filter (proc string)
   "Process and filter the output of \"notmuch search\""
   (let ((results-buf (process-buffer proc))
@@ -877,7 +970,27 @@ sets the :orig-tag property."
 	  (goto-char (point-max))
 	  (insert string))
 	(notmuch-sexp-parse-partial-list 'notmuch-search-append-result
-					 results-buf)))))
+					 results-buf))))
+
+  ;; (message "DEBUG(p): parser position %d" (- (window-end) (point-max) notmuch-search-stop-limit))
+  (if (and (equal notmuch-progressive-search t)              ; progressive search is turned on
+           (not (member proc notmuch--procs-to-flush))       ; proc isn't part of the flush list
+           (<= (- (window-end) (point-max)                   ; window is far from the end of buffer
+                  notmuch--search-stop-limit)
+               0))
+      (unless (member proc notmuch--paused-procs)
+        (unless notmuch--paused-procs
+            ;; First buffer to be stopped, add the scroll hook
+            (add-hook 'window-scroll-functions 'notmuch--scroller))
+        ;; Add to the paused list
+        (add-to-list 'notmuch--paused-procs proc)
+        ;; Send SIGSTOP signal
+        (message "DEBUG(p): Sending SIGSTOP to %d (%s))" (process-id proc) (current-buffer))
+        (signal-process proc 19)
+        (message "DEBUG(p): paused-procs are: %S" notmuch--paused-procs)
+        (unless (timerp notmuch--search-gc-timer)
+          ;; Ensure gc is started
+          (setq notmuch--search-gc-timer (run-at-time 60 60 #'notmuch--paused-procs-gc))))))
 
 (defun notmuch-search-tag-all (tag-changes)
   "Add/remove tags from all messages in current search buffer.
@@ -990,6 +1103,7 @@ the configured default sort order."
 	(set-buffer buffer)
       (switch-to-buffer buffer))
     (notmuch-search-mode)
+    (notmuch--paused-procs-gc)
     ;; Don't track undo information for this buffer
     (set 'buffer-undo-list t)
     (set 'notmuch-search-query-string query)
@@ -1000,8 +1114,9 @@ the configured default sort order."
     (let ((proc (get-buffer-process (current-buffer)))
 	  (inhibit-read-only t))
       (if proc
-	  (error "notmuch search process already running for query `%s'" query)
-	)
+          (if (not (member proc notmuch--paused-procs))
+              (error "notmuch search process already running for query `%s'" query)
+            (notmuch--terminate proc)))
       (erase-buffer)
       (goto-char (point-min))
       (save-excursion
-- 
2.19.2


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]