From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from localhost (localhost [127.0.0.1]) by arlo.cworth.org (Postfix) with ESMTP id 482456DE0140 for ; Sun, 16 Dec 2018 23:58:48 -0800 (PST) X-Virus-Scanned: Debian amavisd-new at cworth.org X-Spam-Flag: NO X-Spam-Score: -0.001 X-Spam-Level: X-Spam-Status: No, score=-0.001 tagged_above=-999 required=5 tests=[RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001] autolearn=disabled Received: from arlo.cworth.org ([127.0.0.1]) by localhost (arlo.cworth.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 8s_rNMtsi5h4 for ; Sun, 16 Dec 2018 23:58:46 -0800 (PST) Received: from mail-pl1-f170.google.com (mail-pl1-f170.google.com [209.85.214.170]) by arlo.cworth.org (Postfix) with ESMTPS id 4C4896DE00F5 for ; Sun, 16 Dec 2018 23:58:46 -0800 (PST) Received: by mail-pl1-f170.google.com with SMTP id 101so5743340pld.6 for ; Sun, 16 Dec 2018 23:58:46 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:subject:to:mime-version:message-id; bh=EiRNWbqokluBUo2JV9PK7VY0wzbL+wzKpapZdiVBUa0=; b=ibncgNdPhm8nKJPxa8KmYcyehVHxM9pgQXXepJjXTq5EsWIfAn8ZLslJG76WUR2mr0 idjw0PVT5ibfgsBh02ne9rToSpolT5+TojQp/BQ9Fw1/A3sk045ytY6wkH+06xi/dv13 cecXVpzUhB4BmIPIOCBgH2373HgQYRLGgmtf5zcKUZxPiMXZ+QYSn82V6gy5Pv/dAWGP 2xnvlrTjhfVS+FYxOAb6BH55PYFoIkxgK9z6Rsvp8+Evf0VDG1G/dYuh9o12Lj/X50gP hegWK5PVwDlLDAsPaIFEITlYNZka0kZJpc0zbZmw1NjqF7GHamHzwutts1zHH2tNu4Z2 JETg== X-Gm-Message-State: AA+aEWZSxdCC+khGac2U1p5mxXLwd4eOFuW/0eaZhZ8zRyGUWFAMsqX6 jwsm/clHzRF405Ad4KiKgNntPnKFNwA= X-Google-Smtp-Source: AFSGD/WPCH5740A3XjlnNp5zDRIpj1astHRh7YiDI2ufh11/ZsCZJBuVtC+Yf7aIzkGL0wDxm7YWUA== X-Received: by 2002:a17:902:b48b:: with SMTP id y11mr11415587plr.200.1545033523219; Sun, 16 Dec 2018 23:58:43 -0800 (PST) Received: from localhost ([59.8.199.230]) by smtp.gmail.com with ESMTPSA id p67sm20468816pfg.44.2018.12.16.23.58.41 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Sun, 16 Dec 2018 23:58:41 -0800 (PST) Date: Mon, 17 Dec 2018 07:59:22 +0000 From: Tristan Cacqueray Subject: [emacs] optimizing notmuch-search to only parse displayed messages To: notmuch@notmuchmail.org MIME-Version: 1.0 Message-Id: <1545033221.mk29aqtk3r.tristanC@fedora> Content-Type: multipart/signed; protocol="application/pgp-signature"; micalg=pgp-sha256; boundary="=-CmyOks8MwwxWy5pid82N" X-BeenThere: notmuch@notmuchmail.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 17 Dec 2018 07:58:48 -0000 --=-CmyOks8MwwxWy5pid82N Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hi, I have been testing emacs as a mail client recently, so first, thank you fo= r all the work on notmuch and the emacs mode, it's really neat :) My main issue is that, while notmuch-search is super fast to show the first messages, it seems like it is processing the whole query output in the background, which is resulting in a long cpu load for large search... Reading through the notmuch.el code, it seems like the search process outpu= t is ingested by a scratch buffer, and iiuc we can't control how fast the std= out gets processed. I tried adding some delay on the process-filter but that doesn't seems to work. I'm pretty much a lisp beginer, thus I don't know how doable this is, but can we replace that scratch buffer or the make-process usage by something that enables stdout's reading to be controled based on the window status (e.g. only read more when the window reaches the end of the buffer) ? As an alternative solution, I've looked into sending STOP and CONT signal t= o the process using the patch below. It's a bit buggy when changing windows, and it assumes the active window is the one being scrolled, but it does make the notmuch-search window reads and parses the message when needed... What would be the best way to improve notmuch-search efficiency? (beside adding search limit to the queries...) Cheers, -Tristan [PATCH] wip: stop notmuch-search process until window reach end of buffer This change updates the process-filter function to send a SIG_STOP signal to the notmuch-search process when the window is close to the end of the buffer. Then a scroll-functions is used to send the SIG_CONT when the window reaches the end of the buffer. --- emacs/notmuch.el | 43 ++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 42 insertions(+), 1 deletion(-) diff --git a/emacs/notmuch.el b/emacs/notmuch.el index 804e78ab..e42396e3 100644 --- a/emacs/notmuch.el +++ b/emacs/notmuch.el @@ -864,8 +864,29 @@ sets the :orig-tag property." (setq notmuch-search-target-thread "found") (goto-char pos)))) +;; notmuch-windows alist (window-name . process) +(setq notmuch-windows nil) +(defun notmuch-scroller (window window-start) + ;; (message "DEBUG: scroller position %d (%s)" (- (point-max) (window-en= d) 8192) (selected-window)) + (if (<=3D (- (point-max) (window-end) 8192) 0) + ;; Window is near the end of the buffer + (progn + (let ((proc (alist-get (selected-window) notmuch-windows))) + (if (not (equal proc nil)) + ;; Window is in notmuch-windows list + (progn + (message "DEBUG: Sending SIG_CONT signal to %d (%s)" (proc= ess-id proc) (selected-window)) + ;; Resume the process + (signal-process proc 18) + ;; Remove it from the notmuch-windows list + (setq notmuch-windows (delq (assoc (selected-window) notmu= ch-windows) notmuch-windows)) + )))))) +;; Install the hook +(add-hook 'window-scroll-functions 'notmuch-scroller) + (defun notmuch-search-process-filter (proc string) "Process and filter the output of \"notmuch search\"" + ;; (message "DEBUG: notmuch-search-process-filter [%d]" (length string)) (let ((results-buf (process-buffer proc)) (parse-buf (process-get proc 'parse-buf)) (inhibit-read-only t) @@ -877,7 +898,27 @@ sets the :orig-tag property." (goto-char (point-max)) (insert string)) (notmuch-sexp-parse-partial-list 'notmuch-search-append-result - results-buf))))) + results-buf)))) + + ;; (message "DEBUG: parser position %d" (- (window-end) (point-max) -163= 84)) + (if (<=3D (- (window-end) (point-max) -16384) 0) + ;; Buffer is past the end of the window + (let ((old-proc (alist-get (selected-window) notmuch-windows))) + ;; Remove previous proc window association + (if (and old-proc (not (equal old-proc proc))) + (progn + (message "DEBUG: Removing stalled association") + (setq notmuch-windows (delq (assoc (selected-window) notmuch= -windows) notmuch-windows)) + )) + ;; Stop proc and associate to selected window + (if (not old-proc) + (progn + (add-to-list 'notmuch-windows `(,(selected-window) . ,proc)) + (message "DEBUG: Sending SIG_STOP signal to %d (%s))" (proce= ss-id proc) (selected-window)) + (signal-process proc 19) + )))) + ) + (defun notmuch-search-tag-all (tag-changes) "Add/remove tags from all messages in current search buffer. -- 2.19.2 = --=-CmyOks8MwwxWy5pid82N Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- iQEzBAABCAAdFiEE6xA96LXmnmMcb/F5IrmgXJJcxdgFAlwXV1oACgkQIrmgXJJc xdiOcAf+IB8482zU6VdYxfo7dSI5hSfjmDnTZDrAYeDvlyierE0qGbolPJ2cO1ND O6iUjA2kBJADwJ9EPXHr7BYrjPvaae/jGycUbPQMDSgdrk9yzDT26r3EIbFGtTDR GQMRToD448Giemm+5hXFbynQ3nZMsRVNovznfHyxmWY8HbKFkSHqpU3JPq1B1eCa eBZJFCvzpENSSHa9vk8rX45zsMM+f9imWixh8Ofg1VCMlSKXuIpfGNG6yGs9xWb6 JE/dXQfcNt2YpD9mUbpqAqVl+XyltqLXYkhqNZQ6q8+az+f1BJjlCw+dQCACD7ol vuKEMbav5l0De40ocunSanFGO5/mSw== =S5Jp -----END PGP SIGNATURE----- --=-CmyOks8MwwxWy5pid82N--