From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from localhost (localhost [127.0.0.1]) by arlo.cworth.org (Postfix) with ESMTP id 8041A6DE0229 for ; Thu, 10 Jan 2019 00:10:40 -0800 (PST) X-Virus-Scanned: Debian amavisd-new at cworth.org X-Spam-Flag: NO X-Spam-Score: 0.001 X-Spam-Level: X-Spam-Status: No, score=0.001 tagged_above=-999 required=5 tests=[RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_PASS=-0.001] autolearn=disabled Received: from arlo.cworth.org ([127.0.0.1]) by localhost (arlo.cworth.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id dMRrTykl3AtA for ; Thu, 10 Jan 2019 00:10:38 -0800 (PST) Received: from mail-pl1-f196.google.com (mail-pl1-f196.google.com [209.85.214.196]) by arlo.cworth.org (Postfix) with ESMTPS id 77A596DE00C6 for ; Thu, 10 Jan 2019 00:10:38 -0800 (PST) Received: by mail-pl1-f196.google.com with SMTP id z23so4880411plo.0 for ; Thu, 10 Jan 2019 00:10:38 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:in-reply-to:references:date :message-id:mime-version; bh=XRX/JbDPavFm3qA+vTCdu0t2WMeVU24HQQRGkMpi7cQ=; b=a755Vk5/F1YocORt8c83Do7zyVAayDgj9tjRc0WoNg2JWCRsvkMTn9XvDfI6BGTsIg ShwGWpZfPxH3g35tTBQwpv6nz3ASI3eLKgM7zj7pET23/fVdIoh64mAydkb8hQfCWRFQ KSYt00uhIYZcb2HMmovJaTBgsjWmQLVh0F+eovCgFKbYr9pHdCvYx/Tfxiqu8K9N1BOk JPIsPD30cAnkNxPPZ3gdK369QCMGZgy1AkG7SpyJTzXuDoWTRXd5+tg+4yaKth7JhpT0 1jvlx20RxMPlyd+hNh6qElbn4zNAvn3Lj6VQ1qxU4OgNZpskHh0VXD4Rvfcqq2uaJ6Dm 1eGA== X-Gm-Message-State: AJcUukchEIhjNnyec9s+/uRUcyNfFH7lIq0h1xLRE+GH11AL29sUz9vo uZMDwIsCfHmDNxd09NyJSYcO8M8OTGw= X-Google-Smtp-Source: ALg8bN4nrq5VVPoR9o4LG2Oq7Op2jDUnA4/nAriFBUKZidBVuYBveHe59Op9Ron1Ad0zNjHbKV0Reg== X-Received: by 2002:a17:902:3124:: with SMTP id w33mr9450727plb.241.1547107836868; Thu, 10 Jan 2019 00:10:36 -0800 (PST) Received: from localhost ([121.165.117.232]) by smtp.gmail.com with ESMTPSA id m3sm96605341pgl.69.2019.01.10.00.10.35 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 10 Jan 2019 00:10:35 -0800 (PST) From: Tristan Cacqueray To: notmuch@notmuchmail.org Subject: Re: [emacs] optimizing notmuch-search to only parse displayed messages In-Reply-To: <1545033221.mk29aqtk3r.tristanC@fedora> References: <1545033221.mk29aqtk3r.tristanC@fedora> Date: Thu, 10 Jan 2019 17:10:03 +0900 Message-ID: <87va2x7x2c.tristanC@fedora> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="==-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" X-BeenThere: notmuch@notmuchmail.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Jan 2019 08:10:40 -0000 --==-=-= Content-Type: multipart/mixed; boundary="=-=-=" --=-=-= Content-Type: text/plain On Mon, Dec 17, 2018 at 07:59 Tristan Cacqueray wrote: > Hi, > > I have been testing emacs as a mail client recently, so first, thank you for > all the work on notmuch and the emacs mode, it's really neat :) > > My main issue is that, while notmuch-search is super fast to show the first > messages, it seems like it is processing the whole query output in the > background, which is resulting in a long cpu load for large search... > > Reading through the notmuch.el code, it seems like the search process output > is ingested by a scratch buffer, and iiuc we can't control how fast the stdout > gets processed. I tried adding some delay on the process-filter but that > doesn't seems to work. > > I'm pretty much a lisp beginer, thus I don't know how doable this is, but > can we replace that scratch buffer or the make-process usage by something > that enables stdout's reading to be controled based on the window status > (e.g. only read more when the window reaches the end of the buffer) ? > > As an alternative solution, I've looked into sending STOP and CONT signal to > the process using the patch below. It's a bit buggy when changing windows, > and it assumes the active window is the one being scrolled, but it does > make the notmuch-search window reads and parses the message when needed... > > What would be the best way to improve notmuch-search efficiency? > (beside adding search limit to the queries...) > Hi, Here is a quick follow-up from the irc discussions. The SIGSTOP trick doesn't work well with ssh wrapper, because the signal doesn't propagate to the remote process with the standard ssh client. Moreover, resuming a search after a db update may not work too. However this works well for me, and below is an improved implementation as other may find it useful. Alternative solutions are: - the query limit patch[0], but extending reload the whole search, - use pagination, but this needs a get_msgs() patch. Thank you all for the suggestions. -Tristan [0]: https://notmuchmail.org/pipermail/notmuch/2011/006297.html --=-=-= Content-Type: text/x-diff Content-Disposition: inline; filename=0001-wip-add-notmuch-progressive-search-custom.patch Content-Transfer-Encoding: quoted-printable From=20d0738252bb62929581ea863d40af0fb6c1827543 Mon Sep 17 00:00:00 2001 From: Tristan Cacqueray Date: Mon, 17 Dec 2018 05:01:17 +0000 Subject: [PATCH] wip: add notmuch-progressive-search custom This change updates the process-filter function to send a SIGSTOP signal to the notmuch-search process when the window is close to the end of the buffer. Then a scroll-functions is used to send a SIGCONT signal when the window reaches the end of the buffer. This change also adds a notmuch-flush-buffer function to read the whole search process outputs. This change also ignore the SIGCHLD handler for the notmuch-search process to properly reap gpg child process. =2D-- emacs/notmuch.el | 121 +++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 118 insertions(+), 3 deletions(-) diff --git a/emacs/notmuch.el b/emacs/notmuch.el index 804e78ab..fd2bc7ef 100644 =2D-- a/emacs/notmuch.el +++ b/emacs/notmuch.el @@ -864,6 +864,99 @@ sets the :orig-tag property." (setq notmuch-search-target-thread "found") (goto-char pos)))) =20 +(defcustom notmuch-progressive-search nil + "Enable progressive search. +When set to non nil, notmuch-search process is paused until viewport +reach the end of the buffer. Use the notmuch-flush-buffer to force +read all the messages." + :type 'boolean + :group 'notmuch-search) + +(defvar notmuch--paused-procs nil + "The list of notmuch search buffer that are paused.") +(defvar notmuch--procs-to-flush nil + "The list of notmuch search buffer the user requested to be flushed.") +(defvar notmuch--search-gc-timer nil + "The buffer process garbage collector timer.") + +(defconst notmuch--search-stop-limit -16384 + "The number of points between buffer and window to stop the search proce= ss.") +(defconst notmuch--search-cont-limit 8192 + "The number of points between window and buffer to cont the search proce= ss.") +(defconst notmuch--search-process-ttl 3600 + "The number of seconds before a paused search process is killed.") + +(defun notmuch--terminate (proc) + "Send sigterm to process group" + (message "DEBUG(term): Sending SIGTERM to %d (%s)" (process-id proc) (cu= rrent-buffer)) + (signal-process proc 15) + (notmuch--paused-procs-remove proc t) + (delete-process proc)) + +(defun notmuch--paused-procs-gc () + "Cleanup old paused processes and the one associated with killed buffers= ." + (message "DEBUG(gc): called at %s" (current-time-string)) + (dolist (proc notmuch--paused-procs) + (let ((buffer (process-buffer proc))) + (message "DEBUG(gc): processing %S associated with %S (starttime %S)" + proc buffer (car (cdr (alist-get 'etime (process-attributes= (process-id proc)))))) + (unless (buffer-live-p buffer) + (message "DEBUG(gc): Deleting because buffer is killed") + (notmuch--terminate proc)) + (if (member (process-status proc) '(run stop)) + (unless (<=3D (car (cdr (alist-get 'etime (process-attributes (p= rocess-id proc))))) + notmuch--search-process-ttl) + (message "DEBUG(gc): Deleting because process is too old") + (notmuch--terminate proc)) + (message "DEBUG(gc): Deleting because process is dead") + (notmuch--paused-procs-remove proc nil)))) + (dolist (proc notmuch--procs-to-flush) + (unless (process-live-p proc) + (setq notmuch--procs-to-flush (delete proc notmuch--procs-to-flush= )))) + (when (and (timerp notmuch--search-gc-timer) + (not notmuch--paused-procs) + (not notmuch--procs-to-flush)) + (message "DEBUG(gc): removing the timer") + ;; Remove un-needed timer + (cancel-timer notmuch--search-gc-timer))) + +(defun notmuch--paused-procs-remove (proc resume) + "Remove PROC from the paused list and send SIGCONT when resume is t" + (when resume + ;; Send SIGCONT signal + (message "DEBUG(r): Sending SIGCONT to %d (%s)" (process-id proc) (cur= rent-buffer)) + (signal-process proc 18)) + (setq notmuch--paused-procs (delete proc notmuch--paused-procs)) + (message "DEBUG(r) paused-procs are %S" notmuch--paused-procs) + (unless notmuch--paused-procs + ;; Remove un-needed scroll hook + (remove-hook 'window-scroll-functions 'notmuch--scroller))) + +(defun notmuch-flush-buffer () + "Manually flush the search process stdout associated with the current bu= ffer." + (interactive) + (let ((proc (get-buffer-process (current-buffer)))) + (if (not (member proc notmuch--paused-procs)) + (error "Notmuch search process is not paused") + (message "DEBUG(fb): Adding %S to the flush list for buffer %S" proc= (current-buffer)) + (add-to-list 'notmuch--procs-to-flush proc) + (notmuch--paused-procs-remove proc t))) + (message "DEBUG(fb): procs-to-flush are %S" notmuch--procs-to-flush)) + +(defun notmuch--scroller (window window-start) + "Resume search process when WINDOW reaches the end of the buffer. + +This is added to the 'window-scroll-functions' when a search process is st= opped. +WINDOW-START unused." + ;; (message "DEBUG(s): scroller position %d (%s)" + ;; (- (point-max) (window-end) notmuch--search-cont-limit) (current-bu= ffer)) + (when (<=3D (- (point-max) (window-end) notmuch--search-cont-limit) 0) + ;; Window is near the end of the buffer + (let ((proc (get-buffer-process (current-buffer)))) + (when (and proc (member proc notmuch--paused-procs)) + ;; Buffer is a stopped notmuch buffer + (notmuch--paused-procs-remove proc t))))) + (defun notmuch-search-process-filter (proc string) "Process and filter the output of \"notmuch search\"" (let ((results-buf (process-buffer proc)) @@ -877,7 +970,27 @@ sets the :orig-tag property." (goto-char (point-max)) (insert string)) (notmuch-sexp-parse-partial-list 'notmuch-search-append-result =2D results-buf))))) + results-buf)))) + + ;; (message "DEBUG(p): parser position %d" (- (window-end) (point-max) n= otmuch-search-stop-limit)) + (if (and (equal notmuch-progressive-search t) ; progressive= search is turned on + (not (member proc notmuch--procs-to-flush)) ; proc isn't = part of the flush list + (<=3D (- (window-end) (point-max) ; window is= far from the end of buffer + notmuch--search-stop-limit) + 0)) + (unless (member proc notmuch--paused-procs) + (unless notmuch--paused-procs + ;; First buffer to be stopped, add the scroll hook + (add-hook 'window-scroll-functions 'notmuch--scroller)) + ;; Add to the paused list + (add-to-list 'notmuch--paused-procs proc) + ;; Send SIGSTOP signal + (message "DEBUG(p): Sending SIGSTOP to %d (%s))" (process-id proc)= (current-buffer)) + (signal-process proc 19) + (message "DEBUG(p): paused-procs are: %S" notmuch--paused-procs) + (unless (timerp notmuch--search-gc-timer) + ;; Ensure gc is started + (setq notmuch--search-gc-timer (run-at-time 60 60 #'notmuch--pau= sed-procs-gc)))))) =20 (defun notmuch-search-tag-all (tag-changes) "Add/remove tags from all messages in current search buffer. @@ -990,6 +1103,7 @@ the configured default sort order." (set-buffer buffer) (switch-to-buffer buffer)) (notmuch-search-mode) + (notmuch--paused-procs-gc) ;; Don't track undo information for this buffer (set 'buffer-undo-list t) (set 'notmuch-search-query-string query) @@ -1000,8 +1114,9 @@ the configured default sort order." (let ((proc (get-buffer-process (current-buffer))) (inhibit-read-only t)) (if proc =2D (error "notmuch search process already running for query `%s'" query) =2D ) + (if (not (member proc notmuch--paused-procs)) + (error "notmuch search process already running for query `%s= '" query) + (notmuch--terminate proc))) (erase-buffer) (goto-char (point-min)) (save-excursion =2D-=20 2.19.2 --=-=-=-- --==-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQEzBAEBCAAdFiEE6xA96LXmnmMcb/F5IrmgXJJcxdgFAlw2/dsACgkQIrmgXJJc xdgSqQf/ZaCuMrYNiiqKzR0IyOIHldwIlu5nwKnMBdw+DjKolaf1Gqhydbcf/BsV nAmg0Ui8VR5Z3fyuVJxXEO0iIEHN5F10C+yECwGcXLMAWLwCVivwfDsKM8MtQgTc V8acRo/yqBtSogUuWetFyz+dN+ju5Iyqk2aeB9B/xLH5jjCKYt4YwgzNTDjaWM4i 73ohbrUX3ykCsgqS3slZGzhYJvpXbOMIQcw/8c2whGhrhOiSrmI1R5S9cKxLkPqE kGpimfxezpwlAIRVVPIkRrNimIAfO5jTC25P7xYUPEGbARyUnfkmgPkefEQvfl/G fdVTa2yMOMIli6lRqjIWkufIdjzSDQ== =M/10 -----END PGP SIGNATURE----- --==-=-=--