Selection bug

Selection bug

[Justus Winter]

[-- Attachment #1.1.1: Type: text/plain, Size: 877 bytes --]

Moin,

there is a bug in notmuch emacs that is mildly annoying.  Sometimes when
I refresh my search buffer, and there are new mails, the topmost
(i.e. newest) message seems to be selected (i.e. with green background),
but in fact if i hit enter (or do any other state changing action like
tagging), notmuch emacs acts on the previously selected message.  There
is a visual clue to that, the block cursor is still on that previously
selected message.

Still, I think it is a bug, and because the green background dominates
the block cursor, I run into this all the time.

I'll attach a screenshot that shows the bug.

This is notmuch straight from Debian bookworm.

ii  elpa-notmuch   0.37-1       all          thread-based email index, search and tagging (emacs interface)
ii  notmuch        0.37-1       amd64        thread-based email index, search and tagging

Best,
Justus

[-- Attachment #1.1.2: The selection bug as it happens. --]
[-- Type: image/png, Size: 6944 bytes --]

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 519 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: Selection bug

[David Bremner]

Justus Winter <justus@sequoia-pgp.org> writes:

> there is a bug in notmuch emacs that is mildly annoying.  Sometimes when
> I refresh my search buffer, and there are new mails, the topmost
> (i.e. newest) message seems to be selected (i.e. with green background),
> but in fact if i hit enter (or do any other state changing action like
> tagging), notmuch emacs acts on the previously selected message.  There
> is a visual clue to that, the block cursor is still on that previously
> selected message.
>
> Still, I think it is a bug, and because the green background dominates
> the block cursor, I run into this all the time.

Hi Justus;

That should be hl-line-mode, not sure why it's not tracking your
cursor. Can you duplicate the issue in emacs -q? Also, what version of
emacs is this?

d

Re: Selection bug

[Justus Winter]

[-- Attachment #1.1.1: Type: text/plain, Size: 2142 bytes --]

Hi David :)

David Bremner <david@tethera.net> writes:

> Justus Winter <justus@sequoia-pgp.org> writes:
>
>> there is a bug in notmuch emacs that is mildly annoying.  Sometimes when
>> I refresh my search buffer, and there are new mails, the topmost
>> (i.e. newest) message seems to be selected (i.e. with green background),
>> but in fact if i hit enter (or do any other state changing action like
>> tagging), notmuch emacs acts on the previously selected message.  There
>> is a visual clue to that, the block cursor is still on that previously
>> selected message.
>>
>> Still, I think it is a bug, and because the green background dominates
>> the block cursor, I run into this all the time.
>
> That should be hl-line-mode, not sure why it's not tracking your
> cursor. Can you duplicate the issue in emacs -q? Also, what version of
> emacs is this?

This is emacs 27.1 from bookworm:

ii  emacs          1:27.1+1-3.1 all          GNU Emacs editor (metapackage)

emacs --version
GNU Emacs 27.1
Copyright (C) 2020 Free Software Foundation, Inc.
GNU Emacs comes with ABSOLUTELY NO WARRANTY.
You may redistribute copies of GNU Emacs
under the terms of the GNU General Public License.
For more information about these matters, see the file named COPYING.

I think I found a reproducer:

mkdir -p /tmp/selection-bug/{tmp,new,cur}
echo "[database]
path=/tmp/selection-bug" > /tmp/selection-bug/notmuch-config
NOTMUCH_CONFIG=/tmp/selection-bug/notmuch-config notmuch new
NOTMUCH_CONFIG=/tmp/selection-bug/notmuch-config emacs -q --eval "(require 'notmuch)"
# M-x notmuch-hello, search for '*'
notmuch show --format=raw id:87edvlanuh.fsf@tethera.net | NOTMUCH_CONFIG=/tmp/selection-bug/notmuch-config notmuch insert
# refresh search buffer, press down once, i.e. navigate to "End of search results".
notmuch show --format=raw id:87fsfuuxwn.fsf@thinkbox | NOTMUCH_CONFIG=/tmp/selection-bug/notmuch-config notmuch insert
notmuch show --format=raw id:8735buw7gf.fsf@ngraves.fr | NOTMUCH_CONFIG=/tmp/selection-bug/notmuch-config notmuch insert
# refresh search buffer, bug: Nicolas' mail is green, mine is opened on pressing enter.

Best,
Justus

[-- Attachment #1.1.2: selection-bug-reproducer.png --]
[-- Type: image/png, Size: 23996 bytes --]

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 519 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: Selection bug

[David Bremner]

Justus Winter <justus@sequoia-pgp.org> writes:

> I think I found a reproducer:
>
> mkdir -p /tmp/selection-bug/{tmp,new,cur}
> echo "[database]
> path=/tmp/selection-bug" > /tmp/selection-bug/notmuch-config
> NOTMUCH_CONFIG=/tmp/selection-bug/notmuch-config notmuch new
> NOTMUCH_CONFIG=/tmp/selection-bug/notmuch-config emacs -q --eval "(require 'notmuch)"
> # M-x notmuch-hello, search for '*'
> notmuch show --format=raw id:87edvlanuh.fsf@tethera.net | NOTMUCH_CONFIG=/tmp/selection-bug/notmuch-config notmuch insert
> # refresh search buffer, press down once, i.e. navigate to "End of search results".
> notmuch show --format=raw id:87fsfuuxwn.fsf@thinkbox | NOTMUCH_CONFIG=/tmp/selection-bug/notmuch-config notmuch insert
> notmuch show --format=raw id:8735buw7gf.fsf@ngraves.fr | NOTMUCH_CONFIG=/tmp/selection-bug/notmuch-config notmuch insert
> # refresh search buffer, bug: Nicolas' mail is green, mine is opened on pressing enter.

1) Do you think the particular messages matter? I don't have
   id:8735buw7gf.fsf@ngraves.fr

2) Just to be clear, that script has to be run manually, with the
   comments corresponding to interactive actions in emacs?

Re: Selection bug

[Justus Winter]

[-- Attachment #1.1: Type: text/plain, Size: 3156 bytes --]

David Bremner <david@tethera.net> writes:

> Justus Winter <justus@sequoia-pgp.org> writes:
>
>> I think I found a reproducer:
>>
>> mkdir -p /tmp/selection-bug/{tmp,new,cur}
>> echo "[database]
>> path=/tmp/selection-bug" > /tmp/selection-bug/notmuch-config
>> NOTMUCH_CONFIG=/tmp/selection-bug/notmuch-config notmuch new
>> NOTMUCH_CONFIG=/tmp/selection-bug/notmuch-config emacs -q --eval "(require 'notmuch)"
>> # M-x notmuch-hello, search for '*'
>> notmuch show --format=raw id:87edvlanuh.fsf@tethera.net | NOTMUCH_CONFIG=/tmp/selection-bug/notmuch-config notmuch insert
>> # refresh search buffer, press down once, i.e. navigate to "End of search results".
>> notmuch show --format=raw id:87fsfuuxwn.fsf@thinkbox | NOTMUCH_CONFIG=/tmp/selection-bug/notmuch-config notmuch insert
>> notmuch show --format=raw id:8735buw7gf.fsf@ngraves.fr | NOTMUCH_CONFIG=/tmp/selection-bug/notmuch-config notmuch insert
>> # refresh search buffer, bug: Nicolas' mail is green, mine is opened on pressing enter.
>
> 1) Do you think the particular messages matter? I don't have
>    id:8735buw7gf.fsf@ngraves.fr

Uh, you are right.  I meant to pick mails form notmuch@, but this one is
from another list.  I actually reduced the example further:

mkdir -p /tmp/selection-bug/{tmp,new,cur}
echo "[database]
path=/tmp/selection-bug" > /tmp/selection-bug/notmuch-config
NOTMUCH_CONFIG=/tmp/selection-bug/notmuch-config notmuch new
NOTMUCH_CONFIG=/tmp/selection-bug/notmuch-config emacs -q --eval "(require 'notmuch)"
# M-x notmuch-hello, search for '*'
notmuch show --format=raw id:87edvlanuh.fsf@tethera.net | NOTMUCH_CONFIG=/tmp/selection-bug/notmuch-config notmuch insert
# refresh search buffer, press down once, i.e. navigate to "End of search results".
notmuch show --format=raw id:87fsfuuxwn.fsf@thinkbox | NOTMUCH_CONFIG=/tmp/selection-bug/notmuch-config notmuch insert
# refresh search buffer, bug: your mail is green, mine is opened on pressing enter.

(I.e., just drop Nicolas' mail.)

The bug can be seen even with just one mail, but then its impact is less
severe:

mkdir -p /tmp/selection-bug/{tmp,new,cur}
echo "[database]
path=/tmp/selection-bug" > /tmp/selection-bug/notmuch-config
NOTMUCH_CONFIG=/tmp/selection-bug/notmuch-config notmuch new
NOTMUCH_CONFIG=/tmp/selection-bug/notmuch-config emacs -q --eval "(require 'notmuch)"
# M-x notmuch-hello, search for '*', press down once, i.e. navigate past the line "End of search results".
notmuch show --format=raw id:87edvlanuh.fsf@tethera.net | NOTMUCH_CONFIG=/tmp/selection-bug/notmuch-config notmuch insert
# refresh search buffer, bug: your mail is green, but in fact "End of search results" is selected and is "acted upon" when pressing enter.

> 2) Just to be clear, that script has to be run manually, with the
>    comments corresponding to interactive actions in emacs?

Yes.

As additional data point, I cannot reproduce the problem on my Desktop
which has the same emacs version but uses elpa-notmuch 0.31.4-2.  On
this system, if I run the reproducer, there is no line with a green
background, so it is less misleading.

Best,
Justus

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 519 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: Selection bug

[Justus Winter]

[-- Attachment #1.1: Type: text/plain, Size: 3386 bytes --]

Justus Winter <justus@sequoia-pgp.org> writes:

> David Bremner <david@tethera.net> writes:
>
>> Justus Winter <justus@sequoia-pgp.org> writes:
>>
>>> I think I found a reproducer:
>>>
>>> mkdir -p /tmp/selection-bug/{tmp,new,cur}
>>> echo "[database]
>>> path=/tmp/selection-bug" > /tmp/selection-bug/notmuch-config
>>> NOTMUCH_CONFIG=/tmp/selection-bug/notmuch-config notmuch new
>>> NOTMUCH_CONFIG=/tmp/selection-bug/notmuch-config emacs -q --eval "(require 'notmuch)"
>>> # M-x notmuch-hello, search for '*'
>>> notmuch show --format=raw id:87edvlanuh.fsf@tethera.net | NOTMUCH_CONFIG=/tmp/selection-bug/notmuch-config notmuch insert
>>> # refresh search buffer, press down once, i.e. navigate to "End of search results".
>>> notmuch show --format=raw id:87fsfuuxwn.fsf@thinkbox | NOTMUCH_CONFIG=/tmp/selection-bug/notmuch-config notmuch insert
>>> notmuch show --format=raw id:8735buw7gf.fsf@ngraves.fr | NOTMUCH_CONFIG=/tmp/selection-bug/notmuch-config notmuch insert
>>> # refresh search buffer, bug: Nicolas' mail is green, mine is opened on pressing enter.
>>
>> 1) Do you think the particular messages matter? I don't have
>>    id:8735buw7gf.fsf@ngraves.fr
>
> Uh, you are right.  I meant to pick mails form notmuch@, but this one is
> from another list.  I actually reduced the example further:
>
> mkdir -p /tmp/selection-bug/{tmp,new,cur}
> echo "[database]
> path=/tmp/selection-bug" > /tmp/selection-bug/notmuch-config
> NOTMUCH_CONFIG=/tmp/selection-bug/notmuch-config notmuch new
> NOTMUCH_CONFIG=/tmp/selection-bug/notmuch-config emacs -q --eval "(require 'notmuch)"
> # M-x notmuch-hello, search for '*'
> notmuch show --format=raw id:87edvlanuh.fsf@tethera.net | NOTMUCH_CONFIG=/tmp/selection-bug/notmuch-config notmuch insert
> # refresh search buffer, press down once, i.e. navigate to "End of search results".
> notmuch show --format=raw id:87fsfuuxwn.fsf@thinkbox | NOTMUCH_CONFIG=/tmp/selection-bug/notmuch-config notmuch insert
> # refresh search buffer, bug: your mail is green, mine is opened on pressing enter.
>
> (I.e., just drop Nicolas' mail.)
>
> The bug can be seen even with just one mail, but then its impact is less
> severe:
>
> mkdir -p /tmp/selection-bug/{tmp,new,cur}
> echo "[database]
> path=/tmp/selection-bug" > /tmp/selection-bug/notmuch-config
> NOTMUCH_CONFIG=/tmp/selection-bug/notmuch-config notmuch new
> NOTMUCH_CONFIG=/tmp/selection-bug/notmuch-config emacs -q --eval "(require 'notmuch)"
> # M-x notmuch-hello, search for '*', press down once, i.e. navigate past the line "End of search results".
> notmuch show --format=raw id:87edvlanuh.fsf@tethera.net | NOTMUCH_CONFIG=/tmp/selection-bug/notmuch-config notmuch insert
> # refresh search buffer, bug: your mail is green, but in fact "End of search results" is selected and is "acted upon" when pressing enter.
>
>> 2) Just to be clear, that script has to be run manually, with the
>>    comments corresponding to interactive actions in emacs?
>
> Yes.
>
> As additional data point, I cannot reproduce the problem on my Desktop
> which has the same emacs version but uses elpa-notmuch 0.31.4-2.  On
> this system, if I run the reproducer, there is no line with a green
> background, so it is less misleading.

My desktop now behaves exactly like my laptop, and it is really
annoying.  Surely I'm not the only one with this problem.  Can no one
else reproduce this?

Best,
Justus

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 519 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: Selection bug

[Carl Worth]

On Tue, Apr 04 2023, Justus Winter wrote:
> My desktop now behaves exactly like my laptop, and it is really
> annoying.  Surely I'm not the only one with this problem.  Can no one
> else reproduce this?

I haven't tried using the reproduction test, but I did just refresh my
inbox and I saw the buggy behavior.

So there's a random anecdote which suggests it might not be that hard to
reproduce the bug, (perhaps doesn't require any special messages).

In my case, it's less annoying because I have more than a screenful of
messages in my inbox (for shame!) so the message with the green bar
isn't even visible when emacs' point is moved to the message that is
actually selected.

So if I hadn't been intentionally looking for this behavior I likely
would have never noticed.

-Carl

Re: Selection bug

[David Bremner]

Justus Winter <justus@sequoia-pgp.org> writes:

> David Bremner <david@tethera.net> writes:
>
> Uh, you are right.  I meant to pick mails form notmuch@, but this one is
> from another list.  I actually reduced the example further:
>
> mkdir -p /tmp/selection-bug/{tmp,new,cur}
> echo "[database]
> path=/tmp/selection-bug" > /tmp/selection-bug/notmuch-config
> NOTMUCH_CONFIG=/tmp/selection-bug/notmuch-config notmuch new
> NOTMUCH_CONFIG=/tmp/selection-bug/notmuch-config emacs -q --eval "(require 'notmuch)"
> # M-x notmuch-hello, search for '*'
> notmuch show --format=raw id:87edvlanuh.fsf@tethera.net | NOTMUCH_CONFIG=/tmp/selection-bug/notmuch-config notmuch insert
> # refresh search buffer, press down once, i.e. navigate to "End of search results".
> notmuch show --format=raw id:87fsfuuxwn.fsf@thinkbox | NOTMUCH_CONFIG=/tmp/selection-bug/notmuch-config notmuch insert
> # refresh search buffer, bug: your mail is green, mine is opened on pressing enter

This reproduces the problem for me. Indeed it looks a bit like a similar
"cursor jumping" problem I have seen (and someone else has reported on
IRC). I will see if I can figure anything more out now that I have a
reproducer.

Re: Selection bug

[David Bremner]

David Bremner <david@tethera.net> writes:

>
> This reproduces the problem for me. Indeed it looks a bit like a similar
> "cursor jumping" problem I have seen (and someone else has reported on
> IRC). I will see if I can figure anything more out now that I have a
> reproducer.

I think the following is a (simpler) reproducer of the same problem.

1) Do a search with at least 10 results

2) goto line 10

3) hit 'g'

For me this consistently leaves the point on line 10 but the hl-line bar
on line 1. I have not debugged it yet, but the proximate cause is that
when notmuch-hl-line-mode is called, (point) is 1, so the bar ends up on
the first line. The whys and wherefores I don't know yet.

Re: Selection bug

[David Bremner]

David Bremner <david@tethera.net> writes:

> David Bremner <david@tethera.net> writes:
>
>>
>> This reproduces the problem for me. Indeed it looks a bit like a similar
>> "cursor jumping" problem I have seen (and someone else has reported on
>> IRC). I will see if I can figure anything more out now that I have a
>> reproducer.
>
> I think the following is a (simpler) reproducer of the same problem.
>
> 1) Do a search with at least 10 results
>
> 2) goto line 10
>
> 3) hit 'g'
>
> For me this consistently leaves the point on line 10 but the hl-line bar
> on line 1. I have not debugged it yet, but the proximate cause is that
> when notmuch-hl-line-mode is called, (point) is 1, so the bar ends up on
> the first line. The whys and wherefores I don't know yet.

I have been slowly trying to debug this. It is a kind of heisenbug
because the position of the hl-line-mode bar is affected by using the
debugger (because hl-line-mode uses post-command-hook).

My current understanding is

0) the refresh calls notmuch-search to rebuild the buffer contents.

1) hl-line-highlight is being called via notmuch-search-hook

2) notmuch-search-hook is being invoked asynchronously from
   notmuch-search-process-filter.

3) There is a buffer local variable notmuch--search-hook-run used to
   make sure this happens only once per call to notmuch-search.

4) Unfortunately when the hook is invoked, (point) is 1, and that
   is how the hl-line-mode bar ends up on the first line, rather than
   where the (point) is after the refresh completes.

A) one question is why notmuch-search-hook is being run so early. I
vaguely remember some discussion / reason for this but I have not dug it
up.

B) I have some hope this can be fixed by use of the existing variable
   notmuch-search-target-line, but it's a bit tricky because
   at the point where the hook is invoked, the buffer does not have
   that many lines yet.

[PATCH] emacs: replace use of hook to draw hl-line in search mode

[David Bremner]

In the thread at id:87fsfuuxwn.fsf@thinkbox, a bug is discussed where
the point and hl-line overlay get out of sync, leading the user to
open the wrong message. As far as I can tell this is caused by
notmuch-hl-mode being invoked too early.

This change bypasses the logic preventing notmuch-search-hook being
called only once, so that the overlay is updated later after after the
buffer is full(er).

This change may lead to the overlay being updated multiple times; if
this is annoying we can add a similar buffer local variable to ensure
it is only called once.

The extra logic to check notmuch-search-target-line reduces the
flicker somewhat by not highlighting the first line every time.
---
 emacs/notmuch.el | 13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)

diff --git a/emacs/notmuch.el b/emacs/notmuch.el
index f55e9b42..108bccec 100644
--- a/emacs/notmuch.el
+++ b/emacs/notmuch.el
@@ -126,13 +126,17 @@ there will be called at other points of notmuch execution."
   :type 'file
   :group 'notmuch)
 
-(defcustom notmuch-search-hook '(notmuch-hl-line-mode)
+(defcustom notmuch-search-hook nil
   "List of functions to call when notmuch displays the search results."
   :type 'hook
-  :options '(notmuch-hl-line-mode)
   :group 'notmuch-search
   :group 'notmuch-hooks)
 
+(defcustom notmuch-hl-line t
+  "Use hl-line-mode to highlight current thread / message"
+  :type 'boolean
+  :group 'notmuch)
+
 ;;; Mime Utilities
 
 (defun notmuch-foreach-mime-part (function mm-handle)
@@ -925,6 +929,11 @@ sets the :orig-tag property."
 	(notmuch-sexp-parse-partial-list 'notmuch-search-append-result
 					 results-buf))
       (with-current-buffer results-buf
+	(when (and notmuch-hl-line
+		   (>=
+		    (line-number-at-pos (point-max) t)
+		    (or notmuch-search-target-line -1)))
+	  (notmuch-hl-line-mode))
 	(notmuch--search-hook-wrapper)))))
 
 ;;; Commands (and some helper functions used by them)
-- 
2.43.0