unofficial mirror of notmuch@notmuchmail.org
 help / color / mirror / code / Atom feed
* 'notmuch search thread:<>' lists multiple threads
@ 2018-04-06  9:46 Naveen N. Rao
  2018-04-06 10:23 ` Naveen N. Rao
  2018-04-08  3:04 ` David Bremner
  0 siblings, 2 replies; 16+ messages in thread
From: Naveen N. Rao @ 2018-04-06  9:46 UTC (permalink / raw)
  To: notmuch

Greetings--
If I search for threads matching a specific thread-id, I am seeing 
multiple results:

$ notmuch search --output=threads thread:00000000000c4d20
thread:00000000000c4d1e
thread:00000000000c4d20

If I list the messages from both those threads, they do belong to the 
same original mailing list thread. It isn't clear why notmuch is 
assigning different thread IDs. Is that to be expected under some 
scenarios?

Also, it is a bit weird to see multiple threads being listed when 
searching for a specific thread ID. Again, is this something to be 
expected?


- Naveen


^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: 'notmuch search thread:<>' lists multiple threads
  2018-04-06  9:46 'notmuch search thread:<>' lists multiple threads Naveen N. Rao
@ 2018-04-06 10:23 ` Naveen N. Rao
  2018-04-08  3:04 ` David Bremner
  1 sibling, 0 replies; 16+ messages in thread
From: Naveen N. Rao @ 2018-04-06 10:23 UTC (permalink / raw)
  To: notmuch

Naveen N. Rao wrote:
> Greetings--
> If I search for threads matching a specific thread-id, I am seeing 
> multiple results:
> 
> $ notmuch search --output=threads thread:00000000000c4d20
> thread:00000000000c4d1e
> thread:00000000000c4d20

Expanding on this:

[04/06 15:37:59 ~]$ notmuch search --output=messages thread:00000000000c4d1e
id:CAKTCnzngahex_sL2raoHFuXqTxgVV7a57R9YmcT1TN-ROsFXnw@mail.gmail.com
[04/06 15:49:34 ~]$ 
[04/06 15:38:01 ~]$ notmuch search --output=messages thread:00000000000c4d20
id:CAKTCnzngahex_sL2raoHFuXqTxgVV7a57R9YmcT1TN-ROsFXnw@mail.gmail.com
id:CAOSf1CFy0im+H368Krr2QVteAumq58CRF_Kfatyv7wvj1n4AGw@mail.gmail.com
id:20180405071500.22320-4-bsingharora@gmail.com
id:20180405071500.22320-3-bsingharora@gmail.com
id:20180405071500.22320-2-bsingharora@gmail.com
id:20180405071500.22320-1-bsingharora@gmail.com
[04/06 15:49:34 ~]$ 
[04/06 15:49:26 ~]$ notmuch show --format=raw id:CAKTCnzngahex_sL2raoHFuXqTxgVV7a57R9YmcT1TN-ROsFXnw@mail.gmail.com | grep -e "In-Reply-To" -e "References" -A2
In-Reply-To:
 <CAOSf1CFy0im+H368Krr2QVteAumq58CRF_Kfatyv7wvj1n4AGw@mail.gmail.com>
References: <20180405071500.22320-1-bsingharora@gmail.com>
 <20180405071500.22320-3-bsingharora@gmail.com>
 <CAOSf1CFy0im+H368Krr2QVteAumq58CRF_Kfatyv7wvj1n4AGw@mail.gmail.com>
[04/06 15:50:01 ~]$ 
[04/06 15:50:02 ~]$ notmuch show --format=raw id:CAOSf1CFy0im+H368Krr2QVteAumq58CRF_Kfatyv7wvj1n4AGw@mail.gmail.com | grep -e "In-Reply-To" -e "References" -A1
In-Reply-To: <20180405071500.22320-3-bsingharora@gmail.com>
References: <20180405071500.22320-1-bsingharora@gmail.com>
 <20180405071500.22320-3-bsingharora@gmail.com>


- Naveen


^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: 'notmuch search thread:<>' lists multiple threads
  2018-04-06  9:46 'notmuch search thread:<>' lists multiple threads Naveen N. Rao
  2018-04-06 10:23 ` Naveen N. Rao
@ 2018-04-08  3:04 ` David Bremner
  2018-04-09 11:54   ` David Bremner
  1 sibling, 1 reply; 16+ messages in thread
From: David Bremner @ 2018-04-08  3:04 UTC (permalink / raw)
  To: Naveen N. Rao, notmuch

"Naveen N. Rao" <naveen.n.rao@linux.vnet.ibm.com> writes:

> Greetings--
> If I search for threads matching a specific thread-id, I am seeing 
> multiple results:
>
> $ notmuch search --output=threads thread:00000000000c4d20
> thread:00000000000c4d1e
> thread:00000000000c4d20

This looks like a bug to me. I was able to replicate it in my own mail
store with the script at the end of the message. I haven't completely
analyzed the situation yet, but one thing I noticed is that in all
"bad threads", there are files with duplicate message-ids. Typical
output looks like

╭─ zancas:software/upstream/notmuch/test 
╰─ (git)-[master]-% notmuch search thread:000000000001760a
thread:00000000000175e5  November 03 [1/2(3)] 128@gmx.us; Bug#846042: VTK 8 (unread)
thread:000000000001760a   2016-11-27 [1/2(3)] 128@gmx.us; Bug#846042: virtual/meta package for python-vtk (unread)

At least some of this mail data is public, but I'm not sure if the bad
threading is reproducible or not; I want to run a complete census
overnight before I reindex.

Even if the bug is non-deterministic, it probably lives in lib/add-message.cc

----------------------------------------------------------------------

count=0
success=0
for id in $(notmuch search --output=threads '*'); do
    count=$((count +1))
    matches=$((`notmuch search --output=threads "$id" | wc -l`))
    if [ "$matches" = 1 ]; then
	success=$((success + 1))
    else
        echo "bad thread: $id"
    fi
    if [ $((count % 1000)) -eq 0 ]; then
        echo $count;
    fi
done

echo "count=$count success=$success"

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: 'notmuch search thread:<>' lists multiple threads
  2018-04-08  3:04 ` David Bremner
@ 2018-04-09 11:54   ` David Bremner
  2018-04-10  1:45     ` [PATCH] devel: add new tool to draw thread structure David Bremner
  2018-04-18 10:18     ` 'notmuch search thread:<>' lists multiple threads Naveen N. Rao
  0 siblings, 2 replies; 16+ messages in thread
From: David Bremner @ 2018-04-09 11:54 UTC (permalink / raw)
  To: Naveen N. Rao, notmuch

David Bremner <david@tethera.net> writes:

> At least some of this mail data is public, but I'm not sure if the bad
> threading is reproducible or not; I want to run a complete census
> overnight before I reindex.
>
> Even if the bug is non-deterministic, it probably lives in lib/add-message.cc

I have a reproducible test for this bug now

  http://pivot.cs.unb.ca/git?p=notmuch.git;a=shortlog;h=refs/heads/fix/thread-search

I still need to analyze the mails a bit more, but it looks like at least
one of the strange results is caused by multiple mail files sharing the
same message-id, but with different References headers (and no
In-Reply-To headers).

d

^ permalink raw reply	[flat|nested] 16+ messages in thread

* [PATCH] devel: add new tool to draw thread structure
  2018-04-09 11:54   ` David Bremner
@ 2018-04-10  1:45     ` David Bremner
  2018-10-08  3:30       ` Daniel Kahn Gillmor
  2018-04-18 10:18     ` 'notmuch search thread:<>' lists multiple threads Naveen N. Rao
  1 sibling, 1 reply; 16+ messages in thread
From: David Bremner @ 2018-04-10  1:45 UTC (permalink / raw)
  To: David Bremner, Naveen N. Rao, notmuch

This is useful for understanding the case where different
message-files with the same message-id have distinct reference
headers.
---
 devel/draw-thread | 35 +++++++++++++++++++++++++++++++++++
 1 file changed, 35 insertions(+)
 create mode 100755 devel/draw-thread

diff --git a/devel/draw-thread b/devel/draw-thread
new file mode 100755
index 00000000..628dcff4
--- /dev/null
+++ b/devel/draw-thread
@@ -0,0 +1,35 @@
+#!/bin/bash
+
+# This script can be used like
+# NOTMUCH_CONFIG=test/tmp.T580-thread-search/notmuch-config \
+#    devel/draw-thread thread:0000000000000002 | dot -Tpdf > thread2.pdf
+
+# In addition to notmuch, you will need the following tools installed
+# - graphviz
+# - formail (part of procmail)
+
+threadid=$1
+
+declare -a edges
+
+declare -a dest
+echo "digraph \"$threadid\" {"
+for messageid in $(notmuch search --output=messages $threadid); do
+    echo "subgraph \"cluster_$messageid\" {"
+    printf "\"%s\" [shape=folder];\n" ${messageid#id:}
+    for file in $(notmuch search --output=files $messageid); do
+        node=$(basename $file)
+        printf "\"%s\" [shape=note];\n" $node
+
+        mapfile -t dest < <(formail -x references < $file | tr '<>,' '"" ')
+        edge="\"$node\" -> { ${dest[*]} }"
+        edges+=($edge)
+    done
+    echo "}"
+done
+
+for edge in "${edges[*]}"; do
+    echo $edge
+done
+
+echo "}"
-- 
2.16.3

^ permalink raw reply related	[flat|nested] 16+ messages in thread

* Re: 'notmuch search thread:<>' lists multiple threads
  2018-04-09 11:54   ` David Bremner
  2018-04-10  1:45     ` [PATCH] devel: add new tool to draw thread structure David Bremner
@ 2018-04-18 10:18     ` Naveen N. Rao
  2018-04-22  0:45       ` David Bremner
  2021-12-23 18:00       ` David Bremner
  1 sibling, 2 replies; 16+ messages in thread
From: Naveen N. Rao @ 2018-04-18 10:18 UTC (permalink / raw)
  To: David Bremner, notmuch

David Bremner wrote:
> David Bremner <david@tethera.net> writes:
> 
>> At least some of this mail data is public, but I'm not sure if the bad
>> threading is reproducible or not; I want to run a complete census
>> overnight before I reindex.
>>
>> Even if the bug is non-deterministic, it probably lives in lib/add-message.cc
> 
> I have a reproducible test for this bug now
> 
>   http://pivot.cs.unb.ca/git?p=notmuch.git;a=shortlog;h=refs/heads/fix/thread-search

Thanks for looking into this.

> 
> I still need to analyze the mails a bit more, but it looks like at least
> one of the strange results is caused by multiple mail files sharing the
> same message-id, but with different References headers (and no
> In-Reply-To headers).

In my case, I seem to be having the In-Reply-To headers. I end up with 
two files per message: one from my inbox and one from the gmane archive 
that I pull in. All the messages from the gmane archive seem to have a 
re-written 'In-Reply-To' header, but 'Message-Id' and 'References' are 
the same.

In the problematic email thread, all other files/messages get allotted a 
single thread except for one of the messages. The offending message has 
3 references compared to 1 or 2 references for the rest, but I don't 
know if that's relevant here.

- Naveen


^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: 'notmuch search thread:<>' lists multiple threads
  2018-04-18 10:18     ` 'notmuch search thread:<>' lists multiple threads Naveen N. Rao
@ 2018-04-22  0:45       ` David Bremner
  2018-06-28 10:36         ` Naveen N. Rao
  2021-12-23 18:00       ` David Bremner
  1 sibling, 1 reply; 16+ messages in thread
From: David Bremner @ 2018-04-22  0:45 UTC (permalink / raw)
  To: Naveen N. Rao, notmuch

"Naveen N. Rao" <naveen.n.rao@linux.vnet.ibm.com> writes:

> In my case, I seem to be having the In-Reply-To headers. I end up with 
> two files per message: one from my inbox and one from the gmane archive 
> that I pull in. All the messages from the gmane archive seem to have a 
> re-written 'In-Reply-To' header, but 'Message-Id' and 'References' are 
> the same.

That sounds like essentially the same issue, due to the fact that
notmuch prefers In-Reply-To when choosing a parent for a message.

Currently the database is correct (or at least one not-crazy definition
of correct): all of the reference and in-reply-to terms are attached to
the message document in the database. On the other hand, the in memory
data structures currently assume that In-reply-to is a unique value
(with ties broken at indexing time).

It might be that the solution is to read a list of in-reply-to values
and use all of them in threading. At a quick glance, that looks doable;
I'm just not sure about unintended consequences.

d

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: 'notmuch search thread:<>' lists multiple threads
  2018-04-22  0:45       ` David Bremner
@ 2018-06-28 10:36         ` Naveen N. Rao
  2018-06-30 13:42           ` David Bremner
  0 siblings, 1 reply; 16+ messages in thread
From: Naveen N. Rao @ 2018-06-28 10:36 UTC (permalink / raw)
  To: David Bremner, notmuch

Hi David,

David Bremner wrote:
> "Naveen N. Rao" <naveen.n.rao@linux.vnet.ibm.com> writes:
> 
>> In my case, I seem to be having the In-Reply-To headers. I end up with 
>> two files per message: one from my inbox and one from the gmane archive 
>> that I pull in. All the messages from the gmane archive seem to have a 
>> re-written 'In-Reply-To' header, but 'Message-Id' and 'References' are 
>> the same.
> 
> That sounds like essentially the same issue, due to the fact that
> notmuch prefers In-Reply-To when choosing a parent for a message.
> 
> Currently the database is correct (or at least one not-crazy definition
> of correct): all of the reference and in-reply-to terms are attached to
> the message document in the database. On the other hand, the in memory
> data structures currently assume that In-reply-to is a unique value
> (with ties broken at indexing time).
> 
> It might be that the solution is to read a list of in-reply-to values
> and use all of them in threading. At a quick glance, that looks doable;
> I'm just not sure about unintended consequences.

Were you able to look into this again?
Using a list of in-reply-to values sounds like a good option, though I 
clearly have no idea about other consequences from that. If you have a 
patch, I can help test that.

Thanks,
Naveen


^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: 'notmuch search thread:<>' lists multiple threads
  2018-06-28 10:36         ` Naveen N. Rao
@ 2018-06-30 13:42           ` David Bremner
  2018-08-30 12:52             ` David Bremner
  0 siblings, 1 reply; 16+ messages in thread
From: David Bremner @ 2018-06-30 13:42 UTC (permalink / raw)
  To: Naveen N. Rao, notmuch

"Naveen N. Rao" <naveen.n.rao@linux.vnet.ibm.com> writes:

>
> Were you able to look into this again?
> Using a list of in-reply-to values sounds like a good option, though I 
> clearly have no idea about other consequences from that. If you have a 
> patch, I can help test that.
>

Sorry I haven't made any progress on this. Thanks for the reminder.

d

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: 'notmuch search thread:<>' lists multiple threads
  2018-06-30 13:42           ` David Bremner
@ 2018-08-30 12:52             ` David Bremner
  2018-09-06 10:50               ` Naveen N. Rao
  0 siblings, 1 reply; 16+ messages in thread
From: David Bremner @ 2018-08-30 12:52 UTC (permalink / raw)
  To: Naveen N. Rao, notmuch

David Bremner <david@tethera.net> writes:

> "Naveen N. Rao" <naveen.n.rao@linux.vnet.ibm.com> writes:
>
>>
>> Were you able to look into this again?
>> Using a list of in-reply-to values sounds like a good option, though I 
>> clearly have no idea about other consequences from that. If you have a 
>> patch, I can help test that.
>>
>
> Sorry I haven't made any progress on this. Thanks for the reminder.
>
> d

It's not much progress but I did manage to make a test case.

     id:20180730224555.26047-16-david@tethera.net

As it says in the commit message, its not 100% clear this is your
problem, but it is a bug, and hopefully fixing it will help your issue.

d

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: 'notmuch search thread:<>' lists multiple threads
  2018-08-30 12:52             ` David Bremner
@ 2018-09-06 10:50               ` Naveen N. Rao
  2019-06-27 14:45                 ` Naveen N. Rao
  0 siblings, 1 reply; 16+ messages in thread
From: Naveen N. Rao @ 2018-09-06 10:50 UTC (permalink / raw)
  To: David Bremner, notmuch

David Bremner wrote:
> David Bremner <david@tethera.net> writes:
> 
>> "Naveen N. Rao" <naveen.n.rao@linux.vnet.ibm.com> writes:
>>
>>>
>>> Were you able to look into this again?
>>> Using a list of in-reply-to values sounds like a good option, though I 
>>> clearly have no idea about other consequences from that. If you have a 
>>> patch, I can help test that.
>>>
>>
>> Sorry I haven't made any progress on this. Thanks for the reminder.
>>
>> d
> 
> It's not much progress but I did manage to make a test case.
> 
>      id:20180730224555.26047-16-david@tethera.net
> 
> As it says in the commit message, its not 100% clear this is your
> problem, but it is a bug, and hopefully fixing it will help your issue.

Thanks for continuing to look into this!

The test is close to what I have -- the only difference in my case is 
that the Message-ID and References: fields match in the duplicate mail 
files, but just the In-reply-to headers differ (the gmane one has a 
meaningless/incorrect, re-written header).

Interestingly though, I re-checked the thread I had the original problem 
with and notmuch seems to be able to cope with it better now. So, some 
other changes seem to have helped with my original problem. I will keep 
an eye out to see if any other threads cause problems (I do occasionally 
see astroid crash, but I haven't seen if it is due to this issue with 
notmuch or a different problem).

  [09/06 16:19:19 ~]$ notmuch --version
  notmuch 0.27
  [09/06 16:19:20 ~]$ notmuch search --output=threads thread:00000000000c4d20
  thread:00000000000c4d20
  [09/06 16:19:22 ~]$ notmuch search --output=threads thread:00000000000c4d1e
  thread:00000000000c4d1e
  [09/06 16:19:26 ~]$ notmuch search --output=messages thread:00000000000c4d20
  id:20180813154910.lzrkd2ivdorge3ro@arbab-laptop.localdomain
  id:CAKTCnzngahex_sL2raoHFuXqTxgVV7a57R9YmcT1TN-ROsFXnw@mail.gmail.com
  id:CAOSf1CFy0im+H368Krr2QVteAumq58CRF_Kfatyv7wvj1n4AGw@mail.gmail.com
  id:20180405071500.22320-4-bsingharora@gmail.com
  id:20180405071500.22320-3-bsingharora@gmail.com
  id:20180405071500.22320-2-bsingharora@gmail.com
  id:20180405071500.22320-1-bsingharora@gmail.com
  [09/06 16:19:35 ~]$ notmuch search --output=messages thread:00000000000c4d1e
  id:CAKTCnzngahex_sL2raoHFuXqTxgVV7a57R9YmcT1TN-ROsFXnw@mail.gmail.com


- Naveen


^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: [PATCH] devel: add new tool to draw thread structure
  2018-04-10  1:45     ` [PATCH] devel: add new tool to draw thread structure David Bremner
@ 2018-10-08  3:30       ` Daniel Kahn Gillmor
  0 siblings, 0 replies; 16+ messages in thread
From: Daniel Kahn Gillmor @ 2018-10-08  3:30 UTC (permalink / raw)
  To: notmuch

[-- Attachment #1: Type: text/plain, Size: 643 bytes --]

On Mon 2018-04-09 22:45:39 -0300, David Bremner wrote:
> This is useful for understanding the case where different
> message-files with the same message-id have distinct reference
> headers.
> ---
>  devel/draw-thread | 35 +++++++++++++++++++++++++++++++++++
>  1 file changed, 35 insertions(+)
>  create mode 100755 devel/draw-thread

fwiw, i think this tool is useful enough for debugging and visibility
that we should go ahead and include it in the notmuch source, under
devel/ -- we don't need to ship it publicly, but it'd be great to have
it easily available to all notmuch devs who might be dealing with this
kind of thing.

     --dkg

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: 'notmuch search thread:<>' lists multiple threads
  2018-09-06 10:50               ` Naveen N. Rao
@ 2019-06-27 14:45                 ` Naveen N. Rao
  2019-06-27 14:53                   ` Naveen N. Rao
  0 siblings, 1 reply; 16+ messages in thread
From: Naveen N. Rao @ 2019-06-27 14:45 UTC (permalink / raw)
  To: David Bremner, notmuch

Naveen N. Rao wrote:
> David Bremner wrote:
>> David Bremner <david@tethera.net> writes:
>> 
>>> "Naveen N. Rao" <naveen.n.rao@linux.vnet.ibm.com> writes:
>>>
>>>>
>>>> Were you able to look into this again?
>>>> Using a list of in-reply-to values sounds like a good option, though I 
>>>> clearly have no idea about other consequences from that. If you have a 
>>>> patch, I can help test that.
>>>>
>>>
>>> Sorry I haven't made any progress on this. Thanks for the reminder.
>>>
>>> d
>> 
>> It's not much progress but I did manage to make a test case.
>> 
>>      id:20180730224555.26047-16-david@tethera.net
>> 
>> As it says in the commit message, its not 100% clear this is your
>> problem, but it is a bug, and hopefully fixing it will help your issue.
> 
> Thanks for continuing to look into this!
> 
> The test is close to what I have -- the only difference in my case is 
> that the Message-ID and References: fields match in the duplicate mail 
> files, but just the In-reply-to headers differ (the gmane one has a 
> meaningless/incorrect, re-written header).
> 
> Interestingly though, I re-checked the thread I had the original problem 
> with and notmuch seems to be able to cope with it better now. So, some 
> other changes seem to have helped with my original problem. I will keep 
> an eye out to see if any other threads cause problems (I do occasionally 
> see astroid crash, but I haven't seen if it is due to this issue with 
> notmuch or a different problem).
> 
>   [09/06 16:19:19 ~]$ notmuch --version
>   notmuch 0.27
>   [09/06 16:19:20 ~]$ notmuch search --output=threads thread:00000000000c4d20
>   thread:00000000000c4d20
>   [09/06 16:19:22 ~]$ notmuch search --output=threads thread:00000000000c4d1e
>   thread:00000000000c4d1e
>   [09/06 16:19:26 ~]$ notmuch search --output=messages thread:00000000000c4d20
>   id:20180813154910.lzrkd2ivdorge3ro@arbab-laptop.localdomain
>   id:CAKTCnzngahex_sL2raoHFuXqTxgVV7a57R9YmcT1TN-ROsFXnw@mail.gmail.com
>   id:CAOSf1CFy0im+H368Krr2QVteAumq58CRF_Kfatyv7wvj1n4AGw@mail.gmail.com
>   id:20180405071500.22320-4-bsingharora@gmail.com
>   id:20180405071500.22320-3-bsingharora@gmail.com
>   id:20180405071500.22320-2-bsingharora@gmail.com
>   id:20180405071500.22320-1-bsingharora@gmail.com
>   [09/06 16:19:35 ~]$ notmuch search --output=messages thread:00000000000c4d1e
>   id:CAKTCnzngahex_sL2raoHFuXqTxgVV7a57R9YmcT1TN-ROsFXnw@mail.gmail.com

I've since moved away from Gmane and using public-inbox now and things 
have been good all this while. Today, however, I hit this issue again.  
I pull archives from lkml and linuxppc-dev and this thread is pulled in 
from both those places, as well as from my local IMAP sent folder:
https://lore.kernel.org/linuxppc-dev/cover.1561634177.git.naveen.n.rao@linux.vnet.ibm.com/T/#t
https://lore.kernel.org/lkml/cover.1561634177.git.naveen.n.rao@linux.vnet.ibm.com/T/#t

[06/27 19:56:07 ~]$ notmuch search -- tag:ppcdev and date:today and from:naveen
thread:00000000000e364b  Today 16:57 [1/1(3)] Naveen N. Rao; [PATCH v2 2/7] x86/ftrace: Fix use of flags in ftrace_replace_code() (imap inbox lkml ppcdev sent)
thread:00000000000e364d  Today 16:53 [8/11(31)] Naveen N. Rao| Steven Rostedt, Masami Hiramatsu; [PATCH v2 7/7] powerpc/kprobes: Allow probing on any ftrace address (imap inbox lkml new ppcdev replied sent unread)
thread:00000000000e3636  Today 15:29 [1/1(2)] Naveen N. Rao; [PATCH] powerpc/xmon: Fix disabling tracing while in xmon (imap inbox ppcdev)
thread:00000000000e357e  Today 00:08 [1/4(12)] Naveen N. Rao| Michael Ellerman, Satheesh Rajendran, Steven Rostedt; [PATCH] recordmcount: Fix spurious mcount entries on powerpc (imap inbox lkml ppcdev)
[06/27 19:56:09 ~]$ 
[06/27 19:56:14 ~]$ notmuch search --output=files -- thread:00000000000e364b
/home/naveen/news/news/ltcimap/Sent/cur/1561634919_0.18171.naverao1-tp,U=1356,FMD5=7f8c0283f16925caed8e632086b81b9c:2,S
/home/naveen/news/news/lore.kernel.org/linuxppc-dev/0/223040
/home/naveen/news/news/lore.kernel.org/lkml/7/120234
[06/27 19:56:34 ~]$ notmuch search --output=files -- thread:00000000000e364d
/home/naveen/news/news/ltcimap/INBOX/new/1561645220_0.8024.naverao1-tp,U=151133,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,
/home/naveen/news/news/ltcimap/INBOX/cur/1561643871_0.8024.naverao1-tp,U=151132,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,S
/home/naveen/news/news/lore.kernel.org/linuxppc-dev/0/223042
/home/naveen/news/news/lore.kernel.org/lkml/7/120325
/home/naveen/news/news/ltcimap/Sent/cur/1561634919_0.18171.naverao1-tp,U=1356,FMD5=7f8c0283f16925caed8e632086b81b9c:2,S
/home/naveen/news/news/lore.kernel.org/linuxppc-dev/0/223040
/home/naveen/news/news/lore.kernel.org/lkml/7/120234
/home/naveen/news/news/ltcimap/INBOX/cur/1561634683_0.18171.naverao1-tp,U=151129,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,S
/home/naveen/news/news/lore.kernel.org/linuxppc-dev/0/223038
/home/naveen/news/news/lore.kernel.org/lkml/7/120225
/home/naveen/news/news/ltcimap/INBOX/cur/1561634682_6.18171.naverao1-tp,U=151128,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,S
/home/naveen/news/news/lore.kernel.org/linuxppc-dev/0/223039
/home/naveen/news/news/lore.kernel.org/lkml/7/120224
/home/naveen/news/news/ltcimap/INBOX/cur/1561634682_5.18171.naverao1-tp,U=151127,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,S
/home/naveen/news/news/lore.kernel.org/linuxppc-dev/0/223037
/home/naveen/news/news/lore.kernel.org/lkml/7/120223
/home/naveen/news/news/ltcimap/INBOX/cur/1561634682_0.18171.naverao1-tp,U=151126,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,S
/home/naveen/news/news/lore.kernel.org/linuxppc-dev/0/223036
/home/naveen/news/news/lore.kernel.org/lkml/7/120222
/home/naveen/news/news/ltcimap/INBOX/cur/1561634682_4.18171.naverao1-tp,U=151125,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,S
/home/naveen/news/news/lore.kernel.org/linuxppc-dev/0/223035
/home/naveen/news/news/lore.kernel.org/lkml/7/120221
/home/naveen/news/news/ltcimap/INBOX/cur/1561634682_1.18171.naverao1-tp,U=151124,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,RS
/home/naveen/news/news/lore.kernel.org/linuxppc-dev/0/223034
/home/naveen/news/news/lore.kernel.org/lkml/7/120220
/home/naveen/news/news/ltcimap/INBOX/cur/1561634682_3.18171.naverao1-tp,U=151123,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,S
/home/naveen/news/news/lore.kernel.org/linuxppc-dev/0/223033
/home/naveen/news/news/lore.kernel.org/lkml/7/120219
/home/naveen/news/news/ltcimap/INBOX/cur/1561634682_2.18171.naverao1-tp,U=151122,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,S
/home/naveen/news/news/lore.kernel.org/linuxppc-dev/0/223031
/home/naveen/news/news/lore.kernel.org/lkml/7/120218

- Naveen


^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: 'notmuch search thread:<>' lists multiple threads
  2019-06-27 14:45                 ` Naveen N. Rao
@ 2019-06-27 14:53                   ` Naveen N. Rao
  0 siblings, 0 replies; 16+ messages in thread
From: Naveen N. Rao @ 2019-06-27 14:53 UTC (permalink / raw)
  To: David Bremner, notmuch

Naveen N. Rao wrote:
> Naveen N. Rao wrote:
>> David Bremner wrote:
>>> David Bremner <david@tethera.net> writes:
>>> 
>>>> "Naveen N. Rao" <naveen.n.rao@linux.vnet.ibm.com> writes:
>>>>
>>>>>
>>>>> Were you able to look into this again?
>>>>> Using a list of in-reply-to values sounds like a good option, though I 
>>>>> clearly have no idea about other consequences from that. If you have a 
>>>>> patch, I can help test that.
>>>>>
>>>>
>>>> Sorry I haven't made any progress on this. Thanks for the reminder.
>>>>
>>>> d
>>> 
>>> It's not much progress but I did manage to make a test case.
>>> 
>>>      id:20180730224555.26047-16-david@tethera.net
>>> 
>>> As it says in the commit message, its not 100% clear this is your
>>> problem, but it is a bug, and hopefully fixing it will help your issue.
>> 
>> Thanks for continuing to look into this!
>> 
>> The test is close to what I have -- the only difference in my case is 
>> that the Message-ID and References: fields match in the duplicate mail 
>> files, but just the In-reply-to headers differ (the gmane one has a 
>> meaningless/incorrect, re-written header).
>> 
>> Interestingly though, I re-checked the thread I had the original problem 
>> with and notmuch seems to be able to cope with it better now. So, some 
>> other changes seem to have helped with my original problem. I will keep 
>> an eye out to see if any other threads cause problems (I do occasionally 
>> see astroid crash, but I haven't seen if it is due to this issue with 
>> notmuch or a different problem).
>> 
>>   [09/06 16:19:19 ~]$ notmuch --version
>>   notmuch 0.27
>>   [09/06 16:19:20 ~]$ notmuch search --output=threads thread:00000000000c4d20
>>   thread:00000000000c4d20
>>   [09/06 16:19:22 ~]$ notmuch search --output=threads thread:00000000000c4d1e
>>   thread:00000000000c4d1e
>>   [09/06 16:19:26 ~]$ notmuch search --output=messages thread:00000000000c4d20
>>   id:20180813154910.lzrkd2ivdorge3ro@arbab-laptop.localdomain
>>   id:CAKTCnzngahex_sL2raoHFuXqTxgVV7a57R9YmcT1TN-ROsFXnw@mail.gmail.com
>>   id:CAOSf1CFy0im+H368Krr2QVteAumq58CRF_Kfatyv7wvj1n4AGw@mail.gmail.com
>>   id:20180405071500.22320-4-bsingharora@gmail.com
>>   id:20180405071500.22320-3-bsingharora@gmail.com
>>   id:20180405071500.22320-2-bsingharora@gmail.com
>>   id:20180405071500.22320-1-bsingharora@gmail.com
>>   [09/06 16:19:35 ~]$ notmuch search --output=messages thread:00000000000c4d1e
>>   id:CAKTCnzngahex_sL2raoHFuXqTxgVV7a57R9YmcT1TN-ROsFXnw@mail.gmail.com
> 
> I've since moved away from Gmane and using public-inbox now and things 
> have been good all this while. Today, however, I hit this issue again.  
> I pull archives from lkml and linuxppc-dev and this thread is pulled in 
> from both those places, as well as from my local IMAP sent folder:
> https://lore.kernel.org/linuxppc-dev/cover.1561634177.git.naveen.n.rao@linux.vnet.ibm.com/T/#t
> https://lore.kernel.org/lkml/cover.1561634177.git.naveen.n.rao@linux.vnet.ibm.com/T/#t
> 
> [06/27 19:56:07 ~]$ notmuch search -- tag:ppcdev and date:today and from:naveen
> thread:00000000000e364b  Today 16:57 [1/1(3)] Naveen N. Rao; [PATCH v2 2/7] x86/ftrace: Fix use of flags in ftrace_replace_code() (imap inbox lkml ppcdev sent)
> thread:00000000000e364d  Today 16:53 [8/11(31)] Naveen N. Rao| Steven Rostedt, Masami Hiramatsu; [PATCH v2 7/7] powerpc/kprobes: Allow probing on any ftrace address (imap inbox lkml new ppcdev replied sent unread)
> thread:00000000000e3636  Today 15:29 [1/1(2)] Naveen N. Rao; [PATCH] powerpc/xmon: Fix disabling tracing while in xmon (imap inbox ppcdev)
> thread:00000000000e357e  Today 00:08 [1/4(12)] Naveen N. Rao| Michael Ellerman, Satheesh Rajendran, Steven Rostedt; [PATCH] recordmcount: Fix spurious mcount entries on powerpc (imap inbox lkml ppcdev)
> [06/27 19:56:09 ~]$ 
> [06/27 19:56:14 ~]$ notmuch search --output=files -- thread:00000000000e364b
> /home/naveen/news/news/ltcimap/Sent/cur/1561634919_0.18171.naverao1-tp,U=1356,FMD5=7f8c0283f16925caed8e632086b81b9c:2,S
> /home/naveen/news/news/lore.kernel.org/linuxppc-dev/0/223040
> /home/naveen/news/news/lore.kernel.org/lkml/7/120234
> [06/27 19:56:34 ~]$ notmuch search --output=files -- thread:00000000000e364d
> /home/naveen/news/news/ltcimap/INBOX/new/1561645220_0.8024.naverao1-tp,U=151133,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,
> /home/naveen/news/news/ltcimap/INBOX/cur/1561643871_0.8024.naverao1-tp,U=151132,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,S
> /home/naveen/news/news/lore.kernel.org/linuxppc-dev/0/223042
> /home/naveen/news/news/lore.kernel.org/lkml/7/120325
> /home/naveen/news/news/ltcimap/Sent/cur/1561634919_0.18171.naverao1-tp,U=1356,FMD5=7f8c0283f16925caed8e632086b81b9c:2,S
> /home/naveen/news/news/lore.kernel.org/linuxppc-dev/0/223040
> /home/naveen/news/news/lore.kernel.org/lkml/7/120234
> /home/naveen/news/news/ltcimap/INBOX/cur/1561634683_0.18171.naverao1-tp,U=151129,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,S
> /home/naveen/news/news/lore.kernel.org/linuxppc-dev/0/223038
> /home/naveen/news/news/lore.kernel.org/lkml/7/120225
> /home/naveen/news/news/ltcimap/INBOX/cur/1561634682_6.18171.naverao1-tp,U=151128,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,S
> /home/naveen/news/news/lore.kernel.org/linuxppc-dev/0/223039
> /home/naveen/news/news/lore.kernel.org/lkml/7/120224
> /home/naveen/news/news/ltcimap/INBOX/cur/1561634682_5.18171.naverao1-tp,U=151127,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,S
> /home/naveen/news/news/lore.kernel.org/linuxppc-dev/0/223037
> /home/naveen/news/news/lore.kernel.org/lkml/7/120223
> /home/naveen/news/news/ltcimap/INBOX/cur/1561634682_0.18171.naverao1-tp,U=151126,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,S
> /home/naveen/news/news/lore.kernel.org/linuxppc-dev/0/223036
> /home/naveen/news/news/lore.kernel.org/lkml/7/120222
> /home/naveen/news/news/ltcimap/INBOX/cur/1561634682_4.18171.naverao1-tp,U=151125,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,S
> /home/naveen/news/news/lore.kernel.org/linuxppc-dev/0/223035
> /home/naveen/news/news/lore.kernel.org/lkml/7/120221
> /home/naveen/news/news/ltcimap/INBOX/cur/1561634682_1.18171.naverao1-tp,U=151124,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,RS
> /home/naveen/news/news/lore.kernel.org/linuxppc-dev/0/223034
> /home/naveen/news/news/lore.kernel.org/lkml/7/120220
> /home/naveen/news/news/ltcimap/INBOX/cur/1561634682_3.18171.naverao1-tp,U=151123,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,S
> /home/naveen/news/news/lore.kernel.org/linuxppc-dev/0/223033
> /home/naveen/news/news/lore.kernel.org/lkml/7/120219
> /home/naveen/news/news/ltcimap/INBOX/cur/1561634682_2.18171.naverao1-tp,U=151122,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,S
> /home/naveen/news/news/lore.kernel.org/linuxppc-dev/0/223031
> /home/naveen/news/news/lore.kernel.org/lkml/7/120218

After being unable to open this thread in astroid email client a few 
times, I was able to resolve this by using 'notmuch reindex' _twice_.  
The result quoted above was after I reindexed it once. But, the second 
re-index seems to have fixed this magically!

- Naveen


^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: 'notmuch search thread:<>' lists multiple threads
  2018-04-18 10:18     ` 'notmuch search thread:<>' lists multiple threads Naveen N. Rao
  2018-04-22  0:45       ` David Bremner
@ 2021-12-23 18:00       ` David Bremner
  2022-01-14 11:26         ` Naveen N. Rao
  1 sibling, 1 reply; 16+ messages in thread
From: David Bremner @ 2021-12-23 18:00 UTC (permalink / raw)
  To: Naveen N. Rao, notmuch

"Naveen N. Rao" <naveen.n.rao@linux.vnet.ibm.com> writes:

> David Bremner wrote:
>> David Bremner <david@tethera.net> writes:
>> 
>>> At least some of this mail data is public, but I'm not sure if the bad
>>> threading is reproducible or not; I want to run a complete census
>>> overnight before I reindex.
>>>
>>> Even if the bug is non-deterministic, it probably lives in lib/add-message.cc
>> 
>> I have a reproducible test for this bug now
>> 
>>   http://pivot.cs.unb.ca/git?p=notmuch.git;a=shortlog;h=refs/heads/fix/thread-search
>
> Thanks for looking into this.
>
>> 
>> I still need to analyze the mails a bit more, but it looks like at least
>> one of the strange results is caused by multiple mail files sharing the
>> same message-id, but with different References headers (and no
>> In-Reply-To headers).
>
> In my case, I seem to be having the In-Reply-To headers. I end up with 
> two files per message: one from my inbox and one from the gmane archive 
> that I pull in. All the messages from the gmane archive seem to have a 
> re-written 'In-Reply-To' header, but 'Message-Id' and 'References' are 
> the same.
>
> In the problematic email thread, all other files/messages get allotted a 
> single thread except for one of the messages. The offending message has 
> 3 references compared to 1 or 2 references for the rest, but I don't 
> know if that's relevant here.
>

I _think_ this problem has been resolved with 3f4de98e7c8 /
dab32dc70c15, so in releases after 0.32.2.

At least the tests I had previously for it are passing in current
notmuch.

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: 'notmuch search thread:<>' lists multiple threads
  2021-12-23 18:00       ` David Bremner
@ 2022-01-14 11:26         ` Naveen N. Rao
  0 siblings, 0 replies; 16+ messages in thread
From: Naveen N. Rao @ 2022-01-14 11:26 UTC (permalink / raw)
  To: David Bremner, notmuch

David Bremner wrote:
> "Naveen N. Rao" <naveen.n.rao@linux.vnet.ibm.com> writes:
> 
>> David Bremner wrote:
>>> David Bremner <david@tethera.net> writes:
>>> 
>>>> At least some of this mail data is public, but I'm not sure if the bad
>>>> threading is reproducible or not; I want to run a complete census
>>>> overnight before I reindex.
>>>>
>>>> Even if the bug is non-deterministic, it probably lives in lib/add-message.cc
>>> 
>>> I have a reproducible test for this bug now
>>> 
>>>   http://pivot.cs.unb.ca/git?p=notmuch.git;a=shortlog;h=refs/heads/fix/thread-search
>>
>> Thanks for looking into this.
>>
>>> 
>>> I still need to analyze the mails a bit more, but it looks like at least
>>> one of the strange results is caused by multiple mail files sharing the
>>> same message-id, but with different References headers (and no
>>> In-Reply-To headers).
>>
>> In my case, I seem to be having the In-Reply-To headers. I end up with 
>> two files per message: one from my inbox and one from the gmane archive 
>> that I pull in. All the messages from the gmane archive seem to have a 
>> re-written 'In-Reply-To' header, but 'Message-Id' and 'References' are 
>> the same.
>>
>> In the problematic email thread, all other files/messages get allotted a 
>> single thread except for one of the messages. The offending message has 
>> 3 references compared to 1 or 2 references for the rest, but I don't 
>> know if that's relevant here.
>>
> 
> I _think_ this problem has been resolved with 3f4de98e7c8 /
> dab32dc70c15, so in releases after 0.32.2.
> 
> At least the tests I had previously for it are passing in current
> notmuch.

That's great to hear!

I have moved to using lore.kernel.org for the mailing list archives and 
I haven't had an issue in a while - notmuch has been very stable. Thanks 
for all your work!


- Naveen

^ permalink raw reply	[flat|nested] 16+ messages in thread

end of thread, other threads:[~2022-01-14 13:50 UTC | newest]

Thread overview: 16+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-04-06  9:46 'notmuch search thread:<>' lists multiple threads Naveen N. Rao
2018-04-06 10:23 ` Naveen N. Rao
2018-04-08  3:04 ` David Bremner
2018-04-09 11:54   ` David Bremner
2018-04-10  1:45     ` [PATCH] devel: add new tool to draw thread structure David Bremner
2018-10-08  3:30       ` Daniel Kahn Gillmor
2018-04-18 10:18     ` 'notmuch search thread:<>' lists multiple threads Naveen N. Rao
2018-04-22  0:45       ` David Bremner
2018-06-28 10:36         ` Naveen N. Rao
2018-06-30 13:42           ` David Bremner
2018-08-30 12:52             ` David Bremner
2018-09-06 10:50               ` Naveen N. Rao
2019-06-27 14:45                 ` Naveen N. Rao
2019-06-27 14:53                   ` Naveen N. Rao
2021-12-23 18:00       ` David Bremner
2022-01-14 11:26         ` Naveen N. Rao

Code repositories for project(s) associated with this public inbox

	https://yhetil.org/notmuch.git/

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).