* 'notmuch search thread:<>' lists multiple threads @ 2018-04-06 9:46 Naveen N. Rao 2018-04-06 10:23 ` Naveen N. Rao 2018-04-08 3:04 ` David Bremner 0 siblings, 2 replies; 16+ messages in thread From: Naveen N. Rao @ 2018-04-06 9:46 UTC (permalink / raw) To: notmuch Greetings-- If I search for threads matching a specific thread-id, I am seeing multiple results: $ notmuch search --output=threads thread:00000000000c4d20 thread:00000000000c4d1e thread:00000000000c4d20 If I list the messages from both those threads, they do belong to the same original mailing list thread. It isn't clear why notmuch is assigning different thread IDs. Is that to be expected under some scenarios? Also, it is a bit weird to see multiple threads being listed when searching for a specific thread ID. Again, is this something to be expected? - Naveen ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: 'notmuch search thread:<>' lists multiple threads 2018-04-06 9:46 'notmuch search thread:<>' lists multiple threads Naveen N. Rao @ 2018-04-06 10:23 ` Naveen N. Rao 2018-04-08 3:04 ` David Bremner 1 sibling, 0 replies; 16+ messages in thread From: Naveen N. Rao @ 2018-04-06 10:23 UTC (permalink / raw) To: notmuch Naveen N. Rao wrote: > Greetings-- > If I search for threads matching a specific thread-id, I am seeing > multiple results: > > $ notmuch search --output=threads thread:00000000000c4d20 > thread:00000000000c4d1e > thread:00000000000c4d20 Expanding on this: [04/06 15:37:59 ~]$ notmuch search --output=messages thread:00000000000c4d1e id:CAKTCnzngahex_sL2raoHFuXqTxgVV7a57R9YmcT1TN-ROsFXnw@mail.gmail.com [04/06 15:49:34 ~]$ [04/06 15:38:01 ~]$ notmuch search --output=messages thread:00000000000c4d20 id:CAKTCnzngahex_sL2raoHFuXqTxgVV7a57R9YmcT1TN-ROsFXnw@mail.gmail.com id:CAOSf1CFy0im+H368Krr2QVteAumq58CRF_Kfatyv7wvj1n4AGw@mail.gmail.com id:20180405071500.22320-4-bsingharora@gmail.com id:20180405071500.22320-3-bsingharora@gmail.com id:20180405071500.22320-2-bsingharora@gmail.com id:20180405071500.22320-1-bsingharora@gmail.com [04/06 15:49:34 ~]$ [04/06 15:49:26 ~]$ notmuch show --format=raw id:CAKTCnzngahex_sL2raoHFuXqTxgVV7a57R9YmcT1TN-ROsFXnw@mail.gmail.com | grep -e "In-Reply-To" -e "References" -A2 In-Reply-To: <CAOSf1CFy0im+H368Krr2QVteAumq58CRF_Kfatyv7wvj1n4AGw@mail.gmail.com> References: <20180405071500.22320-1-bsingharora@gmail.com> <20180405071500.22320-3-bsingharora@gmail.com> <CAOSf1CFy0im+H368Krr2QVteAumq58CRF_Kfatyv7wvj1n4AGw@mail.gmail.com> [04/06 15:50:01 ~]$ [04/06 15:50:02 ~]$ notmuch show --format=raw id:CAOSf1CFy0im+H368Krr2QVteAumq58CRF_Kfatyv7wvj1n4AGw@mail.gmail.com | grep -e "In-Reply-To" -e "References" -A1 In-Reply-To: <20180405071500.22320-3-bsingharora@gmail.com> References: <20180405071500.22320-1-bsingharora@gmail.com> <20180405071500.22320-3-bsingharora@gmail.com> - Naveen ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: 'notmuch search thread:<>' lists multiple threads 2018-04-06 9:46 'notmuch search thread:<>' lists multiple threads Naveen N. Rao 2018-04-06 10:23 ` Naveen N. Rao @ 2018-04-08 3:04 ` David Bremner 2018-04-09 11:54 ` David Bremner 1 sibling, 1 reply; 16+ messages in thread From: David Bremner @ 2018-04-08 3:04 UTC (permalink / raw) To: Naveen N. Rao, notmuch "Naveen N. Rao" <naveen.n.rao@linux.vnet.ibm.com> writes: > Greetings-- > If I search for threads matching a specific thread-id, I am seeing > multiple results: > > $ notmuch search --output=threads thread:00000000000c4d20 > thread:00000000000c4d1e > thread:00000000000c4d20 This looks like a bug to me. I was able to replicate it in my own mail store with the script at the end of the message. I haven't completely analyzed the situation yet, but one thing I noticed is that in all "bad threads", there are files with duplicate message-ids. Typical output looks like ╭─ zancas:software/upstream/notmuch/test ╰─ (git)-[master]-% notmuch search thread:000000000001760a thread:00000000000175e5 November 03 [1/2(3)] 128@gmx.us; Bug#846042: VTK 8 (unread) thread:000000000001760a 2016-11-27 [1/2(3)] 128@gmx.us; Bug#846042: virtual/meta package for python-vtk (unread) At least some of this mail data is public, but I'm not sure if the bad threading is reproducible or not; I want to run a complete census overnight before I reindex. Even if the bug is non-deterministic, it probably lives in lib/add-message.cc ---------------------------------------------------------------------- count=0 success=0 for id in $(notmuch search --output=threads '*'); do count=$((count +1)) matches=$((`notmuch search --output=threads "$id" | wc -l`)) if [ "$matches" = 1 ]; then success=$((success + 1)) else echo "bad thread: $id" fi if [ $((count % 1000)) -eq 0 ]; then echo $count; fi done echo "count=$count success=$success" ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: 'notmuch search thread:<>' lists multiple threads 2018-04-08 3:04 ` David Bremner @ 2018-04-09 11:54 ` David Bremner 2018-04-10 1:45 ` [PATCH] devel: add new tool to draw thread structure David Bremner 2018-04-18 10:18 ` 'notmuch search thread:<>' lists multiple threads Naveen N. Rao 0 siblings, 2 replies; 16+ messages in thread From: David Bremner @ 2018-04-09 11:54 UTC (permalink / raw) To: Naveen N. Rao, notmuch David Bremner <david@tethera.net> writes: > At least some of this mail data is public, but I'm not sure if the bad > threading is reproducible or not; I want to run a complete census > overnight before I reindex. > > Even if the bug is non-deterministic, it probably lives in lib/add-message.cc I have a reproducible test for this bug now http://pivot.cs.unb.ca/git?p=notmuch.git;a=shortlog;h=refs/heads/fix/thread-search I still need to analyze the mails a bit more, but it looks like at least one of the strange results is caused by multiple mail files sharing the same message-id, but with different References headers (and no In-Reply-To headers). d ^ permalink raw reply [flat|nested] 16+ messages in thread
* [PATCH] devel: add new tool to draw thread structure 2018-04-09 11:54 ` David Bremner @ 2018-04-10 1:45 ` David Bremner 2018-10-08 3:30 ` Daniel Kahn Gillmor 2018-04-18 10:18 ` 'notmuch search thread:<>' lists multiple threads Naveen N. Rao 1 sibling, 1 reply; 16+ messages in thread From: David Bremner @ 2018-04-10 1:45 UTC (permalink / raw) To: David Bremner, Naveen N. Rao, notmuch This is useful for understanding the case where different message-files with the same message-id have distinct reference headers. --- devel/draw-thread | 35 +++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+) create mode 100755 devel/draw-thread diff --git a/devel/draw-thread b/devel/draw-thread new file mode 100755 index 00000000..628dcff4 --- /dev/null +++ b/devel/draw-thread @@ -0,0 +1,35 @@ +#!/bin/bash + +# This script can be used like +# NOTMUCH_CONFIG=test/tmp.T580-thread-search/notmuch-config \ +# devel/draw-thread thread:0000000000000002 | dot -Tpdf > thread2.pdf + +# In addition to notmuch, you will need the following tools installed +# - graphviz +# - formail (part of procmail) + +threadid=$1 + +declare -a edges + +declare -a dest +echo "digraph \"$threadid\" {" +for messageid in $(notmuch search --output=messages $threadid); do + echo "subgraph \"cluster_$messageid\" {" + printf "\"%s\" [shape=folder];\n" ${messageid#id:} + for file in $(notmuch search --output=files $messageid); do + node=$(basename $file) + printf "\"%s\" [shape=note];\n" $node + + mapfile -t dest < <(formail -x references < $file | tr '<>,' '"" ') + edge="\"$node\" -> { ${dest[*]} }" + edges+=($edge) + done + echo "}" +done + +for edge in "${edges[*]}"; do + echo $edge +done + +echo "}" -- 2.16.3 ^ permalink raw reply related [flat|nested] 16+ messages in thread
* Re: [PATCH] devel: add new tool to draw thread structure 2018-04-10 1:45 ` [PATCH] devel: add new tool to draw thread structure David Bremner @ 2018-10-08 3:30 ` Daniel Kahn Gillmor 0 siblings, 0 replies; 16+ messages in thread From: Daniel Kahn Gillmor @ 2018-10-08 3:30 UTC (permalink / raw) To: notmuch [-- Attachment #1: Type: text/plain, Size: 643 bytes --] On Mon 2018-04-09 22:45:39 -0300, David Bremner wrote: > This is useful for understanding the case where different > message-files with the same message-id have distinct reference > headers. > --- > devel/draw-thread | 35 +++++++++++++++++++++++++++++++++++ > 1 file changed, 35 insertions(+) > create mode 100755 devel/draw-thread fwiw, i think this tool is useful enough for debugging and visibility that we should go ahead and include it in the notmuch source, under devel/ -- we don't need to ship it publicly, but it'd be great to have it easily available to all notmuch devs who might be dealing with this kind of thing. --dkg [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 227 bytes --] ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: 'notmuch search thread:<>' lists multiple threads 2018-04-09 11:54 ` David Bremner 2018-04-10 1:45 ` [PATCH] devel: add new tool to draw thread structure David Bremner @ 2018-04-18 10:18 ` Naveen N. Rao 2018-04-22 0:45 ` David Bremner 2021-12-23 18:00 ` David Bremner 1 sibling, 2 replies; 16+ messages in thread From: Naveen N. Rao @ 2018-04-18 10:18 UTC (permalink / raw) To: David Bremner, notmuch David Bremner wrote: > David Bremner <david@tethera.net> writes: > >> At least some of this mail data is public, but I'm not sure if the bad >> threading is reproducible or not; I want to run a complete census >> overnight before I reindex. >> >> Even if the bug is non-deterministic, it probably lives in lib/add-message.cc > > I have a reproducible test for this bug now > > http://pivot.cs.unb.ca/git?p=notmuch.git;a=shortlog;h=refs/heads/fix/thread-search Thanks for looking into this. > > I still need to analyze the mails a bit more, but it looks like at least > one of the strange results is caused by multiple mail files sharing the > same message-id, but with different References headers (and no > In-Reply-To headers). In my case, I seem to be having the In-Reply-To headers. I end up with two files per message: one from my inbox and one from the gmane archive that I pull in. All the messages from the gmane archive seem to have a re-written 'In-Reply-To' header, but 'Message-Id' and 'References' are the same. In the problematic email thread, all other files/messages get allotted a single thread except for one of the messages. The offending message has 3 references compared to 1 or 2 references for the rest, but I don't know if that's relevant here. - Naveen ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: 'notmuch search thread:<>' lists multiple threads 2018-04-18 10:18 ` 'notmuch search thread:<>' lists multiple threads Naveen N. Rao @ 2018-04-22 0:45 ` David Bremner 2018-06-28 10:36 ` Naveen N. Rao 2021-12-23 18:00 ` David Bremner 1 sibling, 1 reply; 16+ messages in thread From: David Bremner @ 2018-04-22 0:45 UTC (permalink / raw) To: Naveen N. Rao, notmuch "Naveen N. Rao" <naveen.n.rao@linux.vnet.ibm.com> writes: > In my case, I seem to be having the In-Reply-To headers. I end up with > two files per message: one from my inbox and one from the gmane archive > that I pull in. All the messages from the gmane archive seem to have a > re-written 'In-Reply-To' header, but 'Message-Id' and 'References' are > the same. That sounds like essentially the same issue, due to the fact that notmuch prefers In-Reply-To when choosing a parent for a message. Currently the database is correct (or at least one not-crazy definition of correct): all of the reference and in-reply-to terms are attached to the message document in the database. On the other hand, the in memory data structures currently assume that In-reply-to is a unique value (with ties broken at indexing time). It might be that the solution is to read a list of in-reply-to values and use all of them in threading. At a quick glance, that looks doable; I'm just not sure about unintended consequences. d ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: 'notmuch search thread:<>' lists multiple threads 2018-04-22 0:45 ` David Bremner @ 2018-06-28 10:36 ` Naveen N. Rao 2018-06-30 13:42 ` David Bremner 0 siblings, 1 reply; 16+ messages in thread From: Naveen N. Rao @ 2018-06-28 10:36 UTC (permalink / raw) To: David Bremner, notmuch Hi David, David Bremner wrote: > "Naveen N. Rao" <naveen.n.rao@linux.vnet.ibm.com> writes: > >> In my case, I seem to be having the In-Reply-To headers. I end up with >> two files per message: one from my inbox and one from the gmane archive >> that I pull in. All the messages from the gmane archive seem to have a >> re-written 'In-Reply-To' header, but 'Message-Id' and 'References' are >> the same. > > That sounds like essentially the same issue, due to the fact that > notmuch prefers In-Reply-To when choosing a parent for a message. > > Currently the database is correct (or at least one not-crazy definition > of correct): all of the reference and in-reply-to terms are attached to > the message document in the database. On the other hand, the in memory > data structures currently assume that In-reply-to is a unique value > (with ties broken at indexing time). > > It might be that the solution is to read a list of in-reply-to values > and use all of them in threading. At a quick glance, that looks doable; > I'm just not sure about unintended consequences. Were you able to look into this again? Using a list of in-reply-to values sounds like a good option, though I clearly have no idea about other consequences from that. If you have a patch, I can help test that. Thanks, Naveen ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: 'notmuch search thread:<>' lists multiple threads 2018-06-28 10:36 ` Naveen N. Rao @ 2018-06-30 13:42 ` David Bremner 2018-08-30 12:52 ` David Bremner 0 siblings, 1 reply; 16+ messages in thread From: David Bremner @ 2018-06-30 13:42 UTC (permalink / raw) To: Naveen N. Rao, notmuch "Naveen N. Rao" <naveen.n.rao@linux.vnet.ibm.com> writes: > > Were you able to look into this again? > Using a list of in-reply-to values sounds like a good option, though I > clearly have no idea about other consequences from that. If you have a > patch, I can help test that. > Sorry I haven't made any progress on this. Thanks for the reminder. d ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: 'notmuch search thread:<>' lists multiple threads 2018-06-30 13:42 ` David Bremner @ 2018-08-30 12:52 ` David Bremner 2018-09-06 10:50 ` Naveen N. Rao 0 siblings, 1 reply; 16+ messages in thread From: David Bremner @ 2018-08-30 12:52 UTC (permalink / raw) To: Naveen N. Rao, notmuch David Bremner <david@tethera.net> writes: > "Naveen N. Rao" <naveen.n.rao@linux.vnet.ibm.com> writes: > >> >> Were you able to look into this again? >> Using a list of in-reply-to values sounds like a good option, though I >> clearly have no idea about other consequences from that. If you have a >> patch, I can help test that. >> > > Sorry I haven't made any progress on this. Thanks for the reminder. > > d It's not much progress but I did manage to make a test case. id:20180730224555.26047-16-david@tethera.net As it says in the commit message, its not 100% clear this is your problem, but it is a bug, and hopefully fixing it will help your issue. d ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: 'notmuch search thread:<>' lists multiple threads 2018-08-30 12:52 ` David Bremner @ 2018-09-06 10:50 ` Naveen N. Rao 2019-06-27 14:45 ` Naveen N. Rao 0 siblings, 1 reply; 16+ messages in thread From: Naveen N. Rao @ 2018-09-06 10:50 UTC (permalink / raw) To: David Bremner, notmuch David Bremner wrote: > David Bremner <david@tethera.net> writes: > >> "Naveen N. Rao" <naveen.n.rao@linux.vnet.ibm.com> writes: >> >>> >>> Were you able to look into this again? >>> Using a list of in-reply-to values sounds like a good option, though I >>> clearly have no idea about other consequences from that. If you have a >>> patch, I can help test that. >>> >> >> Sorry I haven't made any progress on this. Thanks for the reminder. >> >> d > > It's not much progress but I did manage to make a test case. > > id:20180730224555.26047-16-david@tethera.net > > As it says in the commit message, its not 100% clear this is your > problem, but it is a bug, and hopefully fixing it will help your issue. Thanks for continuing to look into this! The test is close to what I have -- the only difference in my case is that the Message-ID and References: fields match in the duplicate mail files, but just the In-reply-to headers differ (the gmane one has a meaningless/incorrect, re-written header). Interestingly though, I re-checked the thread I had the original problem with and notmuch seems to be able to cope with it better now. So, some other changes seem to have helped with my original problem. I will keep an eye out to see if any other threads cause problems (I do occasionally see astroid crash, but I haven't seen if it is due to this issue with notmuch or a different problem). [09/06 16:19:19 ~]$ notmuch --version notmuch 0.27 [09/06 16:19:20 ~]$ notmuch search --output=threads thread:00000000000c4d20 thread:00000000000c4d20 [09/06 16:19:22 ~]$ notmuch search --output=threads thread:00000000000c4d1e thread:00000000000c4d1e [09/06 16:19:26 ~]$ notmuch search --output=messages thread:00000000000c4d20 id:20180813154910.lzrkd2ivdorge3ro@arbab-laptop.localdomain id:CAKTCnzngahex_sL2raoHFuXqTxgVV7a57R9YmcT1TN-ROsFXnw@mail.gmail.com id:CAOSf1CFy0im+H368Krr2QVteAumq58CRF_Kfatyv7wvj1n4AGw@mail.gmail.com id:20180405071500.22320-4-bsingharora@gmail.com id:20180405071500.22320-3-bsingharora@gmail.com id:20180405071500.22320-2-bsingharora@gmail.com id:20180405071500.22320-1-bsingharora@gmail.com [09/06 16:19:35 ~]$ notmuch search --output=messages thread:00000000000c4d1e id:CAKTCnzngahex_sL2raoHFuXqTxgVV7a57R9YmcT1TN-ROsFXnw@mail.gmail.com - Naveen ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: 'notmuch search thread:<>' lists multiple threads 2018-09-06 10:50 ` Naveen N. Rao @ 2019-06-27 14:45 ` Naveen N. Rao 2019-06-27 14:53 ` Naveen N. Rao 0 siblings, 1 reply; 16+ messages in thread From: Naveen N. Rao @ 2019-06-27 14:45 UTC (permalink / raw) To: David Bremner, notmuch Naveen N. Rao wrote: > David Bremner wrote: >> David Bremner <david@tethera.net> writes: >> >>> "Naveen N. Rao" <naveen.n.rao@linux.vnet.ibm.com> writes: >>> >>>> >>>> Were you able to look into this again? >>>> Using a list of in-reply-to values sounds like a good option, though I >>>> clearly have no idea about other consequences from that. If you have a >>>> patch, I can help test that. >>>> >>> >>> Sorry I haven't made any progress on this. Thanks for the reminder. >>> >>> d >> >> It's not much progress but I did manage to make a test case. >> >> id:20180730224555.26047-16-david@tethera.net >> >> As it says in the commit message, its not 100% clear this is your >> problem, but it is a bug, and hopefully fixing it will help your issue. > > Thanks for continuing to look into this! > > The test is close to what I have -- the only difference in my case is > that the Message-ID and References: fields match in the duplicate mail > files, but just the In-reply-to headers differ (the gmane one has a > meaningless/incorrect, re-written header). > > Interestingly though, I re-checked the thread I had the original problem > with and notmuch seems to be able to cope with it better now. So, some > other changes seem to have helped with my original problem. I will keep > an eye out to see if any other threads cause problems (I do occasionally > see astroid crash, but I haven't seen if it is due to this issue with > notmuch or a different problem). > > [09/06 16:19:19 ~]$ notmuch --version > notmuch 0.27 > [09/06 16:19:20 ~]$ notmuch search --output=threads thread:00000000000c4d20 > thread:00000000000c4d20 > [09/06 16:19:22 ~]$ notmuch search --output=threads thread:00000000000c4d1e > thread:00000000000c4d1e > [09/06 16:19:26 ~]$ notmuch search --output=messages thread:00000000000c4d20 > id:20180813154910.lzrkd2ivdorge3ro@arbab-laptop.localdomain > id:CAKTCnzngahex_sL2raoHFuXqTxgVV7a57R9YmcT1TN-ROsFXnw@mail.gmail.com > id:CAOSf1CFy0im+H368Krr2QVteAumq58CRF_Kfatyv7wvj1n4AGw@mail.gmail.com > id:20180405071500.22320-4-bsingharora@gmail.com > id:20180405071500.22320-3-bsingharora@gmail.com > id:20180405071500.22320-2-bsingharora@gmail.com > id:20180405071500.22320-1-bsingharora@gmail.com > [09/06 16:19:35 ~]$ notmuch search --output=messages thread:00000000000c4d1e > id:CAKTCnzngahex_sL2raoHFuXqTxgVV7a57R9YmcT1TN-ROsFXnw@mail.gmail.com I've since moved away from Gmane and using public-inbox now and things have been good all this while. Today, however, I hit this issue again. I pull archives from lkml and linuxppc-dev and this thread is pulled in from both those places, as well as from my local IMAP sent folder: https://lore.kernel.org/linuxppc-dev/cover.1561634177.git.naveen.n.rao@linux.vnet.ibm.com/T/#t https://lore.kernel.org/lkml/cover.1561634177.git.naveen.n.rao@linux.vnet.ibm.com/T/#t [06/27 19:56:07 ~]$ notmuch search -- tag:ppcdev and date:today and from:naveen thread:00000000000e364b Today 16:57 [1/1(3)] Naveen N. Rao; [PATCH v2 2/7] x86/ftrace: Fix use of flags in ftrace_replace_code() (imap inbox lkml ppcdev sent) thread:00000000000e364d Today 16:53 [8/11(31)] Naveen N. Rao| Steven Rostedt, Masami Hiramatsu; [PATCH v2 7/7] powerpc/kprobes: Allow probing on any ftrace address (imap inbox lkml new ppcdev replied sent unread) thread:00000000000e3636 Today 15:29 [1/1(2)] Naveen N. Rao; [PATCH] powerpc/xmon: Fix disabling tracing while in xmon (imap inbox ppcdev) thread:00000000000e357e Today 00:08 [1/4(12)] Naveen N. Rao| Michael Ellerman, Satheesh Rajendran, Steven Rostedt; [PATCH] recordmcount: Fix spurious mcount entries on powerpc (imap inbox lkml ppcdev) [06/27 19:56:09 ~]$ [06/27 19:56:14 ~]$ notmuch search --output=files -- thread:00000000000e364b /home/naveen/news/news/ltcimap/Sent/cur/1561634919_0.18171.naverao1-tp,U=1356,FMD5=7f8c0283f16925caed8e632086b81b9c:2,S /home/naveen/news/news/lore.kernel.org/linuxppc-dev/0/223040 /home/naveen/news/news/lore.kernel.org/lkml/7/120234 [06/27 19:56:34 ~]$ notmuch search --output=files -- thread:00000000000e364d /home/naveen/news/news/ltcimap/INBOX/new/1561645220_0.8024.naverao1-tp,U=151133,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2, /home/naveen/news/news/ltcimap/INBOX/cur/1561643871_0.8024.naverao1-tp,U=151132,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,S /home/naveen/news/news/lore.kernel.org/linuxppc-dev/0/223042 /home/naveen/news/news/lore.kernel.org/lkml/7/120325 /home/naveen/news/news/ltcimap/Sent/cur/1561634919_0.18171.naverao1-tp,U=1356,FMD5=7f8c0283f16925caed8e632086b81b9c:2,S /home/naveen/news/news/lore.kernel.org/linuxppc-dev/0/223040 /home/naveen/news/news/lore.kernel.org/lkml/7/120234 /home/naveen/news/news/ltcimap/INBOX/cur/1561634683_0.18171.naverao1-tp,U=151129,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,S /home/naveen/news/news/lore.kernel.org/linuxppc-dev/0/223038 /home/naveen/news/news/lore.kernel.org/lkml/7/120225 /home/naveen/news/news/ltcimap/INBOX/cur/1561634682_6.18171.naverao1-tp,U=151128,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,S /home/naveen/news/news/lore.kernel.org/linuxppc-dev/0/223039 /home/naveen/news/news/lore.kernel.org/lkml/7/120224 /home/naveen/news/news/ltcimap/INBOX/cur/1561634682_5.18171.naverao1-tp,U=151127,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,S /home/naveen/news/news/lore.kernel.org/linuxppc-dev/0/223037 /home/naveen/news/news/lore.kernel.org/lkml/7/120223 /home/naveen/news/news/ltcimap/INBOX/cur/1561634682_0.18171.naverao1-tp,U=151126,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,S /home/naveen/news/news/lore.kernel.org/linuxppc-dev/0/223036 /home/naveen/news/news/lore.kernel.org/lkml/7/120222 /home/naveen/news/news/ltcimap/INBOX/cur/1561634682_4.18171.naverao1-tp,U=151125,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,S /home/naveen/news/news/lore.kernel.org/linuxppc-dev/0/223035 /home/naveen/news/news/lore.kernel.org/lkml/7/120221 /home/naveen/news/news/ltcimap/INBOX/cur/1561634682_1.18171.naverao1-tp,U=151124,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,RS /home/naveen/news/news/lore.kernel.org/linuxppc-dev/0/223034 /home/naveen/news/news/lore.kernel.org/lkml/7/120220 /home/naveen/news/news/ltcimap/INBOX/cur/1561634682_3.18171.naverao1-tp,U=151123,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,S /home/naveen/news/news/lore.kernel.org/linuxppc-dev/0/223033 /home/naveen/news/news/lore.kernel.org/lkml/7/120219 /home/naveen/news/news/ltcimap/INBOX/cur/1561634682_2.18171.naverao1-tp,U=151122,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,S /home/naveen/news/news/lore.kernel.org/linuxppc-dev/0/223031 /home/naveen/news/news/lore.kernel.org/lkml/7/120218 - Naveen ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: 'notmuch search thread:<>' lists multiple threads 2019-06-27 14:45 ` Naveen N. Rao @ 2019-06-27 14:53 ` Naveen N. Rao 0 siblings, 0 replies; 16+ messages in thread From: Naveen N. Rao @ 2019-06-27 14:53 UTC (permalink / raw) To: David Bremner, notmuch Naveen N. Rao wrote: > Naveen N. Rao wrote: >> David Bremner wrote: >>> David Bremner <david@tethera.net> writes: >>> >>>> "Naveen N. Rao" <naveen.n.rao@linux.vnet.ibm.com> writes: >>>> >>>>> >>>>> Were you able to look into this again? >>>>> Using a list of in-reply-to values sounds like a good option, though I >>>>> clearly have no idea about other consequences from that. If you have a >>>>> patch, I can help test that. >>>>> >>>> >>>> Sorry I haven't made any progress on this. Thanks for the reminder. >>>> >>>> d >>> >>> It's not much progress but I did manage to make a test case. >>> >>> id:20180730224555.26047-16-david@tethera.net >>> >>> As it says in the commit message, its not 100% clear this is your >>> problem, but it is a bug, and hopefully fixing it will help your issue. >> >> Thanks for continuing to look into this! >> >> The test is close to what I have -- the only difference in my case is >> that the Message-ID and References: fields match in the duplicate mail >> files, but just the In-reply-to headers differ (the gmane one has a >> meaningless/incorrect, re-written header). >> >> Interestingly though, I re-checked the thread I had the original problem >> with and notmuch seems to be able to cope with it better now. So, some >> other changes seem to have helped with my original problem. I will keep >> an eye out to see if any other threads cause problems (I do occasionally >> see astroid crash, but I haven't seen if it is due to this issue with >> notmuch or a different problem). >> >> [09/06 16:19:19 ~]$ notmuch --version >> notmuch 0.27 >> [09/06 16:19:20 ~]$ notmuch search --output=threads thread:00000000000c4d20 >> thread:00000000000c4d20 >> [09/06 16:19:22 ~]$ notmuch search --output=threads thread:00000000000c4d1e >> thread:00000000000c4d1e >> [09/06 16:19:26 ~]$ notmuch search --output=messages thread:00000000000c4d20 >> id:20180813154910.lzrkd2ivdorge3ro@arbab-laptop.localdomain >> id:CAKTCnzngahex_sL2raoHFuXqTxgVV7a57R9YmcT1TN-ROsFXnw@mail.gmail.com >> id:CAOSf1CFy0im+H368Krr2QVteAumq58CRF_Kfatyv7wvj1n4AGw@mail.gmail.com >> id:20180405071500.22320-4-bsingharora@gmail.com >> id:20180405071500.22320-3-bsingharora@gmail.com >> id:20180405071500.22320-2-bsingharora@gmail.com >> id:20180405071500.22320-1-bsingharora@gmail.com >> [09/06 16:19:35 ~]$ notmuch search --output=messages thread:00000000000c4d1e >> id:CAKTCnzngahex_sL2raoHFuXqTxgVV7a57R9YmcT1TN-ROsFXnw@mail.gmail.com > > I've since moved away from Gmane and using public-inbox now and things > have been good all this while. Today, however, I hit this issue again. > I pull archives from lkml and linuxppc-dev and this thread is pulled in > from both those places, as well as from my local IMAP sent folder: > https://lore.kernel.org/linuxppc-dev/cover.1561634177.git.naveen.n.rao@linux.vnet.ibm.com/T/#t > https://lore.kernel.org/lkml/cover.1561634177.git.naveen.n.rao@linux.vnet.ibm.com/T/#t > > [06/27 19:56:07 ~]$ notmuch search -- tag:ppcdev and date:today and from:naveen > thread:00000000000e364b Today 16:57 [1/1(3)] Naveen N. Rao; [PATCH v2 2/7] x86/ftrace: Fix use of flags in ftrace_replace_code() (imap inbox lkml ppcdev sent) > thread:00000000000e364d Today 16:53 [8/11(31)] Naveen N. Rao| Steven Rostedt, Masami Hiramatsu; [PATCH v2 7/7] powerpc/kprobes: Allow probing on any ftrace address (imap inbox lkml new ppcdev replied sent unread) > thread:00000000000e3636 Today 15:29 [1/1(2)] Naveen N. Rao; [PATCH] powerpc/xmon: Fix disabling tracing while in xmon (imap inbox ppcdev) > thread:00000000000e357e Today 00:08 [1/4(12)] Naveen N. Rao| Michael Ellerman, Satheesh Rajendran, Steven Rostedt; [PATCH] recordmcount: Fix spurious mcount entries on powerpc (imap inbox lkml ppcdev) > [06/27 19:56:09 ~]$ > [06/27 19:56:14 ~]$ notmuch search --output=files -- thread:00000000000e364b > /home/naveen/news/news/ltcimap/Sent/cur/1561634919_0.18171.naverao1-tp,U=1356,FMD5=7f8c0283f16925caed8e632086b81b9c:2,S > /home/naveen/news/news/lore.kernel.org/linuxppc-dev/0/223040 > /home/naveen/news/news/lore.kernel.org/lkml/7/120234 > [06/27 19:56:34 ~]$ notmuch search --output=files -- thread:00000000000e364d > /home/naveen/news/news/ltcimap/INBOX/new/1561645220_0.8024.naverao1-tp,U=151133,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2, > /home/naveen/news/news/ltcimap/INBOX/cur/1561643871_0.8024.naverao1-tp,U=151132,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,S > /home/naveen/news/news/lore.kernel.org/linuxppc-dev/0/223042 > /home/naveen/news/news/lore.kernel.org/lkml/7/120325 > /home/naveen/news/news/ltcimap/Sent/cur/1561634919_0.18171.naverao1-tp,U=1356,FMD5=7f8c0283f16925caed8e632086b81b9c:2,S > /home/naveen/news/news/lore.kernel.org/linuxppc-dev/0/223040 > /home/naveen/news/news/lore.kernel.org/lkml/7/120234 > /home/naveen/news/news/ltcimap/INBOX/cur/1561634683_0.18171.naverao1-tp,U=151129,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,S > /home/naveen/news/news/lore.kernel.org/linuxppc-dev/0/223038 > /home/naveen/news/news/lore.kernel.org/lkml/7/120225 > /home/naveen/news/news/ltcimap/INBOX/cur/1561634682_6.18171.naverao1-tp,U=151128,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,S > /home/naveen/news/news/lore.kernel.org/linuxppc-dev/0/223039 > /home/naveen/news/news/lore.kernel.org/lkml/7/120224 > /home/naveen/news/news/ltcimap/INBOX/cur/1561634682_5.18171.naverao1-tp,U=151127,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,S > /home/naveen/news/news/lore.kernel.org/linuxppc-dev/0/223037 > /home/naveen/news/news/lore.kernel.org/lkml/7/120223 > /home/naveen/news/news/ltcimap/INBOX/cur/1561634682_0.18171.naverao1-tp,U=151126,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,S > /home/naveen/news/news/lore.kernel.org/linuxppc-dev/0/223036 > /home/naveen/news/news/lore.kernel.org/lkml/7/120222 > /home/naveen/news/news/ltcimap/INBOX/cur/1561634682_4.18171.naverao1-tp,U=151125,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,S > /home/naveen/news/news/lore.kernel.org/linuxppc-dev/0/223035 > /home/naveen/news/news/lore.kernel.org/lkml/7/120221 > /home/naveen/news/news/ltcimap/INBOX/cur/1561634682_1.18171.naverao1-tp,U=151124,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,RS > /home/naveen/news/news/lore.kernel.org/linuxppc-dev/0/223034 > /home/naveen/news/news/lore.kernel.org/lkml/7/120220 > /home/naveen/news/news/ltcimap/INBOX/cur/1561634682_3.18171.naverao1-tp,U=151123,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,S > /home/naveen/news/news/lore.kernel.org/linuxppc-dev/0/223033 > /home/naveen/news/news/lore.kernel.org/lkml/7/120219 > /home/naveen/news/news/ltcimap/INBOX/cur/1561634682_2.18171.naverao1-tp,U=151122,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,S > /home/naveen/news/news/lore.kernel.org/linuxppc-dev/0/223031 > /home/naveen/news/news/lore.kernel.org/lkml/7/120218 After being unable to open this thread in astroid email client a few times, I was able to resolve this by using 'notmuch reindex' _twice_. The result quoted above was after I reindexed it once. But, the second re-index seems to have fixed this magically! - Naveen ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: 'notmuch search thread:<>' lists multiple threads 2018-04-18 10:18 ` 'notmuch search thread:<>' lists multiple threads Naveen N. Rao 2018-04-22 0:45 ` David Bremner @ 2021-12-23 18:00 ` David Bremner 2022-01-14 11:26 ` Naveen N. Rao 1 sibling, 1 reply; 16+ messages in thread From: David Bremner @ 2021-12-23 18:00 UTC (permalink / raw) To: Naveen N. Rao, notmuch "Naveen N. Rao" <naveen.n.rao@linux.vnet.ibm.com> writes: > David Bremner wrote: >> David Bremner <david@tethera.net> writes: >> >>> At least some of this mail data is public, but I'm not sure if the bad >>> threading is reproducible or not; I want to run a complete census >>> overnight before I reindex. >>> >>> Even if the bug is non-deterministic, it probably lives in lib/add-message.cc >> >> I have a reproducible test for this bug now >> >> http://pivot.cs.unb.ca/git?p=notmuch.git;a=shortlog;h=refs/heads/fix/thread-search > > Thanks for looking into this. > >> >> I still need to analyze the mails a bit more, but it looks like at least >> one of the strange results is caused by multiple mail files sharing the >> same message-id, but with different References headers (and no >> In-Reply-To headers). > > In my case, I seem to be having the In-Reply-To headers. I end up with > two files per message: one from my inbox and one from the gmane archive > that I pull in. All the messages from the gmane archive seem to have a > re-written 'In-Reply-To' header, but 'Message-Id' and 'References' are > the same. > > In the problematic email thread, all other files/messages get allotted a > single thread except for one of the messages. The offending message has > 3 references compared to 1 or 2 references for the rest, but I don't > know if that's relevant here. > I _think_ this problem has been resolved with 3f4de98e7c8 / dab32dc70c15, so in releases after 0.32.2. At least the tests I had previously for it are passing in current notmuch. ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: 'notmuch search thread:<>' lists multiple threads 2021-12-23 18:00 ` David Bremner @ 2022-01-14 11:26 ` Naveen N. Rao 0 siblings, 0 replies; 16+ messages in thread From: Naveen N. Rao @ 2022-01-14 11:26 UTC (permalink / raw) To: David Bremner, notmuch David Bremner wrote: > "Naveen N. Rao" <naveen.n.rao@linux.vnet.ibm.com> writes: > >> David Bremner wrote: >>> David Bremner <david@tethera.net> writes: >>> >>>> At least some of this mail data is public, but I'm not sure if the bad >>>> threading is reproducible or not; I want to run a complete census >>>> overnight before I reindex. >>>> >>>> Even if the bug is non-deterministic, it probably lives in lib/add-message.cc >>> >>> I have a reproducible test for this bug now >>> >>> http://pivot.cs.unb.ca/git?p=notmuch.git;a=shortlog;h=refs/heads/fix/thread-search >> >> Thanks for looking into this. >> >>> >>> I still need to analyze the mails a bit more, but it looks like at least >>> one of the strange results is caused by multiple mail files sharing the >>> same message-id, but with different References headers (and no >>> In-Reply-To headers). >> >> In my case, I seem to be having the In-Reply-To headers. I end up with >> two files per message: one from my inbox and one from the gmane archive >> that I pull in. All the messages from the gmane archive seem to have a >> re-written 'In-Reply-To' header, but 'Message-Id' and 'References' are >> the same. >> >> In the problematic email thread, all other files/messages get allotted a >> single thread except for one of the messages. The offending message has >> 3 references compared to 1 or 2 references for the rest, but I don't >> know if that's relevant here. >> > > I _think_ this problem has been resolved with 3f4de98e7c8 / > dab32dc70c15, so in releases after 0.32.2. > > At least the tests I had previously for it are passing in current > notmuch. That's great to hear! I have moved to using lore.kernel.org for the mailing list archives and I haven't had an issue in a while - notmuch has been very stable. Thanks for all your work! - Naveen ^ permalink raw reply [flat|nested] 16+ messages in thread
end of thread, other threads:[~2022-01-14 13:50 UTC | newest] Thread overview: 16+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2018-04-06 9:46 'notmuch search thread:<>' lists multiple threads Naveen N. Rao 2018-04-06 10:23 ` Naveen N. Rao 2018-04-08 3:04 ` David Bremner 2018-04-09 11:54 ` David Bremner 2018-04-10 1:45 ` [PATCH] devel: add new tool to draw thread structure David Bremner 2018-10-08 3:30 ` Daniel Kahn Gillmor 2018-04-18 10:18 ` 'notmuch search thread:<>' lists multiple threads Naveen N. Rao 2018-04-22 0:45 ` David Bremner 2018-06-28 10:36 ` Naveen N. Rao 2018-06-30 13:42 ` David Bremner 2018-08-30 12:52 ` David Bremner 2018-09-06 10:50 ` Naveen N. Rao 2019-06-27 14:45 ` Naveen N. Rao 2019-06-27 14:53 ` Naveen N. Rao 2021-12-23 18:00 ` David Bremner 2022-01-14 11:26 ` Naveen N. Rao
Code repositories for project(s) associated with this public inbox https://yhetil.org/notmuch.git/ This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).