Weird tagging issue

Weird tagging issue

[Brian May]

Hello,

I am having a problem with certain messages, in that I remove the tag
and it still shows up in search results.

e.g.

$ notmuch search tag:important
[...]
thread:000000000000aaff 40 mins. ago [37/38(43)] Lucas Liendo, Ricardo Perez; [Bitbucket] Pull request #88: Add webdriver support (wspdigital/bupaoshc) (important inbox unread)
[...]

but if I view the thread:

$ notmuch show thread:000000000000aaff

I see one message, it has no tags, and it is not the message referenced
in the search results (i.e. it was sent hours ago, not 40 minutes).

It only happens with this thread. Sometimes if I play around enough I
can fix it, but the fix doesn't last. I get similar problems for other
tags, such as inbox and important also. I suspect the root problem is it
isn't showing the entire thread.

I have tried:

$ notmuch reindex thread:000000000000aaff
$ notmuch reindex 'Pull request #88: Add webdriver support' 

But it doesn't help.

The emacs reader is getting rather confused as a result. It says the
message has the tags when it doesn't and won't let me see the entire
thread.

Is this a sign my tag database is corrupt? If so how do I fix? Or maybe
there is some problem with the message - it looks OK to me. Was
generated from bitbucket.

It is not 100% important that I preserve my existing tags, but would
like to know what is going on.

This is notmuch version 0.28.4-1 from Debian buster using Maildir.

Thanks
-- 
Brian May <brian@linuxpenguins.xyz>
https://linuxpenguins.xyz/brian/

Re: Weird tagging issue

[Brian May]

Brian May <brian@linuxpenguins.xyz> writes:

> I am having a problem with certain messages, in that I remove the tag
> and it still shows up in search results.

I just recreated the entire database, and I still get the same problem.
-- 
Brian May <brian@linuxpenguins.xyz>
https://linuxpenguins.xyz/brian/

Re: Weird tagging issue

[Daniel Kahn Gillmor]

[-- Attachment #1: Type: text/plain, Size: 1248 bytes --]

Hi Brian--

On Fri 2020-03-20 11:06:55 +1100, Brian May wrote:
> Brian May <brian@linuxpenguins.xyz> writes:
>
>> I am having a problem with certain messages, in that I remove the tag
>> and it still shows up in search results.
>
> I just recreated the entire database, and I still get the same problem.

This sounds really confusing, and i'm not sure how to help you debug. :/
when you say "recreated the entire database", do you mean you dropped
all tags and everything, and reindexed the entire datastore?

Does the thread that you were working with contain any sensitive or
private messages?  If not, can you isolate the specific files from the
maildir that compose the thread (e.g. with a manual grep) and try to
recreate the failure with a smaller maildir and dedicated notmuch
database?  I know this is tedious, but if you can do it and share the
maildir, then it would help other folks reproduce the problem and dig
into it further.

if i was debugging this locally and i had a dedicated maildir that
demonstrated the problem, i'd probably try to inspect the xapian
database (e.g. with xapian-delve).  There are several folks on the
#notmuch channel on freenode that can probably give pointers.

Sorry to not be more helpful,

      --dkg


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: Weird tagging issue

[Brian May]

Daniel Kahn Gillmor <dkg@fifthhorseman.net> writes:

> This sounds really confusing, and i'm not sure how to help you debug. :/
> when you say "recreated the entire database", do you mean you dropped
> all tags and everything, and reindexed the entire datastore?

Yes, that is correct. I basically deleted "~/Maildir/.notmuch" and then
ran "notmuch new". After confirming that the problem still existed, I
then restored the backup of my tags,

> Does the thread that you were working with contain any sensitive or
> private messages?  If not, can you isolate the specific files from the
> maildir that compose the thread (e.g. with a manual grep) and try to
> recreate the failure with a smaller maildir and dedicated notmuch
> database?  I know this is tedious, but if you can do it and share the
> maildir, then it would help other folks reproduce the problem and dig
> into it further.

These are messages associated with a pull request for a private
bitbucket repository owned by my employee, So I probably should not make
them public :-(

> if i was debugging this locally and i had a dedicated maildir that
> demonstrated the problem, i'd probably try to inspect the xapian
> database (e.g. with xapian-delve).  There are several folks on the
> #notmuch channel on freenode that can probably give pointers.

Fortunately that particular pull request has been merged now, I probably
don't have to worry anymore. However would be interested to know what
happened.

My suspicion is that bitbucket generated a bad email - maybe bad headers
or something, which has totally confused things,

The Message-ID values look fine. Although oddly enough there are several
different formats, but this shouldn't be an issue:

=== cut ===
$ grep '^Message-ID' $(grep -ri -l 'Pull request #88: Add webdriver support' Maildir)Maildir/cur/1584486508.24564_0.wspdigital:2,S:Message-ID: <prcomment-140593878@bitbucket.org>
Maildir/cur/1584486509.24570_0.wspdigital:2,S:Message-ID: <prcomment-140593988@bitbucket.org>
Maildir/cur/1584487127.27595_0.wspdigital:2,S:Message-ID: <pr-wspdigital/bupaoshc/88/reviewer-added/890f2cb2ad474cc3bb57fc13240d937e@bitbucket.org>
Maildir/cur/1584487437.29402_0.wspdigital:2,S:Message-ID: <pr-wspdigital/bupaoshc/88/updated/30ab9d5a64ad496994135e8b6d1fc728@bitbucket.org>
Maildir/cur/1584494811.24972_0.wspdigital:2,S:Message-ID: <prcomment-140603506@bitbucket.org>
Maildir/cur/1584494813.25016_0.wspdigital:2,S:Message-ID: <prcomment-140603560@bitbucket.org>
Maildir/cur/1584567341.22359_0.wspdigital:2,S:Message-ID: <prcomment-140819210@bitbucket.org>
Maildir/cur/1584505808.24160_0.wspdigital:2,S:Message-ID: <prcomment-140615930@bitbucket.org>
Maildir/cur/1584506116.25523_0.wspdigital:2,S:Message-ID: <prcomment-140615994@bitbucket.org>
[...]
=== cut ===

Hmmmm. Maybe the References header is bad. I see a number of emails in
this thread referencing "pr-wspdigital/bupaoshc/88@bitbucket.org", e.g.:

=== cut ===
References: <pr-wspdigital/bupaoshc/88@bitbucket.org>
 <prcomment-140582662@bitbucket.org>

References: <pr-wspdigital/bupaoshc/88@bitbucket.org>

References: <pr-wspdigital/bupaoshc/88@bitbucket.org>
 <pr-wspdigital/bupaoshc/88@bitbucket.org>
=== cut ===

But "pr-wspdigital/bupaoshc/88@bitbucket.org" doesn't have any
information uniquely identifying a particular message for PR #88, and
furthermore I don't see any message ids that look like this in this
thread.

I could imagine bad references might be confusing notmuch?

I guess I should wait and see if the problem happens again.
-- 
Brian May <brian@linuxpenguins.xyz>
https://linuxpenguins.xyz/brian/

Re: Weird tagging issue

[David Bremner]

[-- Attachment #1: Type: text/plain, Size: 1048 bytes --]

Brian May <brian@linuxpenguins.xyz> writes:

> Hmmmm. Maybe the References header is bad. I see a number of emails in
> this thread referencing "pr-wspdigital/bupaoshc/88@bitbucket.org", e.g.:
>
> === cut ===
> References: <pr-wspdigital/bupaoshc/88@bitbucket.org>
>  <prcomment-140582662@bitbucket.org>
>
> References: <pr-wspdigital/bupaoshc/88@bitbucket.org>
>
> References: <pr-wspdigital/bupaoshc/88@bitbucket.org>
>  <pr-wspdigital/bupaoshc/88@bitbucket.org>
> === cut ===
>
> But "pr-wspdigital/bupaoshc/88@bitbucket.org" doesn't have any
> information uniquely identifying a particular message for PR #88, and
> furthermore I don't see any message ids that look like this in this
> thread.
>
> I could imagine bad references might be confusing notmuch?


If you still have the messages, maybe you could try the attached script
to visualize the reference structure.

I think dkg and I agreed a few years ago we should ship is as one of our
devel tools, but then I didn't follow through. So I'd be interested in
knowing if it works for you.


[-- Attachment #2: draw-thread --]
[-- Type: application/octet-stream, Size: 912 bytes --]

#!/bin/bash

# This script can be used like
# NOTMUCH_CONFIG=test/tmp.T580-thread-search/notmuch-config \
#    devel/draw-thread thread:0000000000000002 | dot -Tpdf > thread2.pdf

# In addition to notmuch, you will need the following tools installed
# - graphviz
# - formail (part of procmail)

threadid=$1

declare -a edges

declare -a dest
echo "digraph \"$threadid\" {"
for messageid in $(notmuch search --output=messages $threadid); do
    echo "subgraph \"cluster_$messageid\" {"
    printf "\"%s\" [shape=folder];\n" ${messageid#id:}
    for file in $(notmuch search --output=files $messageid); do
        node=$(basename $file)
        printf "\"%s\" [shape=note];\n" $node

        mapfile -t dest < <(formail -x references < $file | tr '<>,' '"" ')
        edge="\"$node\" -> { ${dest[*]} }"
        edges+=($edge)
    done
    echo "}"
done

for edge in "${edges[*]}"; do
    echo $edge
done

echo "}"

Re: Weird tagging issue

[Daniel Kahn Gillmor]

[-- Attachment #1: Type: text/plain, Size: 520 bytes --]

On Sun 2020-03-22 22:12:14 -0300, David Bremner wrote:
> I think dkg and I agreed a few years ago we should ship [draw-thread]
> as one of our devel tools, but then I didn't follow through. So I'd be
> interested in knowing if it works for you.

fwiw, i still think it's worth shipping it in devel/ sooner rather than
later :)

I wonder whether Sean Whitton would consider something like it in
mailscripts; i suppose we'd need better documentation and a reasonable
test suite for it to do that responsibly.

      --dkg

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: Weird tagging issue

[Brian May]

[-- Attachment #1: Type: text/plain, Size: 517 bytes --]

David Bremner <david@tethera.net> writes:

> If you still have the messages, maybe you could try the attached script
> to visualize the reference structure.
>
> I think dkg and I agreed a few years ago we should ship is as one of our
> devel tools, but then I didn't follow through. So I'd be interested in
> knowing if it works for you.

I have attached the dot output.

There seem to be a number of root nodes, wonder if this is the problem.
-- 
Brian May <brian@linuxpenguins.xyz>
https://linuxpenguins.xyz/brian/

[-- Attachment #2: thread.dot --]
[-- Type: text/plain, Size: 15775 bytes --]

digraph "thread:000000000000abfd" {
subgraph "cluster_id:pr-wspdigital/bupaoshc/88/merged/001a8cd53f774894873ffe7a8479dc4d@bitbucket.org" {
"pr-wspdigital/bupaoshc/88/merged/001a8cd53f774894873ffe7a8479dc4d@bitbucket.org" [shape=folder];
"1584673728.1368_0.wspdigital:2,S" [shape=note];
}
subgraph "cluster_id:prcomment-141066745@bitbucket.org" {
"prcomment-141066745@bitbucket.org" [shape=folder];
"1584663662.29414_0.wspdigital:2,S" [shape=note];
}
subgraph "cluster_id:pr-wspdigital/bupaoshc/88/approved/fb98b1eefd37419297a914b9487905f5@bitbucket.org" {
"pr-wspdigital/bupaoshc/88/approved/fb98b1eefd37419297a914b9487905f5@bitbucket.org" [shape=folder];
"1584663351.27930_0.wspdigital:2,S" [shape=note];
}
subgraph "cluster_id:prcomment-141063018@bitbucket.org" {
"prcomment-141063018@bitbucket.org" [shape=folder];
"1584663350.27927_0.wspdigital:2,S" [shape=note];
}
subgraph "cluster_id:prcomment-141062842@bitbucket.org" {
"prcomment-141062842@bitbucket.org" [shape=folder];
"1584663347.27885_0.wspdigital:2,S" [shape=note];
}
subgraph "cluster_id:prcomment-141062786@bitbucket.org" {
"prcomment-141062786@bitbucket.org" [shape=folder];
"1584663345.27882_0.wspdigital:2,S" [shape=note];
}
subgraph "cluster_id:prcomment-141062716@bitbucket.org" {
"prcomment-141062716@bitbucket.org" [shape=folder];
"1584663343.27879_0.wspdigital:2,S" [shape=note];
}
subgraph "cluster_id:prcomment-141061641@bitbucket.org" {
"prcomment-141061641@bitbucket.org" [shape=folder];
"1584659728.14367_0.wspdigital:2,S" [shape=note];
}
subgraph "cluster_id:prcomment-141061519@bitbucket.org" {
"prcomment-141061519@bitbucket.org" [shape=folder];
"1584659726.14353_0.wspdigital:2,S" [shape=note];
}
subgraph "cluster_id:prcomment-141061033@bitbucket.org" {
"prcomment-141061033@bitbucket.org" [shape=folder];
"1584659115.9877_0.wspdigital:2,S" [shape=note];
}
subgraph "cluster_id:prcomment-141060933@bitbucket.org" {
"prcomment-141060933@bitbucket.org" [shape=folder];
"1584659113.9834_0.wspdigital:2,S" [shape=note];
}
subgraph "cluster_id:pr-wspdigital/bupaoshc/88/updated/fe6d4f1a6ce54ad5abbcc15f0bf38cfe@bitbucket.org" {
"pr-wspdigital/bupaoshc/88/updated/fe6d4f1a6ce54ad5abbcc15f0bf38cfe@bitbucket.org" [shape=folder];
"1584659106.9812_0.wspdigital:2,S" [shape=note];
}
subgraph "cluster_id:prcomment-140847518@bitbucket.org" {
"prcomment-140847518@bitbucket.org" [shape=folder];
"1584589075.6347_0.wspdigital:2,S" [shape=note];
}
subgraph "cluster_id:pr-wspdigital/bupaoshc/88/updated/1f9840dfb3c34368b4e91eaf74b03914@bitbucket.org" {
"pr-wspdigital/bupaoshc/88/updated/1f9840dfb3c34368b4e91eaf74b03914@bitbucket.org" [shape=folder];
"1584589073.6344_0.wspdigital:2,S" [shape=note];
}
subgraph "cluster_id:pr-wspdigital/bupaoshc/88/approved/994ccc1ba0c1486183545fa2bdb69319@bitbucket.org" {
"pr-wspdigital/bupaoshc/88/approved/994ccc1ba0c1486183545fa2bdb69319@bitbucket.org" [shape=folder];
"1584588766.4940_0.wspdigital:2,S" [shape=note];
}
subgraph "cluster_id:prcomment-140846703@bitbucket.org" {
"prcomment-140846703@bitbucket.org" [shape=folder];
"1584588153.1776_0.wspdigital:2,S" [shape=note];
"1584659109.9828_0.wspdigital:2,S" [shape=note];
}
subgraph "cluster_id:prcomment-140846643@bitbucket.org" {
"prcomment-140846643@bitbucket.org" [shape=folder];
"1584588152.1729_0.wspdigital:2,S" [shape=note];
}
subgraph "cluster_id:pr-wspdigital/bupaoshc/88/updated/d3ee1775fc9446f9aaae1367de092046@bitbucket.org" {
"pr-wspdigital/bupaoshc/88/updated/d3ee1775fc9446f9aaae1367de092046@bitbucket.org" [shape=folder];
"1584588150.1726_0.wspdigital:2,S" [shape=note];
}
subgraph "cluster_id:prcomment-140835105@bitbucket.org" {
"prcomment-140835105@bitbucket.org" [shape=folder];
"1584577134.11651_0.wspdigital:2,S" [shape=note];
"1584589077.6352_0.wspdigital:2,S" [shape=note];
}
subgraph "cluster_id:prcomment-140830910@bitbucket.org" {
"prcomment-140830910@bitbucket.org" [shape=folder];
"1584574083.26186_0.wspdigital:2,S" [shape=note];
}
subgraph "cluster_id:prcomment-140830855@bitbucket.org" {
"prcomment-140830855@bitbucket.org" [shape=folder];
"1584574081.26143_0.wspdigital:2,S" [shape=note];
}
subgraph "cluster_id:prcomment-140830758@bitbucket.org" {
"prcomment-140830758@bitbucket.org" [shape=folder];
"1584573774.24600_0.wspdigital:2,S" [shape=note];
}
subgraph "cluster_id:prcomment-140830689@bitbucket.org" {
"prcomment-140830689@bitbucket.org" [shape=folder];
"1584573772.24596_0.wspdigital:2,S" [shape=note];
}
subgraph "cluster_id:prcomment-140830682@bitbucket.org" {
"prcomment-140830682@bitbucket.org" [shape=folder];
"1584573771.24568_0.wspdigital:2,S" [shape=note];
}
subgraph "cluster_id:prcomment-140830574@bitbucket.org" {
"prcomment-140830574@bitbucket.org" [shape=folder];
"1584573769.24554_0.wspdigital:2,S" [shape=note];
}
subgraph "cluster_id:prcomment-140830179@bitbucket.org" {
"prcomment-140830179@bitbucket.org" [shape=folder];
"1584573461.22621_0.wspdigital:2,S" [shape=note];
}
subgraph "cluster_id:prcomment-140823647@bitbucket.org" {
"prcomment-140823647@bitbucket.org" [shape=folder];
"1584569488.663_0.wspdigital:2,S" [shape=note];
}
subgraph "cluster_id:prcomment-140821508@bitbucket.org" {
"prcomment-140821508@bitbucket.org" [shape=folder];
"1584568266.26679_0.wspdigital:2,S" [shape=note];
}
subgraph "cluster_id:prcomment-140821377@bitbucket.org" {
"prcomment-140821377@bitbucket.org" [shape=folder];
"1584568263.26673_0.wspdigital:2,S" [shape=note];
}
subgraph "cluster_id:prcomment-140819957@bitbucket.org" {
"prcomment-140819957@bitbucket.org" [shape=folder];
"1584567647.23967_0.wspdigital:2,S" [shape=note];
}
subgraph "cluster_id:prcomment-140819210@bitbucket.org" {
"prcomment-140819210@bitbucket.org" [shape=folder];
"1584567341.22359_0.wspdigital:2,S" [shape=note];
}
subgraph "cluster_id:prcomment-140818467@bitbucket.org" {
"prcomment-140818467@bitbucket.org" [shape=folder];
"1584566731.18493_0.wspdigital:2,S" [shape=note];
}
subgraph "cluster_id:prcomment-140817965@bitbucket.org" {
"prcomment-140817965@bitbucket.org" [shape=folder];
"1584566729.18490_0.wspdigital:2,S" [shape=note];
"1584568264.26676_0.wspdigital:2,S" [shape=note];
}
subgraph "cluster_id:prcomment-140817006@bitbucket.org" {
"prcomment-140817006@bitbucket.org" [shape=folder];
"1584566117.10837_0.wspdigital:2,S" [shape=note];
}
subgraph "cluster_id:prcomment-140814496@bitbucket.org" {
"prcomment-140814496@bitbucket.org" [shape=folder];
"1584565505.6032_0.wspdigital:2,S" [shape=note];
}
subgraph "cluster_id:prcomment-140814050@bitbucket.org" {
"prcomment-140814050@bitbucket.org" [shape=folder];
"1584565503.6029_0.wspdigital:2,S" [shape=note];
}
subgraph "cluster_id:prcomment-140812674@bitbucket.org" {
"prcomment-140812674@bitbucket.org" [shape=folder];
"1584565500.6006_0.wspdigital:2,S" [shape=note];
}
subgraph "cluster_id:prcomment-140812543@bitbucket.org" {
"prcomment-140812543@bitbucket.org" [shape=folder];
"1584565499.6003_0.wspdigital:2,S" [shape=note];
}
subgraph "cluster_id:prcomment-140812446@bitbucket.org" {
"prcomment-140812446@bitbucket.org" [shape=folder];
"1584565497.5997_0.wspdigital:2,S" [shape=note];
}
subgraph "cluster_id:prcomment-140812073@bitbucket.org" {
"prcomment-140812073@bitbucket.org" [shape=folder];
"1584565495.5991_0.wspdigital:2,S" [shape=note];
}
subgraph "cluster_id:prcomment-140812051@bitbucket.org" {
"prcomment-140812051@bitbucket.org" [shape=folder];
"1584565493.5988_0.wspdigital:2,S" [shape=note];
}
subgraph "cluster_id:prcomment-140615994@bitbucket.org" {
"prcomment-140615994@bitbucket.org" [shape=folder];
"1584506116.25523_0.wspdigital:2,S" [shape=note];
}
subgraph "cluster_id:prcomment-140615930@bitbucket.org" {
"prcomment-140615930@bitbucket.org" [shape=folder];
"1584505808.24160_0.wspdigital:2,S" [shape=note];
}
subgraph "cluster_id:prcomment-140603560@bitbucket.org" {
"prcomment-140603560@bitbucket.org" [shape=folder];
"1584494813.25016_0.wspdigital:2,S" [shape=note];
}
subgraph "cluster_id:prcomment-140603506@bitbucket.org" {
"prcomment-140603506@bitbucket.org" [shape=folder];
"1584494811.24972_0.wspdigital:2,S" [shape=note];
}
subgraph "cluster_id:pr-wspdigital/bupaoshc/88/updated/30ab9d5a64ad496994135e8b6d1fc728@bitbucket.org" {
"pr-wspdigital/bupaoshc/88/updated/30ab9d5a64ad496994135e8b6d1fc728@bitbucket.org" [shape=folder];
"1584487437.29402_0.wspdigital:2,S" [shape=note];
}
subgraph "cluster_id:pr-wspdigital/bupaoshc/88/reviewer-added/890f2cb2ad474cc3bb57fc13240d937e@bitbucket.org" {
"pr-wspdigital/bupaoshc/88/reviewer-added/890f2cb2ad474cc3bb57fc13240d937e@bitbucket.org" [shape=folder];
"1584487127.27595_0.wspdigital:2,S" [shape=note];
}
subgraph "cluster_id:prcomment-140593988@bitbucket.org" {
"prcomment-140593988@bitbucket.org" [shape=folder];
"1584486509.24570_0.wspdigital:2,S" [shape=note];
}
subgraph "cluster_id:prcomment-140593878@bitbucket.org" {
"prcomment-140593878@bitbucket.org" [shape=folder];
"1584486508.24564_0.wspdigital:2,S" [shape=note];
}
subgraph "cluster_id:pr-wspdigital/bupaoshc/88@bitbucket.org" {
"pr-wspdigital/bupaoshc/88@bitbucket.org" [shape=folder];
"1584434229.8921_0.wspdigital:2,S" [shape=note];
"1584434231.8924_0.wspdigital:2,S" [shape=note];
"1584434232.8927_0.wspdigital:2,S" [shape=note];
"1584434234.8930_0.wspdigital:2,S" [shape=note];
}
"1584673728.1368_0.wspdigital:2,S" -> { "pr-wspdigital/bupaoshc/88@bitbucket.org" } "1584663662.29414_0.wspdigital:2,S" -> { "pr-wspdigital/bupaoshc/88@bitbucket.org" "prcomment-141064787@bitbucket.org" } "1584663351.27930_0.wspdigital:2,S" -> { "pr-wspdigital/bupaoshc/88@bitbucket.org" } "1584663350.27927_0.wspdigital:2,S" -> { "pr-wspdigital/bupaoshc/88@bitbucket.org" "prcomment-141062842@bitbucket.org" } "1584663347.27885_0.wspdigital:2,S" -> { "pr-wspdigital/bupaoshc/88@bitbucket.org" "prcomment-141062716@bitbucket.org" } "1584663345.27882_0.wspdigital:2,S" -> { "pr-wspdigital/bupaoshc/88@bitbucket.org" "prcomment-141061519@bitbucket.org" } "1584663343.27879_0.wspdigital:2,S" -> { "pr-wspdigital/bupaoshc/88@bitbucket.org" "prcomment-141061641@bitbucket.org" } "1584659728.14367_0.wspdigital:2,S" -> { "pr-wspdigital/bupaoshc/88@bitbucket.org" "prcomment-141061033@bitbucket.org" } "1584659726.14353_0.wspdigital:2,S" -> { "pr-wspdigital/bupaoshc/88@bitbucket.org" "prcomment-141060933@bitbucket.org" } "1584659115.9877_0.wspdigital:2,S" -> { "pr-wspdigital/bupaoshc/88@bitbucket.org" "prcomment-140846703@bitbucket.org" } "1584659113.9834_0.wspdigital:2,S" -> { "pr-wspdigital/bupaoshc/88@bitbucket.org" "prcomment-140846643@bitbucket.org" } "1584659106.9812_0.wspdigital:2,S" -> { "pr-wspdigital/bupaoshc/88@bitbucket.org" } "1584589075.6347_0.wspdigital:2,S" -> { "pr-wspdigital/bupaoshc/88@bitbucket.org" "prcomment-140814050@bitbucket.org" } "1584589073.6344_0.wspdigital:2,S" -> { "pr-wspdigital/bupaoshc/88@bitbucket.org" } "1584588766.4940_0.wspdigital:2,S" -> { "pr-wspdigital/bupaoshc/88@bitbucket.org" } "1584588153.1776_0.wspdigital:2,S" -> { "pr-wspdigital/bupaoshc/88@bitbucket.org" "prcomment-140812446@bitbucket.org" } "1584659109.9828_0.wspdigital:2,S" -> { "pr-wspdigital/bupaoshc/88@bitbucket.org" "prcomment-140812446@bitbucket.org" } "1584588152.1729_0.wspdigital:2,S" -> { "pr-wspdigital/bupaoshc/88@bitbucket.org" "prcomment-140819957@bitbucket.org" } "1584588150.1726_0.wspdigital:2,S" -> { "pr-wspdigital/bupaoshc/88@bitbucket.org" } "1584577134.11651_0.wspdigital:2,S" -> { "pr-wspdigital/bupaoshc/88@bitbucket.org" "prcomment-140830758@bitbucket.org" } "1584589077.6352_0.wspdigital:2,S" -> { "pr-wspdigital/bupaoshc/88@bitbucket.org" "prcomment-140830758@bitbucket.org" } "1584574083.26186_0.wspdigital:2,S" -> { "pr-wspdigital/bupaoshc/88@bitbucket.org" "prcomment-140821632@bitbucket.org" } "1584574081.26143_0.wspdigital:2,S" -> { "pr-wspdigital/bupaoshc/88@bitbucket.org" "prcomment-140818467@bitbucket.org" } "1584573774.24600_0.wspdigital:2,S" -> { "pr-wspdigital/bupaoshc/88@bitbucket.org" "prcomment-140830574@bitbucket.org" } "1584573772.24596_0.wspdigital:2,S" -> { "pr-wspdigital/bupaoshc/88@bitbucket.org" "prcomment-140812051@bitbucket.org" } "1584573771.24568_0.wspdigital:2,S" -> { "pr-wspdigital/bupaoshc/88@bitbucket.org" "prcomment-140830179@bitbucket.org" } "1584573769.24554_0.wspdigital:2,S" -> { "pr-wspdigital/bupaoshc/88@bitbucket.org" "pr-wspdigital/bupaoshc/88@bitbucket.org" } "1584573461.22621_0.wspdigital:2,S" -> { "pr-wspdigital/bupaoshc/88@bitbucket.org" "prcomment-140819210@bitbucket.org" } "1584569488.663_0.wspdigital:2,S" -> { "pr-wspdigital/bupaoshc/88@bitbucket.org" "prcomment-140823021@bitbucket.org" } "1584568266.26679_0.wspdigital:2,S" -> { "pr-wspdigital/bupaoshc/88@bitbucket.org" "prcomment-140821372@bitbucket.org" } "1584568263.26673_0.wspdigital:2,S" -> { "pr-wspdigital/bupaoshc/88@bitbucket.org" "prcomment-140821080@bitbucket.org" } "1584567647.23967_0.wspdigital:2,S" -> { "pr-wspdigital/bupaoshc/88@bitbucket.org" "pr-wspdigital/bupaoshc/88@bitbucket.org" } "1584567341.22359_0.wspdigital:2,S" -> { "pr-wspdigital/bupaoshc/88@bitbucket.org" "prcomment-140406108@bitbucket.org" } "1584566731.18493_0.wspdigital:2,S" -> { "pr-wspdigital/bupaoshc/88@bitbucket.org" "prcomment-140817965@bitbucket.org" } "1584566729.18490_0.wspdigital:2,S" -> { "pr-wspdigital/bupaoshc/88@bitbucket.org" "prcomment-140603560@bitbucket.org" } "1584568264.26676_0.wspdigital:2,S" -> { "pr-wspdigital/bupaoshc/88@bitbucket.org" "prcomment-140603560@bitbucket.org" } "1584566117.10837_0.wspdigital:2,S" -> { "pr-wspdigital/bupaoshc/88@bitbucket.org" "prcomment-140616831@bitbucket.org" } "1584565505.6032_0.wspdigital:2,S" -> { "pr-wspdigital/bupaoshc/88@bitbucket.org" "pr-wspdigital/bupaoshc/88@bitbucket.org" } "1584565503.6029_0.wspdigital:2,S" -> { "pr-wspdigital/bupaoshc/88@bitbucket.org" "pr-wspdigital/bupaoshc/88@bitbucket.org" } "1584565500.6006_0.wspdigital:2,S" -> { "pr-wspdigital/bupaoshc/88@bitbucket.org" "prcomment-140406725@bitbucket.org" } "1584565499.6003_0.wspdigital:2,S" -> { "pr-wspdigital/bupaoshc/88@bitbucket.org" "prcomment-140408368@bitbucket.org" } "1584565497.5997_0.wspdigital:2,S" -> { "pr-wspdigital/bupaoshc/88@bitbucket.org" "pr-wspdigital/bupaoshc/88@bitbucket.org" } "1584565495.5991_0.wspdigital:2,S" -> { "pr-wspdigital/bupaoshc/88@bitbucket.org" "pr-wspdigital/bupaoshc/88@bitbucket.org" } "1584565493.5988_0.wspdigital:2,S" -> { "pr-wspdigital/bupaoshc/88@bitbucket.org" "pr-wspdigital/bupaoshc/88@bitbucket.org" } "1584506116.25523_0.wspdigital:2,S" -> { "pr-wspdigital/bupaoshc/88@bitbucket.org" "prcomment-140607163@bitbucket.org" } "1584505808.24160_0.wspdigital:2,S" -> { "pr-wspdigital/bupaoshc/88@bitbucket.org" "prcomment-140604942@bitbucket.org" } "1584494813.25016_0.wspdigital:2,S" -> { "pr-wspdigital/bupaoshc/88@bitbucket.org" "pr-wspdigital/bupaoshc/88@bitbucket.org" } "1584494811.24972_0.wspdigital:2,S" -> { "pr-wspdigital/bupaoshc/88@bitbucket.org" "pr-wspdigital/bupaoshc/88@bitbucket.org" } "1584487437.29402_0.wspdigital:2,S" -> { "pr-wspdigital/bupaoshc/88@bitbucket.org" } "1584487127.27595_0.wspdigital:2,S" -> { "pr-wspdigital/bupaoshc/88@bitbucket.org" } "1584486509.24570_0.wspdigital:2,S" -> { "pr-wspdigital/bupaoshc/88@bitbucket.org" "prcomment-140582429@bitbucket.org" } "1584486508.24564_0.wspdigital:2,S" -> { "pr-wspdigital/bupaoshc/88@bitbucket.org" "prcomment-140582662@bitbucket.org" } "1584434229.8921_0.wspdigital:2,S" -> { "pr-wspdigital/bupaoshc/88@bitbucket.org" } "1584434231.8924_0.wspdigital:2,S" -> { "pr-wspdigital/bupaoshc/88@bitbucket.org" } "1584434232.8927_0.wspdigital:2,S" -> { "pr-wspdigital/bupaoshc/88@bitbucket.org" } "1584434234.8930_0.wspdigital:2,S" -> { "pr-wspdigital/bupaoshc/88@bitbucket.org" }
}

Re: Weird tagging issue

[Daniel Kahn Gillmor]

[-- Attachment #1: Type: text/plain, Size: 1307 bytes --]

Hi Brian--

whew, what a mess!

I'm having a hard time identifying a specific problem, but I did notice
this surprising sequence:

On Mon 2020-03-30 08:03:20 +1100, Brian May wrote:
> subgraph "cluster_id:pr-wspdigital/bupaoshc/88@bitbucket.org" {
> "pr-wspdigital/bupaoshc/88@bitbucket.org" [shape=folder];
> "1584434229.8921_0.wspdigital:2,S" [shape=note];
> "1584434231.8924_0.wspdigital:2,S" [shape=note];
> "1584434232.8927_0.wspdigital:2,S" [shape=note];
> "1584434234.8930_0.wspdigital:2,S" [shape=note];
> }

[…]
> "1584434229.8921_0.wspdigital:2,S" -> { "pr-wspdigital/bupaoshc/88@bitbucket.org" } 
> "1584434231.8924_0.wspdigital:2,S" -> { "pr-wspdigital/bupaoshc/88@bitbucket.org" }
> "1584434232.8927_0.wspdigital:2,S" -> { "pr-wspdigital/bupaoshc/88@bitbucket.org" }
> "1584434234.8930_0.wspdigital:2,S" -> { "pr-wspdigital/bupaoshc/88@bitbucket.org" }

The surprising thing about this output is that there appear to be 4
copies of message that References: itself!

And, many many other messages also reference that message too.

Would you be willing to send me the full headers from one of the above 4
files so i can verify that this is correct?

I have no idea whether this is part of the problem you're seeing, but it
is certainly unusual.

        --dkg

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: Weird tagging issue

[Brian May]

Daniel Kahn Gillmor <dkg@fifthhorseman.net> writes:

>> "1584434229.8921_0.wspdigital:2,S" -> { "pr-wspdigital/bupaoshc/88@bitbucket.org" } 
>> "1584434231.8924_0.wspdigital:2,S" -> { "pr-wspdigital/bupaoshc/88@bitbucket.org" }
>> "1584434232.8927_0.wspdigital:2,S" -> { "pr-wspdigital/bupaoshc/88@bitbucket.org" }
>> "1584434234.8930_0.wspdigital:2,S" -> { "pr-wspdigital/bupaoshc/88@bitbucket.org" }
>
> The surprising thing about this output is that there appear to be 4
> copies of message that References: itself!
>
> And, many many other messages also reference that message too.
>
> Would you be willing to send me the full headers from one of the above 4
> files so i can verify that this is correct?

I suspect you only need the Message-ID and the References headers,
right? If not, can send more:

Message-ID: <pr-wspdigital/bupaoshc/88@bitbucket.org>
References: <pr-wspdigital/bupaoshc/88@bitbucket.org>

This applies to all 4 of those messages. Yes, they all look like they
have the same Message-ID

Also I am not seeing any duplicate headers or anything like that.
-- 
Brian May <brian@linuxpenguins.xyz>
https://linuxpenguins.xyz/brian/

Re: Weird tagging issue

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 398 bytes --]

On Mon, Mar 30 2020, Brian May wrote:
> This applies to all 4 of those messages. Yes, they all look like they
> have the same Message-ID

4 different messages all with the same message ID definitely violates a
core assumption of notmuch (that each message as a unique message ID).

I think it would be reasonable to state that the behavior of notmuch is
officially "undefined" in that case.

-Carl

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Weird tagging issue

[Daniel Kahn Gillmor]

[-- Attachment #1: Type: text/plain, Size: 1068 bytes --]

On Sun 2020-03-29 17:53:01 -0700, Carl Worth wrote:
> On Mon, Mar 30 2020, Brian May wrote:
>> This applies to all 4 of those messages. Yes, they all look like they
>> have the same Message-ID
>
> 4 different messages all with the same message ID definitely violates a
> core assumption of notmuch (that each message as a unique message ID).

Well, not if they're actually all the same message, right?

Brian, can you confirm whether the body of these 4 messages are the same
or not?  (e.g. you might have gotten different copies due to receiving
mail at different e-mail addresses, or through a mailing list, or just
through an SMTP hiccup)

> I think it would be reasonable to state that the behavior of notmuch is
> officially "undefined" in that case.

What about for a message that References: its own Message-ID:?  Do we
expect to handle that case sensibly?  I'm not sure where i'd look in the
test suite to confirm that we intend to make that work OK, and i can
definitely imagine that being a goofy corner case when trying to
manipulate threads.

       --dkg

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: Weird tagging issue

[Brian May]

Daniel Kahn Gillmor <dkg@fifthhorseman.net> writes:

> Well, not if they're actually all the same message, right?
>
> Brian, can you confirm whether the body of these 4 messages are the same
> or not?  (e.g. you might have gotten different copies due to receiving
> mail at different e-mail addresses, or through a mailing list, or just
> through an SMTP hiccup)

All emails appear to be different. I think bitbucket stuffed up.

> What about for a message that References: its own Message-ID:?  Do we
> expect to handle that case sensibly?  I'm not sure where i'd look in the
> test suite to confirm that we intend to make that work OK, and i can
> definitely imagine that being a goofy corner case when trying to
> manipulate threads.

I can't help here. I do have a vague recollection that I have seen this
situation in the past (any easy way to search for such emails?), but I
really can't remember how well notmuch copes (the fact I noticed such an
issue might suggest I encountered difficulties - or maybe this was
something else entirely).
-- 
Brian May <brian@linuxpenguins.xyz>
https://linuxpenguins.xyz/brian/