Re: Synchronization success stories?

[dm-list-email-notmuch@scs.stanford.edu]

Tilmann Singer <tils@tils.net> writes:

> David Mazieres <dm-list-email-notmuch@scs.stanford.edu> writes:
>> What happens if you get a message that's been stuck in a queue for a few
>> days and has an old Date: header?
>
> It would be missed.  I have set the timespan to look backwards for new
> mail to one month to be a bit safer against the stuck-in-queue cases,
> but mails with older Date: headers would definitely get missed.
>
> The current output of notmuch count "*" is the same on both the client
> and the server, so it seems I didn't run into this problem yet (maybe I
> was just lucky).

I've been playing around with reorganizing my maildir, and found a
couple of messages (on mailing lists) with clearly invalid dates years
in the past.  But checking with notmuch count is a good idea.  Then you
can always fall back to the slow path in the unlikely event that your
counts don't match up.  Well, except that A) count is just unique
message-IDs, not messages, and B) when synchronizing in both directions
you could still miss something.  You have to assume that the invalid
dates are only ever going to occur at one end of a synchronization
event.

>> Or if you get new messages that have
>> the same Message-ID as old ones?
>
> Is that even possible?  I thought that notmuch guarantees the uniqueness
> of indexed message ids.  The only reference I could find without trying
> to read the code was this thread id:87mwyz3s9d.fsf@star.eba from 2012,
> which supports the assumption.

Sadly, yes it is quite possible, and even opens up a slight security
issue.  Suppose I know you are on a mailing list, and some message
appears on that mailing list that I don't want you to see.  I can send
you an innocuous-looking message that just happens to have the same
message-id, and you may never see the original mailing list message.
Even better, depending on how your spam filtering is setup, if I include
the GTUBE string in my message you may never see mine or the original.

That's why with muchsync, I replicate actual mail messages, rather than
message-IDs.  Then you can always periodically check for message-IDs
that appear in more than one file.  (In fact, thought I haven't
published an interface for this, the SQL database kept my muchsync makes
it trivial to check for this and detect certain attacks.)

I understand why notmuch went with message IDs.  For instance you have
sent this reply both directly to me and to a mailing list I am
subscribed to.  So I will get two slightly different copies of the
message (one will have the standard notmuch mailing list signature, the
other won't).  And this way once I've marked it read, the message will
be read even once the second copy comes in.  But personally I'd rather
see the occasional duplicate message than risk not seeing messages.  In
particular, if the goal is to see fewer unread messages, some sort of
feature that pro-actively skips all future messages in a thread or
subthread would be more useful...

> Here is how long they take (on a machine with an SSD, which certainly
> helps):
>
> $ time notmuch dump --format=batch-tag | sort > /tmp/notmuch.dump
> real    0m3.643s
> user    0m3.593s
> sys     0m0.140s
> $ time notmuch restore < /tmp/notmuch.dump
> real    0m3.719s
> user    0m3.357s
> sys     0m0.357s
> $ notmuch count 
> 117118

That's crazy.  I'm jealous.  Then again, this is how fast muchsync runs
(including a full database scan to detect changed messages and tags)
when there is no new mail:

$ time ./muchsync -v
[notmuch] No new mail.
synchronizing muchsync database with Xapian... 0.038506 (+0.038506)
starting scan of Xapian database... 0.039069 (+0.000563)
opened Xapian... 0.040851 (+0.001782)
scanned message IDs... 0.137647 (+0.096796)
scanned tags... 0.170404 (+0.032757)
scanned directories in xapian... 0.172100 (+0.001696)
scanned filenames in xapian... 0.172376 (+0.000276)
adjusted link counts... 0.199461 (+0.027085)
finished synchronizing muchsync database with Xapian... 0.212965 (+0.013505)

real    0m0.220s
user    0m0.173s
sys     0m0.023s

David