viewing duplicate messages

viewing duplicate messages

[drebs]

Hello, sorry for breaking the thread but i just subscribed to this list.
This is a response to:
https://notmuchmail.org/pipermail/notmuch/2019/028189.html

On Thu May 30 16:02:57 PDT 2019, dkg at fifthhorseman.net wrote:
> If you'd care to forward me a zipfile containing examples of at least
> two distinct messages with the same message ID, i'd love to look through
> a copy of this sort of confusion.  It's good to understand the different
> ways these sort of things go wrong.
> 
>     --dkg

I've bounced into this problem when using notmuch-emacs to interact with
a schleuder list (https://schleuder.org).

I know it's very specific, looks like an usability issue more than
anything, and I'm not very sure it should be considered as a notmuch or
a schleuder issue, but I think it's worth to describe it here to have it
noted and maybe get feedback.

---

When I send a message to a schleuder list, it is first encrypted to the
list's OpenPGP certificate before leaving my computer. Schleuder will
then decrypt it and re-encrypt to the subscribers' certificates, and it
will maintain the Message-Id untouched.

After i receive my own message back from the list, I will end up with
2 different files in my maildir having the same Message-Id: the first
one is the message i sent which is encrypted to the list's cert and the
second one is the message the list sent back to me encrypted to my own
cert.

Looks like from notmuch doc that notmuch search counts those 2 files as
the same message, and it shows me that there is 1 match but 2 files (in
parenthesis). notmuch show also shows me the first message only. If
i list --output=files I'll get both files as result.

Problems are:

    1. notmuch-emacs displays to me the message I sent, and that one
       I can't decrypt because it's encrypted to the list's cert.

    2. notmuch does not tag the incoming message as new, so it's hard to
       notice when/if the message has arrived back from the list.

---

Some workarounds I thought:

    - encrypt the outgoing message also to myself, solves (1) but not
      (2).

    - using a post-new hook to find out when the message has been sent
      back and do *something* so notmuch will tag and show the "correct"
      one for me. (thanks to Defolos in IRC; still have to come up with
      *something*)

I wonder if someone goes through the same issue and has comments or
solutions to this.

Sorry that I can't provide a copy of the messages, but I hope the
description is explanatory enough.

Re: viewing duplicate messages

[Georg Faerber]

[-- Attachment #1: Type: text/plain, Size: 593 bytes --]

Hi drebs, all,

Just a short note on this:

On 19-07-17 14:50:50, drebs wrote:
> When I send a message to a schleuder list, it is first encrypted to
> the list's OpenPGP certificate before leaving my computer. Schleuder
> will then decrypt it and re-encrypt to the subscribers' certificates,
> and it will maintain the Message-Id untouched.

FWIW: That's configurable, see [1]. Unfortunately, that's currently the
only place which documents this kind of stuff.

Cheers,
Georg


[1] https://0xacab.org/schleuder/schleuder/blob/191b5122403942496490e532e48a290d7dd0cd98/etc/list-defaults.yml#L49

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: viewing duplicate messages

[Daniel Kahn Gillmor]

[-- Attachment #1: Type: text/plain, Size: 741 bytes --]

On Wed 2019-07-17 15:08:49 +0000, Georg Faerber wrote:
> On 19-07-17 14:50:50, drebs wrote:
>> When I send a message to a schleuder list, it is first encrypted to
>> the list's OpenPGP certificate before leaving my computer. Schleuder
>> will then decrypt it and re-encrypt to the subscribers' certificates,
>> and it will maintain the Message-Id untouched.
>
> FWIW: That's configurable, see [1]. Unfortunately, that's currently the
> only place which documents this kind of stuff.

fwiw, i don't think setting keep_msgid: false in schleuder is a good
answer, because setting it causes pretty nasty threading failures.
hopefully we can think through a better approach for dealing with this
from the client side.

     --dkg

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: viewing duplicate messages

[Georg Faerber]

Hi,

On 19-07-28 21:07:56, Daniel Kahn Gillmor wrote:
> fwiw, i don't think setting keep_msgid: false in schleuder is a good
> answer, because setting it causes pretty nasty threading failures.
> hopefully we can think through a better approach for dealing with this
> from the client side.

I fully agree, and probably should have made this more clearer in my
mail. :)

Cheers,
Georg

viewing duplicate messages

[Rollins, Jameson]

This is essentially a wishlist report...

It would be nice to have an easy way in notmuch-emacs-show to know if
there are multiple messages with the same message-id in the store, and
to cycle through viewing each individually.

It's possible (even likely) that one has multiple messages with the same
message-id in their mail store.  For example, this can happen when sent
messages are saved with an fcc and the same message is received on a
mailing list.  Notmuch indexes all copies, but it only shows you one of
the copies when you ask to view the message.

This behavior would be fine if all copies of the message were identical.
But it's more likely that the copies are in fact different, for example
because the mailing list has altered the received message in some way,
it was mangled in transit, or someone is explicitly trying to attack
your mail store [0].

It would be nice to have better exposure of the fact that multiple
copies exist, and to have an easy way to view alternate copies.  For
notmuch-emacs a key could just cycle through available copies.

jamie.

[0] https://notmuchmail.org/pipermail/notmuch/2012/009965.html

Re: viewing duplicate messages

[Jorge P. de Morais Neto]

"Rollins, Jameson" <jrollins@caltech.edu> writes:

> It would be nice to have an easy way in notmuch-emacs-show to know if
> there are multiple messages with the same message-id in the store, and
> to cycle through viewing each individually.

That would be useful to me too.  I have a Dell laptop and I subscribed
to email notifications about firmware updates.  It turns out that Dell
sends all such notifications (which obviously have different content)
with the same message-id.

Regards
-- 
- I am Brazilian.  I hope my English is correct and I welcome feedback
- Please adopt free formats like PDF, ODF, Org, LaTeX, Opus, WebM and 7z
- Free/libre software for Android: https://f-droid.org/
- [[https://www.gnu.org/philosophy/free-sw.html][What is free software?]]

Re: viewing duplicate messages

[Rollins, Jameson]

On Thu, May 30 2019, Jorge P. de Morais Neto <jorge+list@disroot.org> wrote:
> That would be useful to me too.  I have a Dell laptop and I subscribed
> to email notifications about firmware updates.  It turns out that Dell
> sends all such notifications (which obviously have different content)
> with the same message-id.

[face palm] it's a wonder that the internet works at all...

Re: viewing duplicate messages

[Daniel Kahn Gillmor]

[-- Attachment #1: Type: text/plain, Size: 754 bytes --]

On Thu 2019-05-30 19:11:22 -0300, Jorge P. de Morais Neto wrote:
> I have a Dell laptop and I subscribed to email notifications about
> firmware updates.  It turns out that Dell sends all such notifications
> (which obviously have different content) with the same message-id.

Have you reported this to Dell?  This sounds like something that anyone
at Dell who understands the technical issues would want to fix.  If
you've reported it publicly, i'd appreciate a pointer to the discussion.

If you'd care to forward me a zipfile containing examples of at least
two distinct messages with the same message ID, i'd love to look through
a copy of this sort of confusion.  It's good to understand the different
ways these sort of things go wrong.

    --dkg

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: viewing duplicate messages

[Jorge P. de Morais Neto]

Daniel Kahn Gillmor <dkg@fifthhorseman.net> writes:

> Have you reported this to Dell?

Not yet.

> If you'd care to forward me a zipfile containing examples of at least
> two distinct messages with the same message ID, i'd love to look through
> a copy of this sort of confusion.  It's good to understand the different
> ways these sort of things go wrong.

I have searched and found no copy.  That's because I physically delete
these Dell notification messages after reading and acting on them.  This
way there is always 0 or 1 such message, so Notmuch does not get
confused.  But I will save the next ones and give you.  In fact, in an
attempt to temporarily increase the frequency of notifications, I have
just subscribed to notifications about everything related to my notebook
series, even MS-Windows drivers.

Regards
-- 
- I am Brazilian.  I hope my English is correct and I welcome feedback
- Please adopt free formats like PDF, ODF, Org, LaTeX, Opus, WebM and 7z
- Free/libre software for Android: https://f-droid.org/
- [[https://www.gnu.org/philosophy/free-sw.html][What is free software?]]

Re: viewing duplicate messages

[Daniel Kahn Gillmor]

[-- Attachment #1: Type: text/plain, Size: 678 bytes --]

On Fri 2019-05-31 12:45:46 -0300, Jorge P. de Morais Neto wrote:
> I have searched and found no copy.  That's because I physically delete
> these Dell notification messages after reading and acting on them.  This
> way there is always 0 or 1 such message, so Notmuch does not get
> confused.  But I will save the next ones and give you.  In fact, in an
> attempt to temporarily increase the frequency of notifications, I have
> just subscribed to notifications about everything related to my notebook
> series, even MS-Windows drivers.

thanks!  i don't mean to interrupt your workflow, which makes total
sense to me.  feel free to send them to me as they come in.

      --dkg

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: viewing duplicate messages

[Jorge P. de Morais Neto]

[-- Attachment #1: Type: text/plain, Size: 868 bytes --]

Em 2019-05-30T19:02:57-0400, Daniel Kahn Gillmor escreveu:

> Have you reported this to Dell?  This sounds like something that anyone
> at Dell who understands the technical issues would want to fix.  If
> you've reported it publicly, i'd appreciate a pointer to the discussion.

I have just sent an email to a Dell saleswoman I had previously
corresponded with.  I asked her to forward the report to the appropriate
organizational unit and tell me the ticket identifier should a ticket be
created.  This was the only contact channel I found; I first looked into
Dell's website and found no better option.

> If you'd care to forward me a zipfile containing examples of at least
> two distinct messages with the same message ID, i'd love to look through
> a copy of this sort of confusion.  It's good to understand the different
> ways these sort of things go wrong.


[-- Attachment #2: dell_homonymous_messages.tar.xz --]
[-- Type: application/x-xz, Size: 11252 bytes --]

[-- Attachment #3: Type: text/plain, Size: 582 bytes --]


I have attached a tarball with three homonymous messages from Dell.  The
last (most recent) two have the same subject and bodies, but the first
(earliest) one is different and yet they all have Message-Id 1. I have
included the Notmuch list as a recipient because the tarball is a mere
11252B.

Regards
-- 
- I am Brazilian.  I hope my English is correct and I welcome feedback
- Please adopt free formats like PDF, ODF, Org, LaTeX, Opus, WebM and 7z
- Free/libre software for Android: https://f-droid.org/
- [[https://www.gnu.org/philosophy/free-sw.html][What is free software?]]

Re: viewing duplicate messages

[Daniel Kahn Gillmor]

[-- Attachment #1: Type: text/plain, Size: 2057 bytes --]

On Sat 2019-08-17 19:12:26 -0300, Jorge P. de Morais Neto wrote:
> I have attached a tarball with three homonymous messages from Dell.  The
> last (most recent) two have the same subject and bodies, but the first
> (earliest) one is different and yet they all have Message-Id 1. I have
> included the Notmuch list as a recipient because the tarball is a mere
> 11252B.

thanks for this.  Looking at the headers, it occurs to me that the
problem might actually be that Dell ("idd_messaging_email@dell.com")
might not including a message-id header at all, and it is being added
their IronPort/Sophos AV client as it passes through their mail system.

I suspect this possibility because the placement of the Message-ID
header itself is supiciiously high up in the list of headers (it looks
like it might have been placd there by the initial relaying MTA, rather
than the MUA).

If this is the case, it could be solved in one of two ways: they could
inject a proper unique Message-ID before handing the message off to
IronPort; or they could fix their IronPort appliance to inject a proper
unique Message-ID header.

That's all about fixing it on the sender side though.  Are there
possible fixes on the receiving side?

one thought is that notmuch could treat an obviously low-entropy
message-ID the same way that it treats a message with no Message-ID at
all.  Of course, that raises the question: what is a low-entropy message
ID? A single-character message-id is pretty clearly too low-entropy to
be useful, but if we said "1-character long" was too short, it would at
least avoid this particular mistake.

i also note that NEWS claims (in the section for notmuch 0.17) that
notmuch treats "overlong" message-ids in the same way as missing
message-ids, but i don't see where that distinction is done in the code.
It doesn't appear to be in lib/message-file.c, where the notmuch-sha1-*
generation is done.  But anyway, if we are treating "overlong"
message-ids as missing, it's nicely symmetric to treat "overshort"
message-ids in the same way.

      --dkg

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]