thoughts improving duplicate message handling...

thoughts improving duplicate message handling...

[Eric Wong]

Spinning off of https://public-inbox.org/meta/20201228213139.GA17600@dcvr/

> (*) The downside to this approach is IMAP UIDs (NNTP article
> numbers) get changed, but I think I can workaround that.  The
> workaround I'm thinking of involves capturing exact blob OIDs
> during the unindex phase to create an OID => UID mapping.
> reindex would reuse the OID => UID mapping to keep the same
> IMAP UID.  It could be loosened to use ContentHash, or
> whatever combination of Message-ID/From/Date/etc, too.

I've noticed quite a few messages with identical Message-IDs,
especially with /all/ (*)

a) There are occasionally resent revisions of patches with the
   same Message-ID

b) More often, a cross-posted message has different trailers
   depending on which list it was posted to.  (And the
   PublicInbox::Filter::* API doesn't apply at indexing,
   only at git injection time)

I'm thinking of two new features:

1) the WWW and lei UIs could have a "diff mode"
   that clearly shows differences between rendered texts
   This could be expensive with malicious inputs, but the
   existing "limiter" infrastructure used for serving packs
   should help, here.

2) support PublicInbox::Filter at index-time
   (requires reindexing to take effect, will cause new IMAP
   UIDs+NNTP article numbers to show up, possibly en-mass)

(*) https://public-inbox.org/meta/20201126194543.GA30337@dcvr/

Re: thoughts improving duplicate message handling...

[Konstantin Ryabitsev]

On Mon, Dec 28, 2020 at 09:41:46PM +0000, Eric Wong wrote:
> a) There are occasionally resent revisions of patches with the
>    same Message-ID

This is broken and is unwanted. :)

> b) More often, a cross-posted message has different trailers
>    depending on which list it was posted to.  (And the
>    PublicInbox::Filter::* API doesn't apply at indexing,
>    only at git injection time)

In this case, my preferred solution is "first in wins", with ability to affect
which one shows up first, using a weighting metric such as "comes first in the
config file" or "has a higher priority setting."

-K