Boosts still not quite working

Boosts still not quite working

[Konstantin Ryabitsev]

Eric:

The new x-lore systems are rebuilt and reindexed using the latest master (as
of 2 days ago), but boosts still don't appear to be quite working as intended.
For example, this thread:

https://x-lore.kernel.org/all/20210809175620.720923-1-ltykernel@gmail.com/

It was sent to iommu@lists.linux-foundation.org (mailman) and to a bunch of
vger lists, all of which have higher boosts in the configuration, e.g. netdev:

	[publicinbox "linux-iommu"]
	  address = iommu@lists.linux-foundation.org
	  url = https://x-lore.kernel.org/linux-iommu/
	  inboxdir = /srv/public-inbox/lore.kernel.org/linux-iommu
	  indexlevel = full
	  newsgroup = org.linux-foundation.lists.iommu
	  infourl = https://www.kernel.org/lore.html
	  boost = 1
	  listid = iommu.lists.linux-foundation.org
	[publicinbox "netdev"]
	  address = netdev@vger.kernel.org
	  url = https://x-lore.kernel.org/netdev/
	  inboxdir = /srv/public-inbox/lore.kernel.org/netdev
	  indexlevel = full
	  newsgroup = org.kernel.vger.netdev
	  infourl = https://www.kernel.org/lore.html
	  boost = 10
	  listid = netdev.vger.kernel.org

When retrieving mbox.gz, we get the contents as archived by the iommu list,
though theoretically we should get one of the higher-boosted lists instead.
Anything I can help troubleshoot?

This was indexed from scratch with all repositories already in place.

Best regards,
-K

Re: Boosts still not quite working

[Eric Wong]

Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> Eric:
> 
> The new x-lore systems are rebuilt and reindexed using the latest master (as
> of 2 days ago), but boosts still don't appear to be quite working as intended.
> For example, this thread:
> 
> https://x-lore.kernel.org/all/20210809175620.720923-1-ltykernel@gmail.com/
> 
> It was sent to iommu@lists.linux-foundation.org (mailman) and to a bunch of
> vger lists, all of which have higher boosts in the configuration, e.g. netdev:

boost doesn't come into effect due to the Mailman footer from
the iommu list.  extindex and v2 both account for content
differences despite sharing the same Message-ID
(deduplicating purely on Message-ID would be open to abuse
 (and many old MUA-side bugs))

> 	[publicinbox "linux-iommu"]
> 	  address = iommu@lists.linux-foundation.org
> 	  url = https://x-lore.kernel.org/linux-iommu/
> 	  inboxdir = /srv/public-inbox/lore.kernel.org/linux-iommu
> 	  indexlevel = full
> 	  newsgroup = org.linux-foundation.lists.iommu
> 	  infourl = https://www.kernel.org/lore.html
> 	  boost = 1
> 	  listid = iommu.lists.linux-foundation.org
> 	[publicinbox "netdev"]
> 	  address = netdev@vger.kernel.org
> 	  url = https://x-lore.kernel.org/netdev/
> 	  inboxdir = /srv/public-inbox/lore.kernel.org/netdev
> 	  indexlevel = full
> 	  newsgroup = org.kernel.vger.netdev
> 	  infourl = https://www.kernel.org/lore.html
> 	  boost = 10
> 	  listid = netdev.vger.kernel.org

> When retrieving mbox.gz, we get the contents as archived by the iommu list,
> though theoretically we should get one of the higher-boosted lists instead.
> Anything I can help troubleshoot?

I'm seeing both the iommu and LKML List-IDs (which is intended
behavior due to aforementioned Mailman footer)
 https://x-lore.kernel.org/all/20210809175620.720923-1-ltykernel@gmail.com/raw
and also
 https://x-lore.kernel.org/all/20210809175620.720923-1-ltykernel@gmail.com/t.mbox.gz

Is LKML boosted more than netdev?  (If so, then it's working as
expected, since the footer-less vger versions should be considered
identical).

I'm planning on having a "diff view" to more easily distinguish
between different messages having the same Message-ID.  It would
make it easier to highlight buggy clients, Mailman misconfigurations,
and malicious attempts to obscure/confuse readers.

Re: Boosts still not quite working

[Konstantin Ryabitsev]

On Sat, Aug 14, 2021 at 08:46:33PM +0000, Eric Wong wrote:
> > It was sent to iommu@lists.linux-foundation.org (mailman) and to a bunch of
> > vger lists, all of which have higher boosts in the configuration, e.g. netdev:
> 
> boost doesn't come into effect due to the Mailman footer from
> the iommu list.  extindex and v2 both account for content
> differences despite sharing the same Message-ID
> (deduplicating purely on Message-ID would be open to abuse
>  (and many old MUA-side bugs))

Ah-ha, okay. I clearly misunderstood its purpose. No worries, I see now that
when we retrieve t.mbox.gz, the mailbox contains both the iommu and the vger
sources, which should allow me to pick the preferred version based on the
criteria I need (e.g. DKIM validation).

> I'm planning on having a "diff view" to more easily distinguish
> between different messages having the same Message-ID.  It would
> make it easier to highlight buggy clients, Mailman misconfigurations,
> and malicious attempts to obscure/confuse readers.

Indeed, I agree that it's best to give access to both instead of always
returning the boosted-list version, as this would allow someone for malicious
message-id stuffing.

Now I just need to hack b4 to define a better criteria for when results have
multiple identical message-id's.

Thanks,
-K