From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <e@80x24.org>
X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net
X-Spam-Level: 
X-Spam-ASN:  
X-Spam-Status: No, score=-4.1 required=3.0 tests=ALL_TRUSTED,AWL,BAYES_00,
	DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,
	T_SCC_BODY_TEXT_LINE shortcircuit=no autolearn=ham autolearn_force=no
	version=3.4.2
Received: from localhost (dcvr.yhbt.net [127.0.0.1])
	by dcvr.yhbt.net (Postfix) with ESMTP id 5786C1F727;
	Wed, 29 Jun 2022 16:30:33 +0000 (UTC)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=80x24.org;
	s=selector1; t=1656520233;
	bh=N9oFKqifGYsUlcNJWQZ3eH73rlemT5ymFwlAlasPuok=;
	h=Date:From:To:Cc:Subject:References:In-Reply-To:From;
	b=gIjYrvlUsNRKOUqUwLWcGa8WIF6ZqSVRR0bGCt1c4LugaN9X+q0rHtJipFSpeoAa3
	 a7DDW9nmLdpX3DlfomO54ojA3mbb9bEuJjdsuY9qKMkJV1KvnP9R6FFw0922+2SkyT
	 LiIO6LXchPSBBTTONl/3gOC6/ZBO2kS9UT57u04w=
Date: Wed, 29 Jun 2022 16:30:33 +0000
From: Eric Wong <e@80x24.org>
To: Rob Herring <robh@kernel.org>
Cc: meta@public-inbox.org
Subject: Re: lei missing mails
Message-ID: <20220629163033.GA14412@dcvr>
References: <CAL_JsqJH8xx_2NyZffNsRXbGXiv3kjmCETvKXt3Yfb0uToLm9Q@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
In-Reply-To: <CAL_JsqJH8xx_2NyZffNsRXbGXiv3kjmCETvKXt3Yfb0uToLm9Q@mail.gmail.com>
List-Id: <meta.public-inbox.org>

Rob Herring <robh@kernel.org> wrote:
> Hi,
> 
> I'm using lei with lore where I have 2 queries which overlap. Really,
> one is a subset of the other. On those overlapping threads, I'm
> finding that sometimes new messages are written to one mailbox and not
> the other. (At least sometimes, the messages may be missing from all
> mailboxes sometimes too. I'm not certain.) Using --remote-fudge-time
> to force refetching seems to get the missing mails. I haven't found
> anything strange in timestamps of the missing mails, but otherwise am
> not sure how to debug this further. The queries are retrieving full
> threads and the missing mails are in the threads, but not direct
> matches to the queries. I realize that's not a lot of detail to go on.
> Suggestions on debugging this further?

Is this with 1.8 or 1.7?

I forgot to note in the release notes, but there were some
SQLite usage-related fixes which could avoid missing messages.

You'll need "lei daemon-kill" after upgrading to 1.8 to ensure
the new code is running.

What might be interesting is to use the URLs lei prints and
comparing the results w/o lei.

I'll have to double-check if overlapping affects things, but it
shouldn't; since the dedupe logic is per-output.

Is this exclusively with HTTPS endpoints and writing to Maildirs
(or something else?)

> It might be helpful if lei could print out message-ids of messages
> written to mailboxes.

That could get very noisy, especially as mailboxes are written
in parallel.

Thanks.