From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: AS54825 145.40.68.0/24 X-Spam-Status: No, score=-3.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE shortcircuit=no autolearn=ham autolearn_force=no version=3.4.2 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by dcvr.yhbt.net (Postfix) with ESMTPS id 133B01F729 for ; Wed, 29 Jun 2022 22:02:13 +0000 (UTC) Authentication-Results: dcvr.yhbt.net; dkim=pass (2048-bit key; unprotected) header.d=kernel.org header.i=@kernel.org header.b="fIbGQqRK"; dkim-atps=neutral Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 61F39B8249B for ; Wed, 29 Jun 2022 22:02:11 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 0568DC34114 for ; Wed, 29 Jun 2022 22:02:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1656540130; bh=v166FFe6/5c76vR20FJmoKf1u9bxE9J5Bd43/Gscecw=; h=References:In-Reply-To:From:Date:Subject:To:Cc:From; b=fIbGQqRKRgP3eDgBsEdPketP7mRNWgkyXvGc4+6/Reb+6Y0NIqnQD3vUOj7Cvq98O jE6+txD6z17ICXD9yNUmOqhjb8awriHWcBR3VYwXH/EiMw1MNteaYqLPMwBf4MKDJX HKVmZBUyEYfhJq3fyrB62WjVogBgIXt1XpharbGBApf/CXe25erWVfdapVzOVKP8C4 sWOtPLyA9CuDW4ME+JC3ZZUlSGotrHuicvKj7rEbETMXi7K7Z5T+rDIpGaDPJAb41R Porhj5jbmomH8gc+UIyTsk+XRw2lOA9bZC3ZzFmHhwMi30oEvmlkjAfVLiHtY/xeqd 037zo663DQnnA== Received: by mail-vs1-f50.google.com with SMTP id k25so16490690vso.6 for ; Wed, 29 Jun 2022 15:02:09 -0700 (PDT) X-Gm-Message-State: AJIora8AiK3b8wymQLV3k1gVMW8CUGF91oWafSrTcZ9Bsx7OQ4TmmmoJ QpzifXl4zI3XX/+IeM1ASE4ePyhq/Tusit2v7g== X-Google-Smtp-Source: AGRyM1sEbXeGkZeLR5U8xuqPWiHsHwUDDE37uF0+9k+b3eezbRBEoP82Os5xng1QIC22bJgUTkBELxhOCSFxUfZIJvg= X-Received: by 2002:a67:c187:0:b0:354:3ab2:ba65 with SMTP id h7-20020a67c187000000b003543ab2ba65mr6442456vsj.53.1656540128885; Wed, 29 Jun 2022 15:02:08 -0700 (PDT) MIME-Version: 1.0 References: <20220629163033.GA14412@dcvr> <20220629172742.M978900@dcvr> In-Reply-To: <20220629172742.M978900@dcvr> From: Rob Herring Date: Wed, 29 Jun 2022 16:01:57 -0600 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: lei missing mails To: Eric Wong Cc: meta@public-inbox.org Content-Type: text/plain; charset="UTF-8" List-Id: On Wed, Jun 29, 2022 at 11:27 AM Eric Wong wrote: > > Rob Herring wrote: > > On Wed, Jun 29, 2022 at 10:30 AM Eric Wong wrote: > > > > > > Rob Herring wrote: > > > > Hi, > > > > > > > > I'm using lei with lore where I have 2 queries which overlap. Really, > > > > one is a subset of the other. On those overlapping threads, I'm > > > > finding that sometimes new messages are written to one mailbox and not > > > > the other. (At least sometimes, the messages may be missing from all > > > > mailboxes sometimes too. I'm not certain.) Using --remote-fudge-time > > > > to force refetching seems to get the missing mails. I haven't found > > > > anything strange in timestamps of the missing mails, but otherwise am > > > > not sure how to debug this further. The queries are retrieving full > > > > threads and the missing mails are in the threads, but not direct > > > > matches to the queries. I realize that's not a lot of detail to go on. > > > > Suggestions on debugging this further? > > > > > > Is this with 1.8 or 1.7? > > > > Commit 68b53c888911 actually. So post 1.8. > > OK, thanks for that info. > > > > I forgot to note in the release notes, but there were some > > > SQLite usage-related fixes which could avoid missing messages. > > > > > > You'll need "lei daemon-kill" after upgrading to 1.8 to ensure > > > the new code is running. > > > > It's possible I haven't done that since updating though I do vaguely > > recall seeing something about needing to do that. Is there any way to > > tell before I restart it? > > Not really, but it's pretty cheap to restart (assuming there's no > long-running jobs). I've restarted and just hit this again. > > > What might be interesting is to use the URLs lei prints and > > > comparing the results w/o lei. $ lei up --all # updating /home/rob/Mail/from-me # updating /home/rob/Mail/missing-cc # updating /home/rob/Mail/my-patches # updating /home/rob/Mail/pci # https://lore.kernel.org/all/ limiting to 2022-06-27 12:42 -0600 and newer # https://lore.kernel.org/all/ limiting to 2022-06-27 9:50 -0600 and newer # https://lore.kernel.org/all/ limiting to 2022-06-27 12:42 -0600 and newer # /usr/bin/curl -Sf -s -d '' https://lore.kernel.org/all/?x=m&t=1&q=(dt%3A20220529211430..+AND+(f%3Arobh%40kernel.org+OR+f%3Arobh%2Bdt%40kernel.org))+AND+dt%3A20220627184226.. # /home/rob/.local/share/lei/store 144/144 # /home/rob/.local/share/lei/store 3/3 # /usr/bin/curl -Sf -s -d '' https://lore.kernel.org/all/?x=m&t=1&q=((dfn%3Adrivers+OR+dfn%3Aarch+OR+dfn%3ADocumentation%2F*+OR+dfn%3Ainclude+OR+dfn%3Ascripts)+AND+f%3Arobh%40kernel.org+AND+rt%3A1640812470..)+AND+dt%3A20220627155025.. # /usr/bin/curl -Sf -s -d '' https://lore.kernel.org/all/?x=m&t=1&q=(l%3Alinux-pci+dfn%3Adrivers%2Fpci%2Fcontroller+dt%3A20220529211430..)+AND+dt%3A20220627184226.. # /home/rob/.local/share/lei/store 0/0 # /home/rob/.local/share/lei/store 362/362 # 0 written to /home/rob/Mail/missing-cc/ (0 matches) # https://lore.kernel.org/all/ 72/72 # https://lore.kernel.org/all/ 4/4 # https://lore.kernel.org/all/ 131/? # https://lore.kernel.org/all/ 184/? # https://lore.kernel.org/all/ 412/? # https://lore.kernel.org/all/ 603/? # https://lore.kernel.org/all/ 853/? # https://lore.kernel.org/all/ 1069/? # https://lore.kernel.org/all/ 1442/? # https://lore.kernel.org/all/ 1443/1443 # 1 written to /home/rob/Mail/pci/ (75 matches) # 2 written to /home/rob/Mail/my-patches/ (148 matches) # 7 written to /home/rob/Mail/from-me/ (1805 matches) What I expected was 3 messages written to 'my-patches'. I think the problem is just simply that the new message missing doesn't match the query, but is a reply to a match. So with a date after the original match in the thread won't pick up anything. The 2nd URL above indeed only has 2 results. I guess I just have to fetch a wider window like a month every time? What's needed is a get any new messages in existing threads. I don't suppose there's an efficient way to do that? > > > > > > I'll have to double-check if overlapping affects things, but it > > > shouldn't; since the dedupe logic is per-output. > > > > > > Is this exclusively with HTTPS endpoints and writing to Maildirs > > > (or something else?) > > > > Yes. It's querying lore and writing to a maildir. Here's one of the queries: > > > > [lei] > > q = (dfn:drivers OR dfn:arch OR dfn:Documentation/* OR > > dfn:include OR dfn:scripts) AND \ > > f:robh@kernel.org AND rt:6.month.ago.. > > [lei "q"] > > include = https://lore.kernel.org/all/ > > external = 1 > > local = 1 > > remote = 1 > > threads = 1 > > dedupe = mid > > output = maildir:/home/rob/Mail/my-patches > > Fwiw, dedupe based on mid could be vulnerable to spoofing, which > is why `content' is the default. But yes, in the past, I've > noticed some messages to meta@public-inbox.org not showing up, > though not recently (I guess lack of activity here is a culprit :x) Does 'content' ignore trailers that mailman lists like to add? I think I switched because of that. Rob