lei missing mails

lei missing mails

[Rob Herring]

Hi,

I'm using lei with lore where I have 2 queries which overlap. Really,
one is a subset of the other. On those overlapping threads, I'm
finding that sometimes new messages are written to one mailbox and not
the other. (At least sometimes, the messages may be missing from all
mailboxes sometimes too. I'm not certain.) Using --remote-fudge-time
to force refetching seems to get the missing mails. I haven't found
anything strange in timestamps of the missing mails, but otherwise am
not sure how to debug this further. The queries are retrieving full
threads and the missing mails are in the threads, but not direct
matches to the queries. I realize that's not a lot of detail to go on.
Suggestions on debugging this further?

It might be helpful if lei could print out message-ids of messages
written to mailboxes.

Rob

Re: lei missing mails

[Eric Wong]

Rob Herring <robh@kernel.org> wrote:
> Hi,
> 
> I'm using lei with lore where I have 2 queries which overlap. Really,
> one is a subset of the other. On those overlapping threads, I'm
> finding that sometimes new messages are written to one mailbox and not
> the other. (At least sometimes, the messages may be missing from all
> mailboxes sometimes too. I'm not certain.) Using --remote-fudge-time
> to force refetching seems to get the missing mails. I haven't found
> anything strange in timestamps of the missing mails, but otherwise am
> not sure how to debug this further. The queries are retrieving full
> threads and the missing mails are in the threads, but not direct
> matches to the queries. I realize that's not a lot of detail to go on.
> Suggestions on debugging this further?

Is this with 1.8 or 1.7?

I forgot to note in the release notes, but there were some
SQLite usage-related fixes which could avoid missing messages.

You'll need "lei daemon-kill" after upgrading to 1.8 to ensure
the new code is running.

What might be interesting is to use the URLs lei prints and
comparing the results w/o lei.

I'll have to double-check if overlapping affects things, but it
shouldn't; since the dedupe logic is per-output.

Is this exclusively with HTTPS endpoints and writing to Maildirs
(or something else?)

> It might be helpful if lei could print out message-ids of messages
> written to mailboxes.

That could get very noisy, especially as mailboxes are written
in parallel.

Thanks.

Re: lei missing mails

[Rob Herring]

On Wed, Jun 29, 2022 at 10:30 AM Eric Wong <e@80x24.org> wrote:
>
> Rob Herring <robh@kernel.org> wrote:
> > Hi,
> >
> > I'm using lei with lore where I have 2 queries which overlap. Really,
> > one is a subset of the other. On those overlapping threads, I'm
> > finding that sometimes new messages are written to one mailbox and not
> > the other. (At least sometimes, the messages may be missing from all
> > mailboxes sometimes too. I'm not certain.) Using --remote-fudge-time
> > to force refetching seems to get the missing mails. I haven't found
> > anything strange in timestamps of the missing mails, but otherwise am
> > not sure how to debug this further. The queries are retrieving full
> > threads and the missing mails are in the threads, but not direct
> > matches to the queries. I realize that's not a lot of detail to go on.
> > Suggestions on debugging this further?
>
> Is this with 1.8 or 1.7?

Commit 68b53c888911 actually. So post 1.8.

> I forgot to note in the release notes, but there were some
> SQLite usage-related fixes which could avoid missing messages.
>
> You'll need "lei daemon-kill" after upgrading to 1.8 to ensure
> the new code is running.

It's possible I haven't done that since updating though I do vaguely
recall seeing something about needing to do that. Is there any way to
tell before I restart it?


> What might be interesting is to use the URLs lei prints and
> comparing the results w/o lei.
>
> I'll have to double-check if overlapping affects things, but it
> shouldn't; since the dedupe logic is per-output.
>
> Is this exclusively with HTTPS endpoints and writing to Maildirs
> (or something else?)

Yes. It's querying lore and writing to a maildir. Here's one of the queries:

[lei]
        q = (dfn:drivers OR dfn:arch OR dfn:Documentation/* OR
dfn:include OR dfn:scripts) AND \
         f:robh@kernel.org AND rt:6.month.ago..
[lei "q"]
        include = https://lore.kernel.org/all/
        external = 1
        local = 1
        remote = 1
        threads = 1
        dedupe = mid
        output = maildir:/home/rob/Mail/my-patches

>
> > It might be helpful if lei could print out message-ids of messages
> > written to mailboxes.
>
> That could get very noisy, especially as mailboxes are written
> in parallel.

Verbose mode already is. Maybe specifying what info you want to be
verbose would help. The network side is mostly uninteresting in this
case for example.

Is there any tool to list new messages in a maildir? I could do that
before and after. I've done the clearing the new flag in mutt between
runs, but that's not really ideal.

Rob

Re: lei missing mails

[Eric Wong]

Rob Herring <robh@kernel.org> wrote:
> On Wed, Jun 29, 2022 at 10:30 AM Eric Wong <e@80x24.org> wrote:
> >
> > Rob Herring <robh@kernel.org> wrote:
> > > Hi,
> > >
> > > I'm using lei with lore where I have 2 queries which overlap. Really,
> > > one is a subset of the other. On those overlapping threads, I'm
> > > finding that sometimes new messages are written to one mailbox and not
> > > the other. (At least sometimes, the messages may be missing from all
> > > mailboxes sometimes too. I'm not certain.) Using --remote-fudge-time
> > > to force refetching seems to get the missing mails. I haven't found
> > > anything strange in timestamps of the missing mails, but otherwise am
> > > not sure how to debug this further. The queries are retrieving full
> > > threads and the missing mails are in the threads, but not direct
> > > matches to the queries. I realize that's not a lot of detail to go on.
> > > Suggestions on debugging this further?
> >
> > Is this with 1.8 or 1.7?
> 
> Commit 68b53c888911 actually. So post 1.8.

OK, thanks for that info.

> > I forgot to note in the release notes, but there were some
> > SQLite usage-related fixes which could avoid missing messages.
> >
> > You'll need "lei daemon-kill" after upgrading to 1.8 to ensure
> > the new code is running.
> 
> It's possible I haven't done that since updating though I do vaguely
> recall seeing something about needing to do that. Is there any way to
> tell before I restart it?

Not really, but it's pretty cheap to restart (assuming there's no
long-running jobs).

> > What might be interesting is to use the URLs lei prints and
> > comparing the results w/o lei.
> >
> > I'll have to double-check if overlapping affects things, but it
> > shouldn't; since the dedupe logic is per-output.
> >
> > Is this exclusively with HTTPS endpoints and writing to Maildirs
> > (or something else?)
> 
> Yes. It's querying lore and writing to a maildir. Here's one of the queries:
> 
> [lei]
>         q = (dfn:drivers OR dfn:arch OR dfn:Documentation/* OR
> dfn:include OR dfn:scripts) AND \
>          f:robh@kernel.org AND rt:6.month.ago..
> [lei "q"]
>         include = https://lore.kernel.org/all/
>         external = 1
>         local = 1
>         remote = 1
>         threads = 1
>         dedupe = mid
>         output = maildir:/home/rob/Mail/my-patches

Fwiw, dedupe based on mid could be vulnerable to spoofing, which
is why `content' is the default.  But yes, in the past, I've
noticed some messages to meta@public-inbox.org not showing up,
though not recently (I guess lack of activity here is a culprit :x)

I also just noticed an inotify-related bug deadlocking the whole
lei-deamon while looking into this :<

> > > It might be helpful if lei could print out message-ids of messages
> > > written to mailboxes.
> >
> > That could get very noisy, especially as mailboxes are written
> > in parallel.
> 
> Verbose mode already is. Maybe specifying what info you want to be
> verbose would help. The network side is mostly uninteresting in this
> case for example.

Yes, I've been struggling with the verbosity, too; and many
other things :<

> Is there any tool to list new messages in a maildir? I could do that
> before and after. I've done the clearing the new flag in mutt between
> runs, but that's not really ideal.

I suppose `ls'.  There are likely other tools more suited for Maildirs
but I'm not familiar with them off the top of my head.

Maybe lei could grow yet another command.

Re: lei missing mails

[Rob Herring]

On Wed, Jun 29, 2022 at 11:27 AM Eric Wong <e@80x24.org> wrote:
>
> Rob Herring <robh@kernel.org> wrote:
> > On Wed, Jun 29, 2022 at 10:30 AM Eric Wong <e@80x24.org> wrote:
> > >
> > > Rob Herring <robh@kernel.org> wrote:
> > > > Hi,
> > > >
> > > > I'm using lei with lore where I have 2 queries which overlap. Really,
> > > > one is a subset of the other. On those overlapping threads, I'm
> > > > finding that sometimes new messages are written to one mailbox and not
> > > > the other. (At least sometimes, the messages may be missing from all
> > > > mailboxes sometimes too. I'm not certain.) Using --remote-fudge-time
> > > > to force refetching seems to get the missing mails. I haven't found
> > > > anything strange in timestamps of the missing mails, but otherwise am
> > > > not sure how to debug this further. The queries are retrieving full
> > > > threads and the missing mails are in the threads, but not direct
> > > > matches to the queries. I realize that's not a lot of detail to go on.
> > > > Suggestions on debugging this further?
> > >
> > > Is this with 1.8 or 1.7?
> >
> > Commit 68b53c888911 actually. So post 1.8.
>
> OK, thanks for that info.
>
> > > I forgot to note in the release notes, but there were some
> > > SQLite usage-related fixes which could avoid missing messages.
> > >
> > > You'll need "lei daemon-kill" after upgrading to 1.8 to ensure
> > > the new code is running.
> >
> > It's possible I haven't done that since updating though I do vaguely
> > recall seeing something about needing to do that. Is there any way to
> > tell before I restart it?
>
> Not really, but it's pretty cheap to restart (assuming there's no
> long-running jobs).

I've restarted and just hit this again.


> > > What might be interesting is to use the URLs lei prints and
> > > comparing the results w/o lei.

$ lei up --all
# updating /home/rob/Mail/from-me
# updating /home/rob/Mail/missing-cc
# updating /home/rob/Mail/my-patches
# updating /home/rob/Mail/pci
# https://lore.kernel.org/all/ limiting to 2022-06-27 12:42 -0600 and newer
# https://lore.kernel.org/all/ limiting to 2022-06-27  9:50 -0600 and newer
# https://lore.kernel.org/all/ limiting to 2022-06-27 12:42 -0600 and newer
# /usr/bin/curl -Sf -s -d ''
https://lore.kernel.org/all/?x=m&t=1&q=(dt%3A20220529211430..+AND+(f%3Arobh%40kernel.org+OR+f%3Arobh%2Bdt%40kernel.org))+AND+dt%3A20220627184226..
# /home/rob/.local/share/lei/store 144/144
# /home/rob/.local/share/lei/store 3/3
# /usr/bin/curl -Sf -s -d ''
https://lore.kernel.org/all/?x=m&t=1&q=((dfn%3Adrivers+OR+dfn%3Aarch+OR+dfn%3ADocumentation%2F*+OR+dfn%3Ainclude+OR+dfn%3Ascripts)+AND+f%3Arobh%40kernel.org+AND+rt%3A1640812470..)+AND+dt%3A20220627155025..
# /usr/bin/curl -Sf -s -d ''
https://lore.kernel.org/all/?x=m&t=1&q=(l%3Alinux-pci+dfn%3Adrivers%2Fpci%2Fcontroller+dt%3A20220529211430..)+AND+dt%3A20220627184226..
# /home/rob/.local/share/lei/store 0/0
# /home/rob/.local/share/lei/store 362/362
# 0 written to /home/rob/Mail/missing-cc/ (0 matches)
# https://lore.kernel.org/all/ 72/72
# https://lore.kernel.org/all/ 4/4
# https://lore.kernel.org/all/ 131/?
# https://lore.kernel.org/all/ 184/?
# https://lore.kernel.org/all/ 412/?
# https://lore.kernel.org/all/ 603/?
# https://lore.kernel.org/all/ 853/?
# https://lore.kernel.org/all/ 1069/?
# https://lore.kernel.org/all/ 1442/?
# https://lore.kernel.org/all/ 1443/1443
# 1 written to /home/rob/Mail/pci/ (75 matches)
# 2 written to /home/rob/Mail/my-patches/ (148 matches)
# 7 written to /home/rob/Mail/from-me/ (1805 matches)


What I expected was 3 messages written to 'my-patches'.

I think the problem is just simply that the new message missing
doesn't match the query, but is a reply to a match. So with a date
after the original match in the thread won't pick up anything. The 2nd
URL above indeed only has 2 results. I guess I just have to fetch a
wider window like a month every time? What's needed is a get any new
messages in existing threads. I don't suppose there's an efficient way
to do that?

> > >
> > > I'll have to double-check if overlapping affects things, but it
> > > shouldn't; since the dedupe logic is per-output.
> > >
> > > Is this exclusively with HTTPS endpoints and writing to Maildirs
> > > (or something else?)
> >
> > Yes. It's querying lore and writing to a maildir. Here's one of the queries:
> >
> > [lei]
> >         q = (dfn:drivers OR dfn:arch OR dfn:Documentation/* OR
> > dfn:include OR dfn:scripts) AND \
> >          f:robh@kernel.org AND rt:6.month.ago..
> > [lei "q"]
> >         include = https://lore.kernel.org/all/
> >         external = 1
> >         local = 1
> >         remote = 1
> >         threads = 1
> >         dedupe = mid
> >         output = maildir:/home/rob/Mail/my-patches
>
> Fwiw, dedupe based on mid could be vulnerable to spoofing, which
> is why `content' is the default.  But yes, in the past, I've
> noticed some messages to meta@public-inbox.org not showing up,
> though not recently (I guess lack of activity here is a culprit :x)

Does 'content' ignore trailers that mailman lists like to add? I think
I switched because of that.

Rob

Re: lei missing mails

[Eric Wong]

Rob Herring <robh@kernel.org> wrote:
> On Wed, Jun 29, 2022 at 11:27 AM Eric Wong <e@80x24.org> wrote:
> > Rob Herring <robh@kernel.org> wrote:
> > > On Wed, Jun 29, 2022 at 10:30 AM Eric Wong <e@80x24.org> wrote:
> > > > Rob Herring <robh@kernel.org> wrote:
> > > > > Hi,
> > > > >
> > > > > I'm using lei with lore where I have 2 queries which overlap. Really,
> > > > > one is a subset of the other. On those overlapping threads, I'm
> > > > > finding that sometimes new messages are written to one mailbox and not
> > > > > the other. (At least sometimes, the messages may be missing from all
> > > > > mailboxes sometimes too. I'm not certain.) Using --remote-fudge-time
> > > > > to force refetching seems to get the missing mails. I haven't found
> > > > > anything strange in timestamps of the missing mails, but otherwise am
> > > > > not sure how to debug this further. The queries are retrieving full
> > > > > threads and the missing mails are in the threads, but not direct
> > > > > matches to the queries. I realize that's not a lot of detail to go on.
> > > > > Suggestions on debugging this further?
> > > >
> > > > Is this with 1.8 or 1.7?
> > >
> > > Commit 68b53c888911 actually. So post 1.8.
> >
> > OK, thanks for that info.
> >
> > > > I forgot to note in the release notes, but there were some
> > > > SQLite usage-related fixes which could avoid missing messages.
> > > >
> > > > You'll need "lei daemon-kill" after upgrading to 1.8 to ensure
> > > > the new code is running.
> > >
> > > It's possible I haven't done that since updating though I do vaguely
> > > recall seeing something about needing to do that. Is there any way to
> > > tell before I restart it?
> >
> > Not really, but it's pretty cheap to restart (assuming there's no
> > long-running jobs).
> 
> I've restarted and just hit this again.

Ugh, sorry to hear that :<

> > > > What might be interesting is to use the URLs lei prints and
> > > > comparing the results w/o lei.
> 
> $ lei up --all
> # updating /home/rob/Mail/from-me
> # updating /home/rob/Mail/missing-cc
> # updating /home/rob/Mail/my-patches
> # updating /home/rob/Mail/pci
> # https://lore.kernel.org/all/ limiting to 2022-06-27 12:42 -0600 and newer
> # https://lore.kernel.org/all/ limiting to 2022-06-27  9:50 -0600 and newer
> # https://lore.kernel.org/all/ limiting to 2022-06-27 12:42 -0600 and newer
> # /usr/bin/curl -Sf -s -d ''
> https://lore.kernel.org/all/?x=m&t=1&q=(dt%3A20220529211430..+AND+(f%3Arobh%40kernel.org+OR+f%3Arobh%2Bdt%40kernel.org))+AND+dt%3A20220627184226..
> # /home/rob/.local/share/lei/store 144/144
> # /home/rob/.local/share/lei/store 3/3
> # /usr/bin/curl -Sf -s -d ''
> https://lore.kernel.org/all/?x=m&t=1&q=((dfn%3Adrivers+OR+dfn%3Aarch+OR+dfn%3ADocumentation%2F*+OR+dfn%3Ainclude+OR+dfn%3Ascripts)+AND+f%3Arobh%40kernel.org+AND+rt%3A1640812470..)+AND+dt%3A20220627155025..
> # /usr/bin/curl -Sf -s -d ''
> https://lore.kernel.org/all/?x=m&t=1&q=(l%3Alinux-pci+dfn%3Adrivers%2Fpci%2Fcontroller+dt%3A20220529211430..)+AND+dt%3A20220627184226..
> # /home/rob/.local/share/lei/store 0/0
> # /home/rob/.local/share/lei/store 362/362
> # 0 written to /home/rob/Mail/missing-cc/ (0 matches)
> # https://lore.kernel.org/all/ 72/72
> # https://lore.kernel.org/all/ 4/4
> # https://lore.kernel.org/all/ 131/?
> # https://lore.kernel.org/all/ 184/?
> # https://lore.kernel.org/all/ 412/?
> # https://lore.kernel.org/all/ 603/?
> # https://lore.kernel.org/all/ 853/?
> # https://lore.kernel.org/all/ 1069/?
> # https://lore.kernel.org/all/ 1442/?
> # https://lore.kernel.org/all/ 1443/1443
> # 1 written to /home/rob/Mail/pci/ (75 matches)
> # 2 written to /home/rob/Mail/my-patches/ (148 matches)
> # 7 written to /home/rob/Mail/from-me/ (1805 matches)
> 
> 
> What I expected was 3 messages written to 'my-patches'.
> 
> I think the problem is just simply that the new message missing
> doesn't match the query, but is a reply to a match. So with a date
> after the original match in the thread won't pick up anything. The 2nd
> URL above indeed only has 2 results. I guess I just have to fetch a
> wider window like a month every time? What's needed is a get any new
> messages in existing threads. I don't suppose there's an efficient way
> to do that?

No, I don't think so.  I think this is a separate issue in lei...
"t=1" in the remote query expands threads in a time-agnostic
way, so I don't think that's the problem (though I may be wrong...).

I'll have to check more closely this week (still stuck with POP3
user account/storage issues :<)

> > > >
> > > > I'll have to double-check if overlapping affects things, but it
> > > > shouldn't; since the dedupe logic is per-output.
> > > >
> > > > Is this exclusively with HTTPS endpoints and writing to Maildirs
> > > > (or something else?)
> > >
> > > Yes. It's querying lore and writing to a maildir. Here's one of the queries:
> > >
> > > [lei]
> > >         q = (dfn:drivers OR dfn:arch OR dfn:Documentation/* OR
> > > dfn:include OR dfn:scripts) AND \
> > >          f:robh@kernel.org AND rt:6.month.ago..
> > > [lei "q"]
> > >         include = https://lore.kernel.org/all/
> > >         external = 1
> > >         local = 1
> > >         remote = 1
> > >         threads = 1
> > >         dedupe = mid
> > >         output = maildir:/home/rob/Mail/my-patches
> >
> > Fwiw, dedupe based on mid could be vulnerable to spoofing, which
> > is why `content' is the default.  But yes, in the past, I've
> > noticed some messages to meta@public-inbox.org not showing up,
> > though not recently (I guess lack of activity here is a culprit :x)
> 
> Does 'content' ignore trailers that mailman lists like to add? I think
> I switched because of that.

No, unfortunately not.  Hopefully the admins can be convinced to
get rid of trailers (I'm happy vger did so a few years back).
But I'd rather deal with duplicates than miss messages (there
have been legitimate messages in the past which reused msgids,
unfortunately).

Re: lei missing mails

[Eric Wong]

Eric Wong <e@80x24.org> wrote:
> Rob Herring <robh@kernel.org> wrote:
> > I've restarted and just hit this again.
> 
> Ugh, sorry to hear that :<

Btw, is this only with HTTP(S) endpoints?  (not local mirrors?)

I wonder if it's something wrong on the HTTP end (server or
client), but did more digging the other day and couldn't
reproduce the problem...

I just posted a very minor diagnostic change which might help
(forgot to Cc you :x)
https://public-inbox.org/meta/20220707094030.1185793-1-e@80x24.org/

As usual, daemon-kill is necessary after upgrading.

Re: lei missing mails

[Rob Herring]

On Thu, Jul 7, 2022 at 3:48 AM Eric Wong <e@80x24.org> wrote:
>
> Eric Wong <e@80x24.org> wrote:
> > Rob Herring <robh@kernel.org> wrote:
> > > I've restarted and just hit this again.
> >
> > Ugh, sorry to hear that :<
>
> Btw, is this only with HTTP(S) endpoints?  (not local mirrors?)

Yes. Only in that https is the only case I've tried. Don't know about
local mirrors because I don't know how to set that up...

Rob

Re: lei missing mails

[Rob Herring]

On Thu, Jun 30, 2022 at 2:55 AM Eric Wong <e@80x24.org> wrote:
>
> Rob Herring <robh@kernel.org> wrote:
> > On Wed, Jun 29, 2022 at 11:27 AM Eric Wong <e@80x24.org> wrote:
> > > Rob Herring <robh@kernel.org> wrote:
> > > > On Wed, Jun 29, 2022 at 10:30 AM Eric Wong <e@80x24.org> wrote:
> > > > > Rob Herring <robh@kernel.org> wrote:
> > > > > > Hi,
> > > > > >
> > > > > > I'm using lei with lore where I have 2 queries which overlap. Really,
> > > > > > one is a subset of the other. On those overlapping threads, I'm
> > > > > > finding that sometimes new messages are written to one mailbox and not
> > > > > > the other. (At least sometimes, the messages may be missing from all
> > > > > > mailboxes sometimes too. I'm not certain.) Using --remote-fudge-time
> > > > > > to force refetching seems to get the missing mails. I haven't found
> > > > > > anything strange in timestamps of the missing mails, but otherwise am
> > > > > > not sure how to debug this further. The queries are retrieving full
> > > > > > threads and the missing mails are in the threads, but not direct
> > > > > > matches to the queries. I realize that's not a lot of detail to go on.
> > > > > > Suggestions on debugging this further?
> > > > >
> > > > > Is this with 1.8 or 1.7?
> > > >
> > > > Commit 68b53c888911 actually. So post 1.8.
> > >
> > > OK, thanks for that info.
> > >
> > > > > I forgot to note in the release notes, but there were some
> > > > > SQLite usage-related fixes which could avoid missing messages.
> > > > >
> > > > > You'll need "lei daemon-kill" after upgrading to 1.8 to ensure
> > > > > the new code is running.
> > > >
> > > > It's possible I haven't done that since updating though I do vaguely
> > > > recall seeing something about needing to do that. Is there any way to
> > > > tell before I restart it?
> > >
> > > Not really, but it's pretty cheap to restart (assuming there's no
> > > long-running jobs).
> >
> > I've restarted and just hit this again.
>
> Ugh, sorry to hear that :<
>
> > > > > What might be interesting is to use the URLs lei prints and
> > > > > comparing the results w/o lei.
> >
> > $ lei up --all
> > # updating /home/rob/Mail/from-me
> > # updating /home/rob/Mail/missing-cc
> > # updating /home/rob/Mail/my-patches
> > # updating /home/rob/Mail/pci
> > # https://lore.kernel.org/all/ limiting to 2022-06-27 12:42 -0600 and newer
> > # https://lore.kernel.org/all/ limiting to 2022-06-27  9:50 -0600 and newer
> > # https://lore.kernel.org/all/ limiting to 2022-06-27 12:42 -0600 and newer
> > # /usr/bin/curl -Sf -s -d ''
> > https://lore.kernel.org/all/?x=m&t=1&q=(dt%3A20220529211430..+AND+(f%3Arobh%40kernel.org+OR+f%3Arobh%2Bdt%40kernel.org))+AND+dt%3A20220627184226..
> > # /home/rob/.local/share/lei/store 144/144
> > # /home/rob/.local/share/lei/store 3/3
> > # /usr/bin/curl -Sf -s -d ''
> > https://lore.kernel.org/all/?x=m&t=1&q=((dfn%3Adrivers+OR+dfn%3Aarch+OR+dfn%3ADocumentation%2F*+OR+dfn%3Ainclude+OR+dfn%3Ascripts)+AND+f%3Arobh%40kernel.org+AND+rt%3A1640812470..)+AND+dt%3A20220627155025..
> > # /usr/bin/curl -Sf -s -d ''
> > https://lore.kernel.org/all/?x=m&t=1&q=(l%3Alinux-pci+dfn%3Adrivers%2Fpci%2Fcontroller+dt%3A20220529211430..)+AND+dt%3A20220627184226..
> > # /home/rob/.local/share/lei/store 0/0
> > # /home/rob/.local/share/lei/store 362/362
> > # 0 written to /home/rob/Mail/missing-cc/ (0 matches)
> > # https://lore.kernel.org/all/ 72/72
> > # https://lore.kernel.org/all/ 4/4
> > # https://lore.kernel.org/all/ 131/?
> > # https://lore.kernel.org/all/ 184/?
> > # https://lore.kernel.org/all/ 412/?
> > # https://lore.kernel.org/all/ 603/?
> > # https://lore.kernel.org/all/ 853/?
> > # https://lore.kernel.org/all/ 1069/?
> > # https://lore.kernel.org/all/ 1442/?
> > # https://lore.kernel.org/all/ 1443/1443
> > # 1 written to /home/rob/Mail/pci/ (75 matches)
> > # 2 written to /home/rob/Mail/my-patches/ (148 matches)
> > # 7 written to /home/rob/Mail/from-me/ (1805 matches)
> >
> >
> > What I expected was 3 messages written to 'my-patches'.
> >
> > I think the problem is just simply that the new message missing
> > doesn't match the query, but is a reply to a match. So with a date
> > after the original match in the thread won't pick up anything. The 2nd
> > URL above indeed only has 2 results. I guess I just have to fetch a
> > wider window like a month every time? What's needed is a get any new
> > messages in existing threads. I don't suppose there's an efficient way
> > to do that?
>
> No, I don't think so.  I think this is a separate issue in lei...
> "t=1" in the remote query expands threads in a time-agnostic
> way, so I don't think that's the problem (though I may be wrong...).

Based on what the web interface presents, it sure seems like 't=1' is
independent of the query. The results listed are only those that match
the query and date range on the match.

For example, this query returns 3 matches:

https://lore.kernel.org/all/?x=m&t=1&q=((dfn%3Adrivers+OR+dfn%3Aarch+OR+dfn%3ADocumentation%2F*+OR+dfn%3Ainclude+OR+dfn%3Ascripts)+AND+f%3Arobh%40kernel.org+AND+rt%3A1641934905..)+AND+dt%3A20220630203819..

If I change 'dt' to 1 day earlier, I get 1 more match:

https://lore.kernel.org/all/?x=m&t=1&q=((dfn%3Adrivers+OR+dfn%3Aarch+OR+dfn%3ADocumentation%2F*+OR+dfn%3Ainclude+OR+dfn%3Ascripts)+AND+f%3Arobh%40kernel.org+AND+rt%3A1641934905..)+AND+dt%3A20220629203819..

That 4th match has a reply after 6/30, but the 1st query will not get
the reply. This is all reproducible without lei involved at all.

What seems to be needed is a 'thread date' which is the latest time
for any message in a thread that matches. Or perhaps some way to
separate the query from what's transferred. IOW, query for X, but only
send results newer than some date.

Rob

Re: lei missing mails

[Eric Wong]

Rob Herring <robh@kernel.org> wrote:
> On Thu, Jun 30, 2022 at 2:55 AM Eric Wong <e@80x24.org> wrote:
> >
> > Rob Herring <robh@kernel.org> wrote:
> > > On Wed, Jun 29, 2022 at 11:27 AM Eric Wong <e@80x24.org> wrote:
> > > > Rob Herring <robh@kernel.org> wrote:
> > > > > On Wed, Jun 29, 2022 at 10:30 AM Eric Wong <e@80x24.org> wrote:
> > > > > > Rob Herring <robh@kernel.org> wrote:
> > > > > > > Hi,
> > > > > > >
> > > > > > > I'm using lei with lore where I have 2 queries which overlap. Really,
> > > > > > > one is a subset of the other. On those overlapping threads, I'm
> > > > > > > finding that sometimes new messages are written to one mailbox and not
> > > > > > > the other. (At least sometimes, the messages may be missing from all
> > > > > > > mailboxes sometimes too. I'm not certain.) Using --remote-fudge-time
> > > > > > > to force refetching seems to get the missing mails. I haven't found
> > > > > > > anything strange in timestamps of the missing mails, but otherwise am
> > > > > > > not sure how to debug this further. The queries are retrieving full
> > > > > > > threads and the missing mails are in the threads, but not direct
> > > > > > > matches to the queries. I realize that's not a lot of detail to go on.
> > > > > > > Suggestions on debugging this further?
> > > > > >
> > > > > > Is this with 1.8 or 1.7?
> > > > >
> > > > > Commit 68b53c888911 actually. So post 1.8.
> > > >
> > > > OK, thanks for that info.
> > > >
> > > > > > I forgot to note in the release notes, but there were some
> > > > > > SQLite usage-related fixes which could avoid missing messages.
> > > > > >
> > > > > > You'll need "lei daemon-kill" after upgrading to 1.8 to ensure
> > > > > > the new code is running.
> > > > >
> > > > > It's possible I haven't done that since updating though I do vaguely
> > > > > recall seeing something about needing to do that. Is there any way to
> > > > > tell before I restart it?
> > > >
> > > > Not really, but it's pretty cheap to restart (assuming there's no
> > > > long-running jobs).
> > >
> > > I've restarted and just hit this again.
> >
> > Ugh, sorry to hear that :<
> >
> > > > > > What might be interesting is to use the URLs lei prints and
> > > > > > comparing the results w/o lei.
> > >
> > > $ lei up --all
> > > # updating /home/rob/Mail/from-me
> > > # updating /home/rob/Mail/missing-cc
> > > # updating /home/rob/Mail/my-patches
> > > # updating /home/rob/Mail/pci
> > > # https://lore.kernel.org/all/ limiting to 2022-06-27 12:42 -0600 and newer
> > > # https://lore.kernel.org/all/ limiting to 2022-06-27  9:50 -0600 and newer
> > > # https://lore.kernel.org/all/ limiting to 2022-06-27 12:42 -0600 and newer
> > > # /usr/bin/curl -Sf -s -d ''
> > > https://lore.kernel.org/all/?x=m&t=1&q=(dt%3A20220529211430..+AND+(f%3Arobh%40kernel.org+OR+f%3Arobh%2Bdt%40kernel.org))+AND+dt%3A20220627184226..
> > > # /home/rob/.local/share/lei/store 144/144
> > > # /home/rob/.local/share/lei/store 3/3
> > > # /usr/bin/curl -Sf -s -d ''
> > > https://lore.kernel.org/all/?x=m&t=1&q=((dfn%3Adrivers+OR+dfn%3Aarch+OR+dfn%3ADocumentation%2F*+OR+dfn%3Ainclude+OR+dfn%3Ascripts)+AND+f%3Arobh%40kernel.org+AND+rt%3A1640812470..)+AND+dt%3A20220627155025..
> > > # /usr/bin/curl -Sf -s -d ''
> > > https://lore.kernel.org/all/?x=m&t=1&q=(l%3Alinux-pci+dfn%3Adrivers%2Fpci%2Fcontroller+dt%3A20220529211430..)+AND+dt%3A20220627184226..
> > > # /home/rob/.local/share/lei/store 0/0
> > > # /home/rob/.local/share/lei/store 362/362
> > > # 0 written to /home/rob/Mail/missing-cc/ (0 matches)
> > > # https://lore.kernel.org/all/ 72/72
> > > # https://lore.kernel.org/all/ 4/4
> > > # https://lore.kernel.org/all/ 131/?
> > > # https://lore.kernel.org/all/ 184/?
> > > # https://lore.kernel.org/all/ 412/?
> > > # https://lore.kernel.org/all/ 603/?
> > > # https://lore.kernel.org/all/ 853/?
> > > # https://lore.kernel.org/all/ 1069/?
> > > # https://lore.kernel.org/all/ 1442/?
> > > # https://lore.kernel.org/all/ 1443/1443
> > > # 1 written to /home/rob/Mail/pci/ (75 matches)
> > > # 2 written to /home/rob/Mail/my-patches/ (148 matches)
> > > # 7 written to /home/rob/Mail/from-me/ (1805 matches)
> > >
> > >
> > > What I expected was 3 messages written to 'my-patches'.
> > >
> > > I think the problem is just simply that the new message missing
> > > doesn't match the query, but is a reply to a match. So with a date
> > > after the original match in the thread won't pick up anything. The 2nd
> > > URL above indeed only has 2 results. I guess I just have to fetch a
> > > wider window like a month every time? What's needed is a get any new
> > > messages in existing threads. I don't suppose there's an efficient way
> > > to do that?
> >
> > No, I don't think so.  I think this is a separate issue in lei...
> > "t=1" in the remote query expands threads in a time-agnostic
> > way, so I don't think that's the problem (though I may be wrong...).
> 
> Based on what the web interface presents, it sure seems like 't=1' is
> independent of the query. The results listed are only those that match
> the query and date range on the match.

Actually, for the HTML results, t=1 is ignored right now...
it's only for mboxrd downloads (via POST) at the moment...
That should be clarified/changed.

> For example, this query returns 3 matches:
> 
> https://lore.kernel.org/all/?x=m&t=1&q=((dfn%3Adrivers+OR+dfn%3Aarch+OR+dfn%3ADocumentation%2F*+OR+dfn%3Ainclude+OR+dfn%3Ascripts)+AND+f%3Arobh%40kernel.org+AND+rt%3A1641934905..)+AND+dt%3A20220630203819..
> 
> If I change 'dt' to 1 day earlier, I get 1 more match:
> 
> https://lore.kernel.org/all/?x=m&t=1&q=((dfn%3Adrivers+OR+dfn%3Aarch+OR+dfn%3ADocumentation%2F*+OR+dfn%3Ainclude+OR+dfn%3Ascripts)+AND+f%3Arobh%40kernel.org+AND+rt%3A1641934905..)+AND+dt%3A20220629203819..
> 
> That 4th match has a reply after 6/30, but the 1st query will not get
> the reply. This is all reproducible without lei involved at all.

Right.  t=1 only expands threads if they're linked via
References/In-Reply-To or (loosely) via matching Subjects.

> What seems to be needed is a 'thread date' which is the latest time
> for any message in a thread that matches. Or perhaps some way to
> separate the query from what's transferred. IOW, query for X, but only
> send results newer than some date.

Yeah, the thread ID info is stored independently of the search
index documents, though; so searching (Xapian) vs filtering
(SQLite) gets a bit tricky

I'll look into JMAP further before making more changes to the
data models to accomodate this.

Apologies for the delays; been stressed over other things :<

[PATCH] www: note "x=m" and "t=1" (mis)use for GET requests

[Eric Wong]

Eric Wong <e@80x24.org> wrote:
> Rob Herring <robh@kernel.org> wrote:
> > Based on what the web interface presents, it sure seems like 't=1' is
> > independent of the query. The results listed are only those that match
> > the query and date range on the match.
> 
> Actually, for the HTML results, t=1 is ignored right now...
> it's only for mboxrd downloads (via POST) at the moment...
> That should be clarified/changed.
> 
> > For example, this query returns 3 matches:
> > 
> > https://lore.kernel.org/all/?x=m&t=1&q=((dfn%3Adrivers+OR+dfn%3Aarch+OR+dfn%3ADocumentation%2F*+OR+dfn%3Ainclude+OR+dfn%3Ascripts)+AND+f%3Arobh%40kernel.org+AND+rt%3A1641934905..)+AND+dt%3A20220630203819..
> > 
> > If I change 'dt' to 1 day earlier, I get 1 more match:
> > 
> > https://lore.kernel.org/all/?x=m&t=1&q=((dfn%3Adrivers+OR+dfn%3Aarch+OR+dfn%3ADocumentation%2F*+OR+dfn%3Ainclude+OR+dfn%3Ascripts)+AND+f%3Arobh%40kernel.org+AND+rt%3A1641934905..)+AND+dt%3A20220629203819..
> > 
> > That 4th match has a reply after 6/30, but the 1st query will not get
> > the reply. This is all reproducible without lei involved at all.
> 
> Right.  t=1 only expands threads if they're linked via
> References/In-Reply-To or (loosely) via matching Subjects.

--------8<--------
Subject: [PATCH] www: note "x=m" and "t=1" (mis)use for GET requests

We require "x=m" (requests for mboxes) to be POST requests to
avoid unnecessary traffic from crawlers.  "t=1" only collapses
threads in the summary view, which isn't normally accessible
from <form> elements.

This also fixes the missing "[summary|nested]" element when
"x=m" is used.
---
 lib/PublicInbox/SearchView.pm | 22 ++++++++++++++--------
 1 file changed, 14 insertions(+), 8 deletions(-)

diff --git a/lib/PublicInbox/SearchView.pm b/lib/PublicInbox/SearchView.pm
index b1cdb480..b025ec96 100644
--- a/lib/PublicInbox/SearchView.pm
+++ b/lib/PublicInbox/SearchView.pm
@@ -1,4 +1,4 @@
-# Copyright (C) 2015-2021 all contributors <meta@public-inbox.org>
+# Copyright (C) all contributors <meta@public-inbox.org>
 # License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
 #
 # Displays search results for the web interface
@@ -193,18 +193,24 @@ sub search_nav_top {
 
 	my $x = $q->{x};
 	my $pfx = "\t\t\t";
-	if ($x eq '') {
-		my $t = $q->qs_html(x => 't');
-		$rv .= qq{<b>summary</b>|<a\nhref="?$t">nested</a>}
-	} elsif ($x eq 't') {
+	if ($x eq 't') {
 		my $s = $q->qs_html(x => '');
 		$rv .= qq{<a\nhref="?$s">summary</a>|<b>nested</b>};
 		$pfx = "thread overview <a\nhref=#t>below</a> | ";
+	} else {
+		my $t = $q->qs_html(x => 't');
+		$rv .= qq{<b>summary</b>|<a\nhref="?$t">nested</a>}
 	}
 	my $A = $q->qs_html(x => 'A', r => undef);
-	$rv .= qq{|<a\nhref="?$A">Atom feed</a>]};
+	$rv .= qq{|<a\nhref="?$A">Atom feed</a>]\n};
+	$rv .= <<EOM if $x ne 't' && $q->{t};
+*** "t=1" collapses threads in summary, "full threads" requires mbox.gz ***
+EOM
+	$rv .= <<EOM if $x eq 'm';
+*** "x=m" ignored for GET requests, use download buttons below ***
+EOM
 	if ($ctx->{ibx}->isrch->has_threadid) {
-		$rv .= qq{\n${pfx}download mbox.gz: } .
+		$rv .= qq{${pfx}download mbox.gz: } .
 			# we set name=z w/o using it since it seems required for
 			# lynx (but works fine for w3m).
 			qq{<input\ntype=submit\nname=z\n} .
@@ -212,7 +218,7 @@ sub search_nav_top {
 			qq{|<input\ntype=submit\nname=x\n} .
 				q{value="full threads"/>};
 	} else { # BOFH needs to --reindex
-		$rv .= qq{\n${pfx}download: } .
+		$rv .= qq{${pfx}download: } .
 			qq{<input\ntype=submit\nname=z\nvalue="mbox.gz"/>}
 	}
 	$rv .= qq{</pre></form><pre>};