From: Eric Wong <e@80x24.org>
To: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
Cc: meta@public-inbox.org
Subject: Re: [PATCH] lei_to_mail+mbox_reader: fix handling of empty/bogus emails
Date: Tue, 7 Sep 2021 20:56:17 +0000 [thread overview]
Message-ID: <20210907205617.GA3923@dcvr> (raw)
In-Reply-To: <20210907181726.ghb6xn4qcsdchquw@meerkat.local>
Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> On Sat, Sep 04, 2021 at 09:36:58PM +0000, Eric Wong wrote:
> > Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> > > Yep, that seems to work fine. Question -- I noticed that lei just issues a
> > > regular query, retrieves results with curl and then parses the output. Is
> > > there a danger of potentially running into issues with parsing the regular
> > > HTML output if it changes in the future?
> >
> > It's actually parsing gzipped mboxrd (&x=m). But you're right
> > we could use stronger safeguards in case we see gzipped HTML or
> > something else...
>
> Ooh, okay, I guess I should actually look at the output of the curl call. :)
> The questions I have, then:
>
> 1. this means that each "lei up" call will be increasingly larger and larger,
> since when we init the search with rt:, it gets resolved into a datestamp
> (e.g. rt:2.weeks.ago becomes rt:1625699031). I'm worried that this will be
> increasingly hard on the server side, especially if someone
> fires-and-forgets a cronjob that ends up downloading ever-growing mboxes
> every 5 minutes.
"rt:2.weeks.ago" stays "rt:2.weeks.ago" in saved searches :>
It was one of my primary annoyances when I initially implemented
this and commit 2e4e4b0d6f30d9d4612066395ba694c7c7d61e6e solved it.
https://public-inbox.org/meta/20210416231035.31807-2-e@80x24.org/
("lei q: --save preserves relative time queries")
> 2. is there some sanity limit on the server side that would prevent someone's
> overly broad search query from gzipping and downloading gigabytes of mail?
Not right now. With public-inbox-httpd, the actual git fetches
are handled fairly w.r.t to other requests (and I could
deprioritize them further, if needed...). The Xapian query OTOH...
next prev parent reply other threads:[~2021-09-07 20:56 UTC|newest]
Thread overview: 14+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-09-02 21:12 Showcasing lei at Linux Plumbers Konstantin Ryabitsev
2021-09-02 21:58 ` Eric Wong
2021-09-03 15:15 ` Konstantin Ryabitsev
2021-09-04 21:36 ` [PATCH] lei_to_mail+mbox_reader: fix handling of empty/bogus emails Eric Wong
2021-09-07 18:17 ` Konstantin Ryabitsev
2021-09-07 20:56 ` Eric Wong [this message]
2021-09-07 21:20 ` Konstantin Ryabitsev
2021-09-07 22:22 ` Eric Wong
2021-09-07 21:33 ` Showcasing lei at Linux Plumbers Konstantin Ryabitsev
2021-09-07 22:14 ` Eric Wong
2021-09-08 13:36 ` Konstantin Ryabitsev
2021-09-08 14:49 ` Eric Wong
2021-09-08 17:17 ` Konstantin Ryabitsev
2021-09-08 17:32 ` Eric Wong
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://public-inbox.org/README
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20210907205617.GA3923@dcvr \
--to=e@80x24.org \
--cc=konstantin@linuxfoundation.org \
--cc=meta@public-inbox.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).