From: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
To: Eric Wong <e@80x24.org>
Cc: meta@public-inbox.org
Subject: Re: [PATCH] lei_to_mail+mbox_reader: fix handling of empty/bogus emails
Date: Tue, 7 Sep 2021 14:17:26 -0400 [thread overview]
Message-ID: <20210907181726.ghb6xn4qcsdchquw@meerkat.local> (raw)
In-Reply-To: <20210904213658.GA27941@dcvr>
On Sat, Sep 04, 2021 at 09:36:58PM +0000, Eric Wong wrote:
> Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> > Yep, that seems to work fine. Question -- I noticed that lei just issues a
> > regular query, retrieves results with curl and then parses the output. Is
> > there a danger of potentially running into issues with parsing the regular
> > HTML output if it changes in the future?
>
> It's actually parsing gzipped mboxrd (&x=m). But you're right
> we could use stronger safeguards in case we see gzipped HTML or
> something else...
Ooh, okay, I guess I should actually look at the output of the curl call. :)
The questions I have, then:
1. this means that each "lei up" call will be increasingly larger and larger,
since when we init the search with rt:, it gets resolved into a datestamp
(e.g. rt:2.weeks.ago becomes rt:1625699031). I'm worried that this will be
increasingly hard on the server side, especially if someone
fires-and-forgets a cronjob that ends up downloading ever-growing mboxes
every 5 minutes.
2. is there some sanity limit on the server side that would prevent someone's
overly broad search query from gzipping and downloading gigabytes of mail?
-K
next prev parent reply other threads:[~2021-09-07 18:17 UTC|newest]
Thread overview: 14+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-09-02 21:12 Showcasing lei at Linux Plumbers Konstantin Ryabitsev
2021-09-02 21:58 ` Eric Wong
2021-09-03 15:15 ` Konstantin Ryabitsev
2021-09-04 21:36 ` [PATCH] lei_to_mail+mbox_reader: fix handling of empty/bogus emails Eric Wong
2021-09-07 18:17 ` Konstantin Ryabitsev [this message]
2021-09-07 20:56 ` Eric Wong
2021-09-07 21:20 ` Konstantin Ryabitsev
2021-09-07 22:22 ` Eric Wong
2021-09-07 21:33 ` Showcasing lei at Linux Plumbers Konstantin Ryabitsev
2021-09-07 22:14 ` Eric Wong
2021-09-08 13:36 ` Konstantin Ryabitsev
2021-09-08 14:49 ` Eric Wong
2021-09-08 17:17 ` Konstantin Ryabitsev
2021-09-08 17:32 ` Eric Wong
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://public-inbox.org/README
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20210907181726.ghb6xn4qcsdchquw@meerkat.local \
--to=konstantin@linuxfoundation.org \
--cc=e@80x24.org \
--cc=meta@public-inbox.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).