From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <e@80x24.org>
X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net
X-Spam-Level: 
X-Spam-ASN:  
X-Spam-Status: No, score=-4.0 required=3.0 tests=ALL_TRUSTED,BAYES_00
	shortcircuit=no autolearn=ham autolearn_force=no version=3.4.2
Received: from localhost (dcvr.yhbt.net [127.0.0.1])
	by dcvr.yhbt.net (Postfix) with ESMTP id 2341A1F8C6;
	Tue,  7 Sep 2021 20:56:17 +0000 (UTC)
Date: Tue, 7 Sep 2021 20:56:17 +0000
From: Eric Wong <e@80x24.org>
To: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
Cc: meta@public-inbox.org
Subject: Re: [PATCH] lei_to_mail+mbox_reader: fix handling of empty/bogus
 emails
Message-ID: <20210907205617.GA3923@dcvr>
References: <20210902211225.pmnykwcwcxeaunt5@meerkat.local>
 <20210902215850.GA5063@dcvr>
 <20210903151500.h72mzcpqixgtytjs@meerkat.local>
 <20210904213658.GA27941@dcvr>
 <20210907181726.ghb6xn4qcsdchquw@meerkat.local>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
In-Reply-To: <20210907181726.ghb6xn4qcsdchquw@meerkat.local>
List-Id: <meta.public-inbox.org>

Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> On Sat, Sep 04, 2021 at 09:36:58PM +0000, Eric Wong wrote:
> > Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> > > Yep, that seems to work fine. Question -- I noticed that lei just issues a
> > > regular query, retrieves results with curl and then parses the output. Is
> > > there a danger of potentially running into issues with parsing the regular
> > > HTML output if it changes in the future?
> > 
> > It's actually parsing gzipped mboxrd (&x=m).  But you're right
> > we could use stronger safeguards in case we see gzipped HTML or
> > something else...
> 
> Ooh, okay, I guess I should actually look at the output of the curl call. :)
> The questions I have, then:
> 
> 1. this means that each "lei up" call will be increasingly larger and larger,
>    since when we init the search with rt:, it gets resolved into a datestamp
>    (e.g. rt:2.weeks.ago becomes rt:1625699031). I'm worried that this will be
>    increasingly hard on the server side, especially if someone
>    fires-and-forgets a cronjob that ends up downloading ever-growing mboxes
>    every 5 minutes.

"rt:2.weeks.ago" stays "rt:2.weeks.ago" in saved searches :>

It was one of my primary annoyances when I initially implemented
this and commit 2e4e4b0d6f30d9d4612066395ba694c7c7d61e6e solved it.
https://public-inbox.org/meta/20210416231035.31807-2-e@80x24.org/
("lei q: --save preserves relative time queries")

> 2. is there some sanity limit on the server side that would prevent someone's
>    overly broad search query from gzipping and downloading gigabytes of mail?

Not right now.  With public-inbox-httpd, the actual git fetches
are handled fairly w.r.t to other requests (and I could
deprioritize them further, if needed...).  The Xapian query OTOH...