Re: RFC: monthly epochs for v2

[Eric Wong <e@80x24.org>]

Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> On Tue, Oct 29, 2019 at 10:03:43AM -0500, Eric W. Biederman wrote:
> > So not monthly epochs.  But it would be very handing to have a
> > public-inbox command command that refreshes git mirrors.  It would
> > be even more awesome if there was something like the IMAP IDLE command
> > in http that would let a process block until an update happened and
> > then fetch the updated data.
> > 
> > ssoma had some of that.

Did it?  It still runs via cronjob for old lists I have to drive mlmmj.

> > I have a very rough script that works, but I periodically need to find
> > the new epochs start a new mirror by hand.  So it would be nice if we
> > could get a more polished tool that I could just tell it mirror this
> > mailing list.  Or mirror all of the mailling lists on lore.
> 
> You can use manifest files to recognize when a new epoch is available
> (and if there are new updates). E.g.:
> 
> http://lore.kernel.org/lkml/manifest.js.gz
> 
> It's written to be consumed by grokmirror, but it can be adapted to be
> used by any other tool. The format is straightforward and simple to
> understand.

To reduce polling, I'm thinking a non-standard "IDLE" HTTP
method could be used and public-inbox-httpd could implement that
inotify/kevent to wake up clients on changes.

The following would wait until the local ./manifest.js.gz is
older than the server:

  curl -z ./manifest.js.gz -X IDLE http://example.com/manifest.js.gz

But getting a non-standard request method "IDLE" through proxies
could be tough, so perhaps using POST and a query parameter
would work, too:

  curl -z ./manifest.js.gz -X POST http://example.com/manifest.js.gz?idle=1

But then again, HTTP proxies probably timeout after a minute or so.
I seem to recall nginx times out after 75s.  Maybe empty gzip fragments
can be sent to keep the connection alive?