From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: X-Spam-Status: No, score=-4.0 required=3.0 tests=ALL_TRUSTED,BAYES_00 shortcircuit=no autolearn=ham autolearn_force=no version=3.4.2 Received: from localhost (dcvr.yhbt.net [127.0.0.1]) by dcvr.yhbt.net (Postfix) with ESMTP id ED9301F4C0; Tue, 29 Oct 2019 22:46:50 +0000 (UTC) Date: Tue, 29 Oct 2019 22:46:50 +0000 From: Eric Wong To: "Eric W. Biederman" , meta@public-inbox.org Subject: Re: RFC: monthly epochs for v2 Message-ID: <20191029224650.GA26085@dcvr> References: <20191024195304.5b7zlx7e3vxfxmtg@chatter.i7.local> <20191024203503.GA31522@dcvr> <20191024212108.zfbwh7bmfbo3cgu5@chatter.i7.local> <20191024223451.GA17949@dcvr> <20191025122214.GA6947@dcvr> <20191025205604.GA23680@chatter.i7.local> <20191025225755.GA8414@dcvr> <87mudjlhkg.fsf@x220.int.ebiederm.org> <20191029155520.w5nnxtt4b6v6dgun@pure.paranoia.local> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <20191029155520.w5nnxtt4b6v6dgun@pure.paranoia.local> List-Id: Konstantin Ryabitsev wrote: > On Tue, Oct 29, 2019 at 10:03:43AM -0500, Eric W. Biederman wrote: > > So not monthly epochs. But it would be very handing to have a > > public-inbox command command that refreshes git mirrors. It would > > be even more awesome if there was something like the IMAP IDLE command > > in http that would let a process block until an update happened and > > then fetch the updated data. > > > > ssoma had some of that. Did it? It still runs via cronjob for old lists I have to drive mlmmj. > > I have a very rough script that works, but I periodically need to find > > the new epochs start a new mirror by hand. So it would be nice if we > > could get a more polished tool that I could just tell it mirror this > > mailing list. Or mirror all of the mailling lists on lore. > > You can use manifest files to recognize when a new epoch is available > (and if there are new updates). E.g.: > > http://lore.kernel.org/lkml/manifest.js.gz > > It's written to be consumed by grokmirror, but it can be adapted to be > used by any other tool. The format is straightforward and simple to > understand. To reduce polling, I'm thinking a non-standard "IDLE" HTTP method could be used and public-inbox-httpd could implement that inotify/kevent to wake up clients on changes. The following would wait until the local ./manifest.js.gz is older than the server: curl -z ./manifest.js.gz -X IDLE http://example.com/manifest.js.gz But getting a non-standard request method "IDLE" through proxies could be tough, so perhaps using POST and a query parameter would work, too: curl -z ./manifest.js.gz -X POST http://example.com/manifest.js.gz?idle=1 But then again, HTTP proxies probably timeout after a minute or so. I seem to recall nginx times out after 75s. Maybe empty gzip fragments can be sent to keep the connection alive?