From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net X-Spam-Level: X-Spam-Status: No, score=-4.0 required=3.0 tests=ALL_TRUSTED,BAYES_00 shortcircuit=no autolearn=ham autolearn_force=no version=3.4.2 Received: from localhost (dcvr.yhbt.net [127.0.0.1]) by dcvr.yhbt.net (Postfix) with ESMTP id C0DBB1F55B; Sat, 16 May 2020 19:12:20 +0000 (UTC) Date: Sat, 16 May 2020 19:12:20 +0000 From: Eric Wong To: "Eric W. Biederman" Cc: meta@public-inbox.org Subject: Re: [PATCH 2/2] imap_fetch: Add a command to continuously fetch from an imap mailbox Message-ID: <20200516191220.GA24349@dcvr> References: <87eeyvmx74.fsf@x220.int.ebiederm.org> <20200513193144.GA9299@dcvr> <87ftc3mrq6.fsf@x220.int.ebiederm.org> <20200513221715.GA11718@dcvr> <877dxelmsr.fsf@x220.int.ebiederm.org> <87ftc0c3r4.fsf_-_@x220.int.ebiederm.org> <87a728c3p3.fsf_-_@x220.int.ebiederm.org> <87o8qoao09.fsf@x220.int.ebiederm.org> <20200515225644.GB4131@dcvr> <87mu6888cw.fsf@x220.int.ebiederm.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <87mu6888cw.fsf@x220.int.ebiederm.org> List-Id: "Eric W. Biederman" wrote: > Eric Wong writes: > > "Eric W. Biederman" wrote: > >> > The email messages are placed without modification into the public > >> > inbox repository so minimize changes of corruption or of loosing > >> > valuable information. I use the command imap_fetch for all of my > >> > email and not just a mailling list mirror so I don't want automation > >> > to accidentally cause something important to be lost. > > > > Btw, Email::MIME usage is gone from 1.5.0 due to nasty > > performance problems and replaced by PublicInbox::Eml. Eml > > should be completely non-destructive unless somebody sends an > > abusive message which exceeds the new safety limits; in which > > case it won't OOM or burn CPU like E::M did. > > > > That said, {-public_inbox_raw} still works and Eml looks > > like a drop-in replacement as far as imap_fetch is concerned. > > I almost did that. But I looked and saw PublicInbox::MIME still present > and a number of other references to Email::MIME so I wasn't certain > exactly how that was being handled. But since Email::MIME still > worked I didn't mess with that. I think the Import .pod documentation is the only place aside from some random comments and maintainer tests in xt/*, right? I'll definitely keep the import + indexing parts compatible with Email::MIME. I'm not sure how much of a Perl API I want to expose in case this gets reimplemented/migrated to another language someday, but maybe exposing the most heavily-used parts is OK. I also don't think another language is likely, at this point. > >> > No email messages are deleted from the server instead IMAPTracker > >> > is used to remember which messages were downloaded. > > > > Yup. I've integrated IMAPTracker into a local branch, already. > > I think it could be reused for tracking NNTP fetches, too, > > since UIDs and NNTP article numbers seem interchangeable. > > True. I guess it is possible to make the equivalent of my imap_fetch > for NNTP queries as well. Does NNTP have the equivalent of IDLE where > it is possible to get timely updates of new messages? I haven't seen anything in NNTP like IDLE. Maybe there's something in the draft stages, but AFAIK nothing widely implemented. > I suppose I should see if the git protocol does as well. I am trying to > figure out what the best way to track other public inbox git repositories. I'll likely implement a custom HTTP endpoint and usable with (curl -XIDLE ...); but the timeout will need to vary depending on firewalls. > Currently I have a script that every N minutes does git remote update. > Which is ok. But I think something like imap_fetch might be nicer. Definitely. > >> Bah. I sent this a little too soon. The patch needs this small > >> incremental fix to actually work. > > > > No worries. > > > > Btw, any reason you create the SSLSocket yourself instead of > > passing (Ssl => \@SSL_Socket_options) to IMAPClient->new? > > When I read the documentation it looked like that was the way to do > things. Even now when I reread the documentation that looks like the > way to go. Especially if I wanted to be certain the connection was > encrypted. There seems more than one way to do it, but `Starttls' and `Ssl' are just as documented from what I tell (in v3.38). Socket/RawSocket seem useful for using an external command to connect/launch an IMAP tunnel or server; so it'll be used to mimic the `imap.tunnel' support of git-imap-send.