Re: [PATCH 2/2] imap_fetch: Add a command to continuously fetch from an imap mailbox

[Eric Wong <e@yhbt.net>]

"Eric W. Biederman" <ebiederm@xmission.com> wrote:
> Eric Wong <e@yhbt.net> writes:
> > "Eric W. Biederman" <ebiederm@xmission.com> wrote:
> >> > The email messages are placed without modification into the public
> >> > inbox repository so minimize changes of corruption or of loosing
> >> > valuable information.  I use the command imap_fetch for all of my
> >> > email and not just a mailling list mirror so I don't want automation
> >> > to accidentally cause something important to be lost.
> >
> > Btw, Email::MIME usage is gone from 1.5.0 due to nasty
> > performance problems and replaced by PublicInbox::Eml.  Eml
> > should be completely non-destructive unless somebody sends an
> > abusive message which exceeds the new safety limits; in which
> > case it won't OOM or burn CPU like E::M did.
> >
> > That said, {-public_inbox_raw} still works and Eml looks
> > like a drop-in replacement as far as imap_fetch is concerned.
> 
> I almost did that. But I looked and saw PublicInbox::MIME still present
> and a number of other references to Email::MIME so I wasn't certain
> exactly how that was being handled.  But since Email::MIME still
> worked I didn't mess with that.

I think the Import .pod documentation is the only place aside
from some random comments and maintainer tests in xt/*, right?

I'll definitely keep the import + indexing parts compatible with
Email::MIME.

I'm not sure how much of a Perl API I want to expose in case
this gets reimplemented/migrated to another language someday,
but maybe exposing the most heavily-used parts is OK.

I also don't think another language is likely, at this point.

> >> > No email messages are deleted from the server instead IMAPTracker
> >> > is used to remember which messages were downloaded.
> >
> > Yup.  I've integrated IMAPTracker into a local branch, already.
> > I think it could be reused for tracking NNTP fetches, too,
> > since UIDs and NNTP article numbers seem interchangeable.
> 
> True.  I guess it is possible to make the equivalent of my imap_fetch
> for NNTP queries as well.  Does NNTP have the equivalent of IDLE where
> it is possible to get timely updates of new messages?

I haven't seen anything in NNTP like IDLE.  Maybe there's
something in the draft stages, but AFAIK nothing widely
implemented.

> I suppose I should see if the git protocol does as well.  I am trying to
> figure out what the best way to track other public inbox git repositories.

I'll likely implement a custom HTTP endpoint and usable with
(curl -XIDLE ...); but the timeout will need to vary depending
on firewalls.

> Currently I have a script that every N minutes does git remote update.
> Which is ok.  But I think something like imap_fetch might be nicer.

Definitely.

> >> Bah.  I sent this a little too soon.  The patch needs this small
> >> incremental fix to actually work.
> >
> > No worries.
> >
> > Btw, any reason you create the SSLSocket yourself instead of
> > passing (Ssl => \@SSL_Socket_options) to IMAPClient->new?
> 
> When I read the documentation it looked like that was the way to do
> things.  Even now when I reread the documentation that looks like the
> way to go.  Especially if I wanted to be certain the connection was
> encrypted.

There seems more than one way to do it, but `Starttls' and `Ssl'
are just as documented from what I tell (in v3.38).
Socket/RawSocket seem useful for using an external command to
connect/launch an IMAP tunnel or server; so it'll be used to
mimic the `imap.tunnel' support of git-imap-send.