[PATCH v2] lei: bail out earlier on IMAP writer failures

[Eric Wong <e@80x24.org>]

Ricardo Ribalda <ribalda@chromium.org> wrote:
> The patch did not seem to have any effect :(, I never  get a "IMAP
> LastError: " message

Yeah, I guess IMAP servers will just shutdown the socket w/o
saying anything. At least I didn't get anything from dovecot...

The below patch is a refinement of what I posted originally
and should stop the process instead of attempting to continue
and spew.

> On the other hand, the -j worked! I can go up to -j ,15 without any error.

Good to know.

I wonder if making the default `-j ,4' for IMAP is reasonable if
unspecified.  That's the default limit for HTTP(S) hosts, and I
seem to recall 4 being a reasonable limit for browsers.

Thanks for the report and followup!

------8<------
From: Eric Wong <e@80x24.org>
Subject: [PATCH] lei: bail out earlier on IMAP writer failures

Excessive IMAP connections can overload IMAP servers and cause
clients to be disconnected without diagnostic messages.
Use $lei->fail on these exceptions to propagate errors to the
CLI ASAP to avoid further errors down the line.

This ought to make problems more apparent for users using IMAP
destinations.

Reported-by: Ricardo Ribalda <ribalda@chromium.org>
Link: https://public-inbox.org/meta/CANiDSCsDfutAUMBLPZbxdyka+_jnhv+4YNYdL9QPRoC=wNUGCQ@mail.gmail.com/
---
 lib/PublicInbox/LeiToMail.pm | 10 +++++++---
 lib/PublicInbox/NetReader.pm |  8 +++++++-
 2 files changed, 14 insertions(+), 4 deletions(-)

diff --git a/lib/PublicInbox/LeiToMail.pm b/lib/PublicInbox/LeiToMail.pm
index 2aa3977e..03cbde3b 100644
--- a/lib/PublicInbox/LeiToMail.pm
+++ b/lib/PublicInbox/LeiToMail.pm
@@ -310,8 +310,11 @@ sub _imap_write_cb ($$) {
 	my $dedupe = $lei->{dedupe};
 	$dedupe->prepare_dedupe if $dedupe;
 	my $append = $lei->{net}->can('imap_append');
-	my $uri = $self->{uri};
-	my $mic = $lei->{net}->mic_get($uri);
+	my $uri = $self->{uri} // die 'BUG: no {uri}';
+	my $mic = $lei->{net}->mic_get($uri) // die <<EOM;
+E: $uri connection failed.
+E: Consider using `--jobs ,1' to limit IMAP connections
+EOM
 	my $folder = $uri->mailbox;
 	$uri->uidvalidity($mic->uidvalidity($folder));
 	my $lse = $lei->{lse}; # may be undef
@@ -749,7 +752,8 @@ sub do_post_auth {
 		$au_peers->[1] = undef;
 		sysread($au_peers->[0], my $barrier1, 1);
 	}
-	$self->{wcb} = $self->write_cb($lei);
+	eval { $self->{wcb} = $self->write_cb($lei) };
+	$lei->fail($@) if $@;
 	if ($au_peers) { # wait for peer l2m to set write_cb
 		$au_peers->[3] = undef;
 		sysread($au_peers->[2], my $barrier2, 1);
diff --git a/lib/PublicInbox/NetReader.pm b/lib/PublicInbox/NetReader.pm
index c1af03a3..4de2583e 100644
--- a/lib/PublicInbox/NetReader.pm
+++ b/lib/PublicInbox/NetReader.pm
@@ -685,7 +685,13 @@ sub mic_get {
 	}
 	my $mic = mic_new($self, $mic_arg, $sec, $uri);
 	$cached //= {}; # invalid placeholder if no cache enabled
-	$mic && $mic->IsConnected ? ($cached->{$sec} = $mic) : undef;
+	if ($mic && $mic->IsConnected) {
+		$cached->{$sec} = $mic;
+	} else {
+		warn 'IMAP LastError: ',$mic->LastError, "\n" if $mic;
+		warn "IMAP errno: $!\n" if $!;
+		undef;
+	}
 }
 
 sub imap_each {