unofficial mirror of meta@public-inbox.org
 help / color / mirror / Atom feed
From: Eric Wong <e@80x24.org>
To: Ricardo Ribalda <ribalda@chromium.org>
Cc: meta@public-inbox.org
Subject: [PATCH v2] lei: bail out earlier on IMAP writer failures
Date: Sat, 10 Sep 2022 01:18:59 +0000	[thread overview]
Message-ID: <20220910011859.M68532@dcvr> (raw)
In-Reply-To: <CANiDSCviEaBwFUEpkxYgRxSOhjjO6g5HqxHoH8DVPtueo83rMg@mail.gmail.com>

Ricardo Ribalda <ribalda@chromium.org> wrote:
> The patch did not seem to have any effect :(, I never  get a "IMAP
> LastError: " message

Yeah, I guess IMAP servers will just shutdown the socket w/o
saying anything. At least I didn't get anything from dovecot...

The below patch is a refinement of what I posted originally
and should stop the process instead of attempting to continue
and spew.

> On the other hand, the -j worked! I can go up to -j ,15 without any error.

Good to know.

I wonder if making the default `-j ,4' for IMAP is reasonable if
unspecified.  That's the default limit for HTTP(S) hosts, and I
seem to recall 4 being a reasonable limit for browsers.

Thanks for the report and followup!

------8<------
From: Eric Wong <e@80x24.org>
Subject: [PATCH] lei: bail out earlier on IMAP writer failures

Excessive IMAP connections can overload IMAP servers and cause
clients to be disconnected without diagnostic messages.
Use $lei->fail on these exceptions to propagate errors to the
CLI ASAP to avoid further errors down the line.

This ought to make problems more apparent for users using IMAP
destinations.

Reported-by: Ricardo Ribalda <ribalda@chromium.org>
Link: https://public-inbox.org/meta/CANiDSCsDfutAUMBLPZbxdyka+_jnhv+4YNYdL9QPRoC=wNUGCQ@mail.gmail.com/
---
 lib/PublicInbox/LeiToMail.pm | 10 +++++++---
 lib/PublicInbox/NetReader.pm |  8 +++++++-
 2 files changed, 14 insertions(+), 4 deletions(-)

diff --git a/lib/PublicInbox/LeiToMail.pm b/lib/PublicInbox/LeiToMail.pm
index 2aa3977e..03cbde3b 100644
--- a/lib/PublicInbox/LeiToMail.pm
+++ b/lib/PublicInbox/LeiToMail.pm
@@ -310,8 +310,11 @@ sub _imap_write_cb ($$) {
 	my $dedupe = $lei->{dedupe};
 	$dedupe->prepare_dedupe if $dedupe;
 	my $append = $lei->{net}->can('imap_append');
-	my $uri = $self->{uri};
-	my $mic = $lei->{net}->mic_get($uri);
+	my $uri = $self->{uri} // die 'BUG: no {uri}';
+	my $mic = $lei->{net}->mic_get($uri) // die <<EOM;
+E: $uri connection failed.
+E: Consider using `--jobs ,1' to limit IMAP connections
+EOM
 	my $folder = $uri->mailbox;
 	$uri->uidvalidity($mic->uidvalidity($folder));
 	my $lse = $lei->{lse}; # may be undef
@@ -749,7 +752,8 @@ sub do_post_auth {
 		$au_peers->[1] = undef;
 		sysread($au_peers->[0], my $barrier1, 1);
 	}
-	$self->{wcb} = $self->write_cb($lei);
+	eval { $self->{wcb} = $self->write_cb($lei) };
+	$lei->fail($@) if $@;
 	if ($au_peers) { # wait for peer l2m to set write_cb
 		$au_peers->[3] = undef;
 		sysread($au_peers->[2], my $barrier2, 1);
diff --git a/lib/PublicInbox/NetReader.pm b/lib/PublicInbox/NetReader.pm
index c1af03a3..4de2583e 100644
--- a/lib/PublicInbox/NetReader.pm
+++ b/lib/PublicInbox/NetReader.pm
@@ -685,7 +685,13 @@ sub mic_get {
 	}
 	my $mic = mic_new($self, $mic_arg, $sec, $uri);
 	$cached //= {}; # invalid placeholder if no cache enabled
-	$mic && $mic->IsConnected ? ($cached->{$sec} = $mic) : undef;
+	if ($mic && $mic->IsConnected) {
+		$cached->{$sec} = $mic;
+	} else {
+		warn 'IMAP LastError: ',$mic->LastError, "\n" if $mic;
+		warn "IMAP errno: $!\n" if $!;
+		undef;
+	}
 }
 
 sub imap_each {

  reply	other threads:[~2022-09-10  1:18 UTC|newest]

Thread overview: 10+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2022-09-09 10:09 imap: "Can't use an undefined value as a subroutine reference" Ricardo Ribalda
2022-09-09 17:44 ` [PATCH] lei: add diagnostics for IMAP writer failures Eric Wong
2022-09-09 18:00   ` [PATCH] doc: document --jobs for `lei q' and `lei up' Eric Wong
2022-09-09 20:35   ` [PATCH] lei: add diagnostics for IMAP writer failures Ricardo Ribalda
2022-09-10  1:18     ` Eric Wong [this message]
2022-09-10 19:34       ` [PATCH v2] lei: bail out earlier on " Ricardo Ribalda
2022-09-10 19:50         ` Eric Wong
2022-09-10 19:53           ` Ricardo Ribalda
2022-09-10 20:19             ` Eric Wong
2022-11-14  8:07               ` [PATCH] lei q|up: limit default write --jobs for IMAP(S) Eric Wong

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: https://public-inbox.org/README

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20220910011859.M68532@dcvr \
    --to=e@80x24.org \
    --cc=meta@public-inbox.org \
    --cc=ribalda@chromium.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).