From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: X-Spam-Status: No, score=-3.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE shortcircuit=no autolearn=ham autolearn_force=no version=3.4.2 Received: from mail-ed1-x529.google.com (mail-ed1-x529.google.com [IPv6:2a00:1450:4864:20::529]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by dcvr.yhbt.net (Postfix) with ESMTPS id 326261F54E for ; Sat, 10 Sep 2022 19:34:16 +0000 (UTC) Authentication-Results: dcvr.yhbt.net; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.b="ifmn0iJN"; dkim-atps=neutral Received: by mail-ed1-x529.google.com with SMTP id t5so7240537edc.11 for ; Sat, 10 Sep 2022 12:34:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date; bh=/YvdNraljDSj/7BVadGgPXaBDIrJDluzkBVPXFysMtg=; b=ifmn0iJNCJxPR/CTdO0x8V/DeIy/OlQiWtu6LFUPjDzZnVbAtcotsKExHieynIHvst W+VC8gXpr94urh0FqephYk3Ecg+tesQYKbdVoRxwjl9m2aGBZSVBgwN+xQAWmyduJ6el RP3iYwGc8BEmu+yfqdajEdCklQUzLMaPnyq6k= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date; bh=/YvdNraljDSj/7BVadGgPXaBDIrJDluzkBVPXFysMtg=; b=mWYG8VcHm5DLQc8NA8Lw6ti1kFu76sydZSuUbuW8n4Dbv85+WZMda+2lsgGcy4nSba RbTHcfPTViaAjhvzd0ufjaceVWYoi2vnabNQBCgPC+QVCNmkYnrM+8B+cy9ssmeI+UyP i0u5OLOPDNssSBICVwy5VIcpUyJX7uQp1pT2PqCSPzaJlcSVKcBeD5schhgZs8uNbfu3 HmVPeePvtuGcup0f8viP0ZTxvC95T5xYahmRR/80wzb/+Dz7a+lc1Pq2ZSPPNrQ6pcls hb5KwRyouhGzk3WZh+uXdti/JSXCL15ofOGKtFQ57dvMlQSSjX2s+K6U1jNNi3lCkde1 EPcA== X-Gm-Message-State: ACgBeo2Km9s+4DoVgkvSjhrbDV60d4kaYlSBDhktvNEU6CzvDbK5eDk2 KIfZsXVmoBm24eFzJ2vkGTo92a61WEM3HT3CeNM= X-Google-Smtp-Source: AA6agR46fSpjdzJu6R5dpj02VZ08Jt3Xl8/lNUH6Jv3qlBVwl1DrqOCYXWFc6QAQJJsX/iPNwWwTog== X-Received: by 2002:a05:6402:3806:b0:450:bad8:8cd5 with SMTP id es6-20020a056402380600b00450bad88cd5mr11704459edb.305.1662838452977; Sat, 10 Sep 2022 12:34:12 -0700 (PDT) Received: from mail-ej1-f50.google.com (mail-ej1-f50.google.com. [209.85.218.50]) by smtp.gmail.com with ESMTPSA id my39-20020a1709065a6700b0073d9a0d0cbcsm2035245ejc.72.2022.09.10.12.34.12 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sat, 10 Sep 2022 12:34:12 -0700 (PDT) Received: by mail-ej1-f50.google.com with SMTP id v16so11458177ejr.10 for ; Sat, 10 Sep 2022 12:34:12 -0700 (PDT) X-Received: by 2002:a17:907:2cd4:b0:73c:9fa8:3ddc with SMTP id hg20-20020a1709072cd400b0073c9fa83ddcmr13709849ejc.40.1662838452101; Sat, 10 Sep 2022 12:34:12 -0700 (PDT) MIME-Version: 1.0 References: <20220909174410.M560915@dcvr> <20220910011859.M68532@dcvr> In-Reply-To: <20220910011859.M68532@dcvr> From: Ricardo Ribalda Date: Sat, 10 Sep 2022 21:34:00 +0200 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: [PATCH v2] lei: bail out earlier on IMAP writer failures To: Eric Wong Cc: meta@public-inbox.org Content-Type: text/plain; charset="UTF-8" List-Id: Hi Eric On Sat, 10 Sept 2022 at 03:19, Eric Wong wrote: > > Ricardo Ribalda wrote: > > The patch did not seem to have any effect :(, I never get a "IMAP > > LastError: " message > > Yeah, I guess IMAP servers will just shutdown the socket w/o > saying anything. At least I didn't get anything from dovecot... > > The below patch is a refinement of what I posted originally > and should stop the process instead of attempting to continue > and spew. > > > On the other hand, the -j worked! I can go up to -j ,15 without any error. > > Good to know. > > I wonder if making the default `-j ,4' for IMAP is reasonable if > unspecified. That's the default limit for HTTP(S) hosts, and I > seem to recall 4 being a reasonable limit for browsers. > > Thanks for the report and followup! Similar output: ribalda@denia:/tmp/public-inbox$ lei up imaps://imap.gmail.com/lei/me # https://lore.kernel.org/all/ limiting to 2022-09-08 19:33 +0000 and newer # /usr/local/google/home/ribalda/.local/share/lei/store 12/12 # /usr/bin/curl -Sf -s -d '' https://lore.kernel.org/all/?x=m&t=1&q=((ribalda)+AND+rt%3A1660159987..)+AND+dt%3A20220908193300.. 1486767 lei2mail 17 wq_worker: do_post_auth: Can't call method "uidvalidity" on an undefined value at /usr/share/perl5/PublicInbox/LeiToMail.pm line 313. 1486762 lei2mail 12 wq_worker: do_post_auth: Can't call method "uidvalidity" on an undefined value at /usr/share/perl5/PublicInbox/LeiToMail.pm line 313. 1486782 lei2mail 32 wq_worker: do_post_auth: Can't call method "uidvalidity" on an undefined value at /usr/share/perl5/PublicInbox/LeiToMail.pm line 313. 1486771 lei2mail 21 wq_worker: do_post_auth: Can't call method "uidvalidity" on an undefined value at /usr/share/perl5/PublicInbox/LeiToMail.pm line 313. 1486766 lei2mail 16 wq_worker: do_post_auth: Can't call method "uidvalidity" on an undefined value at /usr/share/perl5/PublicInbox/LeiToMail.pm line 313. 1486788 lei2mail 38 wq_worker: do_post_auth: Can't call method "uidvalidity" on an undefined value at /usr/share/perl5/PublicInbox/LeiToMail.pm line 313. 1486761 lei2mail 11 wq_worker: do_post_auth: Can't call method "uidvalidity" on an undefined value at /usr/share/perl5/PublicInbox/LeiToMail.pm line 313. 1486791 lei2mail 41 wq_worker: do_post_auth: Can't call method "uidvalidity" on an undefined value at /usr/share/perl5/PublicInbox/LeiToMail.pm line 313. 1486775 lei2mail 25 wq_worker: do_post_auth: Can't call method "uidvalidity" on an undefined value at /usr/share/perl5/PublicInbox/LeiToMail.pm line 313. 1486765 lei2mail 15 wq_worker: do_post_auth: Can't call method "uidvalidity" on an undefined value at /usr/share/perl5/PublicInbox/LeiToMail.pm line 313. 1486755 lei2mail 5 wq_worker: do_post_auth: Can't call method "uidvalidity" on an undefined value at /usr/share/perl5/PublicInbox/LeiToMail.pm line 313. 1486793 lei2mail 43 wq_worker: do_post_auth: Can't call method "uidvalidity" on an undefined value at /usr/share/perl5/PublicInbox/LeiToMail.pm line 313. 1486795 lei2mail 45 wq_worker: do_post_auth: Can't call method "uidvalidity" on an undefined value at /usr/share/perl5/PublicInbox/LeiToMail.pm line 313. 1486763 lei2mail 13 wq_worker: do_post_auth: Can't call method "uidvalidity" on an undefined value at /usr/share/perl5/PublicInbox/LeiToMail.pm line 313. 1486779 lei2mail 29 wq_worker: do_post_auth: Can't call method "uidvalidity" on an undefined value at /usr/share/perl5/PublicInbox/LeiToMail.pm line 313. 1486773 lei2mail 23 wq_worker: do_post_auth: Can't call method "uidvalidity" on an undefined value at /usr/share/perl5/PublicInbox/LeiToMail.pm line 313. 1486751 lei2mail 1 wq_worker: do_post_auth: Can't call method "uidvalidity" on an undefined value at /usr/share/perl5/PublicInbox/LeiToMail.pm line 313. 1486787 lei2mail 37 wq_worker: do_post_auth: Can't call method "uidvalidity" on an undefined value at /usr/share/perl5/PublicInbox/LeiToMail.pm line 313. 1486796 lei2mail 46 wq_worker: do_post_auth: Can't call method "uidvalidity" on an undefined value at /usr/share/perl5/PublicInbox/LeiToMail.pm line 313. 1486776 lei2mail 26 wq_worker: do_post_auth: Can't call method "uidvalidity" on an undefined value at /usr/share/perl5/PublicInbox/LeiToMail.pm line 313. 1486752 lei2mail 2 wq_worker: do_post_auth: Can't call method "uidvalidity" on an undefined value at /usr/share/perl5/PublicInbox/LeiToMail.pm line 313. 1486750 lei2mail 0 wq_worker: do_post_auth: Can't call method "uidvalidity" on an undefined value at /usr/share/perl5/PublicInbox/LeiToMail.pm line 313. 1486786 lei2mail 36 wq_worker: do_post_auth: Can't call method "uidvalidity" on an undefined value at /usr/share/perl5/PublicInbox/LeiToMail.pm line 313. 1486777 lei2mail 27 wq_worker: do_post_auth: Can't call method "uidvalidity" on an undefined value at /usr/share/perl5/PublicInbox/LeiToMail.pm line 313. 1486764 lei2mail 14 wq_worker: do_post_auth: Can't call method "uidvalidity" on an undefined value at /usr/share/perl5/PublicInbox/LeiToMail.pm line 313. 1486794 lei2mail 44 wq_worker: do_post_auth: Can't call method "uidvalidity" on an undefined value at /usr/share/perl5/PublicInbox/LeiToMail.pm line 313. 1486756 lei2mail 6 wq_worker: do_post_auth: Can't call method "uidvalidity" on an undefined value at /usr/share/perl5/PublicInbox/LeiToMail.pm line 313. 1486790 lei2mail 40 wq_worker: do_post_auth: Can't call method "uidvalidity" on an undefined value at /usr/share/perl5/PublicInbox/LeiToMail.pm line 313. 1486784 lei2mail 34 wq_worker: do_post_auth: Can't call method "uidvalidity" on an undefined value at /usr/share/perl5/PublicInbox/LeiToMail.pm line 313. 1486792 lei2mail 42 wq_worker: do_post_auth: Can't call method "uidvalidity" on an undefined value at /usr/share/perl5/PublicInbox/LeiToMail.pm line 313. 1486768 lei2mail 18 wq_worker: do_post_auth: Can't call method "uidvalidity" on an undefined value at /usr/share/perl5/PublicInbox/LeiToMail.pm line 313. 1486774 lei2mail 24 wq_worker: do_post_auth: Can't call method "uidvalidity" on an undefined value at /usr/share/perl5/PublicInbox/LeiToMail.pm line 313. 1486757 lei2mail 7 wq_worker: do_post_auth: Can't call method "uidvalidity" on an undefined value at /usr/share/perl5/PublicInbox/LeiToMail.pm line 313. 1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm line 783. 1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm line 783. 1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm line 783. 1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm line 783. 1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm line 783. 1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm line 783. 1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm line 783. 1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm line 783. 1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm line 783. 1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm line 783. 1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm line 783. 1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm line 783. 1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm line 783. 1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm line 783. 1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm line 783. 1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm line 783. 1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm line 783. 1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm line 783. 1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm line 783. 1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm line 783. # https://lore.kernel.org/all/ 20/20 # 0 written to imaps://imap.gmail.com/lei/me (32 matches) ribalda@denia:/tmp/public-inbox$ lei up imaps://imap.gmail.com/lei/me -j ,15 # https://lore.kernel.org/all/ limiting to 2022-09-08 19:33 +0000 and newer # /usr/local/google/home/ribalda/.local/share/lei/store 12/12 # /usr/bin/curl -Sf -s -d '' https://lore.kernel.org/all/?x=m&t=1&q=((ribalda)+AND+rt%3A1660160023..)+AND+dt%3A20220908193307.. # https://lore.kernel.org/all/ 20/20 # 0 written to imaps://imap.gmail.com/lei/me (32 matches) Thanks! > > ------8<------ > From: Eric Wong > Subject: [PATCH] lei: bail out earlier on IMAP writer failures > > Excessive IMAP connections can overload IMAP servers and cause > clients to be disconnected without diagnostic messages. > Use $lei->fail on these exceptions to propagate errors to the > CLI ASAP to avoid further errors down the line. > > This ought to make problems more apparent for users using IMAP > destinations. > > Reported-by: Ricardo Ribalda > Link: https://public-inbox.org/meta/CANiDSCsDfutAUMBLPZbxdyka+_jnhv+4YNYdL9QPRoC=wNUGCQ@mail.gmail.com/ > --- > lib/PublicInbox/LeiToMail.pm | 10 +++++++--- > lib/PublicInbox/NetReader.pm | 8 +++++++- > 2 files changed, 14 insertions(+), 4 deletions(-) > > diff --git a/lib/PublicInbox/LeiToMail.pm b/lib/PublicInbox/LeiToMail.pm > index 2aa3977e..03cbde3b 100644 > --- a/lib/PublicInbox/LeiToMail.pm > +++ b/lib/PublicInbox/LeiToMail.pm > @@ -310,8 +310,11 @@ sub _imap_write_cb ($$) { > my $dedupe = $lei->{dedupe}; > $dedupe->prepare_dedupe if $dedupe; > my $append = $lei->{net}->can('imap_append'); > - my $uri = $self->{uri}; > - my $mic = $lei->{net}->mic_get($uri); > + my $uri = $self->{uri} // die 'BUG: no {uri}'; > + my $mic = $lei->{net}->mic_get($uri) // die < +E: $uri connection failed. > +E: Consider using `--jobs ,1' to limit IMAP connections > +EOM > my $folder = $uri->mailbox; > $uri->uidvalidity($mic->uidvalidity($folder)); > my $lse = $lei->{lse}; # may be undef > @@ -749,7 +752,8 @@ sub do_post_auth { > $au_peers->[1] = undef; > sysread($au_peers->[0], my $barrier1, 1); > } > - $self->{wcb} = $self->write_cb($lei); > + eval { $self->{wcb} = $self->write_cb($lei) }; > + $lei->fail($@) if $@; > if ($au_peers) { # wait for peer l2m to set write_cb > $au_peers->[3] = undef; > sysread($au_peers->[2], my $barrier2, 1); > diff --git a/lib/PublicInbox/NetReader.pm b/lib/PublicInbox/NetReader.pm > index c1af03a3..4de2583e 100644 > --- a/lib/PublicInbox/NetReader.pm > +++ b/lib/PublicInbox/NetReader.pm > @@ -685,7 +685,13 @@ sub mic_get { > } > my $mic = mic_new($self, $mic_arg, $sec, $uri); > $cached //= {}; # invalid placeholder if no cache enabled > - $mic && $mic->IsConnected ? ($cached->{$sec} = $mic) : undef; > + if ($mic && $mic->IsConnected) { > + $cached->{$sec} = $mic; > + } else { > + warn 'IMAP LastError: ',$mic->LastError, "\n" if $mic; > + warn "IMAP errno: $!\n" if $!; > + undef; > + } > } > > sub imap_each { -- Ricardo Ribalda