unofficial mirror of meta@public-inbox.org
 help / color / mirror / Atom feed
From: Ricardo Ribalda <ribalda@chromium.org>
To: Eric Wong <e@80x24.org>
Cc: meta@public-inbox.org
Subject: Re: [PATCH v2] lei: bail out earlier on IMAP writer failures
Date: Sat, 10 Sep 2022 21:34:00 +0200	[thread overview]
Message-ID: <CANiDSCvSdYY1KkV-XDa5ZH+db0Nq6VDHBfJuh6pr7RghMo0dKQ@mail.gmail.com> (raw)
In-Reply-To: <20220910011859.M68532@dcvr>

Hi Eric

On Sat, 10 Sept 2022 at 03:19, Eric Wong <e@80x24.org> wrote:
>
> Ricardo Ribalda <ribalda@chromium.org> wrote:
> > The patch did not seem to have any effect :(, I never  get a "IMAP
> > LastError: " message
>
> Yeah, I guess IMAP servers will just shutdown the socket w/o
> saying anything. At least I didn't get anything from dovecot...
>
> The below patch is a refinement of what I posted originally
> and should stop the process instead of attempting to continue
> and spew.
>
> > On the other hand, the -j worked! I can go up to -j ,15 without any error.
>
> Good to know.
>
> I wonder if making the default `-j ,4' for IMAP is reasonable if
> unspecified.  That's the default limit for HTTP(S) hosts, and I
> seem to recall 4 being a reasonable limit for browsers.
>
> Thanks for the report and followup!

Similar output:


ribalda@denia:/tmp/public-inbox$ lei up imaps://imap.gmail.com/lei/me
# https://lore.kernel.org/all/ limiting to 2022-09-08 19:33 +0000 and newer
# /usr/local/google/home/ribalda/.local/share/lei/store 12/12
# /usr/bin/curl -Sf -s -d ''
https://lore.kernel.org/all/?x=m&t=1&q=((ribalda)+AND+rt%3A1660159987..)+AND+dt%3A20220908193300..
1486767 lei2mail 17 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486762 lei2mail 12 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486782 lei2mail 32 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486771 lei2mail 21 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486766 lei2mail 16 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486788 lei2mail 38 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486761 lei2mail 11 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486791 lei2mail 41 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486775 lei2mail 25 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486765 lei2mail 15 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486755 lei2mail 5 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486793 lei2mail 43 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486795 lei2mail 45 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486763 lei2mail 13 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486779 lei2mail 29 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486773 lei2mail 23 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486751 lei2mail 1 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486787 lei2mail 37 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486796 lei2mail 46 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486776 lei2mail 26 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486752 lei2mail 2 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486750 lei2mail 0 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486786 lei2mail 36 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486777 lei2mail 27 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486764 lei2mail 14 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486794 lei2mail 44 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486756 lei2mail 6 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486790 lei2mail 40 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486784 lei2mail 34 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486792 lei2mail 42 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486768 lei2mail 18 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486774 lei2mail 24 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486757 lei2mail 7 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
# https://lore.kernel.org/all/ 20/20
# 0 written to imaps://imap.gmail.com/lei/me (32 matches)
ribalda@denia:/tmp/public-inbox$ lei up imaps://imap.gmail.com/lei/me -j ,15
# https://lore.kernel.org/all/ limiting to 2022-09-08 19:33 +0000 and newer
# /usr/local/google/home/ribalda/.local/share/lei/store 12/12
# /usr/bin/curl -Sf -s -d ''
https://lore.kernel.org/all/?x=m&t=1&q=((ribalda)+AND+rt%3A1660160023..)+AND+dt%3A20220908193307..
# https://lore.kernel.org/all/ 20/20
# 0 written to imaps://imap.gmail.com/lei/me (32 matches)


Thanks!


>
> ------8<------
> From: Eric Wong <e@80x24.org>
> Subject: [PATCH] lei: bail out earlier on IMAP writer failures
>
> Excessive IMAP connections can overload IMAP servers and cause
> clients to be disconnected without diagnostic messages.
> Use $lei->fail on these exceptions to propagate errors to the
> CLI ASAP to avoid further errors down the line.
>
> This ought to make problems more apparent for users using IMAP
> destinations.
>
> Reported-by: Ricardo Ribalda <ribalda@chromium.org>
> Link: https://public-inbox.org/meta/CANiDSCsDfutAUMBLPZbxdyka+_jnhv+4YNYdL9QPRoC=wNUGCQ@mail.gmail.com/
> ---
>  lib/PublicInbox/LeiToMail.pm | 10 +++++++---
>  lib/PublicInbox/NetReader.pm |  8 +++++++-
>  2 files changed, 14 insertions(+), 4 deletions(-)
>
> diff --git a/lib/PublicInbox/LeiToMail.pm b/lib/PublicInbox/LeiToMail.pm
> index 2aa3977e..03cbde3b 100644
> --- a/lib/PublicInbox/LeiToMail.pm
> +++ b/lib/PublicInbox/LeiToMail.pm
> @@ -310,8 +310,11 @@ sub _imap_write_cb ($$) {
>         my $dedupe = $lei->{dedupe};
>         $dedupe->prepare_dedupe if $dedupe;
>         my $append = $lei->{net}->can('imap_append');
> -       my $uri = $self->{uri};
> -       my $mic = $lei->{net}->mic_get($uri);
> +       my $uri = $self->{uri} // die 'BUG: no {uri}';
> +       my $mic = $lei->{net}->mic_get($uri) // die <<EOM;
> +E: $uri connection failed.
> +E: Consider using `--jobs ,1' to limit IMAP connections
> +EOM
>         my $folder = $uri->mailbox;
>         $uri->uidvalidity($mic->uidvalidity($folder));
>         my $lse = $lei->{lse}; # may be undef
> @@ -749,7 +752,8 @@ sub do_post_auth {
>                 $au_peers->[1] = undef;
>                 sysread($au_peers->[0], my $barrier1, 1);
>         }
> -       $self->{wcb} = $self->write_cb($lei);
> +       eval { $self->{wcb} = $self->write_cb($lei) };
> +       $lei->fail($@) if $@;
>         if ($au_peers) { # wait for peer l2m to set write_cb
>                 $au_peers->[3] = undef;
>                 sysread($au_peers->[2], my $barrier2, 1);
> diff --git a/lib/PublicInbox/NetReader.pm b/lib/PublicInbox/NetReader.pm
> index c1af03a3..4de2583e 100644
> --- a/lib/PublicInbox/NetReader.pm
> +++ b/lib/PublicInbox/NetReader.pm
> @@ -685,7 +685,13 @@ sub mic_get {
>         }
>         my $mic = mic_new($self, $mic_arg, $sec, $uri);
>         $cached //= {}; # invalid placeholder if no cache enabled
> -       $mic && $mic->IsConnected ? ($cached->{$sec} = $mic) : undef;
> +       if ($mic && $mic->IsConnected) {
> +               $cached->{$sec} = $mic;
> +       } else {
> +               warn 'IMAP LastError: ',$mic->LastError, "\n" if $mic;
> +               warn "IMAP errno: $!\n" if $!;
> +               undef;
> +       }
>  }
>
>  sub imap_each {



-- 
Ricardo Ribalda

  reply	other threads:[~2022-09-10 19:34 UTC|newest]

Thread overview: 10+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2022-09-09 10:09 imap: "Can't use an undefined value as a subroutine reference" Ricardo Ribalda
2022-09-09 17:44 ` [PATCH] lei: add diagnostics for IMAP writer failures Eric Wong
2022-09-09 18:00   ` [PATCH] doc: document --jobs for `lei q' and `lei up' Eric Wong
2022-09-09 20:35   ` [PATCH] lei: add diagnostics for IMAP writer failures Ricardo Ribalda
2022-09-10  1:18     ` [PATCH v2] lei: bail out earlier on " Eric Wong
2022-09-10 19:34       ` Ricardo Ribalda [this message]
2022-09-10 19:50         ` Eric Wong
2022-09-10 19:53           ` Ricardo Ribalda
2022-09-10 20:19             ` Eric Wong
2022-11-14  8:07               ` [PATCH] lei q|up: limit default write --jobs for IMAP(S) Eric Wong

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: https://public-inbox.org/README

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=CANiDSCvSdYY1KkV-XDa5ZH+db0Nq6VDHBfJuh6pr7RghMo0dKQ@mail.gmail.com \
    --to=ribalda@chromium.org \
    --cc=e@80x24.org \
    --cc=meta@public-inbox.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).