imap: "Can't use an undefined value as a subroutine reference"

imap: "Can't use an undefined value as a subroutine reference"

[Ricardo Ribalda]

Hi

I am getting a lot of those messages when using lei in imap mode.

Nonetheless the mail seems to arrive fine to its destination (on the
first run, some of the mail was lost, but for small batches of mails
it seems to work fine).

It is a valid message or just some red-herring.

I am using debian testing/

Regards!

# https://lore.kernel.org/all/ limiting to 2022-09-07 10:03 +0000 and newer
# /usr/local/google/home/ribalda/.local/share/lei/store 13/13
# /usr/bin/curl -Sf -s -d ''
https://lore.kernel.org/all/?x=m&t=1&q=((ribalda)+AND+rt%3A1660039651..)+AND+dt%3A20220907100321..
1285740 lei2mail 6 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1285776 lei2mail 42 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1285748 lei2mail 14 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1285749 lei2mail 15 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1285767 lei2mail 33 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1285756 lei2mail 22 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1285735 lei2mail 1 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1285736 lei2mail 2 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1285755 lei2mail 21 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1285753 lei2mail 19 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1285764 lei2mail 30 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1285739 lei2mail 5 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1285744 lei2mail 10 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1285772 lei2mail 38 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1285747 lei2mail 13 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1285752 lei2mail 18 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1285769 lei2mail 35 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1285763 lei2mail 29 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1285771 lei2mail 37 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1285742 lei2mail 8 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1285780 lei2mail 46 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1285779 lei2mail 45 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1285761 lei2mail 27 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1285743 lei2mail 9 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1285746 lei2mail 12 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1285766 lei2mail 32 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1285758 lei2mail 24 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1285754 lei2mail 20 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1285762 lei2mail 28 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1285774 lei2mail 40 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1285751 lei2mail 17 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1285770 lei2mail 36 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1285781 lei2mail 47 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1285735 lei2mail 1 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1285735 lei2mail 1 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1285735 lei2mail 1 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1285735 lei2mail 1 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1285735 lei2mail 1 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1285735 lei2mail 1 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1285735 lei2mail 1 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1285735 lei2mail 1 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1285735 lei2mail 1 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1285735 lei2mail 1 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1285735 lei2mail 1 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1285735 lei2mail 1 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1285735 lei2mail 1 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1285735 lei2mail 1 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
# https://lore.kernel.org/all/ 18/18
# 14 written to imaps://imap.gmail.com/lei/me (31 matches)

-- 
Ricardo Ribalda

[PATCH] lei: add diagnostics for IMAP writer failures

[Eric Wong]

Ricardo Ribalda <ribalda@chromium.org> wrote:
> Hi
> 
> I am getting a lot of those messages when using lei in imap mode.
> 
> Nonetheless the mail seems to arrive fine to its destination (on the
> first run, some of the mail was lost, but for small batches of mails
> it seems to work fine).
> 
> It is a valid message or just some red-herring.
> 
> I am using debian testing/
> 
> Regards!
> 
> # https://lore.kernel.org/all/ limiting to 2022-09-07 10:03 +0000 and newer
> # /usr/local/google/home/ribalda/.local/share/lei/store 13/13
> # /usr/bin/curl -Sf -s -d ''
> https://lore.kernel.org/all/?x=m&t=1&q=((ribalda)+AND+rt%3A1660039651..)+AND+dt%3A20220907100321..
> 1285740 lei2mail 6 wq_worker: do_post_auth: Can't call method
> "uidvalidity" on an undefined value at
> /usr/share/perl5/PublicInbox/LeiToMail.pm line 313.

<snip>

> 1285735 lei2mail 1 wq_worker: write_mail: Can't use an undefined value
> as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
> line 783.
> # https://lore.kernel.org/all/ 18/18
> # 14 written to imaps://imap.gmail.com/lei/me (31 matches)

I wonder if it's excessive parallelism for gmail's IMAP.
I haven't tested IMAP destinations, much...

Can you try the patch at the bottom?

There's also another patch coming to document the `--jobs|-j' CLI
switch for `lei up' and `lei q', but trying `-j ,1' may help you
if it's parallelism.  Note the comma before `1', it accepts
`-j $Q,$W' since $Q is the number of query processes and $W is
the number of LeiToMail writers.

-------8<-------
From: Eric Wong <e@80x24.org>
Subject: [PATCH] lei: add diagnostics for IMAP writer failures

This may help diagnose the problem with IMAP destinations
encountered at:
https://public-inbox.org/meta/CANiDSCsDfutAUMBLPZbxdyka+_jnhv+4YNYdL9QPRoC=wNUGCQ@mail.gmail.com/
---
 lib/PublicInbox/LeiToMail.pm | 4 ++--
 lib/PublicInbox/NetReader.pm | 8 +++++++-
 2 files changed, 9 insertions(+), 3 deletions(-)

diff --git a/lib/PublicInbox/LeiToMail.pm b/lib/PublicInbox/LeiToMail.pm
index 2aa3977e..bc00b96a 100644
--- a/lib/PublicInbox/LeiToMail.pm
+++ b/lib/PublicInbox/LeiToMail.pm
@@ -310,8 +310,8 @@ sub _imap_write_cb ($$) {
 	my $dedupe = $lei->{dedupe};
 	$dedupe->prepare_dedupe if $dedupe;
 	my $append = $lei->{net}->can('imap_append');
-	my $uri = $self->{uri};
-	my $mic = $lei->{net}->mic_get($uri);
+	my $uri = $self->{uri} // die 'BUG: no {uri}';
+	my $mic = $lei->{net}->mic_get($uri) // die 'BUG: no $mic';
 	my $folder = $uri->mailbox;
 	$uri->uidvalidity($mic->uidvalidity($folder));
 	my $lse = $lei->{lse}; # may be undef
diff --git a/lib/PublicInbox/NetReader.pm b/lib/PublicInbox/NetReader.pm
index c1af03a3..4de2583e 100644
--- a/lib/PublicInbox/NetReader.pm
+++ b/lib/PublicInbox/NetReader.pm
@@ -685,7 +685,13 @@ sub mic_get {
 	}
 	my $mic = mic_new($self, $mic_arg, $sec, $uri);
 	$cached //= {}; # invalid placeholder if no cache enabled
-	$mic && $mic->IsConnected ? ($cached->{$sec} = $mic) : undef;
+	if ($mic && $mic->IsConnected) {
+		$cached->{$sec} = $mic;
+	} else {
+		warn 'IMAP LastError: ',$mic->LastError, "\n" if $mic;
+		warn "IMAP errno: $!\n" if $!;
+		undef;
+	}
 }
 
 sub imap_each {

[PATCH] doc: document --jobs for `lei q' and `lei up'

[Eric Wong]

Eric Wong <e@80x24.org> wrote:
> There's also another patch coming to document the `--jobs|-j' CLI
> switch for `lei up' and `lei q', but trying `-j ,1' may help you
> if it's parallelism.  Note the comma before `1', it accepts
> `-j $Q,$W' since $Q is the number of query processes and $W is
> the number of LeiToMail writers.

-------8<-----
From: Eric Wong <e@80x24.org>
Subject: [PATCH] doc: document --jobs for `lei q' and `lei up'

These may be helpful for users on slow disks or limited IMAP
connections.
---
 Documentation/lei-q.pod  | 17 +++++++++++++++++
 Documentation/lei-up.pod |  4 +++-
 2 files changed, 20 insertions(+), 1 deletion(-)

diff --git a/Documentation/lei-q.pod b/Documentation/lei-q.pod
index 2f0c3bc6..8134223e 100644
--- a/Documentation/lei-q.pod
+++ b/Documentation/lei-q.pod
@@ -124,6 +124,23 @@ of the same thread.
 TODO: Warning: this flag may become persistent and saved in
 lei/store unless an MUA unflags it!  (Behavior undecided)
 
+=item --jobs=QUERY_WORKERS[,WRITE_WORKERS]
+=item --jobs=,WRITE_WORKERS
+
+=item -j QUERY_WORKERS[,WRITE_WORKERS]
+=item -j ,WRITE_WORKERS
+
+Set the number of query and write worker processes for parallelism.
+
+C<QUERY_WORKERS> defaults to the number of CPUs available, but 4 per
+remote (HTTP/HTTPS) host.
+
+C<WRITE_WORKERS> defaults to the number of CPUs available for Maildir,
+IMAP/IMAPS, and mbox* destinations.
+
+Omitting C<QUERY_WORKERS> but leaving the comma (C<,>) allows
+one to only set C<WRITE_WORKERS>
+
 =item --dedupe=STRATEGY
 
 =item -d STRATEGY
diff --git a/Documentation/lei-up.pod b/Documentation/lei-up.pod
index ac644a96..3b7c6f46 100644
--- a/Documentation/lei-up.pod
+++ b/Documentation/lei-up.pod
@@ -64,7 +64,9 @@ specified via C<lei q --only>.
 
 =item --mua=CMD
 
-C<--lock>, C<--alert>, and C<--mua> are all supported and
+=item --jobs QUERY_WORKERS[,WRITE_WORKERS]
+
+C<--lock>, C<--alert>, C<--mua>, and C<--jobs> are all supported and
 documented in L<lei-q(1)>.
 
 C<--mua> is incompatible with C<--all>.

Re: [PATCH] lei: add diagnostics for IMAP writer failures

[Ricardo Ribalda]

Hi Eric

On Fri, 9 Sept 2022 at 19:45, Eric Wong <e@80x24.org> wrote:
>
> Ricardo Ribalda <ribalda@chromium.org> wrote:
> > Hi
> >
> > I am getting a lot of those messages when using lei in imap mode.
> >
> > Nonetheless the mail seems to arrive fine to its destination (on the
> > first run, some of the mail was lost, but for small batches of mails
> > it seems to work fine).
> >
> > It is a valid message or just some red-herring.
> >
> > I am using debian testing/
> >
> > Regards!
> >
> > # https://lore.kernel.org/all/ limiting to 2022-09-07 10:03 +0000 and newer
> > # /usr/local/google/home/ribalda/.local/share/lei/store 13/13
> > # /usr/bin/curl -Sf -s -d ''
> > https://lore.kernel.org/all/?x=m&t=1&q=((ribalda)+AND+rt%3A1660039651..)+AND+dt%3A20220907100321..
> > 1285740 lei2mail 6 wq_worker: do_post_auth: Can't call method
> > "uidvalidity" on an undefined value at
> > /usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
>
> <snip>
>
> > 1285735 lei2mail 1 wq_worker: write_mail: Can't use an undefined value
> > as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
> > line 783.
> > # https://lore.kernel.org/all/ 18/18
> > # 14 written to imaps://imap.gmail.com/lei/me (31 matches)
>
> I wonder if it's excessive parallelism for gmail's IMAP.
> I haven't tested IMAP destinations, much...
>
> Can you try the patch at the bottom?
>
> There's also another patch coming to document the `--jobs|-j' CLI
> switch for `lei up' and `lei q', but trying `-j ,1' may help you
> if it's parallelism.  Note the comma before `1', it accepts
> `-j $Q,$W' since $Q is the number of query processes and $W is
> the number of LeiToMail writers.

The patch did not seem to have any effect :(, I never  get a "IMAP
LastError: " message

On the other hand, the -j worked! I can go up to -j ,15 without any error.

>
> -------8<-------
> From: Eric Wong <e@80x24.org>
> Subject: [PATCH] lei: add diagnostics for IMAP writer failures
>
> This may help diagnose the problem with IMAP destinations
> encountered at:
> https://public-inbox.org/meta/CANiDSCsDfutAUMBLPZbxdyka+_jnhv+4YNYdL9QPRoC=wNUGCQ@mail.gmail.com/
> ---
>  lib/PublicInbox/LeiToMail.pm | 4 ++--
>  lib/PublicInbox/NetReader.pm | 8 +++++++-
>  2 files changed, 9 insertions(+), 3 deletions(-)
>
> diff --git a/lib/PublicInbox/LeiToMail.pm b/lib/PublicInbox/LeiToMail.pm
> index 2aa3977e..bc00b96a 100644
> --- a/lib/PublicInbox/LeiToMail.pm
> +++ b/lib/PublicInbox/LeiToMail.pm
> @@ -310,8 +310,8 @@ sub _imap_write_cb ($$) {
>         my $dedupe = $lei->{dedupe};
>         $dedupe->prepare_dedupe if $dedupe;
>         my $append = $lei->{net}->can('imap_append');
> -       my $uri = $self->{uri};
> -       my $mic = $lei->{net}->mic_get($uri);
> +       my $uri = $self->{uri} // die 'BUG: no {uri}';
> +       my $mic = $lei->{net}->mic_get($uri) // die 'BUG: no $mic';
>         my $folder = $uri->mailbox;
>         $uri->uidvalidity($mic->uidvalidity($folder));
>         my $lse = $lei->{lse}; # may be undef
> diff --git a/lib/PublicInbox/NetReader.pm b/lib/PublicInbox/NetReader.pm
> index c1af03a3..4de2583e 100644
> --- a/lib/PublicInbox/NetReader.pm
> +++ b/lib/PublicInbox/NetReader.pm
> @@ -685,7 +685,13 @@ sub mic_get {
>         }
>         my $mic = mic_new($self, $mic_arg, $sec, $uri);
>         $cached //= {}; # invalid placeholder if no cache enabled
> -       $mic && $mic->IsConnected ? ($cached->{$sec} = $mic) : undef;
> +       if ($mic && $mic->IsConnected) {
> +               $cached->{$sec} = $mic;
> +       } else {
> +               warn 'IMAP LastError: ',$mic->LastError, "\n" if $mic;
> +               warn "IMAP errno: $!\n" if $!;
> +               undef;
> +       }
>  }
>
>  sub imap_each {



-- 
Ricardo Ribalda

[PATCH v2] lei: bail out earlier on IMAP writer failures

[Eric Wong]

Ricardo Ribalda <ribalda@chromium.org> wrote:
> The patch did not seem to have any effect :(, I never  get a "IMAP
> LastError: " message

Yeah, I guess IMAP servers will just shutdown the socket w/o
saying anything. At least I didn't get anything from dovecot...

The below patch is a refinement of what I posted originally
and should stop the process instead of attempting to continue
and spew.

> On the other hand, the -j worked! I can go up to -j ,15 without any error.

Good to know.

I wonder if making the default `-j ,4' for IMAP is reasonable if
unspecified.  That's the default limit for HTTP(S) hosts, and I
seem to recall 4 being a reasonable limit for browsers.

Thanks for the report and followup!

------8<------
From: Eric Wong <e@80x24.org>
Subject: [PATCH] lei: bail out earlier on IMAP writer failures

Excessive IMAP connections can overload IMAP servers and cause
clients to be disconnected without diagnostic messages.
Use $lei->fail on these exceptions to propagate errors to the
CLI ASAP to avoid further errors down the line.

This ought to make problems more apparent for users using IMAP
destinations.

Reported-by: Ricardo Ribalda <ribalda@chromium.org>
Link: https://public-inbox.org/meta/CANiDSCsDfutAUMBLPZbxdyka+_jnhv+4YNYdL9QPRoC=wNUGCQ@mail.gmail.com/
---
 lib/PublicInbox/LeiToMail.pm | 10 +++++++---
 lib/PublicInbox/NetReader.pm |  8 +++++++-
 2 files changed, 14 insertions(+), 4 deletions(-)

diff --git a/lib/PublicInbox/LeiToMail.pm b/lib/PublicInbox/LeiToMail.pm
index 2aa3977e..03cbde3b 100644
--- a/lib/PublicInbox/LeiToMail.pm
+++ b/lib/PublicInbox/LeiToMail.pm
@@ -310,8 +310,11 @@ sub _imap_write_cb ($$) {
 	my $dedupe = $lei->{dedupe};
 	$dedupe->prepare_dedupe if $dedupe;
 	my $append = $lei->{net}->can('imap_append');
-	my $uri = $self->{uri};
-	my $mic = $lei->{net}->mic_get($uri);
+	my $uri = $self->{uri} // die 'BUG: no {uri}';
+	my $mic = $lei->{net}->mic_get($uri) // die <<EOM;
+E: $uri connection failed.
+E: Consider using `--jobs ,1' to limit IMAP connections
+EOM
 	my $folder = $uri->mailbox;
 	$uri->uidvalidity($mic->uidvalidity($folder));
 	my $lse = $lei->{lse}; # may be undef
@@ -749,7 +752,8 @@ sub do_post_auth {
 		$au_peers->[1] = undef;
 		sysread($au_peers->[0], my $barrier1, 1);
 	}
-	$self->{wcb} = $self->write_cb($lei);
+	eval { $self->{wcb} = $self->write_cb($lei) };
+	$lei->fail($@) if $@;
 	if ($au_peers) { # wait for peer l2m to set write_cb
 		$au_peers->[3] = undef;
 		sysread($au_peers->[2], my $barrier2, 1);
diff --git a/lib/PublicInbox/NetReader.pm b/lib/PublicInbox/NetReader.pm
index c1af03a3..4de2583e 100644
--- a/lib/PublicInbox/NetReader.pm
+++ b/lib/PublicInbox/NetReader.pm
@@ -685,7 +685,13 @@ sub mic_get {
 	}
 	my $mic = mic_new($self, $mic_arg, $sec, $uri);
 	$cached //= {}; # invalid placeholder if no cache enabled
-	$mic && $mic->IsConnected ? ($cached->{$sec} = $mic) : undef;
+	if ($mic && $mic->IsConnected) {
+		$cached->{$sec} = $mic;
+	} else {
+		warn 'IMAP LastError: ',$mic->LastError, "\n" if $mic;
+		warn "IMAP errno: $!\n" if $!;
+		undef;
+	}
 }
 
 sub imap_each {

Re: [PATCH v2] lei: bail out earlier on IMAP writer failures

[Ricardo Ribalda]

Hi Eric

On Sat, 10 Sept 2022 at 03:19, Eric Wong <e@80x24.org> wrote:
>
> Ricardo Ribalda <ribalda@chromium.org> wrote:
> > The patch did not seem to have any effect :(, I never  get a "IMAP
> > LastError: " message
>
> Yeah, I guess IMAP servers will just shutdown the socket w/o
> saying anything. At least I didn't get anything from dovecot...
>
> The below patch is a refinement of what I posted originally
> and should stop the process instead of attempting to continue
> and spew.
>
> > On the other hand, the -j worked! I can go up to -j ,15 without any error.
>
> Good to know.
>
> I wonder if making the default `-j ,4' for IMAP is reasonable if
> unspecified.  That's the default limit for HTTP(S) hosts, and I
> seem to recall 4 being a reasonable limit for browsers.
>
> Thanks for the report and followup!

Similar output:


ribalda@denia:/tmp/public-inbox$ lei up imaps://imap.gmail.com/lei/me
# https://lore.kernel.org/all/ limiting to 2022-09-08 19:33 +0000 and newer
# /usr/local/google/home/ribalda/.local/share/lei/store 12/12
# /usr/bin/curl -Sf -s -d ''
https://lore.kernel.org/all/?x=m&t=1&q=((ribalda)+AND+rt%3A1660159987..)+AND+dt%3A20220908193300..
1486767 lei2mail 17 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486762 lei2mail 12 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486782 lei2mail 32 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486771 lei2mail 21 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486766 lei2mail 16 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486788 lei2mail 38 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486761 lei2mail 11 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486791 lei2mail 41 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486775 lei2mail 25 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486765 lei2mail 15 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486755 lei2mail 5 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486793 lei2mail 43 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486795 lei2mail 45 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486763 lei2mail 13 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486779 lei2mail 29 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486773 lei2mail 23 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486751 lei2mail 1 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486787 lei2mail 37 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486796 lei2mail 46 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486776 lei2mail 26 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486752 lei2mail 2 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486750 lei2mail 0 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486786 lei2mail 36 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486777 lei2mail 27 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486764 lei2mail 14 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486794 lei2mail 44 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486756 lei2mail 6 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486790 lei2mail 40 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486784 lei2mail 34 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486792 lei2mail 42 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486768 lei2mail 18 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486774 lei2mail 24 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486757 lei2mail 7 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
# https://lore.kernel.org/all/ 20/20
# 0 written to imaps://imap.gmail.com/lei/me (32 matches)
ribalda@denia:/tmp/public-inbox$ lei up imaps://imap.gmail.com/lei/me -j ,15
# https://lore.kernel.org/all/ limiting to 2022-09-08 19:33 +0000 and newer
# /usr/local/google/home/ribalda/.local/share/lei/store 12/12
# /usr/bin/curl -Sf -s -d ''
https://lore.kernel.org/all/?x=m&t=1&q=((ribalda)+AND+rt%3A1660160023..)+AND+dt%3A20220908193307..
# https://lore.kernel.org/all/ 20/20
# 0 written to imaps://imap.gmail.com/lei/me (32 matches)


Thanks!


>
> ------8<------
> From: Eric Wong <e@80x24.org>
> Subject: [PATCH] lei: bail out earlier on IMAP writer failures
>
> Excessive IMAP connections can overload IMAP servers and cause
> clients to be disconnected without diagnostic messages.
> Use $lei->fail on these exceptions to propagate errors to the
> CLI ASAP to avoid further errors down the line.
>
> This ought to make problems more apparent for users using IMAP
> destinations.
>
> Reported-by: Ricardo Ribalda <ribalda@chromium.org>
> Link: https://public-inbox.org/meta/CANiDSCsDfutAUMBLPZbxdyka+_jnhv+4YNYdL9QPRoC=wNUGCQ@mail.gmail.com/
> ---
>  lib/PublicInbox/LeiToMail.pm | 10 +++++++---
>  lib/PublicInbox/NetReader.pm |  8 +++++++-
>  2 files changed, 14 insertions(+), 4 deletions(-)
>
> diff --git a/lib/PublicInbox/LeiToMail.pm b/lib/PublicInbox/LeiToMail.pm
> index 2aa3977e..03cbde3b 100644
> --- a/lib/PublicInbox/LeiToMail.pm
> +++ b/lib/PublicInbox/LeiToMail.pm
> @@ -310,8 +310,11 @@ sub _imap_write_cb ($$) {
>         my $dedupe = $lei->{dedupe};
>         $dedupe->prepare_dedupe if $dedupe;
>         my $append = $lei->{net}->can('imap_append');
> -       my $uri = $self->{uri};
> -       my $mic = $lei->{net}->mic_get($uri);
> +       my $uri = $self->{uri} // die 'BUG: no {uri}';
> +       my $mic = $lei->{net}->mic_get($uri) // die <<EOM;
> +E: $uri connection failed.
> +E: Consider using `--jobs ,1' to limit IMAP connections
> +EOM
>         my $folder = $uri->mailbox;
>         $uri->uidvalidity($mic->uidvalidity($folder));
>         my $lse = $lei->{lse}; # may be undef
> @@ -749,7 +752,8 @@ sub do_post_auth {
>                 $au_peers->[1] = undef;
>                 sysread($au_peers->[0], my $barrier1, 1);
>         }
> -       $self->{wcb} = $self->write_cb($lei);
> +       eval { $self->{wcb} = $self->write_cb($lei) };
> +       $lei->fail($@) if $@;
>         if ($au_peers) { # wait for peer l2m to set write_cb
>                 $au_peers->[3] = undef;
>                 sysread($au_peers->[2], my $barrier2, 1);
> diff --git a/lib/PublicInbox/NetReader.pm b/lib/PublicInbox/NetReader.pm
> index c1af03a3..4de2583e 100644
> --- a/lib/PublicInbox/NetReader.pm
> +++ b/lib/PublicInbox/NetReader.pm
> @@ -685,7 +685,13 @@ sub mic_get {
>         }
>         my $mic = mic_new($self, $mic_arg, $sec, $uri);
>         $cached //= {}; # invalid placeholder if no cache enabled
> -       $mic && $mic->IsConnected ? ($cached->{$sec} = $mic) : undef;
> +       if ($mic && $mic->IsConnected) {
> +               $cached->{$sec} = $mic;
> +       } else {
> +               warn 'IMAP LastError: ',$mic->LastError, "\n" if $mic;
> +               warn "IMAP errno: $!\n" if $!;
> +               undef;
> +       }
>  }
>
>  sub imap_each {



-- 
Ricardo Ribalda

Re: [PATCH v2] lei: bail out earlier on IMAP writer failures

[Eric Wong]

Ricardo Ribalda <ribalda@chromium.org> wrote:
> Similar output:

Oh, wait, did you run `lei daemon-kill' after applying the
patch?  That's needed to reload the code, I'll try to make it
either auto-reload or warn users in the future.

Thanks.

Re: [PATCH v2] lei: bail out earlier on IMAP writer failures

[Ricardo Ribalda]

On Sat, 10 Sept 2022 at 21:51, Eric Wong <e@80x24.org> wrote:
>
> Ricardo Ribalda <ribalda@chromium.org> wrote:
> > Similar output:
>
> Oh, wait, did you run `lei daemon-kill' after applying the
> patch?  That's needed to reload the code, I'll try to make it
> either auto-reload or warn users in the future.
>
> Thanks.

Ups :S

ribalda@denia:/tmp/public-inbox$ lei up imaps://imap.gmail.com/lei/me
# https://lore.kernel.org/all/ limiting to 2022-09-08 19:52 +0000 and newer
# /usr/local/google/home/ribalda/.local/share/lei/store 12/12
# /usr/bin/curl -Sf -s -d ''
https://lore.kernel.org/all/?x=m&t=1&q=((ribalda)+AND+rt%3A1660161176..)+AND+dt%3A20220908195223..
E: imaps://imap.gmail.com/lei/me connection failed.
E: Consider using `--jobs ,1' to limit IMAP connections
ribalda@denia:/tmp/public-inbox$ lei up imaps://imap.gmail.com/lei/me --jobs ,15
# https://lore.kernel.org/all/ limiting to 2022-09-08 19:52 +0000 and newer
# /usr/local/google/home/ribalda/.local/share/lei/store 12/12
# /usr/bin/curl -Sf -s -d ''
https://lore.kernel.org/all/?x=m&t=1&q=((ribalda)+AND+rt%3A1660161182..)+AND+dt%3A20220908195223..
# https://lore.kernel.org/all/ 20/20
# 0 written to imaps://imap.gmail.com/lei/me (32 matches, 20 duplicates)




-- 
Ricardo Ribalda

Re: [PATCH v2] lei: bail out earlier on IMAP writer failures

[Eric Wong]

Ricardo Ribalda <ribalda@chromium.org> wrote:
> ribalda@denia:/tmp/public-inbox$ lei up imaps://imap.gmail.com/lei/me
> # https://lore.kernel.org/all/ limiting to 2022-09-08 19:52 +0000 and newer
> # /usr/local/google/home/ribalda/.local/share/lei/store 12/12
> # /usr/bin/curl -Sf -s -d ''
> https://lore.kernel.org/all/?x=m&t=1&q=((ribalda)+AND+rt%3A1660161176..)+AND+dt%3A20220908195223..
> E: imaps://imap.gmail.com/lei/me connection failed.
> E: Consider using `--jobs ,1' to limit IMAP connections

Thanks for confirming things work as intended.  I think the
default should be clamped, though... 15 seems a bit high for
smaller IMAP servers *shrug*

> ribalda@denia:/tmp/public-inbox$ lei up imaps://imap.gmail.com/lei/me --jobs ,15

[PATCH] lei q|up: limit default write --jobs for IMAP(S)

[Eric Wong]

Eric Wong <e@80x24.org> wrote:
> Thanks for confirming things work as intended.  I think the
> default should be clamped, though... 15 seems a bit high for
> smaller IMAP servers *shrug*

--------8<-------
Subject: [PATCH] lei q|up: limit default write --jobs for IMAP(S)

IMAP(S) servers often limit per-user connections, so avoid
bumping into limits to improve the out-of-the-box experience.
4 seems like a conservative default, since we already chose
that number for remote HTTP(S) endpoints.

Link: https://public-inbox.org/meta/20220910201958.GA12212@dcvr/
---
  /me having git-repack OOM due to excessive default pack.threads
  reminded me of this issue :x

 Documentation/lei-q.pod     | 4 ++--
 lib/PublicInbox/LeiQuery.pm | 7 +++++--
 2 files changed, 7 insertions(+), 4 deletions(-)

diff --git a/Documentation/lei-q.pod b/Documentation/lei-q.pod
index 8134223e..d52c5b04 100644
--- a/Documentation/lei-q.pod
+++ b/Documentation/lei-q.pod
@@ -135,8 +135,8 @@ Set the number of query and write worker processes for parallelism.
 C<QUERY_WORKERS> defaults to the number of CPUs available, but 4 per
 remote (HTTP/HTTPS) host.
 
-C<WRITE_WORKERS> defaults to the number of CPUs available for Maildir,
-IMAP/IMAPS, and mbox* destinations.
+C<WRITE_WORKERS> defaults to 75% of the number of CPUs available for
+Maildir and mbox* destinations, but 4 per IMAP/IMAPS host.
 
 Omitting C<QUERY_WORKERS> but leaving the comma (C<,>) allows
 one to only set C<WRITE_WORKERS>
diff --git a/lib/PublicInbox/LeiQuery.pm b/lib/PublicInbox/LeiQuery.pm
index df9c32b3..0f839236 100644
--- a/lib/PublicInbox/LeiQuery.pm
+++ b/lib/PublicInbox/LeiQuery.pm
@@ -39,8 +39,11 @@ sub _start_query { # used by "lei q" and "lei up"
 			$lms->lms_write_prepare->lms_pause; # just create
 		}
 	}
-	$l2m and $l2m->{-wq_nr_workers} //= $mj //
-		int($nproc * 0.75 + 0.5); # keep some CPU for git
+	$l2m and $l2m->{-wq_nr_workers} //= $mj // do {
+		# keep some CPU for git, and don't overload IMAP destinations
+		my $n = int($nproc * 0.75 + 0.5);
+		$self->{net} && $n > 4 ? 4 : $n;
+	};
 
 	# descending docid order is cheapest, MUA controls sorting order
 	$self->{mset_opt}->{relevance} //= -2 if $l2m || $opt->{threads};