From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: X-Spam-Status: No, score=-4.2 required=3.0 tests=ALL_TRUSTED,AWL,BAYES_00, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, T_SCC_BODY_TEXT_LINE shortcircuit=no autolearn=ham autolearn_force=no version=3.4.6 Received: from localhost (dcvr.yhbt.net [127.0.0.1]) by dcvr.yhbt.net (Postfix) with ESMTP id 0C0EE1F513 for ; Wed, 15 Nov 2023 09:21:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=80x24.org; s=selector1; t=1700040110; bh=FYm30aJD+X3qludh1xNJ5KR/99HteRZSU1WesWePkmU=; h=From:To:Subject:Date:In-Reply-To:References:From; b=S6mLagPAigbRi83DPaHd6XTCq86pZfm5ayKO3JqSb81+gyv0DMb9x/I2iGv1Z1GPe 8NkHiZZ70b8guwT3PSmDpftgE/zDSgEtLCzClWxJuiHp9tjCBbwZP0GWTJ2ddUMuNH Ck0py63aewSn9zao/b/xY0bcd7Rj+W+y5BUtLO/g= From: Eric Wong To: meta@public-inbox.org Subject: [PATCH 4/4] lei q|up|convert: common finish_output to detect errors Date: Wed, 15 Nov 2023 09:21:45 +0000 Message-Id: <20231115092145.1131822-5-e@80x24.org> In-Reply-To: <20231115092145.1131822-1-e@80x24.org> References: <20231115092145.1131822-1-e@80x24.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit List-Id: We need to consistently check the exit code of pigz|gzip|xz|bzip2 when writing to compressed mboxes (or bad storage). --- lib/PublicInbox/LeiConvert.pm | 4 ++-- lib/PublicInbox/LeiToMail.pm | 11 +++++++++++ lib/PublicInbox/LeiXSearch.pm | 9 +-------- 3 files changed, 14 insertions(+), 10 deletions(-) diff --git a/lib/PublicInbox/LeiConvert.pm b/lib/PublicInbox/LeiConvert.pm index 9d2479b0..8f628562 100644 --- a/lib/PublicInbox/LeiConvert.pm +++ b/lib/PublicInbox/LeiConvert.pm @@ -33,9 +33,9 @@ sub process_inputs { # via wq_do local $PublicInbox::DS::in_loop = 0; # force synchronous awaitpid $self->SUPER::process_inputs; my $lei = $self->{lei}; - delete $lei->{1}; my $l2m = delete $lei->{l2m}; - delete $self->{wcb}; # commit + delete $self->{wcb}; # may close connections + $l2m->finish_output($lei) if $l2m; if (my $v2w = delete $lei->{v2w}) { $v2w->done } # may die my $nr_w = delete($l2m->{-nr_write}) // 0; my $d = (delete($l2m->{-nr_seen}) // 0) - $nr_w; diff --git a/lib/PublicInbox/LeiToMail.pm b/lib/PublicInbox/LeiToMail.pm index 0d62888d..007191bb 100644 --- a/lib/PublicInbox/LeiToMail.pm +++ b/lib/PublicInbox/LeiToMail.pm @@ -609,6 +609,17 @@ sub _pre_augment_mbox { undef; } +sub finish_output { + my ($self, $lei) = @_; + my $out = delete $lei->{1} // die 'BUG: no lei->{1}'; + my $old = delete $lei->{old_1}; + $lei->{1} = $old if $old; + return if $out->close; # reaps gzip|pigz|xz|bzip2 + my $msg = "E: Error closing $lei->{ovv}->{dst}"; + $? ? $lei->child_error($?) : ($msg .= " ($!)"); + die $msg; +} + sub _do_augment_mbox { my ($self, $lei) = @_; return unless $self->{seekable}; diff --git a/lib/PublicInbox/LeiXSearch.pm b/lib/PublicInbox/LeiXSearch.pm index 5e36c11a..cee3ad07 100644 --- a/lib/PublicInbox/LeiXSearch.pm +++ b/lib/PublicInbox/LeiXSearch.pm @@ -393,14 +393,7 @@ sub query_done { # EOF callback for main daemon $lei->sto_done_request; $lei->{ovv}->ovv_end($lei); if ($l2m) { # close() calls LeiToMail reap_compress - if (my $out = delete $lei->{old_1}) { - if (my $mbout = $lei->{1}) { # compressor pipe process - $mbout->close or die <<""; -Error closing $lei->{ovv}->{dst}: \$!=$! \$?=$? - - } - $lei->{1} = $out; - } + $l2m->finish_output($lei); if ($l2m->lock_free) { $l2m->poke_dst; $lei->poke_mua;