From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net X-Spam-Level: X-Spam-Status: No, score=-4.0 required=3.0 tests=ALL_TRUSTED,BAYES_00 shortcircuit=no autolearn=ham autolearn_force=no version=3.4.2 Received: from localhost (dcvr.yhbt.net [127.0.0.1]) by dcvr.yhbt.net (Postfix) with ESMTP id E49D51F4B4 for ; Mon, 1 Feb 2021 08:28:33 +0000 (UTC) From: Eric Wong To: meta@public-inbox.org Subject: [PATCH 00/21] lei2mail worker segfault finally fixed Date: Sun, 31 Jan 2021 22:28:12 -1000 Message-Id: <20210201082833.3293-1-e@80x24.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit List-Id: This lei2mail segfault turned out to be an old Perl 5 quirk which plagued many before me. It was not consistently reproducible, and random changes seemed to make it happen more or less frequently. There were several times when I thought I fixed it (and maybe this is still one of them!) only to have it pop up again. Still, I found many other little bugs and improvements worth doing along the way. Hope things go more smoothly in the future... Anyways, [PATCH 18/21] is the fix (and I'll followup with more on how I found the fix). 19/21 is purely defensive future-proofing. Eric Wong (21): lei: more consistent dedupe and ovv_buf init ipc: switch wq to use the event loop lei: remove per-child SIG{__WARN__} lei: remove SIGPIPE handler ipc: more helpful ETOOMANYREFS error messages lei: remove syslog dependency sharedkv: release {dbh} before rmtree lei: keep $lei around until workers are reaped lei_dedupe: use Digest::SHA lei_xsearch: load PublicInbox::Smsg lei: deep clone {ovv} for l2m workers sharedkv: lock and explicitly disconnect {dbh} lei: increase initial timeout sharedkv: use lock_for_scope_fast lei_to_mail: reduce spew on Maildir removal sharedkv: do not set cache_size by default import: reap git-config(1) synchronously ds: guard against stack-not-refcounted quirk of Perl 5 ds: next_tick: avoid $_ in top-level loop iterator lei: avoid ETOOMANYREFS, cleanup imports doc: note optional BSD::Resource use Documentation/public-inbox-config.pod | 2 +- INSTALL | 6 ++ MANIFEST | 2 + lib/PublicInbox/DS.pm | 12 ++-- lib/PublicInbox/IPC.pm | 43 +++++++----- lib/PublicInbox/Import.pm | 1 + lib/PublicInbox/LEI.pm | 95 +++++++++++++++------------ lib/PublicInbox/LeiDedupe.pm | 6 +- lib/PublicInbox/LeiExternal.pm | 3 +- lib/PublicInbox/LeiOverview.pm | 51 +++++++------- lib/PublicInbox/LeiToMail.pm | 84 +++++++++++------------ lib/PublicInbox/LeiXSearch.pm | 36 +++++----- lib/PublicInbox/Lock.pm | 17 +++++ lib/PublicInbox/SharedKV.pm | 33 +++++++--- lib/PublicInbox/WQWorker.pm | 34 ++++++++++ script/lei | 28 +++++--- t/lei_to_mail.t | 31 +++++---- xt/stress-sharedkv.t | 50 ++++++++++++++ 18 files changed, 342 insertions(+), 192 deletions(-) create mode 100644 lib/PublicInbox/WQWorker.pm create mode 100644 xt/stress-sharedkv.t