From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: X-Spam-Status: No, score=-4.2 required=3.0 tests=ALL_TRUSTED,AWL,BAYES_00, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, T_SCC_BODY_TEXT_LINE shortcircuit=no autolearn=ham autolearn_force=no version=3.4.6 Received: from localhost (dcvr.yhbt.net [127.0.0.1]) by dcvr.yhbt.net (Postfix) with ESMTP id AACFA1F566; Tue, 4 Jun 2024 22:25:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=80x24.org; s=selector1; t=1717539920; bh=lRyJfow/vqsbf7Kvs6Zs3+d5ceF0TLkxvYp8CPvP3OY=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=l6DO8aWdgY2sK6EEf869DG1pb4PmmYS5bm4LGb9GOuUulVnzryNOBjSg6S6Nhx5V2 5mC4FzBCHLd5tFmh2oWBn2ocoSXBoI1R5QBC5kKe/xWVIW3nm5oAIq6InhwNuH0vYS OAV4Zks5ZsMRKkLMZVg7bk4Vb2MFCO7TT9H2AoYk= Date: Tue, 4 Jun 2024 22:25:20 +0000 From: Eric Wong To: Jonathan Corbet Cc: meta@public-inbox.org Subject: [PATCH] mda: do not auto-create Xapian indices Message-ID: <20240604222520.M507257@dcvr> References: <87ed9c1ywx.fsf@meer.lwn.net> <20240604170735.M954932@dcvr> <87wmn4zkx4.fsf@meer.lwn.net> <20240604172403.M156885@dcvr> <87sexszk6s.fsf@meer.lwn.net> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <87sexszk6s.fsf@meer.lwn.net> List-Id: Jonathan Corbet wrote: > Eric Wong writes: > > -mda does the detection, too, just every time. I haven't used > > it much with -extindex but I think it there's tests for it so it works... > > Interesting...for me it recreates the 0/1/2 directories unless I edit > the config file explicitly. Oops, yeah, that's a bug. -learn did it right, but not -mda This should fix it: -------8<------ Subject: [PATCH] mda: do not auto-create Xapian indices As with -learn, -mda now detects indexlevel=basic without an explicit config setting for inboxes which only have SQLite files. Omitting indexlevel=basic in the config file allows users to reduce configuration file size (and RAM usage). We'll also ensure completely unindexed v1 inboxes can stay unindexed despite the default being indexlevel=full. --- script/public-inbox-mda | 1 + t/mda.t | 30 ++++++++++++++++++++++++++++++ t/v2mda.t | 14 ++++++++++++++ 3 files changed, 45 insertions(+) diff --git a/script/public-inbox-mda b/script/public-inbox-mda index b463b07b..74202912 100755 --- a/script/public-inbox-mda +++ b/script/public-inbox-mda @@ -72,6 +72,7 @@ if (!scalar(@$dests)) { my $err; @$dests = grep { my $ibx = PublicInbox::InboxWritable->new($_); + $ibx->{indexlevel} = $ibx->detect_indexlevel; eval { $ibx->assert_usable_dir }; if ($@) { warn $@; diff --git a/t/mda.t b/t/mda.t index 1d9e237b..51aae231 100644 --- a/t/mda.t +++ b/t/mda.t @@ -396,6 +396,36 @@ EOM @xap = grep(!m!/over\.sqlite3!, glob("$maindir/public-inbox/xapian*/*")); is_deeply(\@xap, [], 'no Xapian files created by -learn'); + + $in = <<'EOM'; +From: a@example.com +To: updated-address@example.com +Subject: basic message for mda +Date: Fri, 02 Oct 1993 00:00:00 +0000 +Message-ID: + +basic +EOM + local $ENV{ORIGINAL_RECIPIENT} = $addr; + ok run_script(['-mda'], undef, $rdr), '-mda for basic'; + @xap = grep(!m!/over\.sqlite3!, + glob("$maindir/public-inbox/xapian*/*")); + is_deeply \@xap, [], 'no Xapian files created by -mda'; + + # try ensure completely unindexed v1 stays unindexed + remove_tree "$maindir/public-inbox"; + $in = <<'EOM'; +From: a@example.com +To: updated-address@example.com +Subject: unnidexed message for mda +Date: Fri, 02 Oct 1993 00:00:00 +0000 +Message-ID: + +unindexed +EOM + + ok run_script(['-mda'], undef, $rdr), '-mda for unindexed'; + ok !-e "$maindir/public-inbox", 'no v1 index created by default'; }; done_testing(); diff --git a/t/v2mda.t b/t/v2mda.t index b7d177b2..cecc9722 100644 --- a/t/v2mda.t +++ b/t/v2mda.t @@ -119,6 +119,20 @@ EOM ok($smsg, 'ham message learned w/ indexlevel=basic'); @shards = grep(m!/[0-9]+\z!, glob("$ibx->{inboxdir}/xap*/*")); is_deeply(\@shards, [], 'not converted to medium/full after learn'); + + $rdr->{0} = \<<'EOM'; +From: a@example.com +To: test@example.com +Subject: this is a message for -mda to stay basic +Date: Fri, 02 Oct 1993 00:00:00 +0000 +Message-ID: + +yum +EOM + ok run_script(['-mda'], undef, $rdr), '-learn runs on basic' + or diag $err; + @shards = grep m!/[0-9]+\z!, glob("$ibx->{inboxdir}/xap*/*"); + is_deeply \@shards, [], 'not converted to medium/full after -mda'; } done_testing();