* [PATCH 0/4] lei reindex-related stuff
@ 2022-08-19 9:07 71% Eric Wong
2022-08-19 9:07 62% ` [PATCH 2/4] tests: add some basic "lei reindex" tests Eric Wong
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2022-08-19 9:07 UTC (permalink / raw)
To: meta
1/4 is an important fix
And I'm still unusually stressed out about how to deal with
ancient, pre-release stuff from lei. --rethread may happen
another time, since AFAIK it's not necessary, right now as there
were no threading fixes required since lei...
Eric Wong (4):
lei reindex: account for parallel lei/store users
tests: add some basic "lei reindex" tests
smsg: ->populate falls back to old {ds}/{ts} values
lei/store: reindex culls over-indexed messages
MANIFEST | 1 +
lib/PublicInbox/LeiStore.pm | 14 ++++++++++++--
lib/PublicInbox/Smsg.pm | 6 ++++--
t/lei-index.t | 12 +++++++++++-
t/lei-reindex.t | 12 ++++++++++++
5 files changed, 40 insertions(+), 5 deletions(-)
create mode 100644 t/lei-reindex.t
^ permalink raw reply [relevance 71%]
* [PATCH 2/4] tests: add some basic "lei reindex" tests
2022-08-19 9:07 71% [PATCH 0/4] lei reindex-related stuff Eric Wong
@ 2022-08-19 9:07 62% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2022-08-19 9:07 UTC (permalink / raw)
To: meta
This is a bit hard-to-test, but at least we must ensure
volatile-metadata is preserved.
---
MANIFEST | 1 +
t/lei-index.t | 12 +++++++++++-
t/lei-reindex.t | 12 ++++++++++++
3 files changed, 24 insertions(+), 1 deletion(-)
create mode 100644 t/lei-reindex.t
diff --git a/MANIFEST b/MANIFEST
index 27e4c4e0..43128382 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -477,6 +477,7 @@ t/lei-q-remote-import.t
t/lei-q-save.t
t/lei-q-thread.t
t/lei-refresh-mail-sync.t
+t/lei-reindex.t
t/lei-sigpipe.t
t/lei-tag.t
t/lei-up.t
diff --git a/t/lei-index.t b/t/lei-index.t
index aab8f7e6..c31b1c3c 100644
--- a/t/lei-index.t
+++ b/t/lei-index.t
@@ -1,5 +1,5 @@
#!perl -w
-# Copyright (C) 2021 all contributors <meta@public-inbox.org>
+# Copyright (C) all contributors <meta@public-inbox.org>
# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
use strict; use v5.10.1; use PublicInbox::TestCommon;
use File::Spec;
@@ -85,6 +85,10 @@ test_lei({ tmpdir => $tmpdir }, sub {
lei_ok qw(q m:multipart-html-sucks@11);
is_deeply(json_utf8->decode($lei_out)->[0]->{'kw'},
['seen'], 'keyword set');
+ lei_ok 'reindex';
+ lei_ok qw(q m:multipart-html-sucks@11);
+ is_deeply(json_utf8->decode($lei_out)->[0]->{'kw'},
+ ['seen'], 'keyword still set after reindex');
$srv->{nntpd} and
lei_ok('index', "nntp://$srv->{nntp_host_port}/t.v2");
@@ -104,6 +108,12 @@ test_lei({ tmpdir => $tmpdir }, sub {
my $t = xqx(['git', "--git-dir=$store_path/ALL.git",
qw(cat-file -t), $res_b->{blob}]);
is($t, "blob\n", 'got blob');
+
+ lei_ok('reindex');
+ lei_ok qw(q m:multipart-html-sucks@11);
+ $res_a = json_utf8->decode($lei_out)->[0];
+ is_deeply($res_a->{'kw'}, ['seen'],
+ 'keywords still set after reindex');
});
done_testing;
diff --git a/t/lei-reindex.t b/t/lei-reindex.t
new file mode 100644
index 00000000..73346ee8
--- /dev/null
+++ b/t/lei-reindex.t
@@ -0,0 +1,12 @@
+#!perl -w
+# Copyright (C) all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+use v5.12; use PublicInbox::TestCommon;
+require_mods(qw(lei));
+my ($tmpdir, $for_destroy) = tmpdir;
+test_lei(sub {
+ ok(!lei('reindex'), 'reindex fails w/o store');
+ like $lei_err, qr/nothing indexed/, "`nothing indexed' noted";
+});
+
+done_testing;
^ permalink raw reply related [relevance 62%]
* Re: [PATCH 4/4] lei reindex: new command to reindex lei/store
2022-08-17 9:33 69% ` [PATCH 4/4] lei reindex: new command to reindex lei/store Eric Wong
@ 2022-08-18 7:22 90% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2022-08-18 7:22 UTC (permalink / raw)
To: meta
Eric Wong <e@80x24.org> wrote:
> index d49746cb..277ed6bd 100644
> --- a/lib/PublicInbox/LeiStore.pm
> +++ b/lib/PublicInbox/LeiStore.pm
> @@ -335,6 +335,36 @@ sub _docids_and_maybe_kw ($$) {
> ($docids, [ sort keys %$kw ]);
> }
>
> +sub _reindex_1 { # git->cat_async callback
> + my ($bref, $hex, $type, $size, $smsg) = @_;
> + my ($self, $eidx, $tl) = delete @$smsg{qw(-self -eidx -tl)};
> + $bref //= _lms_rw($self)->local_blob($hex, 1);
> + if ($bref) {
> + my $eml = PublicInbox::Eml->new($bref);
> + $smsg->{-merge_vmd} = 1; # preserve existing keywords
> + $eidx->idx_shard($smsg->{num})->index_eml($eml, $smsg);
> + } else {
> + warn("E: $type $hex\n");
This path has been worrying me a bit, I hit it quite a bit on
one of my systems since there was a time when external-only
messages were fully-indexed inside lei/store. Nowadays,
duplicate indexing is avoided...
^ permalink raw reply [relevance 90%]
* [PATCH 3/4] lei/store: reduce work when accessing mail_sync.sqlite3
2022-08-17 9:33 71% [PATCH 0/4] lei reindex, minor tweaks Eric Wong
2022-08-17 9:33 71% ` [PATCH 2/4] lei inspect: less scary exception for invalid "docid:" inspect Eric Wong
@ 2022-08-17 9:33 71% ` Eric Wong
2022-08-17 9:33 69% ` [PATCH 4/4] lei reindex: new command to reindex lei/store Eric Wong
2 siblings, 0 replies; 200+ results
From: Eric Wong @ 2022-08-17 9:33 UTC (permalink / raw)
To: meta
There's no need to initialize eidx if we already have an open
handle for mail_sync.sqlite3
---
lib/PublicInbox/LeiStore.pm | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/lib/PublicInbox/LeiStore.pm b/lib/PublicInbox/LeiStore.pm
index 66049dfe..d49746cb 100644
--- a/lib/PublicInbox/LeiStore.pm
+++ b/lib/PublicInbox/LeiStore.pm
@@ -255,13 +255,13 @@ sub remove_eml_vmd { # remove just the VMD
sub _lms_rw ($) { # it is important to have eidx processes open before lms
my ($self) = @_;
- my ($eidx, $tl) = eidx_init($self);
- $self->{lms} //= do {
+ $self->{lms} // do {
require PublicInbox::LeiMailSync;
+ my ($eidx, $tl) = eidx_init($self);
my $f = "$self->{priv_eidx}->{topdir}/mail_sync.sqlite3";
my $lms = PublicInbox::LeiMailSync->new($f);
$lms->lms_write_prepare;
- $lms;
+ $self->{lms} = $lms;
};
}
^ permalink raw reply related [relevance 71%]
* [PATCH 4/4] lei reindex: new command to reindex lei/store
2022-08-17 9:33 71% [PATCH 0/4] lei reindex, minor tweaks Eric Wong
2022-08-17 9:33 71% ` [PATCH 2/4] lei inspect: less scary exception for invalid "docid:" inspect Eric Wong
2022-08-17 9:33 71% ` [PATCH 3/4] lei/store: reduce work when accessing mail_sync.sqlite3 Eric Wong
@ 2022-08-17 9:33 69% ` Eric Wong
2022-08-18 7:22 90% ` Eric Wong
2 siblings, 1 reply; 200+ results
From: Eric Wong @ 2022-08-17 9:33 UTC (permalink / raw)
To: meta
---
Documentation/lei-reindex.pod | 47 +++++++++++++++++++++++++++++++++
MANIFEST | 2 ++
lib/PublicInbox/LEI.pm | 2 ++
lib/PublicInbox/LeiReindex.pm | 49 +++++++++++++++++++++++++++++++++++
lib/PublicInbox/LeiStore.pm | 32 ++++++++++++++++++++++-
5 files changed, 131 insertions(+), 1 deletion(-)
create mode 100644 Documentation/lei-reindex.pod
create mode 100644 lib/PublicInbox/LeiReindex.pm
diff --git a/Documentation/lei-reindex.pod b/Documentation/lei-reindex.pod
new file mode 100644
index 00000000..3a5861c4
--- /dev/null
+++ b/Documentation/lei-reindex.pod
@@ -0,0 +1,47 @@
+=head1 NAME
+
+lei-reindex - reindex messages already in lei/store
+
+=head1 SYNOPSIS
+
+lei reindex [OPTIONS]
+
+=head1 DESCRIPTION
+
+Forces a re-index of all messages previously-indexed by L<lei-import(1)>
+or L<lei-index(1)>. This can be used for in-place upgrades and bugfixes
+while other processes are querying the store. Keep in mind this roughly
+doubles the size of the already-large Xapian database.
+
+It does not re-index messages in externals, using the C<--reindex>
+switch of L<public-inbox-index(1)> or L<public-inbox-extindex(1)> is
+needed for that.
+
+=head1 OPTIONS
+
+=over
+
+=item -q
+
+=item --quiet
+
+Suppress feedback messages.
+
+=back
+
+=head1 CONTACT
+
+Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org>
+
+The mail archives are hosted at L<https://public-inbox.org/meta/> and
+L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/>
+
+=head1 COPYRIGHT
+
+Copyright all contributors L<mailto:meta@public-inbox.org>
+
+License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>
+
+=head1 SEE ALSO
+
+L<lei-index(1)>, L<lei-import(1)>
diff --git a/MANIFEST b/MANIFEST
index cc0a9a4c..27e4c4e0 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -56,6 +56,7 @@ Documentation/lei-p2q.pod
Documentation/lei-q.pod
Documentation/lei-rediff.pod
Documentation/lei-refresh-mail-sync.pod
+Documentation/lei-reindex.pod
Documentation/lei-rm-watch.pod
Documentation/lei-rm.pod
Documentation/lei-security.pod
@@ -256,6 +257,7 @@ lib/PublicInbox/LeiPmdir.pm
lib/PublicInbox/LeiQuery.pm
lib/PublicInbox/LeiRediff.pm
lib/PublicInbox/LeiRefreshMailSync.pm
+lib/PublicInbox/LeiReindex.pm
lib/PublicInbox/LeiRemote.pm
lib/PublicInbox/LeiRm.pm
lib/PublicInbox/LeiRmWatch.pm
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 595b3fa9..8a3a3ab6 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -253,6 +253,8 @@ our %CMD = ( # sorted in order of importance/use:
'forget-watch' => [ '{WATCH_NUMBER|--prune}', 'stop and forget a watch',
qw(prune), @c_opt ],
+'reindex' => [ '', 'reindex all locally-indexed messages', @c_opt ],
+
'index' => [ 'LOCATION...', 'one-time index from URL or filesystem',
qw(in-format|F=s kw! offset=i recursive|r exclude=s include|I=s
verbose|v+ incremental!), @net_opt, # mainly for --proxy=
diff --git a/lib/PublicInbox/LeiReindex.pm b/lib/PublicInbox/LeiReindex.pm
new file mode 100644
index 00000000..3f109f33
--- /dev/null
+++ b/lib/PublicInbox/LeiReindex.pm
@@ -0,0 +1,49 @@
+# Copyright all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+
+# "lei reindex" command to reindex everything in lei/store
+package PublicInbox::LeiReindex;
+use v5.12;
+use parent qw(PublicInbox::IPC);
+
+sub reindex_full {
+ my ($lei) = @_;
+ my $sto = $lei->{sto};
+ my $max = $sto->search->over(1)->max;
+ $lei->qerr("# reindexing 1..$max");
+ $sto->wq_do('reindex_art', $_) for (1..$max);
+}
+
+sub reindex_store { # via wq_do
+ my ($self) = @_;
+ my ($lei, $argv) = delete @$self{qw(lei argv)};
+ if (!@$argv) {
+ reindex_full($lei);
+ }
+}
+
+sub lei_reindex {
+ my ($lei, @argv) = @_;
+ my $sto = $lei->_lei_store or return $lei->fail('nothing indexed');
+ $sto->write_prepare($lei);
+ my $self = bless { lei => $lei, argv => \@argv }, __PACKAGE__;
+ my ($op_c, $ops) = $lei->workers_start($self, 1);
+ $lei->{wq1} = $self;
+ $lei->wait_wq_events($op_c, $ops);
+ $self->wq_do('reindex_store');
+ $self->wq_close;
+}
+
+sub _lei_wq_eof { # EOF callback for main lei daemon
+ my ($lei) = @_;
+ $lei->{sto}->wq_do('reindex_done');
+ $lei->wq_eof;
+}
+
+sub ipc_atfork_child {
+ my ($self) = @_;
+ $self->{lei}->_lei_atfork_child;
+ $self->SUPER::ipc_atfork_child;
+}
+
+1;
diff --git a/lib/PublicInbox/LeiStore.pm b/lib/PublicInbox/LeiStore.pm
index d49746cb..277ed6bd 100644
--- a/lib/PublicInbox/LeiStore.pm
+++ b/lib/PublicInbox/LeiStore.pm
@@ -1,4 +1,4 @@
-# Copyright (C) 2020-2021 all contributors <meta@public-inbox.org>
+# Copyright (C) all contributors <meta@public-inbox.org>
# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
#
# Local storage (cache/memo) for lei(1), suitable for personal/private
@@ -335,6 +335,36 @@ sub _docids_and_maybe_kw ($$) {
($docids, [ sort keys %$kw ]);
}
+sub _reindex_1 { # git->cat_async callback
+ my ($bref, $hex, $type, $size, $smsg) = @_;
+ my ($self, $eidx, $tl) = delete @$smsg{qw(-self -eidx -tl)};
+ $bref //= _lms_rw($self)->local_blob($hex, 1);
+ if ($bref) {
+ my $eml = PublicInbox::Eml->new($bref);
+ $smsg->{-merge_vmd} = 1; # preserve existing keywords
+ $eidx->idx_shard($smsg->{num})->index_eml($eml, $smsg);
+ } else {
+ warn("E: $type $hex\n");
+ }
+}
+
+sub reindex_art {
+ my ($self, $art) = @_;
+ my ($eidx, $tl) = eidx_init($self);
+ my $smsg = $eidx->{oidx}->get_art($art) // return;
+ return if $smsg->{bytes} == 0; # external-only message
+ @$smsg{qw(-self -eidx -tl)} = ($self, $eidx, $tl);
+ $eidx->git->cat_async($smsg->{blob} // die("no blob (#$art)"),
+ \&_reindex_1, $smsg);
+}
+
+sub reindex_done {
+ my ($self) = @_;
+ my ($eidx, $tl) = eidx_init($self);
+ $eidx->git->async_wait_all;
+ # ->done to be called via sto_done_request
+}
+
sub add_eml {
my ($self, $eml, $vmd, $xoids) = @_;
my $im = $self->{-fake_im} // $self->importer; # may create new epoch
^ permalink raw reply related [relevance 69%]
* [PATCH 0/4] lei reindex, minor tweaks
@ 2022-08-17 9:33 71% Eric Wong
2022-08-17 9:33 71% ` [PATCH 2/4] lei inspect: less scary exception for invalid "docid:" inspect Eric Wong
` (2 more replies)
0 siblings, 3 replies; 200+ results
From: Eric Wong @ 2022-08-17 9:33 UTC (permalink / raw)
To: meta
Reindex is far from complete, and probably needs a compact, and
some other fixups for old data + rethread support.
But avoiding false positives from base-85 is nice.
Eric Wong (4):
searchidx: fix spelling error in comment
lei inspect: less scary exception for invalid "docid:" inspect
lei/store: reduce work when accessing mail_sync.sqlite3
lei reindex: new command to reindex lei/store
Documentation/lei-reindex.pod | 47 +++++++++++++++++++++++++++++++++
MANIFEST | 2 ++
lib/PublicInbox/LEI.pm | 2 ++
lib/PublicInbox/LeiInspect.pm | 5 ++--
lib/PublicInbox/LeiReindex.pm | 49 +++++++++++++++++++++++++++++++++++
lib/PublicInbox/LeiStore.pm | 38 ++++++++++++++++++++++++---
lib/PublicInbox/SearchIdx.pm | 2 +-
7 files changed, 138 insertions(+), 7 deletions(-)
create mode 100644 Documentation/lei-reindex.pod
create mode 100644 lib/PublicInbox/LeiReindex.pm
^ permalink raw reply [relevance 71%]
* [PATCH 2/4] lei inspect: less scary exception for invalid "docid:" inspect
2022-08-17 9:33 71% [PATCH 0/4] lei reindex, minor tweaks Eric Wong
@ 2022-08-17 9:33 71% ` Eric Wong
2022-08-17 9:33 71% ` [PATCH 3/4] lei/store: reduce work when accessing mail_sync.sqlite3 Eric Wong
2022-08-17 9:33 69% ` [PATCH 4/4] lei reindex: new command to reindex lei/store Eric Wong
2 siblings, 0 replies; 200+ results
From: Eric Wong @ 2022-08-17 9:33 UTC (permalink / raw)
To: meta
It still says "Exception:", but doesn't pointlessly print out
the line number and file of the exception when it's a data/input
problem, and not a code problem on our end.
---
lib/PublicInbox/LeiInspect.pm | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/lib/PublicInbox/LeiInspect.pm b/lib/PublicInbox/LeiInspect.pm
index d7775d4b..d1dca4ef 100644
--- a/lib/PublicInbox/LeiInspect.pm
+++ b/lib/PublicInbox/LeiInspect.pm
@@ -1,4 +1,4 @@
-# Copyright (C) 2021 all contributors <meta@public-inbox.org>
+# Copyright (C) all contributors <meta@public-inbox.org>
# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
# "lei inspect" general purpose inspector for stuff in SQLite and
@@ -235,7 +235,8 @@ sub inspect_argv { # via wq_do
$lei->{1}->autoflush(0);
$lei->out('[') if $multi;
while (defined(my $x = shift @$argv)) {
- inspect1($lei, $x, scalar(@$argv)) or return;
+ eval { inspect1($lei, $x, scalar(@$argv)) or return };
+ warn "E: $@\n" if $@;
}
$lei->out(']') if $multi;
}
^ permalink raw reply related [relevance 71%]
* [PATCH] lei: do not wait for sto->done on disconnected EOF
@ 2022-08-16 3:44 71% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2022-08-16 3:44 UTC (permalink / raw)
To: meta
lei-daemon (the top-level daemon process) should not have
synchronous waits, and this was causing a deadlock with
interrupted commands. There may still be a bug lurking in
lei/store despite this fix, though. I originally thought commit
fd261b9e65674505 (lei_store_err: use level-trigger for error pipe, 2022-08-15)
was sufficient, but at least this change is needed, as well.
---
lib/PublicInbox/LEI.pm | 11 ++++-------
1 file changed, 4 insertions(+), 7 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index d81ca296..595b3fa9 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -1520,13 +1520,10 @@ sub sto_done_request {
return unless $lei->{sto};
local $current_lei = $lei;
my $sock = $wq ? $wq->{lei_sock} : undef;
- eval {
- if ($sock //= $lei->{sock}) { # issue, async wait
- $lei->{sto}->wq_io_do('done', [ $sock ]);
- } else { # forcibly wait
- my $wait = $lei->{sto}->wq_do('done');
- }
- };
+ $sock //= $lei->{sock};
+ my @io;
+ push(@io, $sock) if $sock; # async wait iff possible
+ eval { $lei->{sto}->wq_io_do('done', \@io) };
warn($@) if $@;
}
^ permalink raw reply related [relevance 71%]
* [PATCH 0/2] lei inotify/EVFILT_VNODE deadlock fix
@ 2022-07-19 22:42 71% Eric Wong
2022-07-19 22:42 51% ` [PATCH 1/2] lei: avoid deadlock on inotify/EVFILT_VNODE wakeups Eric Wong
2022-07-19 22:42 71% ` [PATCH 2/2] lei note-event: inline note_event_arm_done Eric Wong
0 siblings, 2 replies; 200+ results
From: Eric Wong @ 2022-07-19 22:42 UTC (permalink / raw)
To: meta
For every bug I fix, I feel dumber for introducing it :<
Eric Wong (2):
lei: avoid deadlock on inotify/EVFILT_VNODE wakeups
lei note-event: inline note_event_arm_done
MANIFEST | 1 +
lib/PublicInbox/IPC.pm | 26 +++++++++++++++--
lib/PublicInbox/LeiNoteEvent.pm | 15 ++++------
lib/PublicInbox/WQBlocked.pm | 49 +++++++++++++++++++++++++++++++++
4 files changed, 79 insertions(+), 12 deletions(-)
create mode 100644 lib/PublicInbox/WQBlocked.pm
^ permalink raw reply [relevance 71%]
* [PATCH 2/2] lei note-event: inline note_event_arm_done
2022-07-19 22:42 71% [PATCH 0/2] lei inotify/EVFILT_VNODE deadlock fix Eric Wong
2022-07-19 22:42 51% ` [PATCH 1/2] lei: avoid deadlock on inotify/EVFILT_VNODE wakeups Eric Wong
@ 2022-07-19 22:42 71% ` Eric Wong
1 sibling, 0 replies; 200+ results
From: Eric Wong @ 2022-07-19 22:42 UTC (permalink / raw)
To: meta
This was a single-caller sub since 47d4e53734820b4e
(lei_mail_sync: rely on flock(2), avoid IPC, 2021-09-18)
and unlikely to be used further, so inline it and save
a few KB of memory.
---
lib/PublicInbox/LeiNoteEvent.pm | 10 ++--------
1 file changed, 2 insertions(+), 8 deletions(-)
diff --git a/lib/PublicInbox/LeiNoteEvent.pm b/lib/PublicInbox/LeiNoteEvent.pm
index 93f80116..8581bd9a 100644
--- a/lib/PublicInbox/LeiNoteEvent.pm
+++ b/lib/PublicInbox/LeiNoteEvent.pm
@@ -27,13 +27,6 @@ sub flush_task { # PublicInbox::DS timer callback
for my $lei (values %$todo) { flush_lei($lei) }
}
-# sets a timer to flush
-sub note_event_arm_done ($) {
- my ($lei) = @_;
- PublicInbox::DS::add_uniq_timer('flush_timer', 5, \&flush_task);
- $to_flush->{$lei->{cfg}->{'-f'}} //= $lei;
-}
-
sub eml_event ($$$$) {
my ($self, $eml, $vmd, $state) = @_;
my $sto = $self->{lei}->{sto};
@@ -92,7 +85,8 @@ sub lei_note_event {
$jobs = 4 if $jobs > 4; # same default as V2Writable
my ($op_c, $ops) = $lei->workers_start($wq, $jobs);
$lei->wait_wq_events($op_c, $ops);
- note_event_arm_done($lei);
+ PublicInbox::DS::add_uniq_timer('flush_timer', 5, \&flush_task);
+ $to_flush->{$lei->{cfg}->{'-f'}} //= $lei;
$wq->prepare_nonblock;
$lei->{lne} = $wq;
};
^ permalink raw reply related [relevance 71%]
* [PATCH 1/2] lei: avoid deadlock on inotify/EVFILT_VNODE wakeups
2022-07-19 22:42 71% [PATCH 0/2] lei inotify/EVFILT_VNODE deadlock fix Eric Wong
@ 2022-07-19 22:42 51% ` Eric Wong
2022-07-19 22:42 71% ` [PATCH 2/2] lei note-event: inline note_event_arm_done Eric Wong
1 sibling, 0 replies; 200+ results
From: Eric Wong @ 2022-07-19 22:42 UTC (permalink / raw)
To: meta
Enqueuing "note-event" requests from the DS event loop must
not wait on workers being able to drain the queue quickly
enough. Thus we make the SOCK_SEQPACKET writes nonblocking
and rely on the lei-daemon event loop to enqueue writes.
This is a unique problem for "note-event" since it reuses
workers in between commands, while most lei commands currently
fork off new workers.
---
MANIFEST | 1 +
lib/PublicInbox/IPC.pm | 26 +++++++++++++++--
lib/PublicInbox/LeiNoteEvent.pm | 5 ++--
lib/PublicInbox/WQBlocked.pm | 49 +++++++++++++++++++++++++++++++++
4 files changed, 77 insertions(+), 4 deletions(-)
create mode 100644 lib/PublicInbox/WQBlocked.pm
diff --git a/MANIFEST b/MANIFEST
index 2cbe66b7..923f5147 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -337,6 +337,7 @@ lib/PublicInbox/V2Writable.pm
lib/PublicInbox/View.pm
lib/PublicInbox/ViewDiff.pm
lib/PublicInbox/ViewVCS.pm
+lib/PublicInbox/WQBlocked.pm
lib/PublicInbox/WQWorker.pm
lib/PublicInbox/WWW.pm
lib/PublicInbox/WWW.pod
diff --git a/lib/PublicInbox/IPC.pm b/lib/PublicInbox/IPC.pm
index 67e86a43..74862673 100644
--- a/lib/PublicInbox/IPC.pm
+++ b/lib/PublicInbox/IPC.pm
@@ -1,4 +1,4 @@
-# Copyright (C) 2020-2021 all contributors <meta@public-inbox.org>
+# Copyright (C) all contributors <meta@public-inbox.org>
# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
# base class for remote IPC calls and workqueues, requires Storable or Sereal
@@ -43,7 +43,7 @@ if ($enc && $dec) { # should be custom ops
}
my $recv_cmd = PublicInbox::Spawn->can('recv_cmd4');
-my $send_cmd = PublicInbox::Spawn->can('send_cmd4') // do {
+our $send_cmd = PublicInbox::Spawn->can('send_cmd4') // do {
require PublicInbox::CmdIPC4;
$recv_cmd //= PublicInbox::CmdIPC4->can('recv_cmd4');
PublicInbox::CmdIPC4->can('send_cmd4');
@@ -348,6 +348,24 @@ sub wq_do {
}
}
+sub prepare_nonblock {
+ ($_[0]->{-wq_s1} // die 'BUG: no {-wq_s1}')->blocking(0);
+ $_[0]->{-reap_async} or die 'BUG: {-reap_async} needed for nonblock';
+ require PublicInbox::WQBlocked;
+}
+
+sub wq_nonblock_do { # always async
+ my ($self, $sub, @args) = @_;
+ my $buf = ipc_freeze([$sub, @args]);
+ if ($self->{wqb}) { # saturated once, assume saturated forever
+ $self->{wqb}->flush_send($buf);
+ } else {
+ $send_cmd->($self->{-wq_s1}, [], $buf, MSG_EOR) //
+ ($!{EAGAIN} ? PublicInbox::WQBlocked->new($self, $buf)
+ : croak("sendmsg: $!"));
+ }
+}
+
sub _wq_worker_start ($$$$) {
my ($self, $oldset, $fields, $one) = @_;
my ($bcast1, $bcast2);
@@ -405,6 +423,10 @@ sub wq_workers_start {
sub wq_close {
my ($self) = @_;
+ if (my $wqb = delete $self->{wqb}) {
+ $self->{-reap_async} or die 'BUG: {-reap_async} unset';
+ $wqb->enq_close;
+ }
delete @$self{qw(-wq_s1 -wq_s2)} or return;
return if $self->{-reap_async};
my @pids = keys %{$self->{-wq_workers}};
diff --git a/lib/PublicInbox/LeiNoteEvent.pm b/lib/PublicInbox/LeiNoteEvent.pm
index db387633..93f80116 100644
--- a/lib/PublicInbox/LeiNoteEvent.pm
+++ b/lib/PublicInbox/LeiNoteEvent.pm
@@ -58,7 +58,7 @@ sub eml_event ($$$$) {
}
}
-sub maildir_event { # via wq_io_do
+sub maildir_event { # via wq_nonblock_do
my ($self, $fn, $vmd, $state) = @_;
if (my $eml = PublicInbox::InboxWritable::eml_from_path($fn)) {
eml_event($self, $eml, $vmd, $state);
@@ -93,6 +93,7 @@ sub lei_note_event {
my ($op_c, $ops) = $lei->workers_start($wq, $jobs);
$lei->wait_wq_events($op_c, $ops);
note_event_arm_done($lei);
+ $wq->prepare_nonblock;
$lei->{lne} = $wq;
};
if ($folder =~ /\Amaildir:/i) {
@@ -101,7 +102,7 @@ sub lei_note_event {
return if index($fl, 'T') >= 0;
my $kw = PublicInbox::MdirReader::flags2kw($fl);
my $vmd = { kw => $kw, sync_info => [ $folder, \$bn ] };
- $self->wq_do('maildir_event', $fn, $vmd, $state);
+ $self->wq_nonblock_do('maildir_event', $fn, $vmd, $state);
} # else: TODO: imap
}
diff --git a/lib/PublicInbox/WQBlocked.pm b/lib/PublicInbox/WQBlocked.pm
new file mode 100644
index 00000000..fbb43600
--- /dev/null
+++ b/lib/PublicInbox/WQBlocked.pm
@@ -0,0 +1,49 @@
+# Copyright (C) all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+
+# non-blocking workqueues, currently used by LeiNoteEvent to track renames
+package PublicInbox::WQBlocked;
+use v5.12;
+use parent qw(PublicInbox::DS);
+use PublicInbox::Syscall qw(EPOLLOUT EPOLLONESHOT);
+use PublicInbox::IPC;
+use Carp ();
+use Socket qw(MSG_EOR);
+
+sub new {
+ my ($cls, $wq, $buf) = @_;
+ my $self = bless { msgq => [$buf], }, $cls;
+ $wq->{wqb} = $self->SUPER::new($wq->{-wq_s1}, EPOLLOUT|EPOLLONESHOT);
+}
+
+sub flush_send {
+ my ($self) = @_;
+ push(@{$self->{msgq}}, $_[1]) if defined($_[1]);
+ while (defined(my $buf = shift @{$self->{msgq}})) {
+ if (ref($buf) eq 'CODE') {
+ $buf->($self); # could be \&PublicInbox::DS::close
+ } else {
+ my $wq_s1 = $self->{sock};
+ my $n = $PublicInbox::IPC::send_cmd->($wq_s1, [], $buf,
+ MSG_EOR);
+ next if defined($n);
+ Carp::croak("sendmsg: $!") unless $!{EAGAIN};
+ PublicInbox::DS::epwait($wq_s1, EPOLLOUT|EPOLLONESHOT);
+ unshift @{$self->{msgq}}, $buf;
+ last; # wait for ->event_step
+ }
+ }
+}
+
+sub enq_close { flush_send($_[0], $_[0]->can('close')) }
+
+sub event_step { # called on EPOLLOUT wakeup
+ my ($self) = @_;
+ eval { flush_send($self) } if $self->{sock};
+ if ($@) {
+ warn $@;
+ $self->close;
+ }
+}
+
+1;
^ permalink raw reply related [relevance 51%]
* Re: lei missing mails
2022-07-11 21:59 68% ` Rob Herring
@ 2022-07-18 23:41 71% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2022-07-18 23:41 UTC (permalink / raw)
To: Rob Herring; +Cc: meta
Rob Herring <robh@kernel.org> wrote:
> On Thu, Jun 30, 2022 at 2:55 AM Eric Wong <e@80x24.org> wrote:
> >
> > Rob Herring <robh@kernel.org> wrote:
> > > On Wed, Jun 29, 2022 at 11:27 AM Eric Wong <e@80x24.org> wrote:
> > > > Rob Herring <robh@kernel.org> wrote:
> > > > > On Wed, Jun 29, 2022 at 10:30 AM Eric Wong <e@80x24.org> wrote:
> > > > > > Rob Herring <robh@kernel.org> wrote:
> > > > > > > Hi,
> > > > > > >
> > > > > > > I'm using lei with lore where I have 2 queries which overlap. Really,
> > > > > > > one is a subset of the other. On those overlapping threads, I'm
> > > > > > > finding that sometimes new messages are written to one mailbox and not
> > > > > > > the other. (At least sometimes, the messages may be missing from all
> > > > > > > mailboxes sometimes too. I'm not certain.) Using --remote-fudge-time
> > > > > > > to force refetching seems to get the missing mails. I haven't found
> > > > > > > anything strange in timestamps of the missing mails, but otherwise am
> > > > > > > not sure how to debug this further. The queries are retrieving full
> > > > > > > threads and the missing mails are in the threads, but not direct
> > > > > > > matches to the queries. I realize that's not a lot of detail to go on.
> > > > > > > Suggestions on debugging this further?
> > > > > >
> > > > > > Is this with 1.8 or 1.7?
> > > > >
> > > > > Commit 68b53c888911 actually. So post 1.8.
> > > >
> > > > OK, thanks for that info.
> > > >
> > > > > > I forgot to note in the release notes, but there were some
> > > > > > SQLite usage-related fixes which could avoid missing messages.
> > > > > >
> > > > > > You'll need "lei daemon-kill" after upgrading to 1.8 to ensure
> > > > > > the new code is running.
> > > > >
> > > > > It's possible I haven't done that since updating though I do vaguely
> > > > > recall seeing something about needing to do that. Is there any way to
> > > > > tell before I restart it?
> > > >
> > > > Not really, but it's pretty cheap to restart (assuming there's no
> > > > long-running jobs).
> > >
> > > I've restarted and just hit this again.
> >
> > Ugh, sorry to hear that :<
> >
> > > > > > What might be interesting is to use the URLs lei prints and
> > > > > > comparing the results w/o lei.
> > >
> > > $ lei up --all
> > > # updating /home/rob/Mail/from-me
> > > # updating /home/rob/Mail/missing-cc
> > > # updating /home/rob/Mail/my-patches
> > > # updating /home/rob/Mail/pci
> > > # https://lore.kernel.org/all/ limiting to 2022-06-27 12:42 -0600 and newer
> > > # https://lore.kernel.org/all/ limiting to 2022-06-27 9:50 -0600 and newer
> > > # https://lore.kernel.org/all/ limiting to 2022-06-27 12:42 -0600 and newer
> > > # /usr/bin/curl -Sf -s -d ''
> > > https://lore.kernel.org/all/?x=m&t=1&q=(dt%3A20220529211430..+AND+(f%3Arobh%40kernel.org+OR+f%3Arobh%2Bdt%40kernel.org))+AND+dt%3A20220627184226..
> > > # /home/rob/.local/share/lei/store 144/144
> > > # /home/rob/.local/share/lei/store 3/3
> > > # /usr/bin/curl -Sf -s -d ''
> > > https://lore.kernel.org/all/?x=m&t=1&q=((dfn%3Adrivers+OR+dfn%3Aarch+OR+dfn%3ADocumentation%2F*+OR+dfn%3Ainclude+OR+dfn%3Ascripts)+AND+f%3Arobh%40kernel.org+AND+rt%3A1640812470..)+AND+dt%3A20220627155025..
> > > # /usr/bin/curl -Sf -s -d ''
> > > https://lore.kernel.org/all/?x=m&t=1&q=(l%3Alinux-pci+dfn%3Adrivers%2Fpci%2Fcontroller+dt%3A20220529211430..)+AND+dt%3A20220627184226..
> > > # /home/rob/.local/share/lei/store 0/0
> > > # /home/rob/.local/share/lei/store 362/362
> > > # 0 written to /home/rob/Mail/missing-cc/ (0 matches)
> > > # https://lore.kernel.org/all/ 72/72
> > > # https://lore.kernel.org/all/ 4/4
> > > # https://lore.kernel.org/all/ 131/?
> > > # https://lore.kernel.org/all/ 184/?
> > > # https://lore.kernel.org/all/ 412/?
> > > # https://lore.kernel.org/all/ 603/?
> > > # https://lore.kernel.org/all/ 853/?
> > > # https://lore.kernel.org/all/ 1069/?
> > > # https://lore.kernel.org/all/ 1442/?
> > > # https://lore.kernel.org/all/ 1443/1443
> > > # 1 written to /home/rob/Mail/pci/ (75 matches)
> > > # 2 written to /home/rob/Mail/my-patches/ (148 matches)
> > > # 7 written to /home/rob/Mail/from-me/ (1805 matches)
> > >
> > >
> > > What I expected was 3 messages written to 'my-patches'.
> > >
> > > I think the problem is just simply that the new message missing
> > > doesn't match the query, but is a reply to a match. So with a date
> > > after the original match in the thread won't pick up anything. The 2nd
> > > URL above indeed only has 2 results. I guess I just have to fetch a
> > > wider window like a month every time? What's needed is a get any new
> > > messages in existing threads. I don't suppose there's an efficient way
> > > to do that?
> >
> > No, I don't think so. I think this is a separate issue in lei...
> > "t=1" in the remote query expands threads in a time-agnostic
> > way, so I don't think that's the problem (though I may be wrong...).
>
> Based on what the web interface presents, it sure seems like 't=1' is
> independent of the query. The results listed are only those that match
> the query and date range on the match.
Actually, for the HTML results, t=1 is ignored right now...
it's only for mboxrd downloads (via POST) at the moment...
That should be clarified/changed.
> For example, this query returns 3 matches:
>
> https://lore.kernel.org/all/?x=m&t=1&q=((dfn%3Adrivers+OR+dfn%3Aarch+OR+dfn%3ADocumentation%2F*+OR+dfn%3Ainclude+OR+dfn%3Ascripts)+AND+f%3Arobh%40kernel.org+AND+rt%3A1641934905..)+AND+dt%3A20220630203819..
>
> If I change 'dt' to 1 day earlier, I get 1 more match:
>
> https://lore.kernel.org/all/?x=m&t=1&q=((dfn%3Adrivers+OR+dfn%3Aarch+OR+dfn%3ADocumentation%2F*+OR+dfn%3Ainclude+OR+dfn%3Ascripts)+AND+f%3Arobh%40kernel.org+AND+rt%3A1641934905..)+AND+dt%3A20220629203819..
>
> That 4th match has a reply after 6/30, but the 1st query will not get
> the reply. This is all reproducible without lei involved at all.
Right. t=1 only expands threads if they're linked via
References/In-Reply-To or (loosely) via matching Subjects.
> What seems to be needed is a 'thread date' which is the latest time
> for any message in a thread that matches. Or perhaps some way to
> separate the query from what's transferred. IOW, query for X, but only
> send results newer than some date.
Yeah, the thread ID info is stored independently of the search
index documents, though; so searching (Xapian) vs filtering
(SQLite) gets a bit tricky
I'll look into JMAP further before making more changes to the
data models to accomodate this.
Apologies for the delays; been stressed over other things :<
^ permalink raw reply [relevance 71%]
* Re: lei missing mails
2022-06-30 8:55 71% ` Eric Wong
2022-07-07 9:48 71% ` Eric Wong
@ 2022-07-11 21:59 68% ` Rob Herring
2022-07-18 23:41 71% ` Eric Wong
1 sibling, 1 reply; 200+ results
From: Rob Herring @ 2022-07-11 21:59 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
On Thu, Jun 30, 2022 at 2:55 AM Eric Wong <e@80x24.org> wrote:
>
> Rob Herring <robh@kernel.org> wrote:
> > On Wed, Jun 29, 2022 at 11:27 AM Eric Wong <e@80x24.org> wrote:
> > > Rob Herring <robh@kernel.org> wrote:
> > > > On Wed, Jun 29, 2022 at 10:30 AM Eric Wong <e@80x24.org> wrote:
> > > > > Rob Herring <robh@kernel.org> wrote:
> > > > > > Hi,
> > > > > >
> > > > > > I'm using lei with lore where I have 2 queries which overlap. Really,
> > > > > > one is a subset of the other. On those overlapping threads, I'm
> > > > > > finding that sometimes new messages are written to one mailbox and not
> > > > > > the other. (At least sometimes, the messages may be missing from all
> > > > > > mailboxes sometimes too. I'm not certain.) Using --remote-fudge-time
> > > > > > to force refetching seems to get the missing mails. I haven't found
> > > > > > anything strange in timestamps of the missing mails, but otherwise am
> > > > > > not sure how to debug this further. The queries are retrieving full
> > > > > > threads and the missing mails are in the threads, but not direct
> > > > > > matches to the queries. I realize that's not a lot of detail to go on.
> > > > > > Suggestions on debugging this further?
> > > > >
> > > > > Is this with 1.8 or 1.7?
> > > >
> > > > Commit 68b53c888911 actually. So post 1.8.
> > >
> > > OK, thanks for that info.
> > >
> > > > > I forgot to note in the release notes, but there were some
> > > > > SQLite usage-related fixes which could avoid missing messages.
> > > > >
> > > > > You'll need "lei daemon-kill" after upgrading to 1.8 to ensure
> > > > > the new code is running.
> > > >
> > > > It's possible I haven't done that since updating though I do vaguely
> > > > recall seeing something about needing to do that. Is there any way to
> > > > tell before I restart it?
> > >
> > > Not really, but it's pretty cheap to restart (assuming there's no
> > > long-running jobs).
> >
> > I've restarted and just hit this again.
>
> Ugh, sorry to hear that :<
>
> > > > > What might be interesting is to use the URLs lei prints and
> > > > > comparing the results w/o lei.
> >
> > $ lei up --all
> > # updating /home/rob/Mail/from-me
> > # updating /home/rob/Mail/missing-cc
> > # updating /home/rob/Mail/my-patches
> > # updating /home/rob/Mail/pci
> > # https://lore.kernel.org/all/ limiting to 2022-06-27 12:42 -0600 and newer
> > # https://lore.kernel.org/all/ limiting to 2022-06-27 9:50 -0600 and newer
> > # https://lore.kernel.org/all/ limiting to 2022-06-27 12:42 -0600 and newer
> > # /usr/bin/curl -Sf -s -d ''
> > https://lore.kernel.org/all/?x=m&t=1&q=(dt%3A20220529211430..+AND+(f%3Arobh%40kernel.org+OR+f%3Arobh%2Bdt%40kernel.org))+AND+dt%3A20220627184226..
> > # /home/rob/.local/share/lei/store 144/144
> > # /home/rob/.local/share/lei/store 3/3
> > # /usr/bin/curl -Sf -s -d ''
> > https://lore.kernel.org/all/?x=m&t=1&q=((dfn%3Adrivers+OR+dfn%3Aarch+OR+dfn%3ADocumentation%2F*+OR+dfn%3Ainclude+OR+dfn%3Ascripts)+AND+f%3Arobh%40kernel.org+AND+rt%3A1640812470..)+AND+dt%3A20220627155025..
> > # /usr/bin/curl -Sf -s -d ''
> > https://lore.kernel.org/all/?x=m&t=1&q=(l%3Alinux-pci+dfn%3Adrivers%2Fpci%2Fcontroller+dt%3A20220529211430..)+AND+dt%3A20220627184226..
> > # /home/rob/.local/share/lei/store 0/0
> > # /home/rob/.local/share/lei/store 362/362
> > # 0 written to /home/rob/Mail/missing-cc/ (0 matches)
> > # https://lore.kernel.org/all/ 72/72
> > # https://lore.kernel.org/all/ 4/4
> > # https://lore.kernel.org/all/ 131/?
> > # https://lore.kernel.org/all/ 184/?
> > # https://lore.kernel.org/all/ 412/?
> > # https://lore.kernel.org/all/ 603/?
> > # https://lore.kernel.org/all/ 853/?
> > # https://lore.kernel.org/all/ 1069/?
> > # https://lore.kernel.org/all/ 1442/?
> > # https://lore.kernel.org/all/ 1443/1443
> > # 1 written to /home/rob/Mail/pci/ (75 matches)
> > # 2 written to /home/rob/Mail/my-patches/ (148 matches)
> > # 7 written to /home/rob/Mail/from-me/ (1805 matches)
> >
> >
> > What I expected was 3 messages written to 'my-patches'.
> >
> > I think the problem is just simply that the new message missing
> > doesn't match the query, but is a reply to a match. So with a date
> > after the original match in the thread won't pick up anything. The 2nd
> > URL above indeed only has 2 results. I guess I just have to fetch a
> > wider window like a month every time? What's needed is a get any new
> > messages in existing threads. I don't suppose there's an efficient way
> > to do that?
>
> No, I don't think so. I think this is a separate issue in lei...
> "t=1" in the remote query expands threads in a time-agnostic
> way, so I don't think that's the problem (though I may be wrong...).
Based on what the web interface presents, it sure seems like 't=1' is
independent of the query. The results listed are only those that match
the query and date range on the match.
For example, this query returns 3 matches:
https://lore.kernel.org/all/?x=m&t=1&q=((dfn%3Adrivers+OR+dfn%3Aarch+OR+dfn%3ADocumentation%2F*+OR+dfn%3Ainclude+OR+dfn%3Ascripts)+AND+f%3Arobh%40kernel.org+AND+rt%3A1641934905..)+AND+dt%3A20220630203819..
If I change 'dt' to 1 day earlier, I get 1 more match:
https://lore.kernel.org/all/?x=m&t=1&q=((dfn%3Adrivers+OR+dfn%3Aarch+OR+dfn%3ADocumentation%2F*+OR+dfn%3Ainclude+OR+dfn%3Ascripts)+AND+f%3Arobh%40kernel.org+AND+rt%3A1641934905..)+AND+dt%3A20220629203819..
That 4th match has a reply after 6/30, but the 1st query will not get
the reply. This is all reproducible without lei involved at all.
What seems to be needed is a 'thread date' which is the latest time
for any message in a thread that matches. Or perhaps some way to
separate the query from what's transferred. IOW, query for X, but only
send results newer than some date.
Rob
^ permalink raw reply [relevance 68%]
* Re: lei missing mails
2022-07-07 9:48 71% ` Eric Wong
@ 2022-07-11 21:17 71% ` Rob Herring
0 siblings, 0 replies; 200+ results
From: Rob Herring @ 2022-07-11 21:17 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
On Thu, Jul 7, 2022 at 3:48 AM Eric Wong <e@80x24.org> wrote:
>
> Eric Wong <e@80x24.org> wrote:
> > Rob Herring <robh@kernel.org> wrote:
> > > I've restarted and just hit this again.
> >
> > Ugh, sorry to hear that :<
>
> Btw, is this only with HTTP(S) endpoints? (not local mirrors?)
Yes. Only in that https is the only case I've tried. Don't know about
local mirrors because I don't know how to set that up...
Rob
^ permalink raw reply [relevance 71%]
* Re: lei missing mails
2022-06-30 8:55 71% ` Eric Wong
@ 2022-07-07 9:48 71% ` Eric Wong
2022-07-11 21:17 71% ` Rob Herring
2022-07-11 21:59 68% ` Rob Herring
1 sibling, 1 reply; 200+ results
From: Eric Wong @ 2022-07-07 9:48 UTC (permalink / raw)
To: Rob Herring; +Cc: meta
Eric Wong <e@80x24.org> wrote:
> Rob Herring <robh@kernel.org> wrote:
> > I've restarted and just hit this again.
>
> Ugh, sorry to hear that :<
Btw, is this only with HTTP(S) endpoints? (not local mirrors?)
I wonder if it's something wrong on the HTTP end (server or
client), but did more digging the other day and couldn't
reproduce the problem...
I just posted a very minor diagnostic change which might help
(forgot to Cc you :x)
https://public-inbox.org/meta/20220707094030.1185793-1-e@80x24.org/
As usual, daemon-kill is necessary after upgrading.
^ permalink raw reply [relevance 71%]
* [PATCH 1/2] lei_xsearch: simplify lei/store import check
2022-07-07 9:40 71% [PATCH 0/2] lei: minor diagnostic improvement Eric Wong
@ 2022-07-07 9:40 71% ` Eric Wong
2022-07-07 9:40 53% ` [PATCH 2/2] lei: track seen messages to note duplicates Eric Wong
1 sibling, 0 replies; 200+ results
From: Eric Wong @ 2022-07-07 9:40 UTC (permalink / raw)
To: meta
There's no need to check for two fields when one will suffice.
---
lib/PublicInbox/LeiXSearch.pm | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/lib/PublicInbox/LeiXSearch.pm b/lib/PublicInbox/LeiXSearch.pm
index 2958d3f9..41e79856 100644
--- a/lib/PublicInbox/LeiXSearch.pm
+++ b/lib/PublicInbox/LeiXSearch.pm
@@ -285,7 +285,7 @@ sub each_remote_eml { # callback for MboxReader->mboxrd
my ($res, $kw) = $self->{import_sto}->wq_do('add_eml', $eml);
if (ref($res) eq ref($smsg)) { # totally new message
$smsg = $res;
- $self->{-imported} = 1;
+ $self->{-sto_imported} = 1;
}
$smsg->{kw} = $kw; # short-circuit xsmsg_vmd
}
@@ -376,7 +376,7 @@ sub query_remote_mboxrd {
$fh = IO::Uncompress::Gunzip->new($fh, MultiStream => 1);
PublicInbox::MboxReader->mboxrd($fh, \&each_remote_eml, $self,
$lei, $each_smsg);
- if ($self->{import_sto} && delete($self->{-imported})) {
+ if (delete($self->{-sto_imported})) {
my $wait = $self->{import_sto}->wq_do('done');
}
$reap_curl->join;
^ permalink raw reply related [relevance 71%]
* [PATCH 2/2] lei: track seen messages to note duplicates
2022-07-07 9:40 71% [PATCH 0/2] lei: minor diagnostic improvement Eric Wong
2022-07-07 9:40 71% ` [PATCH 1/2] lei_xsearch: simplify lei/store import check Eric Wong
@ 2022-07-07 9:40 53% ` Eric Wong
1 sibling, 0 replies; 200+ results
From: Eric Wong @ 2022-07-07 9:40 UTC (permalink / raw)
To: meta
This may help track down deduplication or other bugs in lei
which lead to occasionally missing messages.
Link: https://public-inbox.org/meta/CAL_JsqJH8xx_2NyZffNsRXbGXiv3kjmCETvKXt3Yfb0uToLm9Q@mail.gmail.com/
---
lib/PublicInbox/LeiConvert.pm | 8 +++++---
lib/PublicInbox/LeiToMail.pm | 13 ++++++++++---
lib/PublicInbox/LeiXSearch.pm | 20 ++++++++++++--------
3 files changed, 27 insertions(+), 14 deletions(-)
diff --git a/lib/PublicInbox/LeiConvert.pm b/lib/PublicInbox/LeiConvert.pm
index 906f3026..59af40de 100644
--- a/lib/PublicInbox/LeiConvert.pm
+++ b/lib/PublicInbox/LeiConvert.pm
@@ -1,4 +1,4 @@
-# Copyright (C) 2021 all contributors <meta@public-inbox.org>
+# Copyright (C) all contributors <meta@public-inbox.org>
# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
# front-end for the "lei convert" sub-command
@@ -35,8 +35,10 @@ sub process_inputs { # via wq_do
my $lei = $self->{lei};
delete $lei->{1};
delete $self->{wcb}; # commit
- my $nr = delete($lei->{-nr_write}) // 0;
- $lei->qerr("# converted $nr messages");
+ my $nr_w = delete($lei->{-nr_write}) // 0;
+ my $d = (delete($lei->{-nr_seen}) // 0) - $nr_w;
+ $d = $d ? " ($d duplicates)" : '';
+ $lei->qerr("# converted $nr_w messages$d");
}
sub lei_convert { # the main "lei convert" method
diff --git a/lib/PublicInbox/LeiToMail.pm b/lib/PublicInbox/LeiToMail.pm
index 3c5e7e59..2aa3977e 100644
--- a/lib/PublicInbox/LeiToMail.pm
+++ b/lib/PublicInbox/LeiToMail.pm
@@ -1,4 +1,4 @@
-# Copyright (C) 2020-2021 all contributors <meta@public-inbox.org>
+# Copyright (C) all contributors <meta@public-inbox.org>
# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
# Writes PublicInbox::Eml objects atomically to a mbox variant or Maildir
@@ -197,6 +197,7 @@ sub _mbox_write_cb ($$) {
sub { # for git_to_mail
my ($buf, $smsg, $eml) = @_;
$eml //= PublicInbox::Eml->new($buf);
+ ++$lei->{-nr_seen};
return if $dedupe->is_dup($eml, $smsg);
$lse->xsmsg_vmd($smsg) if $lse;
$smsg->{-recent} = 1 if $set_recent;
@@ -291,6 +292,8 @@ sub _maildir_write_cb ($$) {
sub { # for git_to_mail
my ($bref, $smsg, $eml) = @_;
$dst // return $lei->fail; # dst may be undef-ed in last run
+
+ ++$lei->{-nr_seen};
return if $dedupe && $dedupe->is_dup($eml //
PublicInbox::Eml->new($$bref),
$smsg);
@@ -317,6 +320,8 @@ sub _imap_write_cb ($$) {
sub { # for git_to_mail
my ($bref, $smsg, $eml) = @_;
$mic // return $lei->fail; # mic may be undef-ed in last run
+
+ ++$lei->{-nr_seen};
return if $dedupe && $dedupe->is_dup($eml //
PublicInbox::Eml->new($$bref),
$smsg);
@@ -360,6 +365,7 @@ sub _v2_write_cb ($$) {
sub { # for git_to_mail
my ($bref, $smsg, $eml) = @_;
$eml //= PublicInbox::Eml->new($bref);
+ ++$lei->{-nr_seen};
return if $dedupe && $dedupe->is_dup($eml, $smsg);
$lei->{v2w}->wq_do('add', $eml); # V2Writable->add
++$lei->{-nr_write};
@@ -792,9 +798,10 @@ sub wq_atexit_child {
my $lei = $self->{lei};
delete $self->{wcb};
$lei->{ale}->git->async_wait_all;
- my $nr = delete($lei->{-nr_write}) or return;
+ my ($nr_w, $nr_s) = delete(@$lei{qw(-nr_write -nr_seen)});
+ $nr_s or return;
return if $lei->{early_mua} || !$lei->{-progress} || !$lei->{pkt_op_p};
- $lei->{pkt_op_p}->pkt_do('l2m_progress', $nr);
+ $lei->{pkt_op_p}->pkt_do('l2m_progress', $nr_w, $nr_s);
}
# runs on a 1s timer in lei-daemon
diff --git a/lib/PublicInbox/LeiXSearch.pm b/lib/PublicInbox/LeiXSearch.pm
index 41e79856..6f877019 100644
--- a/lib/PublicInbox/LeiXSearch.pm
+++ b/lib/PublicInbox/LeiXSearch.pm
@@ -1,4 +1,4 @@
-# Copyright (C) 2020-2021 all contributors <meta@public-inbox.org>
+# Copyright (C) all contributors <meta@public-inbox.org>
# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
# Combine any combination of PublicInbox::Search,
@@ -163,8 +163,9 @@ sub mset_progress {
}
sub l2m_progress {
- my ($lei, $nr) = @_;
- $lei->{-nr_write} += $nr;
+ my ($lei, $nr_write, $nr_seen) = @_;
+ $lei->{-nr_write} += $nr_write;
+ $lei->{-nr_seen} += $nr_seen;
}
sub query_one_mset { # for --threads and l2m w/o sort
@@ -447,13 +448,16 @@ Error closing $lei->{ovv}->{dst}: \$!=$! \$?=$?
}
if ($lei->{-progress}) {
my $tot = $lei->{-mset_total} // 0;
- my $nr = $lei->{-nr_write} // 0;
+ my $nr_w = $lei->{-nr_write} // 0;
+ my $d = ($lei->{-nr_seen} // 0) - $nr_w;
+ my $x = "$tot matches";
+ $x .= ", $d duplicates" if $d;
if ($l2m) {
- my $m = "# $nr written to " .
- "$lei->{ovv}->{dst} ($tot matches)";
- $nr ? $lei->qfin($m) : $lei->qerr($m);
+ my $m = "# $nr_w written to " .
+ "$lei->{ovv}->{dst} ($x)";
+ $nr_w ? $lei->qfin($m) : $lei->qerr($m);
} else {
- $lei->qerr("# $tot matches");
+ $lei->qerr("# $x");
}
}
$lei->start_mua if $l2m && !$l2m->lock_free;
^ permalink raw reply related [relevance 53%]
* [PATCH 0/2] lei: minor diagnostic improvement
@ 2022-07-07 9:40 71% Eric Wong
2022-07-07 9:40 71% ` [PATCH 1/2] lei_xsearch: simplify lei/store import check Eric Wong
2022-07-07 9:40 53% ` [PATCH 2/2] lei: track seen messages to note duplicates Eric Wong
0 siblings, 2 replies; 200+ results
From: Eric Wong @ 2022-07-07 9:40 UTC (permalink / raw)
To: meta
Still trying to consistently reproduce the source of missing
messages. It may be isolated to HTTP(S) remotes, or not, but
noting the number of seen vs. written messages ought to be a
reasonable start.
patch 1/2 is just a tiny simplification I noticed along the way
Eric Wong (2):
lei_xsearch: simplify lei/store import check
lei: track seen messages to note duplicates
lib/PublicInbox/LeiConvert.pm | 8 +++++---
lib/PublicInbox/LeiToMail.pm | 13 ++++++++++---
lib/PublicInbox/LeiXSearch.pm | 24 ++++++++++++++----------
3 files changed, 29 insertions(+), 16 deletions(-)
^ permalink raw reply [relevance 71%]
* Re: lei missing mails
2022-06-29 22:01 59% ` Rob Herring
@ 2022-06-30 8:55 71% ` Eric Wong
2022-07-07 9:48 71% ` Eric Wong
2022-07-11 21:59 68% ` Rob Herring
0 siblings, 2 replies; 200+ results
From: Eric Wong @ 2022-06-30 8:55 UTC (permalink / raw)
To: Rob Herring; +Cc: meta
Rob Herring <robh@kernel.org> wrote:
> On Wed, Jun 29, 2022 at 11:27 AM Eric Wong <e@80x24.org> wrote:
> > Rob Herring <robh@kernel.org> wrote:
> > > On Wed, Jun 29, 2022 at 10:30 AM Eric Wong <e@80x24.org> wrote:
> > > > Rob Herring <robh@kernel.org> wrote:
> > > > > Hi,
> > > > >
> > > > > I'm using lei with lore where I have 2 queries which overlap. Really,
> > > > > one is a subset of the other. On those overlapping threads, I'm
> > > > > finding that sometimes new messages are written to one mailbox and not
> > > > > the other. (At least sometimes, the messages may be missing from all
> > > > > mailboxes sometimes too. I'm not certain.) Using --remote-fudge-time
> > > > > to force refetching seems to get the missing mails. I haven't found
> > > > > anything strange in timestamps of the missing mails, but otherwise am
> > > > > not sure how to debug this further. The queries are retrieving full
> > > > > threads and the missing mails are in the threads, but not direct
> > > > > matches to the queries. I realize that's not a lot of detail to go on.
> > > > > Suggestions on debugging this further?
> > > >
> > > > Is this with 1.8 or 1.7?
> > >
> > > Commit 68b53c888911 actually. So post 1.8.
> >
> > OK, thanks for that info.
> >
> > > > I forgot to note in the release notes, but there were some
> > > > SQLite usage-related fixes which could avoid missing messages.
> > > >
> > > > You'll need "lei daemon-kill" after upgrading to 1.8 to ensure
> > > > the new code is running.
> > >
> > > It's possible I haven't done that since updating though I do vaguely
> > > recall seeing something about needing to do that. Is there any way to
> > > tell before I restart it?
> >
> > Not really, but it's pretty cheap to restart (assuming there's no
> > long-running jobs).
>
> I've restarted and just hit this again.
Ugh, sorry to hear that :<
> > > > What might be interesting is to use the URLs lei prints and
> > > > comparing the results w/o lei.
>
> $ lei up --all
> # updating /home/rob/Mail/from-me
> # updating /home/rob/Mail/missing-cc
> # updating /home/rob/Mail/my-patches
> # updating /home/rob/Mail/pci
> # https://lore.kernel.org/all/ limiting to 2022-06-27 12:42 -0600 and newer
> # https://lore.kernel.org/all/ limiting to 2022-06-27 9:50 -0600 and newer
> # https://lore.kernel.org/all/ limiting to 2022-06-27 12:42 -0600 and newer
> # /usr/bin/curl -Sf -s -d ''
> https://lore.kernel.org/all/?x=m&t=1&q=(dt%3A20220529211430..+AND+(f%3Arobh%40kernel.org+OR+f%3Arobh%2Bdt%40kernel.org))+AND+dt%3A20220627184226..
> # /home/rob/.local/share/lei/store 144/144
> # /home/rob/.local/share/lei/store 3/3
> # /usr/bin/curl -Sf -s -d ''
> https://lore.kernel.org/all/?x=m&t=1&q=((dfn%3Adrivers+OR+dfn%3Aarch+OR+dfn%3ADocumentation%2F*+OR+dfn%3Ainclude+OR+dfn%3Ascripts)+AND+f%3Arobh%40kernel.org+AND+rt%3A1640812470..)+AND+dt%3A20220627155025..
> # /usr/bin/curl -Sf -s -d ''
> https://lore.kernel.org/all/?x=m&t=1&q=(l%3Alinux-pci+dfn%3Adrivers%2Fpci%2Fcontroller+dt%3A20220529211430..)+AND+dt%3A20220627184226..
> # /home/rob/.local/share/lei/store 0/0
> # /home/rob/.local/share/lei/store 362/362
> # 0 written to /home/rob/Mail/missing-cc/ (0 matches)
> # https://lore.kernel.org/all/ 72/72
> # https://lore.kernel.org/all/ 4/4
> # https://lore.kernel.org/all/ 131/?
> # https://lore.kernel.org/all/ 184/?
> # https://lore.kernel.org/all/ 412/?
> # https://lore.kernel.org/all/ 603/?
> # https://lore.kernel.org/all/ 853/?
> # https://lore.kernel.org/all/ 1069/?
> # https://lore.kernel.org/all/ 1442/?
> # https://lore.kernel.org/all/ 1443/1443
> # 1 written to /home/rob/Mail/pci/ (75 matches)
> # 2 written to /home/rob/Mail/my-patches/ (148 matches)
> # 7 written to /home/rob/Mail/from-me/ (1805 matches)
>
>
> What I expected was 3 messages written to 'my-patches'.
>
> I think the problem is just simply that the new message missing
> doesn't match the query, but is a reply to a match. So with a date
> after the original match in the thread won't pick up anything. The 2nd
> URL above indeed only has 2 results. I guess I just have to fetch a
> wider window like a month every time? What's needed is a get any new
> messages in existing threads. I don't suppose there's an efficient way
> to do that?
No, I don't think so. I think this is a separate issue in lei...
"t=1" in the remote query expands threads in a time-agnostic
way, so I don't think that's the problem (though I may be wrong...).
I'll have to check more closely this week (still stuck with POP3
user account/storage issues :<)
> > > >
> > > > I'll have to double-check if overlapping affects things, but it
> > > > shouldn't; since the dedupe logic is per-output.
> > > >
> > > > Is this exclusively with HTTPS endpoints and writing to Maildirs
> > > > (or something else?)
> > >
> > > Yes. It's querying lore and writing to a maildir. Here's one of the queries:
> > >
> > > [lei]
> > > q = (dfn:drivers OR dfn:arch OR dfn:Documentation/* OR
> > > dfn:include OR dfn:scripts) AND \
> > > f:robh@kernel.org AND rt:6.month.ago..
> > > [lei "q"]
> > > include = https://lore.kernel.org/all/
> > > external = 1
> > > local = 1
> > > remote = 1
> > > threads = 1
> > > dedupe = mid
> > > output = maildir:/home/rob/Mail/my-patches
> >
> > Fwiw, dedupe based on mid could be vulnerable to spoofing, which
> > is why `content' is the default. But yes, in the past, I've
> > noticed some messages to meta@public-inbox.org not showing up,
> > though not recently (I guess lack of activity here is a culprit :x)
>
> Does 'content' ignore trailers that mailman lists like to add? I think
> I switched because of that.
No, unfortunately not. Hopefully the admins can be convinced to
get rid of trailers (I'm happy vger did so a few years back).
But I'd rather deal with duplicates than miss messages (there
have been legitimate messages in the past which reused msgids,
unfortunately).
^ permalink raw reply [relevance 71%]
* Re: lei missing mails
2022-06-29 17:27 71% ` Eric Wong
@ 2022-06-29 22:01 59% ` Rob Herring
2022-06-30 8:55 71% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Rob Herring @ 2022-06-29 22:01 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
On Wed, Jun 29, 2022 at 11:27 AM Eric Wong <e@80x24.org> wrote:
>
> Rob Herring <robh@kernel.org> wrote:
> > On Wed, Jun 29, 2022 at 10:30 AM Eric Wong <e@80x24.org> wrote:
> > >
> > > Rob Herring <robh@kernel.org> wrote:
> > > > Hi,
> > > >
> > > > I'm using lei with lore where I have 2 queries which overlap. Really,
> > > > one is a subset of the other. On those overlapping threads, I'm
> > > > finding that sometimes new messages are written to one mailbox and not
> > > > the other. (At least sometimes, the messages may be missing from all
> > > > mailboxes sometimes too. I'm not certain.) Using --remote-fudge-time
> > > > to force refetching seems to get the missing mails. I haven't found
> > > > anything strange in timestamps of the missing mails, but otherwise am
> > > > not sure how to debug this further. The queries are retrieving full
> > > > threads and the missing mails are in the threads, but not direct
> > > > matches to the queries. I realize that's not a lot of detail to go on.
> > > > Suggestions on debugging this further?
> > >
> > > Is this with 1.8 or 1.7?
> >
> > Commit 68b53c888911 actually. So post 1.8.
>
> OK, thanks for that info.
>
> > > I forgot to note in the release notes, but there were some
> > > SQLite usage-related fixes which could avoid missing messages.
> > >
> > > You'll need "lei daemon-kill" after upgrading to 1.8 to ensure
> > > the new code is running.
> >
> > It's possible I haven't done that since updating though I do vaguely
> > recall seeing something about needing to do that. Is there any way to
> > tell before I restart it?
>
> Not really, but it's pretty cheap to restart (assuming there's no
> long-running jobs).
I've restarted and just hit this again.
> > > What might be interesting is to use the URLs lei prints and
> > > comparing the results w/o lei.
$ lei up --all
# updating /home/rob/Mail/from-me
# updating /home/rob/Mail/missing-cc
# updating /home/rob/Mail/my-patches
# updating /home/rob/Mail/pci
# https://lore.kernel.org/all/ limiting to 2022-06-27 12:42 -0600 and newer
# https://lore.kernel.org/all/ limiting to 2022-06-27 9:50 -0600 and newer
# https://lore.kernel.org/all/ limiting to 2022-06-27 12:42 -0600 and newer
# /usr/bin/curl -Sf -s -d ''
https://lore.kernel.org/all/?x=m&t=1&q=(dt%3A20220529211430..+AND+(f%3Arobh%40kernel.org+OR+f%3Arobh%2Bdt%40kernel.org))+AND+dt%3A20220627184226..
# /home/rob/.local/share/lei/store 144/144
# /home/rob/.local/share/lei/store 3/3
# /usr/bin/curl -Sf -s -d ''
https://lore.kernel.org/all/?x=m&t=1&q=((dfn%3Adrivers+OR+dfn%3Aarch+OR+dfn%3ADocumentation%2F*+OR+dfn%3Ainclude+OR+dfn%3Ascripts)+AND+f%3Arobh%40kernel.org+AND+rt%3A1640812470..)+AND+dt%3A20220627155025..
# /usr/bin/curl -Sf -s -d ''
https://lore.kernel.org/all/?x=m&t=1&q=(l%3Alinux-pci+dfn%3Adrivers%2Fpci%2Fcontroller+dt%3A20220529211430..)+AND+dt%3A20220627184226..
# /home/rob/.local/share/lei/store 0/0
# /home/rob/.local/share/lei/store 362/362
# 0 written to /home/rob/Mail/missing-cc/ (0 matches)
# https://lore.kernel.org/all/ 72/72
# https://lore.kernel.org/all/ 4/4
# https://lore.kernel.org/all/ 131/?
# https://lore.kernel.org/all/ 184/?
# https://lore.kernel.org/all/ 412/?
# https://lore.kernel.org/all/ 603/?
# https://lore.kernel.org/all/ 853/?
# https://lore.kernel.org/all/ 1069/?
# https://lore.kernel.org/all/ 1442/?
# https://lore.kernel.org/all/ 1443/1443
# 1 written to /home/rob/Mail/pci/ (75 matches)
# 2 written to /home/rob/Mail/my-patches/ (148 matches)
# 7 written to /home/rob/Mail/from-me/ (1805 matches)
What I expected was 3 messages written to 'my-patches'.
I think the problem is just simply that the new message missing
doesn't match the query, but is a reply to a match. So with a date
after the original match in the thread won't pick up anything. The 2nd
URL above indeed only has 2 results. I guess I just have to fetch a
wider window like a month every time? What's needed is a get any new
messages in existing threads. I don't suppose there's an efficient way
to do that?
> > >
> > > I'll have to double-check if overlapping affects things, but it
> > > shouldn't; since the dedupe logic is per-output.
> > >
> > > Is this exclusively with HTTPS endpoints and writing to Maildirs
> > > (or something else?)
> >
> > Yes. It's querying lore and writing to a maildir. Here's one of the queries:
> >
> > [lei]
> > q = (dfn:drivers OR dfn:arch OR dfn:Documentation/* OR
> > dfn:include OR dfn:scripts) AND \
> > f:robh@kernel.org AND rt:6.month.ago..
> > [lei "q"]
> > include = https://lore.kernel.org/all/
> > external = 1
> > local = 1
> > remote = 1
> > threads = 1
> > dedupe = mid
> > output = maildir:/home/rob/Mail/my-patches
>
> Fwiw, dedupe based on mid could be vulnerable to spoofing, which
> is why `content' is the default. But yes, in the past, I've
> noticed some messages to meta@public-inbox.org not showing up,
> though not recently (I guess lack of activity here is a culprit :x)
Does 'content' ignore trailers that mailman lists like to add? I think
I switched because of that.
Rob
^ permalink raw reply [relevance 59%]
* Re: lei missing mails
2022-06-29 16:53 69% ` Rob Herring
@ 2022-06-29 17:27 71% ` Eric Wong
2022-06-29 22:01 59% ` Rob Herring
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2022-06-29 17:27 UTC (permalink / raw)
To: Rob Herring; +Cc: meta
Rob Herring <robh@kernel.org> wrote:
> On Wed, Jun 29, 2022 at 10:30 AM Eric Wong <e@80x24.org> wrote:
> >
> > Rob Herring <robh@kernel.org> wrote:
> > > Hi,
> > >
> > > I'm using lei with lore where I have 2 queries which overlap. Really,
> > > one is a subset of the other. On those overlapping threads, I'm
> > > finding that sometimes new messages are written to one mailbox and not
> > > the other. (At least sometimes, the messages may be missing from all
> > > mailboxes sometimes too. I'm not certain.) Using --remote-fudge-time
> > > to force refetching seems to get the missing mails. I haven't found
> > > anything strange in timestamps of the missing mails, but otherwise am
> > > not sure how to debug this further. The queries are retrieving full
> > > threads and the missing mails are in the threads, but not direct
> > > matches to the queries. I realize that's not a lot of detail to go on.
> > > Suggestions on debugging this further?
> >
> > Is this with 1.8 or 1.7?
>
> Commit 68b53c888911 actually. So post 1.8.
OK, thanks for that info.
> > I forgot to note in the release notes, but there were some
> > SQLite usage-related fixes which could avoid missing messages.
> >
> > You'll need "lei daemon-kill" after upgrading to 1.8 to ensure
> > the new code is running.
>
> It's possible I haven't done that since updating though I do vaguely
> recall seeing something about needing to do that. Is there any way to
> tell before I restart it?
Not really, but it's pretty cheap to restart (assuming there's no
long-running jobs).
> > What might be interesting is to use the URLs lei prints and
> > comparing the results w/o lei.
> >
> > I'll have to double-check if overlapping affects things, but it
> > shouldn't; since the dedupe logic is per-output.
> >
> > Is this exclusively with HTTPS endpoints and writing to Maildirs
> > (or something else?)
>
> Yes. It's querying lore and writing to a maildir. Here's one of the queries:
>
> [lei]
> q = (dfn:drivers OR dfn:arch OR dfn:Documentation/* OR
> dfn:include OR dfn:scripts) AND \
> f:robh@kernel.org AND rt:6.month.ago..
> [lei "q"]
> include = https://lore.kernel.org/all/
> external = 1
> local = 1
> remote = 1
> threads = 1
> dedupe = mid
> output = maildir:/home/rob/Mail/my-patches
Fwiw, dedupe based on mid could be vulnerable to spoofing, which
is why `content' is the default. But yes, in the past, I've
noticed some messages to meta@public-inbox.org not showing up,
though not recently (I guess lack of activity here is a culprit :x)
I also just noticed an inotify-related bug deadlocking the whole
lei-deamon while looking into this :<
> > > It might be helpful if lei could print out message-ids of messages
> > > written to mailboxes.
> >
> > That could get very noisy, especially as mailboxes are written
> > in parallel.
>
> Verbose mode already is. Maybe specifying what info you want to be
> verbose would help. The network side is mostly uninteresting in this
> case for example.
Yes, I've been struggling with the verbosity, too; and many
other things :<
> Is there any tool to list new messages in a maildir? I could do that
> before and after. I've done the clearing the new flag in mutt between
> runs, but that's not really ideal.
I suppose `ls'. There are likely other tools more suited for Maildirs
but I'm not familiar with them off the top of my head.
Maybe lei could grow yet another command.
^ permalink raw reply [relevance 71%]
* Re: lei missing mails
2022-06-29 16:30 71% ` Eric Wong
@ 2022-06-29 16:53 69% ` Rob Herring
2022-06-29 17:27 71% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Rob Herring @ 2022-06-29 16:53 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
On Wed, Jun 29, 2022 at 10:30 AM Eric Wong <e@80x24.org> wrote:
>
> Rob Herring <robh@kernel.org> wrote:
> > Hi,
> >
> > I'm using lei with lore where I have 2 queries which overlap. Really,
> > one is a subset of the other. On those overlapping threads, I'm
> > finding that sometimes new messages are written to one mailbox and not
> > the other. (At least sometimes, the messages may be missing from all
> > mailboxes sometimes too. I'm not certain.) Using --remote-fudge-time
> > to force refetching seems to get the missing mails. I haven't found
> > anything strange in timestamps of the missing mails, but otherwise am
> > not sure how to debug this further. The queries are retrieving full
> > threads and the missing mails are in the threads, but not direct
> > matches to the queries. I realize that's not a lot of detail to go on.
> > Suggestions on debugging this further?
>
> Is this with 1.8 or 1.7?
Commit 68b53c888911 actually. So post 1.8.
> I forgot to note in the release notes, but there were some
> SQLite usage-related fixes which could avoid missing messages.
>
> You'll need "lei daemon-kill" after upgrading to 1.8 to ensure
> the new code is running.
It's possible I haven't done that since updating though I do vaguely
recall seeing something about needing to do that. Is there any way to
tell before I restart it?
> What might be interesting is to use the URLs lei prints and
> comparing the results w/o lei.
>
> I'll have to double-check if overlapping affects things, but it
> shouldn't; since the dedupe logic is per-output.
>
> Is this exclusively with HTTPS endpoints and writing to Maildirs
> (or something else?)
Yes. It's querying lore and writing to a maildir. Here's one of the queries:
[lei]
q = (dfn:drivers OR dfn:arch OR dfn:Documentation/* OR
dfn:include OR dfn:scripts) AND \
f:robh@kernel.org AND rt:6.month.ago..
[lei "q"]
include = https://lore.kernel.org/all/
external = 1
local = 1
remote = 1
threads = 1
dedupe = mid
output = maildir:/home/rob/Mail/my-patches
>
> > It might be helpful if lei could print out message-ids of messages
> > written to mailboxes.
>
> That could get very noisy, especially as mailboxes are written
> in parallel.
Verbose mode already is. Maybe specifying what info you want to be
verbose would help. The network side is mostly uninteresting in this
case for example.
Is there any tool to list new messages in a maildir? I could do that
before and after. I've done the clearing the new flag in mutt between
runs, but that's not really ideal.
Rob
^ permalink raw reply [relevance 69%]
* Re: lei missing mails
2022-06-29 16:15 71% lei missing mails Rob Herring
@ 2022-06-29 16:30 71% ` Eric Wong
2022-06-29 16:53 69% ` Rob Herring
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2022-06-29 16:30 UTC (permalink / raw)
To: Rob Herring; +Cc: meta
Rob Herring <robh@kernel.org> wrote:
> Hi,
>
> I'm using lei with lore where I have 2 queries which overlap. Really,
> one is a subset of the other. On those overlapping threads, I'm
> finding that sometimes new messages are written to one mailbox and not
> the other. (At least sometimes, the messages may be missing from all
> mailboxes sometimes too. I'm not certain.) Using --remote-fudge-time
> to force refetching seems to get the missing mails. I haven't found
> anything strange in timestamps of the missing mails, but otherwise am
> not sure how to debug this further. The queries are retrieving full
> threads and the missing mails are in the threads, but not direct
> matches to the queries. I realize that's not a lot of detail to go on.
> Suggestions on debugging this further?
Is this with 1.8 or 1.7?
I forgot to note in the release notes, but there were some
SQLite usage-related fixes which could avoid missing messages.
You'll need "lei daemon-kill" after upgrading to 1.8 to ensure
the new code is running.
What might be interesting is to use the URLs lei prints and
comparing the results w/o lei.
I'll have to double-check if overlapping affects things, but it
shouldn't; since the dedupe logic is per-output.
Is this exclusively with HTTPS endpoints and writing to Maildirs
(or something else?)
> It might be helpful if lei could print out message-ids of messages
> written to mailboxes.
That could get very noisy, especially as mailboxes are written
in parallel.
Thanks.
^ permalink raw reply [relevance 71%]
* lei missing mails
@ 2022-06-29 16:15 71% Rob Herring
2022-06-29 16:30 71% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Rob Herring @ 2022-06-29 16:15 UTC (permalink / raw)
To: meta
Hi,
I'm using lei with lore where I have 2 queries which overlap. Really,
one is a subset of the other. On those overlapping threads, I'm
finding that sometimes new messages are written to one mailbox and not
the other. (At least sometimes, the messages may be missing from all
mailboxes sometimes too. I'm not certain.) Using --remote-fudge-time
to force refetching seems to get the missing mails. I haven't found
anything strange in timestamps of the missing mails, but otherwise am
not sure how to debug this further. The queries are retrieving full
threads and the missing mails are in the threads, but not direct
matches to the queries. I realize that's not a lot of detail to go on.
Suggestions on debugging this further?
It might be helpful if lei could print out message-ids of messages
written to mailboxes.
Rob
^ permalink raw reply [relevance 71%]
* [PATCH 5/3] doc: lei-q: regenerate for patchid: help
@ 2022-06-22 7:50 58% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2022-06-22 7:50 UTC (permalink / raw)
To: meta; +Cc: Kyle Meyer
---
Oops, left this out of the original patch :x
Documentation/lei-q.pod | 49 +++++++++++++++++++++--------------------
1 file changed, 25 insertions(+), 24 deletions(-)
diff --git a/Documentation/lei-q.pod b/Documentation/lei-q.pod
index fd829655..1cbffba4 100644
--- a/Documentation/lei-q.pod
+++ b/Documentation/lei-q.pod
@@ -253,30 +253,31 @@ instances:
=for comment
AUTO-GENERATED-SEARCH-TERMS-BEGIN
- s: match within Subject e.g. s:"a quick brown fox"
- d: match date-time range, git "approxidate" formats supported
- Open-ended ranges such as `d:last.week..' and
- `d:..2.days.ago' are supported
- b: match within message body, including text attachments
- nq: match non-quoted text within message body
- q: match quoted text within message body
- n: match filename of attachment(s)
- t: match within the To header
- c: match within the Cc header
- f: match within the From header
- a: match within the To, Cc, and From headers
- tc: match within the To and Cc headers
- l: match contents of the List-Id header
- bs: match within the Subject and body
- dfn: match filename from diff
- dfa: match diff removed (-) lines
- dfb: match diff added (+) lines
- dfhh: match diff hunk header context (usually a function name)
- dfctx: match diff context lines
- dfpre: match pre-image git blob ID
- dfpost: match post-image git blob ID
- dfblob: match either pre or post-image git blob ID
- rt: match received time, like `d:' if sender's clock was correct
+ s: match within Subject e.g. s:"a quick brown fox"
+ d: match date-time range, git "approxidate" formats supported
+ Open-ended ranges such as `d:last.week..' and
+ `d:..2.days.ago' are supported
+ b: match within message body, including text attachments
+ nq: match non-quoted text within message body
+ q: match quoted text within message body
+ n: match filename of attachment(s)
+ t: match within the To header
+ c: match within the Cc header
+ f: match within the From header
+ a: match within the To, Cc, and From headers
+ tc: match within the To and Cc headers
+ l: match contents of the List-Id header
+ bs: match within the Subject and body
+ dfn: match filename from diff
+ dfa: match diff removed (-) lines
+ dfb: match diff added (+) lines
+ dfhh: match diff hunk header context (usually a function name)
+ dfctx: match diff context lines
+ dfpre: match pre-image git blob ID
+ dfpost: match post-image git blob ID
+ dfblob: match either pre or post-image git blob ID
+ patchid: match `git patch-id --stable' output
+ rt: match received time, like `d:' if sender's clock was correct
=for comment
AUTO-GENERATED-SEARCH-TERMS-END
^ permalink raw reply related [relevance 58%]
* Re: Trouble running lei
2022-05-03 15:24 71% ` Konstantin Ryabitsev
@ 2022-05-03 20:32 71% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2022-05-03 20:32 UTC (permalink / raw)
To: Konstantin Ryabitsev; +Cc: Filipe Manana, meta
Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> On Tue, May 03, 2022 at 01:50:52PM +0100, Filipe Manana wrote:
> > > Perhaps it's already running lei-daemon as an older version?
> > > "lei daemon-kill" should kill it and it'll restart on the next
> > > command, unless something else got wedged.
> >
> > Ah, running "lei daemon-kill" fixed it.
> > I don't know if I did something wrong before, but after running that,
> > lei is now working fine.
Good to know!
> I think this is actually a common occurrence. Any way lei-daemon can recognize
> when there's a version mismatch between itself and the binary talking to it?
Sorta, but even I manage to lose track of which commits cause
protocol or internal API changes since everything is lazy-loaded.
I think the best way would be to have inotify/kqueue watch the /PublicInbox/
directory and auto-restart iff no active commands are running...
^ permalink raw reply [relevance 71%]
* Re: Trouble running lei
2022-05-03 12:50 71% ` Filipe Manana
@ 2022-05-03 15:24 71% ` Konstantin Ryabitsev
2022-05-03 20:32 71% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Konstantin Ryabitsev @ 2022-05-03 15:24 UTC (permalink / raw)
To: Filipe Manana; +Cc: Eric Wong, meta
On Tue, May 03, 2022 at 01:50:52PM +0100, Filipe Manana wrote:
> > Perhaps it's already running lei-daemon as an older version?
> > "lei daemon-kill" should kill it and it'll restart on the next
> > command, unless something else got wedged.
>
> Ah, running "lei daemon-kill" fixed it.
> I don't know if I did something wrong before, but after running that,
> lei is now working fine.
I think this is actually a common occurrence. Any way lei-daemon can recognize
when there's a version mismatch between itself and the binary talking to it?
Regards,
-K'
^ permalink raw reply [relevance 71%]
* Re: Trouble running lei
2022-05-03 11:37 71% ` Eric Wong
@ 2022-05-03 12:50 71% ` Filipe Manana
2022-05-03 15:24 71% ` Konstantin Ryabitsev
0 siblings, 1 reply; 200+ results
From: Filipe Manana @ 2022-05-03 12:50 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
On Tue, May 3, 2022 at 12:37 PM Eric Wong <e@80x24.org> wrote:
>
> Filipe Manana <fdmanana@kernel.org> wrote:
> > Hello,
> >
> > I tried both 'master' branch and tag v1.8.0, I did the documented build steps:
> >
> > $ git clone https://public-inbox.org/public-inbox.git/
> > $ cd public-inbox
> >
> > $ perl Makefile.PL
> > $ make
> > $ echo $? # success, prints 0
> >
> > $ cd certs
> > $ /usr/bin/perl ./create-certs.perl
> > $ cd ..
> > $ make test
> > (...)
>
> Any messages about missing dependencies when running tests?
> (e.g. Xapian)
>
> Inline::C or Socket::Msghdr shouldn't be required with 1.8
> on most arches...
>
> > All tests successful.
> > Files=157, Tests=6785, 491 wallclock secs ( 0.77 usr 0.13 sys + 39.65
> > cusr 14.90 csys = 55.45 CPU)
>
> Fwiw, "make check" or "make check-run" can be a lot faster on SMP.
Oh, I didn't know about it. Good to know.
>
> > Result: PASS
> >
> > $ make symlink-install
> >
> > $HOME/bin is included in my $PATH, but when I run 'lei', I get an error:
> >
> > $ lei q -o ~/Mail/overlay -I https://lore.kernel.org/all -t
> > 'dfn:fs/btrfs/* AND rt:3.month.ago..'
> > Attempt to reload PublicInbox/LeiXSearch.pm aborted.
>
> Odd, "Attempt to reload %s aborted" coming from perl itself means it
> already tried and failed before. (man perldiag)
>
> Were there previous errors from other commands?
>
> Perhaps it's already running lei-daemon as an older version?
> "lei daemon-kill" should kill it and it'll restart on the next
> command, unless something else got wedged.
Ah, running "lei daemon-kill" fixed it.
I don't know if I did something wrong before, but after running that,
lei is now working fine.
Thanks!
>
> I've been meaning to make the daemon stuff "worth it"
> w.r.t. automatic updates/flag/keyword sync (inotify/IDLE) but
> haven't gotten around to it, yet :x
>
> > Compilation failed in require at
> > /home/fdmanana/git/hub/public-inbox/lib/PublicInbox/LeiQuery.pm line
> > 74.
> >
> > This is on a Ubuntu 20.04.3 LTS distro.
> > I have it working on a Debian SID box without any problems (it was set
> > up several months ago).
> >
> > Any ideas about what's wrong?
>
> Probably daemon-kill will get rid of an old version that's
> already running (and verifying that it's actually dead).
^ permalink raw reply [relevance 71%]
* Re: Trouble running lei
2022-05-03 11:15 70% Trouble running lei Filipe Manana
@ 2022-05-03 11:37 71% ` Eric Wong
2022-05-03 12:50 71% ` Filipe Manana
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2022-05-03 11:37 UTC (permalink / raw)
To: Filipe Manana; +Cc: meta
Filipe Manana <fdmanana@kernel.org> wrote:
> Hello,
>
> I tried both 'master' branch and tag v1.8.0, I did the documented build steps:
>
> $ git clone https://public-inbox.org/public-inbox.git/
> $ cd public-inbox
>
> $ perl Makefile.PL
> $ make
> $ echo $? # success, prints 0
>
> $ cd certs
> $ /usr/bin/perl ./create-certs.perl
> $ cd ..
> $ make test
> (...)
Any messages about missing dependencies when running tests?
(e.g. Xapian)
Inline::C or Socket::Msghdr shouldn't be required with 1.8
on most arches...
> All tests successful.
> Files=157, Tests=6785, 491 wallclock secs ( 0.77 usr 0.13 sys + 39.65
> cusr 14.90 csys = 55.45 CPU)
Fwiw, "make check" or "make check-run" can be a lot faster on SMP.
> Result: PASS
>
> $ make symlink-install
>
> $HOME/bin is included in my $PATH, but when I run 'lei', I get an error:
>
> $ lei q -o ~/Mail/overlay -I https://lore.kernel.org/all -t
> 'dfn:fs/btrfs/* AND rt:3.month.ago..'
> Attempt to reload PublicInbox/LeiXSearch.pm aborted.
Odd, "Attempt to reload %s aborted" coming from perl itself means it
already tried and failed before. (man perldiag)
Were there previous errors from other commands?
Perhaps it's already running lei-daemon as an older version?
"lei daemon-kill" should kill it and it'll restart on the next
command, unless something else got wedged.
I've been meaning to make the daemon stuff "worth it"
w.r.t. automatic updates/flag/keyword sync (inotify/IDLE) but
haven't gotten around to it, yet :x
> Compilation failed in require at
> /home/fdmanana/git/hub/public-inbox/lib/PublicInbox/LeiQuery.pm line
> 74.
>
> This is on a Ubuntu 20.04.3 LTS distro.
> I have it working on a Debian SID box without any problems (it was set
> up several months ago).
>
> Any ideas about what's wrong?
Probably daemon-kill will get rid of an old version that's
already running (and verifying that it's actually dead).
^ permalink raw reply [relevance 71%]
* Trouble running lei
@ 2022-05-03 11:15 70% Filipe Manana
2022-05-03 11:37 71% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Filipe Manana @ 2022-05-03 11:15 UTC (permalink / raw)
To: meta
Hello,
I tried both 'master' branch and tag v1.8.0, I did the documented build steps:
$ git clone https://public-inbox.org/public-inbox.git/
$ cd public-inbox
$ perl Makefile.PL
$ make
$ echo $? # success, prints 0
$ cd certs
$ /usr/bin/perl ./create-certs.perl
$ cd ..
$ make test
(...)
All tests successful.
Files=157, Tests=6785, 491 wallclock secs ( 0.77 usr 0.13 sys + 39.65
cusr 14.90 csys = 55.45 CPU)
Result: PASS
$ make symlink-install
$HOME/bin is included in my $PATH, but when I run 'lei', I get an error:
$ lei q -o ~/Mail/overlay -I https://lore.kernel.org/all -t
'dfn:fs/btrfs/* AND rt:3.month.ago..'
Attempt to reload PublicInbox/LeiXSearch.pm aborted.
Compilation failed in require at
/home/fdmanana/git/hub/public-inbox/lib/PublicInbox/LeiQuery.pm line
74.
This is on a Ubuntu 20.04.3 LTS distro.
I have it working on a Debian SID box without any problems (it was set
up several months ago).
Any ideas about what's wrong?
Thank you.
^ permalink raw reply [relevance 70%]
* [PATCH] lei import: add label completions (+L:$LABEL)
@ 2022-05-02 18:10 71% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2022-05-02 18:10 UTC (permalink / raw)
To: meta
This can probably be added for "lei q", too, but we typically
import first. Labels can probably be made persistent on a
per-folder basis in the future.
---
lib/PublicInbox/LeiImport.pm | 2 ++
1 file changed, 2 insertions(+)
diff --git a/lib/PublicInbox/LeiImport.pm b/lib/PublicInbox/LeiImport.pm
index b9865829..2d91e4c4 100644
--- a/lib/PublicInbox/LeiImport.pm
+++ b/lib/PublicInbox/LeiImport.pm
@@ -117,6 +117,8 @@ sub _complete_import {
my ($lei, @argv) = @_;
my ($re, $cur, $match_cb) = $lei->complete_url_prepare(\@argv);
my @k = $lei->url_folder_cache->keys($argv[-1] // undef, 1);
+ my @L = eval { $lei->_lei_store->search->all_terms('L') };
+ push(@k, map { "+L:$_" } @L);
my @m = map { $match_cb->($_) } @k;
my %f = map { $_ => 1 } (@m ? @m : @k);
if (my $lms = $lei->lms) {
^ permalink raw reply related [relevance 71%]
* [PATCH] lei refresh-mail-sync: filter NNTP(S) from --all
@ 2022-04-30 21:29 71% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2022-04-30 21:29 UTC (permalink / raw)
To: meta
We currently do not support refresh from NNTP since deletes are
rare with public-inbox NNTP servers; but traditional Usenet
servers do delete/expire messages and we should probably support
that at some point.
---
lib/PublicInbox/LeiRefreshMailSync.pm | 2 ++
1 file changed, 2 insertions(+)
diff --git a/lib/PublicInbox/LeiRefreshMailSync.pm b/lib/PublicInbox/LeiRefreshMailSync.pm
index 7821008f..a60a9a5e 100644
--- a/lib/PublicInbox/LeiRefreshMailSync.pm
+++ b/lib/PublicInbox/LeiRefreshMailSync.pm
@@ -73,6 +73,8 @@ lei mail_sync.sqlite3 uninitialized, see lei-import(1)
EOM
if (defined(my $all = $lei->{opt}->{all})) {
$lms->group2folders($lei, $all, \@folders) or return;
+ # TODO: handle NNTP servers which delete messages
+ @folders = grep(!m!\Anntps?://!, @folders);
} else {
$lms->arg2folder($lei, \@folders); # may die
}
^ permalink raw reply related [relevance 71%]
* [PATCH] lei: improve diagnosis of errors from children
@ 2022-04-30 21:04 71% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2022-04-30 21:04 UTC (permalink / raw)
To: meta
Not 100% sure what's going on, but maybe this helps.
---
Anybody else see this?
lib/PublicInbox/LEI.pm | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 89aa4119..d81ca296 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -1394,7 +1394,7 @@ sub wq_done_wait { # dwaitpid callback
local $current_lei = $lei;
my $err_type = $lei->{-err_type};
$? and $lei->child_error($?,
- $err_type ? "$err_type errors during $lei->{cmd}" : ());
+ $err_type ? "$err_type errors during $lei->{cmd} \$?=$?" : ());
$lei->dclose;
}
^ permalink raw reply related [relevance 71%]
* [PATCH 2/2] lei: move to v5.12 to avoid "use strict"
@ 2022-04-23 22:03 61% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2022-04-23 22:03 UTC (permalink / raw)
To: meta
Socket.pm still loads strict.pm, unfortunately, which hurts
startup time; but we'll save some LoC this way.
---
lib/PublicInbox/CmdIPC4.pm | 3 +--
lib/PublicInbox/LEI.pm | 3 +--
lib/PublicInbox/Spawn.pm | 2 +-
lib/PublicInbox/Syscall.pm | 3 +--
script/lei | 3 +--
5 files changed, 5 insertions(+), 9 deletions(-)
diff --git a/lib/PublicInbox/CmdIPC4.pm b/lib/PublicInbox/CmdIPC4.pm
index 76938b6d..e368d032 100644
--- a/lib/PublicInbox/CmdIPC4.pm
+++ b/lib/PublicInbox/CmdIPC4.pm
@@ -5,8 +5,7 @@
# first choice for script/lei front-end and 2nd choice for lei backend
# libsocket-msghdr-perl is in Debian but not many other distros as of 2021.
package PublicInbox::CmdIPC4;
-use strict;
-use v5.10.1;
+use v5.12;
use Socket qw(SOL_SOCKET SCM_RIGHTS);
BEGIN { eval {
require Socket::MsgHdr; # XS
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 93b4ea03..89aa4119 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -6,8 +6,7 @@
# local clients with read/write access to the FS and use as many
# system resources as the local user has access to.
package PublicInbox::LEI;
-use strict;
-use v5.10.1;
+use v5.12;
use parent qw(PublicInbox::DS PublicInbox::LeiExternal
PublicInbox::LeiQuery);
use Getopt::Long ();
diff --git a/lib/PublicInbox/Spawn.pm b/lib/PublicInbox/Spawn.pm
index 137b8087..3f69108a 100644
--- a/lib/PublicInbox/Spawn.pm
+++ b/lib/PublicInbox/Spawn.pm
@@ -15,7 +15,7 @@
# We don't want too many DSOs: https://udrepper.livejournal.com/8790.html
package PublicInbox::Spawn;
-use strict;
+use v5.12;
use parent qw(Exporter);
use Symbol qw(gensym);
use Fcntl qw(LOCK_EX SEEK_SET);
diff --git a/lib/PublicInbox/Syscall.pm b/lib/PublicInbox/Syscall.pm
index 22b779ad..777c44d0 100644
--- a/lib/PublicInbox/Syscall.pm
+++ b/lib/PublicInbox/Syscall.pm
@@ -16,8 +16,7 @@
# You may distribute under the terms of either the GNU General Public
# License or the Artistic License, as specified in the Perl README file.
package PublicInbox::Syscall;
-use strict;
-use v5.10.1;
+use v5.12;
use parent qw(Exporter);
use POSIX qw(ENOENT ENOSYS EINVAL O_NONBLOCK);
use Socket qw(SOL_SOCKET SCM_RIGHTS);
diff --git a/script/lei b/script/lei
index adef9944..5feb7751 100755
--- a/script/lei
+++ b/script/lei
@@ -1,8 +1,7 @@
#!perl -w
# Copyright (C) all contributors <meta@public-inbox.org>
# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
-use strict;
-use v5.10.1;
+use v5.12;
use Socket qw(AF_UNIX SOCK_SEQPACKET MSG_EOR pack_sockaddr_un);
use PublicInbox::CmdIPC4;
my $narg = 5;
^ permalink raw reply related [relevance 61%]
* [PATCH] lei: commit store on interrupted partial imports
@ 2022-04-21 11:59 58% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2022-04-21 11:59 UTC (permalink / raw)
To: meta
This change prevents lingering shard and git-fast-import
processes from remaining after interrupted "lei import" (and
similar). It also reduces the likelyhood of data-loss in case
of subsequent abnormal termination of the daemon.
I think this is the least surprising way to handle users
prematurely aborting imports or other similar operations which
write to lei/store and will result in reduced bandwidth waste
for users with intermittent connections. This is because the
lei/store processes may be shared by parallel "lei import"
callers, and commits done by any "lei import" caller will
inevitably trigger writes for all of them.
---
lib/PublicInbox/LEI.pm | 11 ++++++++---
lib/PublicInbox/LeiImportKw.pm | 4 +---
lib/PublicInbox/LeiNoteEvent.pm | 4 +---
lib/PublicInbox/LeiPmdir.pm | 6 ++----
4 files changed, 12 insertions(+), 13 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 4bd9183e..93b4ea03 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -660,6 +660,7 @@ sub wait_wq_events {
for my $wq (grep(defined, @$lei{qw(ikw pmd)})) { # auxiliary WQs
$wq->wq_close;
}
+ $wq1->{lei_sock} = $lei->{sock} if $wq1;
$op_c->{ops} = $ops;
}
@@ -1405,9 +1406,11 @@ sub fchdir {
}
sub wq_eof { # EOF callback for main daemon
- my ($lei) = @_;
+ my ($lei, $wq_fld) = @_;
local $current_lei = $lei;
- delete $lei->{wq1} // return $lei->fail; # already failed
+ my $wq = delete $lei->{$wq_fld // 'wq1'};
+ $lei->sto_done_request($wq);
+ $wq // $lei->fail; # already failed
}
sub watch_state_ok ($) {
@@ -1514,8 +1517,10 @@ sub lms {
}
sub sto_done_request {
- my ($lei, $sock) = @_;
+ my ($lei, $wq) = @_;
+ return unless $lei->{sto};
local $current_lei = $lei;
+ my $sock = $wq ? $wq->{lei_sock} : undef;
eval {
if ($sock //= $lei->{sock}) { # issue, async wait
$lei->{sto}->wq_io_do('done', [ $sock ]);
diff --git a/lib/PublicInbox/LeiImportKw.pm b/lib/PublicInbox/LeiImportKw.pm
index 52fd4043..4dd938f5 100644
--- a/lib/PublicInbox/LeiImportKw.pm
+++ b/lib/PublicInbox/LeiImportKw.pm
@@ -47,9 +47,7 @@ sub ck_update_kw { # via wq_io_do
}
sub _lei_wq_eof { # EOF callback for main lei daemon
- my ($lei) = @_;
- my $ikw = delete $lei->{ikw} or return $lei->fail;
- $lei->sto_done_request($ikw->{lei_sock});
+ $_[0]->wq_eof('ikw');
}
1;
diff --git a/lib/PublicInbox/LeiNoteEvent.pm b/lib/PublicInbox/LeiNoteEvent.pm
index 22d6ffac..db387633 100644
--- a/lib/PublicInbox/LeiNoteEvent.pm
+++ b/lib/PublicInbox/LeiNoteEvent.pm
@@ -114,9 +114,7 @@ sub ipc_atfork_child {
}
sub _lei_wq_eof { # EOF callback for main lei daemon
- my ($lei) = @_;
- my $lne = delete $lei->{lne} or return $lei->fail;
- $lei->sto_done_request($lne->{lei_sock});
+ $_[0]->wq_eof('lne');
}
1;
diff --git a/lib/PublicInbox/LeiPmdir.pm b/lib/PublicInbox/LeiPmdir.pm
index f9b68fc2..d4aa0212 100644
--- a/lib/PublicInbox/LeiPmdir.pm
+++ b/lib/PublicInbox/LeiPmdir.pm
@@ -1,4 +1,4 @@
-# Copyright (C) 2021 all contributors <meta@public-inbox.org>
+# Copyright (C) all contributors <meta@public-inbox.org>
# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
# WQ worker for dealing with parallel Maildir reads;
@@ -48,9 +48,7 @@ sub mdir_iter { # via wq_io_do
}
sub _lei_wq_eof { # EOF callback for main lei daemon
- my ($lei) = @_;
- my $pmd = delete $lei->{pmd} or return $lei->fail;
- $lei->sto_done_request($pmd->{lei_sock});
+ $_[0]->wq_eof('pmd');
}
1;
^ permalink raw reply related [relevance 58%]
* [PATCH 1/4] lei: clobber recvmsg buffer on errors
2022-04-18 9:50 71% [PATCH 0/4] lei: finish wiring up pure-Perl stuff for Linux Eric Wong
@ 2022-04-18 9:50 71% ` Eric Wong
2022-04-18 9:50 54% ` [PATCH 3/4] lei: wire up pure Perl sendmsg/recvmsg for Linux users Eric Wong
1 sibling, 0 replies; 200+ results
From: Eric Wong @ 2022-04-18 9:50 UTC (permalink / raw)
To: meta
It will be necessary when we drop the Inline::C requirement
since the pure Perl Linux syscall recvmsg implementation.
This likely would've caused errors for Socket::MsgHdr users
without Inline::C, but I haven't tested it since it's a rare
configuration.
---
lib/PublicInbox/LEI.pm | 1 +
lib/PublicInbox/LeiSelfSocket.pm | 1 +
2 files changed, 2 insertions(+)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index a7ddc21f..9ab91714 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -1149,6 +1149,7 @@ sub event_step {
if (scalar(@fds) == 1 && !defined($fds[0])) {
return if $! == EAGAIN;
die "recvmsg: $!" if $! != ECONNRESET;
+ $buf = '';
@fds = (); # for open loop below:
}
for (@fds) { open my $rfh, '+<&=', $_ }
diff --git a/lib/PublicInbox/LeiSelfSocket.pm b/lib/PublicInbox/LeiSelfSocket.pm
index dd64b6cf..860020cb 100644
--- a/lib/PublicInbox/LeiSelfSocket.pm
+++ b/lib/PublicInbox/LeiSelfSocket.pm
@@ -29,6 +29,7 @@ sub event_step {
if (scalar(@fds) == 1 && !defined($fds[0])) {
return if $!{EAGAIN};
die "recvmsg: $!" unless $!{ECONNRESET};
+ $buf = '';
} else { # just in case open so perl can auto-close them:
for (@fds) { open my $fh, '+<&=', $_ };
}
^ permalink raw reply related [relevance 71%]
* [PATCH 0/4] lei: finish wiring up pure-Perl stuff for Linux
@ 2022-04-18 9:50 71% Eric Wong
2022-04-18 9:50 71% ` [PATCH 1/4] lei: clobber recvmsg buffer on errors Eric Wong
2022-04-18 9:50 54% ` [PATCH 3/4] lei: wire up pure Perl sendmsg/recvmsg for Linux users Eric Wong
0 siblings, 2 replies; 200+ results
From: Eric Wong @ 2022-04-18 9:50 UTC (permalink / raw)
To: meta
Some cleanups here, too; but this hopefully makes lei more
accessible to users who don't have space/bandwidth to use
Inline::C.
I thought I sent these out weeks ago, but Real-Life, power outages
and other lei problems got in the way, I guess :x
Eric Wong (4):
lei: clobber recvmsg buffer on errors
syscall: more idiomatic cmsghdr space allocation
lei: wire up pure Perl sendmsg/recvmsg for Linux users
syscall: golf + more idiomatic buffer initialization
lib/PublicInbox/IPC.pm | 4 ++++
lib/PublicInbox/LEI.pm | 4 ++++
lib/PublicInbox/LeiSelfSocket.pm | 1 +
lib/PublicInbox/Syscall.pm | 13 ++++++-------
lib/PublicInbox/TestCommon.pm | 16 ++++++++++++----
t/lei-daemon.t | 6 +++++-
t/lei-externals.t | 1 +
7 files changed, 33 insertions(+), 12 deletions(-)
^ permalink raw reply [relevance 71%]
* [PATCH 3/4] lei: wire up pure Perl sendmsg/recvmsg for Linux users
2022-04-18 9:50 71% [PATCH 0/4] lei: finish wiring up pure-Perl stuff for Linux Eric Wong
2022-04-18 9:50 71% ` [PATCH 1/4] lei: clobber recvmsg buffer on errors Eric Wong
@ 2022-04-18 9:50 54% ` Eric Wong
1 sibling, 0 replies; 200+ results
From: Eric Wong @ 2022-04-18 9:50 UTC (permalink / raw)
To: meta
This enables lei-daemon to work without Inline::C nor
Socket::MsgHdr installed. Prior to this, only the `lei' client
was using the pure Perl implementation. Either C implementation
is still marginally faster, however.
---
lib/PublicInbox/IPC.pm | 4 ++++
lib/PublicInbox/LEI.pm | 3 +++
lib/PublicInbox/TestCommon.pm | 16 ++++++++++++----
t/lei-daemon.t | 6 +++++-
t/lei-externals.t | 1 +
5 files changed, 25 insertions(+), 5 deletions(-)
diff --git a/lib/PublicInbox/IPC.pm b/lib/PublicInbox/IPC.pm
index 8376275e..67e86a43 100644
--- a/lib/PublicInbox/IPC.pm
+++ b/lib/PublicInbox/IPC.pm
@@ -47,6 +47,10 @@ my $send_cmd = PublicInbox::Spawn->can('send_cmd4') // do {
require PublicInbox::CmdIPC4;
$recv_cmd //= PublicInbox::CmdIPC4->can('recv_cmd4');
PublicInbox::CmdIPC4->can('send_cmd4');
+} // do {
+ require PublicInbox::Syscall;
+ $recv_cmd //= PublicInbox::Syscall->can('recv_cmd4');
+ PublicInbox::Syscall->can('send_cmd4');
};
sub _get_rec ($) {
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 9ab91714..4bd9183e 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -1277,6 +1277,9 @@ sub lazy_start {
require PublicInbox::CmdIPC4;
$send_cmd = PublicInbox::CmdIPC4->can('send_cmd4');
PublicInbox::CmdIPC4->can('recv_cmd4');
+ } // do {
+ $send_cmd = PublicInbox::Syscall->can('send_cmd4');
+ PublicInbox::Syscall->can('recv_cmd4');
};
}
$recv_cmd or die <<"";
diff --git a/lib/PublicInbox/TestCommon.pm b/lib/PublicInbox/TestCommon.pm
index ca732811..943dd2fa 100644
--- a/lib/PublicInbox/TestCommon.pm
+++ b/lib/PublicInbox/TestCommon.pm
@@ -1,4 +1,4 @@
-# Copyright (C) 2015-2021 all contributors <meta@public-inbox.org>
+# Copyright (C) all contributors <meta@public-inbox.org>
# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
# internal APIs used only for tests
@@ -19,7 +19,8 @@ BEGIN {
run_script start_script key2sub xsys xsys_e xqx eml_load tick
have_xapian_compact json_utf8 setup_public_inboxes create_inbox
tcp_host_port test_lei lei lei_ok $lei_out $lei_err $lei_opt
- test_httpd xbail require_cmd is_xdeeply tail_f);
+ test_httpd xbail require_cmd is_xdeeply tail_f
+ ignore_inline_c_missing);
require Test::More;
my @methods = grep(!/\W/, @Test::More::EXPORT);
eval(join('', map { "*$_=\\&Test::More::$_;" } @methods));
@@ -547,6 +548,11 @@ sub is_xdeeply ($$$) {
$ok;
}
+sub ignore_inline_c_missing {
+ $_[0] = join('', grep(/\S/, grep(!/compilation aborted/,
+ grep(!/\bInline\b/, split(/^/m, $_[0])))));
+}
+
sub test_lei {
SKIP: {
my ($cb) = pop @_;
@@ -571,8 +577,10 @@ SKIP: {
$ENV{LANG} = $ENV{LC_ALL} = 'C';
my (undef, $fn, $lineno) = caller(0);
my $t = "$fn:$lineno";
- state $lei_daemon = PublicInbox::Spawn->can('send_cmd4') ||
- eval { require Socket::MsgHdr; 1 };
+ state $lei_daemon = PublicInbox::Spawn->can('send_cmd4') || do {
+ require PublicInbox::Syscall;
+ PublicInbox::Syscall->can('send_cmd4');
+ } || eval { require Socket::MsgHdr; 1 };
unless ($lei_daemon) {
skip('Inline::C unconfigured/missing '.
'(mkdir -p ~/.cache/public-inbox/inline-c) OR Socket::MsgHdr missing',
diff --git a/t/lei-daemon.t b/t/lei-daemon.t
index b60c7ce6..e11105bc 100644
--- a/t/lei-daemon.t
+++ b/t/lei-daemon.t
@@ -1,5 +1,5 @@
#!perl -w
-# Copyright (C) 2020-2021 all contributors <meta@public-inbox.org>
+# Copyright (C) all contributors <meta@public-inbox.org>
# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
use strict; use v5.10.1; use PublicInbox::TestCommon;
use Socket qw(AF_UNIX SOCK_SEQPACKET MSG_EOR pack_sockaddr_un);
@@ -8,12 +8,16 @@ test_lei({ daemon_only => 1 }, sub {
my $send_cmd = PublicInbox::Spawn->can('send_cmd4') // do {
require PublicInbox::CmdIPC4;
PublicInbox::CmdIPC4->can('send_cmd4');
+ } // do {
+ require PublicInbox::Syscall;
+ PublicInbox::Syscall->can('send_cmd4');
};
$send_cmd or BAIL_OUT 'started testing lei-daemon w/o send_cmd4!';
my $sock = "$ENV{XDG_RUNTIME_DIR}/lei/5.seq.sock";
my $err_log = "$ENV{XDG_RUNTIME_DIR}/lei/errors.log";
lei_ok('daemon-pid');
+ ignore_inline_c_missing($lei_err);
is($lei_err, '', 'no error from daemon-pid');
like($lei_out, qr/\A[0-9]+\n\z/s, 'pid returned') or BAIL_OUT;
chomp(my $pid = $lei_out);
diff --git a/t/lei-externals.t b/t/lei-externals.t
index fed57789..284be1b9 100644
--- a/t/lei-externals.t
+++ b/t/lei-externals.t
@@ -76,6 +76,7 @@ test_lei(sub {
my $config_file = "$home/.config/lei/config";
my $store_dir = "$home/.local/share/lei";
lei_ok 'ls-external', \'ls-external on fresh install';
+ ignore_inline_c_missing($lei_err);
is($lei_out.$lei_err, '', 'ls-external no output, yet');
ok(!-e $config_file && !-e $store_dir,
'nothing created by ls-external');
^ permalink raw reply related [relevance 54%]
* [PATCH] lei: always open mail_sync.sqlite3 R/W
@ 2022-04-05 8:18 37% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2022-04-05 8:18 UTC (permalink / raw)
To: meta
This will make transparently upgrading from 1.7.0 -> 1.8.x
easier. Only a single user has access to mail_sync.sqlite3,
and R/W at the kernel-level is required for WAL, anyways.
---
lib/PublicInbox/LEI.pm | 6 +++---
lib/PublicInbox/LeiImport.pm | 8 ++++----
lib/PublicInbox/LeiImportKw.pm | 6 +++---
lib/PublicInbox/LeiLcat.pm | 10 +++++-----
lib/PublicInbox/LeiMailSync.pm | 34 +++++++++++++++++-----------------
lib/PublicInbox/LeiSearch.pm | 4 ++--
lib/PublicInbox/NetReader.pm | 4 ++--
7 files changed, 36 insertions(+), 36 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 4e0295fa..a7ddc21f 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -1,4 +1,4 @@
-# Copyright (C) 2020-2021 all contributors <meta@public-inbox.org>
+# Copyright (C) all contributors <meta@public-inbox.org>
# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
# Backend for `lei' (local email interface). Unlike the C10K-oriented
@@ -1502,11 +1502,11 @@ sub git_oid {
}
sub lms {
- my ($lei, $rw) = @_;
+ my ($lei, $creat) = @_;
my $sto = $lei->{sto} // _lei_store($lei) // return;
require PublicInbox::LeiMailSync;
my $f = "$sto->{priv_eidx}->{topdir}/mail_sync.sqlite3";
- (-f $f || $rw) ? PublicInbox::LeiMailSync->new($f) : undef;
+ (-f $f || $creat) ? PublicInbox::LeiMailSync->new($f) : undef;
}
sub sto_done_request {
diff --git a/lib/PublicInbox/LeiImport.pm b/lib/PublicInbox/LeiImport.pm
index bbc0634e..b9865829 100644
--- a/lib/PublicInbox/LeiImport.pm
+++ b/lib/PublicInbox/LeiImport.pm
@@ -1,4 +1,4 @@
-# Copyright (C) 2021 all contributors <meta@public-inbox.org>
+# Copyright (C) all contributors <meta@public-inbox.org>
# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
# front-end for the "lei import" sub-command
@@ -36,7 +36,7 @@ sub pmdir_cb { # called via wq_io_do from LeiPmdir->each_mdir_fn
my $kw = PublicInbox::MdirReader::flags2kw($fl);
substr($folder, 0, 0) = 'maildir:'; # add prefix
my $lse = $self->{lse} //= $self->{lei}->{sto}->search;
- my $lms = $self->{-lms_ro} //= $self->{lei}->lms; # may be 0 or undef
+ my $lms = $self->{-lms_rw} //= $self->{lei}->lms; # may be 0 or undef
my @oidbin = $lms ? $lms->name_oidbin($folder, $bn) : ();
@oidbin > 1 and warn("W: $folder/*/$$bn not unique:\n",
map { "\t".unpack('H*', $_)."\n" } @oidbin);
@@ -87,8 +87,8 @@ sub do_import_index ($$@) {
# $j = $net->net_concurrency($j); TODO
if ($lei->{opt}->{incremental} // 1) {
$net->{incremental} = 1;
- $net->{-lms_ro} = $lei->lms // 0;
- if ($self->{-import_kw} && $net->{-lms_ro} &&
+ $net->{-lms_rw} = $lei->lms // 0;
+ if ($self->{-import_kw} && $net->{-lms_rw} &&
!$lei->{opt}->{'new-only'} &&
$net->{imap_order}) {
require PublicInbox::LeiImportKw;
diff --git a/lib/PublicInbox/LeiImportKw.pm b/lib/PublicInbox/LeiImportKw.pm
index 54454511..52fd4043 100644
--- a/lib/PublicInbox/LeiImportKw.pm
+++ b/lib/PublicInbox/LeiImportKw.pm
@@ -1,4 +1,4 @@
-# Copyright (C) 2021 all contributors <meta@public-inbox.org>
+# Copyright (C) all contributors <meta@public-inbox.org>
# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
# WQ worker for dealing with LeiImport IMAP flags on already-imported messages
@@ -28,13 +28,13 @@ sub ipc_atfork_child {
$self->{verbose} = $lei->{opt}->{verbose};
$self->{lse} = $self->{sto}->search;
$self->{over} = $self->{lse}->over;
- $self->{-lms_ro} = $net->{-lms_ro} || die 'BUG: net->{-lms_ro} FALSE';
+ $self->{-lms_rw} = $net->{-lms_rw} || die 'BUG: net->{-lms_rw} FALSE';
$self->SUPER::ipc_atfork_child;
}
sub ck_update_kw { # via wq_io_do
my ($self, $url, $uid, $kw) = @_;
- my @oidbin = $self->{-lms_ro}->num_oidbin($url, $uid);
+ my @oidbin = $self->{-lms_rw}->num_oidbin($url, $uid);
my $uid_url = "$url/;UID=$uid";
@oidbin > 1 and warn("W: $uid_url not unique:\n",
map { "\t".unpack('H*', $_)."\n" } @oidbin);
diff --git a/lib/PublicInbox/LeiLcat.pm b/lib/PublicInbox/LeiLcat.pm
index 191f6f24..8d89cb73 100644
--- a/lib/PublicInbox/LeiLcat.pm
+++ b/lib/PublicInbox/LeiLcat.pm
@@ -1,4 +1,4 @@
-# Copyright (C) 2021 all contributors <meta@public-inbox.org>
+# Copyright (C) all contributors <meta@public-inbox.org>
# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
# lcat: local cat, display a local message by Message-ID or blob,
@@ -13,7 +13,7 @@ use PublicInbox::MID qw($MID_EXTRACT);
sub lcat_folder ($$;$$) {
my ($lei, $folder, $beg, $end) = @_;
- my $lms = $lei->{-lms_ro} //= $lei->lms // return;
+ my $lms = $lei->{-lms_rw} //= $lei->lms // return;
my $folders = [ $folder ];
eval { $lms->arg2folder($lei, $folders) };
return $lei->child_error(0, "# unknown folder: $folder") if $@;
@@ -31,7 +31,7 @@ sub lcat_folder ($$;$$) {
sub lcat_imap_uri ($$) {
my ($lei, $uri) = @_;
# cf. LeiXSearch->lcat_dump
- my $lms = $lei->{-lms_ro} //= $lei->lms // return;
+ my $lms = $lei->{-lms_rw} //= $lei->lms // return;
if (defined $uri->uid) {
push @{$lei->{lcat_todo}}, $lms->imap_oidhex($lei, $uri);
} elsif (defined(my $fid = $lms->fid_for($$uri))) {
@@ -45,7 +45,7 @@ sub lcat_nntp_uri ($$) {
my ($lei, $uri) = @_;
my $mid = $uri->message; # already unescaped by URI::news
return "mid:$mid" if defined($mid);
- my $lms = $lei->{-lms_ro} //= $lei->lms // return;
+ my $lms = $lei->{-lms_rw} //= $lei->lms // return;
my ($ng, $beg, $end) = $uri->group;
$uri->group($ng);
lcat_folder($lei, $$uri, $beg, $end);
@@ -118,7 +118,7 @@ could not extract Message-ID from $x
}
}
- delete $lei->{-lms_ro};
+ delete $lei->{-lms_rw};
@q ? join(' OR ', @q) : $lei->fail("no Message-ID in: @argv");
}
diff --git a/lib/PublicInbox/LeiMailSync.pm b/lib/PublicInbox/LeiMailSync.pm
index a9a65fd6..85480599 100644
--- a/lib/PublicInbox/LeiMailSync.pm
+++ b/lib/PublicInbox/LeiMailSync.pm
@@ -11,9 +11,9 @@ use PublicInbox::ContentHash qw(git_sha);
use Carp ();
sub dbh_new {
- my ($self, $rw) = @_;
+ my ($self) = @_;
my $f = $self->{filename};
- my $creat = $rw && !-s $f;
+ my $creat = !-s $f;
if ($creat) {
require PublicInbox::Syscall;
open my $fh, '+>>', $f or Carp::croak "open($f): $!";
@@ -23,11 +23,10 @@ sub dbh_new {
AutoCommit => 1,
RaiseError => 1,
PrintError => 0,
- ReadOnly => !$rw,
sqlite_use_immediate_transaction => 1,
});
# no sqlite_unicode, here, all strings are binary
- create_tables($self, $dbh) if $rw;
+ create_tables($self, $dbh);
$dbh->do('PRAGMA journal_mode = WAL') if $creat;
$dbh->do('PRAGMA case_sensitive_like = ON');
$dbh;
@@ -42,7 +41,7 @@ sub new {
}, $cls;
}
-sub lms_write_prepare { ($_[0]->{dbh} //= dbh_new($_[0], 1)); $_[0] }
+sub lms_write_prepare { ($_[0]->{dbh} //= dbh_new($_[0])); $_[0] }
sub lms_pause {
my ($self) = @_;
@@ -102,7 +101,7 @@ UPDATE folders SET loc = ? WHERE fid = ?
}
sub get_fid ($$$) {
- my ($sth, $folder, $dbh) = @_; # $dbh is set iff RW
+ my ($sth, $folder, $dbh) = @_;
$sth->bind_param(1, $folder, SQL_BLOB);
$sth->execute;
my ($fid) = $sth->fetchrow_array;
@@ -118,36 +117,37 @@ sub get_fid ($$$) {
}
sub fid_for {
- my ($self, $folder, $rw) = @_;
- my $dbh = $self->{dbh} //= dbh_new($self, $rw);
+ my ($self, $folder, $creat) = @_;
+ my $dbh = $self->{dbh} //= dbh_new($self);
my $sth = $dbh->prepare_cached(<<'', undef, 1);
SELECT fid FROM folders WHERE loc = ? LIMIT 1
- my $rw_dbh = $dbh->{ReadOnly} ? undef : $dbh;
- my $fid = get_fid($sth, $folder, $rw_dbh);
+ my $fid = get_fid($sth, $folder, $dbh);
return $fid if defined($fid);
# caller had trailing slash (LeiToMail)
if ($folder =~ s!\A((?:maildir|mh):.*?)/+\z!$1!i) {
- $fid = get_fid($sth, $folder, $rw_dbh);
+ $fid = get_fid($sth, $folder, $dbh);
if (defined $fid) {
- update_fid($dbh, $fid, $folder) if $rw;
+ update_fid($dbh, $fid, $folder);
return $fid;
}
# sometimes we stored trailing slash..
} elsif ($folder =~ m!\A(?:maildir|mh):!i) {
- $fid = get_fid($sth, $folder, $rw_dbh);
+ $fid = get_fid($sth, $folder, $dbh);
if (defined $fid) {
- update_fid($dbh, $fid, $folder) if $rw;
+ update_fid($dbh, $fid, $folder);
return $fid;
}
- } elsif ($rw && $folder =~ m!\Aimaps?://!i) {
+ } elsif ($creat && $folder =~ m!\Aimaps?://!i) {
require PublicInbox::URIimap;
- PublicInbox::URIimap->new($folder)->uidvalidity //
+ my $uri = PublicInbox::URIimap->new($folder);
+ $uri->uidvalidity //
Carp::croak("BUG: $folder has no UIDVALIDITY");
+ defined($uri->uid) and Carp::confess("BUG: $folder has UID");
}
- return unless $rw;
+ return unless $creat;
($fid) = $dbh->selectrow_array('SELECT MAX(fid) FROM folders');
$fid += 1;
diff --git a/lib/PublicInbox/LeiSearch.pm b/lib/PublicInbox/LeiSearch.pm
index d0ca13f0..936c2751 100644
--- a/lib/PublicInbox/LeiSearch.pm
+++ b/lib/PublicInbox/LeiSearch.pm
@@ -1,4 +1,4 @@
-# Copyright (C) 2020-2021 all contributors <meta@public-inbox.org>
+# Copyright (C) all contributors <meta@public-inbox.org>
# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
# read-only counterpart for PublicInbox::LeiStore
@@ -101,7 +101,7 @@ sub xoids_for {
my $git = $self->git;
my $xoids = {};
# no lms when used via {ale}:
- my $lms = $self->{-lms_ro} //= lms($self) if defined($self->{topdir});
+ my $lms = $self->{-lms_rw} //= lms($self) if defined($self->{topdir});
for my $mid (@$mids) {
for my $o (@overs) {
my ($id, $prev);
diff --git a/lib/PublicInbox/NetReader.pm b/lib/PublicInbox/NetReader.pm
index 032b4fda..c1af03a3 100644
--- a/lib/PublicInbox/NetReader.pm
+++ b/lib/PublicInbox/NetReader.pm
@@ -1,4 +1,4 @@
-# Copyright (C) 2021 all contributors <meta@public-inbox.org>
+# Copyright (C) all contributors <meta@public-inbox.org>
# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
# common reader code for IMAP and NNTP (and maybe JMAP)
@@ -481,7 +481,7 @@ sub itrk_last ($$;$$) {
my ($self, $uri, $r_uidval, $mic) = @_;
return (undef, undef, $r_uidval) unless $self->{incremental};
my ($itrk, $l_uid, $l_uidval);
- if (defined(my $lms = $self->{-lms_ro})) { # LeiMailSync or 0
+ if (defined(my $lms = $self->{-lms_rw})) { # LeiMailSync or 0
$uri->uidvalidity($r_uidval) if defined $r_uidval;
if ($mic) {
my $auth = $mic->Authmechanism // '';
^ permalink raw reply related [relevance 37%]
* Re: [PATCH] t/lei-sigpipe.t: ensure SIGPIPE is not ignored instead of not blocked
2022-03-14 22:14 71% ` Eric Wong
@ 2022-03-15 2:56 71% ` Julien Moutinho
0 siblings, 0 replies; 200+ results
From: Julien Moutinho @ 2022-03-15 2:56 UTC (permalink / raw)
To: Eric Wong; +Cc: meta, Dominique Martinet
Le lun. 14 mars 2022 22h14 +0000, Eric Wong a écrit :
> I (and any other project maintainer) would appreciate more
> complete explanations in the future as to why a patch is needed.
> Thanks again.
Oh sorry, being immersed into the testing I tought
that it was obvious. I should have clarified
that the unblocking fix was not working.
Thanks for the merge!
^ permalink raw reply [relevance 71%]
* Re: [PATCH] t/lei-sigpipe.t: ensure SIGPIPE is not ignored instead of not blocked
2022-03-11 10:42 69% ` [PATCH] t/lei-sigpipe.t: ensure SIGPIPE is not ignored instead of not blocked Julien Moutinho
@ 2022-03-14 22:14 71% ` Eric Wong
2022-03-15 2:56 71% ` Julien Moutinho
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2022-03-14 22:14 UTC (permalink / raw)
To: Julien Moutinho; +Cc: meta, Dominique Martinet
Julien Moutinho <julm+public-inbox@sourcephile.fr> wrote:
> Ignoring a signal is different than blocking a signal.
Thanks, pushed as commit f1e4e14793d155ea7d6ed7a6858b668e97c7e5d8
I wasn't sure why this patch was necessary at first. Thus I
took take a closer look at systemd behavior to confirm, and
updated the message:
Ignoring a signal is different than blocking a signal, and the
"IgnoreSIGPIPE" option of systemd ignores.
[ew: note systemd behavior]
Acked-by: Eric Wong <e@80x24.org>
I (and any other project maintainer) would appreciate more
complete explanations in the future as to why a patch is needed.
Thanks again.
^ permalink raw reply [relevance 71%]
* Re: Failed 'lei q' blocks 'lei init' from working
2022-03-12 21:04 87% Failed 'lei q' blocks 'lei init' from working Nícolas F. R. A. Prado
@ 2022-03-13 0:06 90% ` Kyle Meyer
0 siblings, 0 replies; 200+ results
From: Kyle Meyer @ 2022-03-13 0:06 UTC (permalink / raw)
To: Nícolas F. R. A. Prado; +Cc: meta
Nícolas F. R. A. Prado writes:
> Hi,
>
> Right after building lei from source, I get an error running the 'q' command:
>
> [nfraprado@notapiano public-inbox]$ ./lei.sh q -I https://lore.kernel.org/all/ -o ~/.mail/lei/test --threads --dedupe=mid 'dfn:mt8192.dtsi rt:1.month.ago..'
> open(/home/nfraprado/.local/share/lei/store/mail_sync.sqlite3): No such file or directory at /home/nfraprado/ext/git/public-inbox/lib/PublicInbox/LeiQuery.pm line 37.
>
> That's somewhat reasonable since I haven't issued an init first. But if I then
> try to init [...]:
Hmm, an explicit lei-init call shouldn't be needed (i.e. lei-q should
already behave as you propose at the end of your message, initializing
the store/config if necessary).
Eric has even considered dropping lei-init entirely:
https://public-inbox.org/meta/20210325083207.GA30551@dcvr/
> Not quite sure if it's a Perl thing or a bug in public-inbox, but since it's
> unusual behavior (and took me some time to find out) I figured I'd report it.
This reminded me a bit of
https://public-inbox.org/meta/20211120153054.xq6jfad2m2buoic6@gmail.com/
Perhaps a lei-daemon process was running from some previous
experimentation?
^ permalink raw reply [relevance 90%]
* Failed 'lei q' blocks 'lei init' from working
@ 2022-03-12 21:04 87% Nícolas F. R. A. Prado
2022-03-13 0:06 90% ` Kyle Meyer
0 siblings, 1 reply; 200+ results
From: Nícolas F. R. A. Prado @ 2022-03-12 21:04 UTC (permalink / raw)
To: meta
Hi,
Right after building lei from source, I get an error running the 'q' command:
[nfraprado@notapiano public-inbox]$ ./lei.sh q -I https://lore.kernel.org/all/ -o ~/.mail/lei/test --threads --dedupe=mid 'dfn:mt8192.dtsi rt:1.month.ago..'
open(/home/nfraprado/.local/share/lei/store/mail_sync.sqlite3): No such file or directory at /home/nfraprado/ext/git/public-inbox/lib/PublicInbox/LeiQuery.pm line 37.
That's somewhat reasonable since I haven't issued an init first. But if I then
try to init:
[nfraprado@notapiano public-inbox]$ ./lei.sh init
# leistore.dir=/home/nfraprado/.local/share/lei/store newly initialized
[nfraprado@notapiano public-inbox]$ ./lei.sh init
# leistore.dir=/home/nfraprado/.local/share/lei/store newly initialized
No store is ever created. If I reboot (or issue the 'e' sysrq to basically
restart userspace) and then run the init command right away, it does work.
So it seems to me that there's some state that is kept after running the q
command which blocks init from working, hence requiring the reboot to clear that
state.
Not quite sure if it's a Perl thing or a bug in public-inbox, but since it's
unusual behavior (and took me some time to find out) I figured I'd report it.
Making 'q' run 'init' if not already done so could also help to work around this
issue.
Thanks,
Nícolas
^ permalink raw reply [relevance 87%]
* [PATCH] t/lei-sigpipe.t: ensure SIGPIPE is not ignored instead of not blocked
2022-02-27 11:17 62% ` [PATCH] t/lei-sigpipe: ensure SIGPIPE is unblocked for this test Eric Wong
@ 2022-03-11 10:42 69% ` Julien Moutinho
2022-03-14 22:14 71% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Julien Moutinho @ 2022-03-11 10:42 UTC (permalink / raw)
To: meta; +Cc: Dominique Martinet, Eric Wong, Julien Moutinho
Ignoring a signal is different than blocking a signal.
---
t/lei-sigpipe.t | 14 +++++---------
1 file changed, 5 insertions(+), 9 deletions(-)
diff --git a/t/lei-sigpipe.t b/t/lei-sigpipe.t
index 7fab9aeb..55c208e2 100644
--- a/t/lei-sigpipe.t
+++ b/t/lei-sigpipe.t
@@ -4,20 +4,16 @@
use strict;
use v5.10.1;
use PublicInbox::TestCommon;
-use POSIX qw(WTERMSIG WIFSIGNALED SIGPIPE SIG_UNBLOCK SIG_SETMASK sigprocmask);
+use POSIX qw(WTERMSIG WIFSIGNALED SIGPIPE);
use PublicInbox::OnDestroy;
-# undo systemd (and similar) blocking SIGPIPE, since lei expects to be run
+# undo systemd (and similar) ignoring SIGPIPE, since lei expects to be run
# from an interactive terminal:
# https://public-inbox.org/meta/20220227080422.gyqowrxomzu6gyin@sourcephile.fr/
-my $set = POSIX::SigSet->new;
-my $old = POSIX::SigSet->new;
-$set->emptyset or xbail "sigemptyset $!";
-$old->emptyset or xbail "sigemptyset $!";
-$set->addset(SIGPIPE);
-sigprocmask(SIG_UNBLOCK, $set, $old) or xbail "SIG_UNBLOCK: $!";
+my $oldSIGPIPE = $SIG{PIPE};
+$SIG{PIPE} = 'DEFAULT';
my $cleanup = PublicInbox::OnDestroy->new($$, sub {
- sigprocmask(SIG_SETMASK, $old);
+ $SIG{PIPE} = $oldSIGPIPE;
});
test_lei(sub {
--
2.35.1
^ permalink raw reply related [relevance 69%]
* [PATCH] t/lei-sigpipe: ensure SIGPIPE is unblocked for this test
2022-02-27 8:04 71% ` Julien Moutinho
@ 2022-02-27 11:17 62% ` Eric Wong
2022-03-11 10:42 69% ` [PATCH] t/lei-sigpipe.t: ensure SIGPIPE is not ignored instead of not blocked Julien Moutinho
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2022-02-27 11:17 UTC (permalink / raw)
To: Julien Moutinho; +Cc: Dominique Martinet, meta
Julien Moutinho <julm+public-inbox@sourcephile.fr> wrote:
> Le dim. 27 févr. 2022 16h23 +0900, Dominique Martinet a écrit :
> > I guess we can now open a nix issue asking to reset signal handlers
> > after fork, before executing its builder?...
> Well spotted Dominique, thanks!
>
> Turns out this is actually systemd which ignores SIGPIPE.
> I'm able to pass t/lei-sigpipe.t when my `nix build`
> connects to a nix-daemon run manually outside systemd.
> There is already a 3yo issue on nix's tracker for that:
> https://github.com/NixOS/nix/issues/2803
> suggesting to add to nix-daemon.service:
> > [Service]
> > IgnoreSIGPIPE=no
> And that indeed also makes t/lei-sigpipe.t succeed.
>
> In the meantime I'll disable t/lei-sigpipe.t
> to finally get public-inbox added to Nixpkgs.
>
> Thank you Dominique and Eric!
You're welcome and thank you for tracking this down.
Yes, blocking SIGPIPE makes sense for most daemons so I think
systemd is being reasonable, here. However, this lei test is
for simulating an interactive environment, so re-enabling
SIGPIPE under t/lei-sigpipe.t seems best... I initially
considered adding this for all test_lei(), but most lei tests
don't care and I didn't want to waste CPU time.
------8<-----
Subject: [PATCH] t/lei-sigpipe: ensure SIGPIPE is unblocked for this test
Tests run under systemd (and similar) have SIGPIPE blocked by
default. This was causing this SIGPIPE test to get stuck when
run by automated builders used by Nix. Thanks to Julien
Moutinho and Dominique Martinet for tracking down this failure.
Reported-by: Julien Moutinho <julm+public-inbox@sourcephile.fr>
Reported-by: Dominique Martinet <asmadeus@codewreck.org>
Link: https://public-inbox.org/meta/20220227080422.gyqowrxomzu6gyin@sourcephile.fr/
---
t/lei-sigpipe.t | 19 +++++++++++++++++--
1 file changed, 17 insertions(+), 2 deletions(-)
diff --git a/t/lei-sigpipe.t b/t/lei-sigpipe.t
index 6b2772a6..7fab9aeb 100644
--- a/t/lei-sigpipe.t
+++ b/t/lei-sigpipe.t
@@ -1,10 +1,25 @@
#!perl -w
-# Copyright (C) 2021 all contributors <meta@public-inbox.org>
+# Copyright (C) all contributors <meta@public-inbox.org>
# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
use strict;
use v5.10.1;
use PublicInbox::TestCommon;
-use POSIX qw(WTERMSIG WIFSIGNALED SIGPIPE);
+use POSIX qw(WTERMSIG WIFSIGNALED SIGPIPE SIG_UNBLOCK SIG_SETMASK sigprocmask);
+use PublicInbox::OnDestroy;
+
+# undo systemd (and similar) blocking SIGPIPE, since lei expects to be run
+# from an interactive terminal:
+# https://public-inbox.org/meta/20220227080422.gyqowrxomzu6gyin@sourcephile.fr/
+my $set = POSIX::SigSet->new;
+my $old = POSIX::SigSet->new;
+$set->emptyset or xbail "sigemptyset $!";
+$old->emptyset or xbail "sigemptyset $!";
+$set->addset(SIGPIPE);
+sigprocmask(SIG_UNBLOCK, $set, $old) or xbail "SIG_UNBLOCK: $!";
+my $cleanup = PublicInbox::OnDestroy->new($$, sub {
+ sigprocmask(SIG_SETMASK, $old);
+});
+
test_lei(sub {
my $f = "$ENV{HOME}/big.eml";
my $imported;
^ permalink raw reply related [relevance 62%]
* Re: [PATCH] t/lei-sigpipe: attempt to improve diagnostics for stuck test
2022-02-27 7:23 71% ` Dominique Martinet
@ 2022-02-27 8:04 71% ` Julien Moutinho
2022-02-27 11:17 62% ` [PATCH] t/lei-sigpipe: ensure SIGPIPE is unblocked for this test Eric Wong
0 siblings, 1 reply; 200+ results
From: Julien Moutinho @ 2022-02-27 8:04 UTC (permalink / raw)
To: Dominique Martinet; +Cc: Eric Wong, meta
Le dim. 27 févr. 2022 16h23 +0900, Dominique Martinet a écrit :
> I guess we can now open a nix issue asking to reset signal handlers
> after fork, before executing its builder?...
Well spotted Dominique, thanks!
Turns out this is actually systemd which ignores SIGPIPE.
I'm able to pass t/lei-sigpipe.t when my `nix build`
connects to a nix-daemon run manually outside systemd.
There is already a 3yo issue on nix's tracker for that:
https://github.com/NixOS/nix/issues/2803
suggesting to add to nix-daemon.service:
> [Service]
> IgnoreSIGPIPE=no
And that indeed also makes t/lei-sigpipe.t succeed.
In the meantime I'll disable t/lei-sigpipe.t
to finally get public-inbox added to Nixpkgs.
Thank you Dominique and Eric!
^ permalink raw reply [relevance 71%]
* Re: [PATCH] t/lei-sigpipe: attempt to improve diagnostics for stuck test
2022-02-27 6:41 71% ` Julien Moutinho
@ 2022-02-27 7:23 71% ` Dominique Martinet
2022-02-27 8:04 71% ` Julien Moutinho
0 siblings, 1 reply; 200+ results
From: Dominique Martinet @ 2022-02-27 7:23 UTC (permalink / raw)
To: Julien Moutinho; +Cc: Eric Wong, meta
Hello,
Julien Moutinho wrote on Sun, Feb 27, 2022 at 07:41:44AM +0100:
> Le dim. 27 févr. 2022 05h15 +0100, Julien Moutinho a écrit :
> > Yes, I'm using:
> > $ nix-daemon --version
> > > nix-daemon (Nix) 2.7.0pre20220127_558c4ee
> Dominique reminded me that that fix was first intended for
> `nix develop`, so I've asked whether it also applies to `nix build`:
> https://github.com/NixOS/nix/pull/5683#issuecomment-1053239025
I've replied over there, I don't think leaked fd is a problem.
However Julien's latest traces gave a good enough hint: the lei process
that should die from sigpipe just didn't die...
And surely enough, nix adds SIGPIPE to ignores.
This can be confirmed by looking at SigIgn in /proc/xxx/status of one of
the builder PIDs:
# grep SigIgn /proc/3124202/status
SigIgn: 0000000000001004
Where 0x1000 is the 13th bit:
$ kill -l 13
PIPE
Running tests manually won't inherit masked signals and thus won't
reproduce.
I guess we can now open a nix issue asking to reset signal handlers
after fork, before executing its builder?...
--
Dominique
^ permalink raw reply [relevance 71%]
* Re: [PATCH] t/lei-sigpipe: attempt to improve diagnostics for stuck test
2022-02-27 4:15 71% ` Julien Moutinho
@ 2022-02-27 6:41 71% ` Julien Moutinho
2022-02-27 7:23 71% ` Dominique Martinet
0 siblings, 1 reply; 200+ results
From: Julien Moutinho @ 2022-02-27 6:41 UTC (permalink / raw)
To: Eric Wong; +Cc: Dominique Martinet, meta
Le dim. 27 févr. 2022 05h15 +0100, Julien Moutinho a écrit :
> Yes, I'm using:
> $ nix-daemon --version
> > nix-daemon (Nix) 2.7.0pre20220127_558c4ee
Dominique reminded me that that fix was first intended for
`nix develop`, so I've asked whether it also applies to `nix build`:
https://github.com/NixOS/nix/pull/5683#issuecomment-1053239025
^ permalink raw reply [relevance 71%]
* Re: [PATCH] t/lei-sigpipe: attempt to improve diagnostics for stuck test
2022-02-22 6:44 71% ` Eric Wong
@ 2022-02-27 4:15 71% ` Julien Moutinho
2022-02-27 6:41 71% ` Julien Moutinho
0 siblings, 1 reply; 200+ results
From: Julien Moutinho @ 2022-02-27 4:15 UTC (permalink / raw)
To: Eric Wong; +Cc: Dominique Martinet, meta
Le mar. 22 févr. 2022 06h44 +0000, Eric Wong a écrit :
> Julien Moutinho <julm+public-inbox@sourcephile.fr> wrote:
> > This deadlock appears only when the test is run by nix,
> > it doesn't when I run it manually from a terminal
> > by entering the sandbox:
>
> Oh, btw, is this with the FD_CLOEXEC fixes to nix shell itself?
> https://public-inbox.org/meta/YfnoOVOG3TrbqZFs@codewreck.org/
Yes, I'm using:
$ nix-daemon --version
> nix-daemon (Nix) 2.7.0pre20220127_558c4ee
Which contains the commit from https://github.com/NixOS/nix/pull/5683 :
$ git merge-base --is-ancestor 6e0cbc666b60515b5e201dd28855f5fe1de9a107 558c4ee
$ echo $?
> 0
^ permalink raw reply [relevance 71%]
* Re: [PATCH] t/lei-sigpipe: attempt to improve diagnostics for stuck test
2022-02-20 1:38 51% ` Julien Moutinho
@ 2022-02-22 6:44 71% ` Eric Wong
2022-02-27 4:15 71% ` Julien Moutinho
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2022-02-22 6:44 UTC (permalink / raw)
To: Julien Moutinho; +Cc: Dominique Martinet, meta
Julien Moutinho <julm+public-inbox@sourcephile.fr> wrote:
> This deadlock appears only when the test is run by nix,
> it doesn't when I run it manually from a terminal
> by entering the sandbox:
Oh, btw, is this with the FD_CLOEXEC fixes to nix shell itself?
https://public-inbox.org/meta/YfnoOVOG3TrbqZFs@codewreck.org/
^ permalink raw reply [relevance 71%]
* Re: [PATCH] t/lei-sigpipe: attempt to improve diagnostics for stuck test
2022-02-17 21:02 63% ` [PATCH] t/lei-sigpipe: attempt to improve diagnostics for stuck test Eric Wong
@ 2022-02-20 1:38 51% ` Julien Moutinho
2022-02-22 6:44 71% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Julien Moutinho @ 2022-02-20 1:38 UTC (permalink / raw)
To: Eric Wong; +Cc: Dominique Martinet, meta
[-- Attachment #1: Type: text/plain, Size: 3484 bytes --]
Hi Eric,
Thank you for following up on this
despite your having more important matters on your shoulders :\
Le jeu. 17 févr. 2022 21h02 +0000, Eric Wong a écrit :
> > Looks like it is waiting on sysread() in
> > https://public-inbox.org/public-inbox.git/tree/t/lei-sigpipe.t#n35
>
> OK, that makes more sense...
My tracing with print statements shows that sysread is passed
and that it is $tp->join which hangs:
[nixbld@localhost:/build]$ cat stderr.log
> lei-sigpipe: begin
> lei-sigpipe: start_script: done
> lei-sigpipe: close w: done
> lei-sigpipe: sysread
> lei-sigpipe: sysread: done
> lei-sigpipe: close r: done
[nixbld@localhost:/build]$ cat start_script.log
> start_script: begin
> start_script: fork
> start_script: pid!=0
> start_script: pid==0
> start_script: pid!=0: tail
> start_script: pid==0: if sub
> start_script: pid==0: exec lei
ps auxwwf | grep nixbld
> nixbld1 491489 0.1 0.0 5312 4036 ? Ss 02:12 0:01 \_ bash -e /nix/store/qf3mzpvsmkrw963xchbivcci06078n13-builder.sh
> nixbld1 494307 0.0 0.1 16672 12616 ? S 02:12 0:00 \_ perl -MExtUtils::Command::MM -MTest::Harness -e undef *Test::Harness::Switches; test_harness(0, 'blib/lib', 'blib/arch') t/lei-sigpipe.t
> nixbld1 494311 0.2 0.5 49320 40068 ? S 02:12 0:02 | \_ t/lei-sigpipe.t
> nixbld1 494445 0.0 0.0 0 0 ? Z 02:12 0:00 | \_ [/nix/store/176g] <defunct>
> nixbld1 494447 0.0 0.1 14388 10176 ? S 02:12 0:00 | \_ /nix/store/6wcdxh3h5jp9qgrccq3n71wnalca5mf5-perl-5.34.0/bin/perl -w -I/nix/store/6wcdxh3h5jp9qgrccq3n71wnalca5mf5-perl-5.34.0/lib/perl5/site_perl -I/nix/store/jibsc747pfzmyxp5lsqgzyvrs2lnkws6-perl5.34.0-Mail-IMAPClient-3.42/lib/perl5/site_perl -I/nix/st blib/script/lei q -q -t z:1..
> nixbld1 494427 0.1 0.5 50664 41892 ? S 02:12 0:00 \_ lei-daemon /build/pi-lei-sigpipe-2750-0bv8/lei-daemon/xdg_run/lei/5.seq.sock
> nixbld1 494433 0.0 0.4 50260 39580 ? S 02:12 0:00 \_ lei/store /build/pi-lei-sigpipe-2750-0bv8/lei-daemon/.local/share
The defunct process with truncated path "/nix/store/176g"
is points to /nix/store/176gh50y24c0lx2bnnmsvf9wazb73php-coreutils-9.0
It appears only when I export TAIL=${coreutils}/bin/tail
which does not workaround the hang.
This deadlock appears only when the test is run by nix,
it doesn't when I run it manually from a terminal
by entering the sandbox:
[nixbld@localhost:/]$ (cd /build/public-inbox-1.7.0; export PERL_INLINE_DIRECTORY=$PWD/inline-c; rm -rf $PERL_INLINE_DIRECTORY; mkdir $PERL_INLINE_DIRECTORY; prove -bvw t/lei-sigpipe.t )
> t/lei-sigpipe.t ..
> ok 1 - lei import $TMPDIR/big.eml
> ok 2 - read one byte
> ok 3 - signaled
> ok 4 - got SIGPIPE
> ok 5 - read one byte
> ok 6 - signaled -f mboxcl2
> ok 7 - got SIGPIPE -f mboxcl2
> ok 8 - read one byte
> ok 9 - signaled -f text
> ok 10 - got SIGPIPE -f text
> ok 11 - lei daemon-pid (daemon-pid after t/lei-sigpipe.t:58)
> ok 12 - daemon running after t/lei-sigpipe.t:58
> ok 13 - lei daemon-kill (daemon-kill after t/lei-sigpipe.t:58)
> ok 14 - t/lei-sigpipe.t:58 daemon stopped
> ok 15 - t/lei-sigpipe.t:58 daemon XDG_RUNTIME_DIR/lei/errors.log empty
> 1..15
> ok
> All tests successful.
> Files=1, Tests=15, 8 wallclock secs ( 0.06 usr 0.08 sys + 3.55 cusr 2.81 csys = 6.50 CPU)
> Result: PASS
[-- Attachment #2: 0001-trace-t-lei-sigpipe.t.patch --]
[-- Type: text/plain, Size: 3841 bytes --]
From d919b3441bdf61e8dfae0c08005b8a77662ddb91 Mon Sep 17 00:00:00 2001
From: Julien Moutinho <julm+public-inbox@sourcephile.fr>
Date: Sun, 20 Feb 2022 01:33:04 +0100
Subject: [PATCH] trace t/lei-sigpipe.t
---
lib/PublicInbox/TestCommon.pm | 10 ++++++++++
t/lei-sigpipe.t | 22 +++++++++++++++-------
2 files changed, 25 insertions(+), 7 deletions(-)
diff --git a/lib/PublicInbox/TestCommon.pm b/lib/PublicInbox/TestCommon.pm
index ca732811..d8416f90 100644
--- a/lib/PublicInbox/TestCommon.pm
+++ b/lib/PublicInbox/TestCommon.pm
@@ -439,6 +439,9 @@ sub start_script {
my $run_mode = $ENV{TEST_RUN_MODE} // $opt->{run_mode} // 2;
my $sub = $run_mode == 0 ? undef : key2sub($key);
my $tail;
+ open my $errfh, '+>>', "/build/start_script.log" or xbail $!;
+ $errfh->autoflush(1);
+ print $errfh "\nstart_script: begin\n";
if ($tail_cmd) {
my @paths;
for (@argv) {
@@ -459,8 +462,10 @@ sub start_script {
}
$tail = tail_f(@paths);
}
+ print $errfh "\nstart_script: fork\n";
my $pid = fork // die "fork: $!\n";
if ($pid == 0) {
+ print $errfh "\nstart_script: pid==0\n";
eval { PublicInbox::DS->Reset };
# pretend to be systemd (cf. sd_listen_fds(3))
# 3 == SD_LISTEN_FDS_START
@@ -484,18 +489,23 @@ sub start_script {
}
if ($opt->{-C}) { chdir($opt->{-C}) or die "chdir: $!" }
$0 = join(' ', @$cmd);
+ print $errfh "\nstart_script: pid==0: if sub\n";
if ($sub) {
+ print $errfh "\nstart_script: pid==0: sub\n";
eval { PublicInbox::DS->Reset };
_run_sub($sub, $key, \@argv);
POSIX::_exit($? >> 8);
} else {
+ print $errfh "\nstart_script: pid==0: exec $key\n";
exec(key2script($key), @argv);
die "FAIL: ",join(' ', $key, @argv), ": $!\n";
}
}
+ print $errfh "\nstart_script: pid!=0\n";
require PublicInbox::AutoReap;
my $td = PublicInbox::AutoReap->new($pid);
$td->{-extra} = $tail;
+ print $errfh "\nstart_script: pid!=0: tail\n";
$td;
}
diff --git a/t/lei-sigpipe.t b/t/lei-sigpipe.t
index 6b2772a6..e5e701dd 100644
--- a/t/lei-sigpipe.t
+++ b/t/lei-sigpipe.t
@@ -6,7 +6,7 @@ use v5.10.1;
use PublicInbox::TestCommon;
use POSIX qw(WTERMSIG WIFSIGNALED SIGPIPE);
test_lei(sub {
- my $f = "$ENV{HOME}/big.eml";
+ my $f = "/build/big.eml";
my $imported;
for my $out ([], [qw(-f mboxcl2)], [qw(-f text)]) {
pipe(my ($r, $w)) or BAIL_OUT $!;
@@ -27,25 +27,33 @@ EOM
}
lei_ok(qw(import), $f) if $imported++ == 0;
- open my $errfh, '+>>', "$ENV{HOME}/stderr.log" or xbail $!;
+ open my $errfh, '+>>', "/build/stderr.log" or xbail $!;
+ $errfh->autoflush(1);
+ print $errfh "\nlei-sigpipe: begin\n";
my $opt = { run_mode => 0, 2 => $errfh, 1 => $w };
my $cmd = [qw(lei q -q -t), @$out, 'z:1..'];
my $tp = start_script($cmd, undef, $opt);
+ print $errfh "\nlei-sigpipe: start_script: done\n";
close $w;
+ print $errfh "\nlei-sigpipe: close w: done\n";
vec(my $rvec = '', fileno($r), 1) = 1;
- if (!select($rvec, undef, undef, 30)) {
+ if (!select($rvec, undef, undef, 3)) {
seek($errfh, 0, 0) or xbail $!;
my $s = do { local $/; <$errfh> };
- xbail "lei q had no output after 30s, stderr=$s";
+ xbail "lei q had no output after 3s, stderr=$s";
}
+ print $errfh "\nlei-sigpipe: sysread\n";
is(sysread($r, my $buf, 1), 1, 'read one byte');
+ print $errfh "\nlei-sigpipe: sysread: done\n";
close $r; # trigger SIGPIPE
+ print $errfh "\nlei-sigpipe: close r: done\n";
$tp->join;
+ print $errfh "\nlei-sigpipe: join: done\n";
ok(WIFSIGNALED($?), "signaled @$out");
is(WTERMSIG($?), SIGPIPE, "got SIGPIPE @$out");
- seek($errfh, 0, 0) or xbail $!;
- my $s = do { local $/; <$errfh> };
- is($s, '', "quiet after sigpipe @$out");
+ #seek($errfh, 0, 0) or xbail $!;
+ #my $s = do { local $/; <$errfh> };
+ #is($s, '', "quiet after sigpipe @$out");
}
});
--
2.34.1
^ permalink raw reply related [relevance 51%]
* [PATCH] t/lei-sigpipe: attempt to improve diagnostics for stuck test
@ 2022-02-17 21:02 63% ` Eric Wong
2022-02-20 1:38 51% ` Julien Moutinho
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2022-02-17 21:02 UTC (permalink / raw)
To: Julien Moutinho; +Cc: Dominique Martinet, meta
Julien Moutinho <julm+public-inbox@sourcephile.fr> wrote:
> With Inline::C, and with Nix's sandbox
> --------------------------------------
>
> Using (in pkgs/servers/mail/public-inbox/default.nix):
> > preCheck = ''
> > perl certs/create-certs.perl
> > export HOME=$(mktemp -d)
> > mkdir -p $HOME/.cache/public-inbox/inline-c
> > '';
>
> $ nix -L build -f . public-inbox
> > [...]
> > perl5.34.0-public-inbox> t/lei-refresh-mail-sync.t .... ok
> Hangs like forever after this line, while on t/lei-sigpipe.t:
>
> $ ps auxwwf
> > root 1755997 0.0 0.0 453172 2800 ? Ssl déc.04 0:03 nix-daemon --daemon
> > root 1811429 0.0 0.0 453652 1928 ? Ssl déc.04 0:00 \_ nix-daemon 1757028
> > root 1811455 0.0 0.0 453652 2004 ? Ssl déc.04 0:00 \_ nix-daemon 1757031
> > root 1839837 0.0 0.0 453652 1832 ? Ssl déc.04 0:00 \_ nix-daemon 1757027
> > root 1839841 0.0 0.0 453652 1996 ? Ssl déc.04 0:00 \_ nix-daemon 1757030
> > root 1839856 0.0 0.0 453652 1832 ? Ssl déc.04 0:00 \_ nix-daemon 1757029
> > root 3135741 0.0 0.2 603352 17080 ? Ssl 01:05 0:00 \_ nix-daemon 3135672
> > nixbld1 3137110 0.0 0.0 5200 4072 ? Ss 01:05 0:01 \_ bash -e /nix/store/qf3mzpvsmkrw963xchbivcci06078n13-builder.sh
> > nixbld1 3140619 0.0 0.0 4996 4012 ? S 01:05 0:00 \_ make SHELL=/nix/store/l0wlqpbsvh1pgvhcdhw7qkka3d31si7k-bash-5.1-p8/bin/bash VERBOSE=y test
> > nixbld1 3140633 0.0 0.1 17188 13352 ? S 01:05 0:01 | \_ /nix/store/vslsa0l17xjcrdgm2knwj0z5hlvf73m7-perl-5.34.0/bin/perl -MExtUtils::Command::MM -MTest::Harness -e undef *Test::Harness::Switches; test_harness(0, 'blib/lib', 'blib/arch') t/admin.t t/altid.t t/altid_v2.t t/cgi.t t/cmd_ipc.t t/content_hash.t t/dir_idle.t t/edit.t t/eml_content_disposition.t t/eml_content_type.t t/epoll.t t/extindex-psgi.t t/extsearch.t t/fake_inotify.t t/feed.t t/gcf2.t t/gcf2_client.t t/httpd.t t/idx_stack.t t/imap.t t/imap_searchqp.t t/imapd-tls.t t/imapd.t t/inbox_idle.t t/index-git-times.t t/indexlevels-mirror.t t/ipc.t t/kqnotify.t t/lei-auto-watch.t t/lei-convert.t t/lei-daemon.t t/lei-export-kw.t t/lei-externals.t t/lei-import-http.t t/lei-import-imap.t t/lei-import-maildir.t t/lei-import-nntp.t t/lei-import.t t/lei-index.t t/lei-inspect.t t/lei-lcat.t t/lei-p2q.t t/lei-q-kw.t t/lei-q-remote-import.t t/lei-q-save.t t/lei-q-thread.t t/lei-refresh-mail-sync.t t/lei-sigpipe.t t/lei-tag.t t/lei-up.t t/lei-watch.t t/lei.t t/lei_dedupe.t t/lei_external.t t/lei_lcat.t t/lei_mail_sync.t t/lei_overview.t t/lei_saved_search.t t/lei_store.t t/lei_to_mail.t t/lei_xsearch.t t/mbox_lock.t t/mbox_reader.t t/mdir_reader.t t/mime.t t/miscsearch.t t/net_reader-imap.t t/nntpd-tls.t t/nntpd.t t/nodatacow.t t/on_destroy.t t/plack.t t/psgi_attach.t t/psgi_bad_mids.t t/psgi_mount.t t/psgi_multipart_not.t t/psgi_scan_all.t t/psgi_search.t t/psgi_v2.t t/rename_noreplace.t t/reply.t t/search-thr-index.t t/shared_kv.t t/solver_git.t t/thread-index-gap.t t/uri_imap.t t/uri_nntps.t t/v2dupindex.t t/www_altid.t t/xcpdb-reshard.t t/www_static.t t/watch_multiple_headers.t t/watch_maildir_v2.t t/watch_maildir.t t/watch_imap.t t/watch_filter_rubylang.t t/view.t t/v2reindex.t t/v2mirror.t t/v2mda.t t/v2index-late-dupe.t t/v2-add-remove-add.t t/v1reindex.t t/v1-add-remove-add.t t/time.t t/thread-cycle.t t/spamcheck_spamc.t t/sigfd.t t/replace.t t/reindex-time-range.t t/qspawn.t t/purge.t t/psgi_text.t t/precheck.t t/over.t t/nulsubject.t t/nntpd-v2.t t/nntp.t t/multi-mid.t t/msgtime.t t/msgmap.t t/msg_iter.t t/mid.t t/mda_filter_rubylang.t t/mda.t t/linkify.t t/init.t t/indexlevels-mirror-v1.t t/inbox.t t/import.t t/imap_tracker.t t/hval.t t/httpd-unix.t t/httpd-https.t t/httpd-corner.t t/gzip_filter.t t/git.t t/filter_vger.t t/filter_subjecttag.t t/filter_rubylang.t t/filter_mirror.t t/filter_base.t t/emergency.t t/ds-poll.t t/ds-leak.t t/ds-kqxs.t t/config_limiter.t t/config.t t/address.t
> > nixbld1 3145719 0.0 0.4 44576 35640 ? S 01:09 0:01 | \_ t/lei-sigpipe.t
> > nixbld1 3145743 0.0 0.1 14388 10388 ? S 01:09 0:00 | \_ /nix/store/vslsa0l17xjcrdgm2knwj0z5hlvf73m7-perl-5.34.0/bin/perl -w -I/nix/store/vslsa0l17xjcrdgm2knwj0z5hlvf73m7-perl-5.34.0/lib/perl5/site_perl -I/nix/store/1hdx7bxjwqrfnalalkgbwi32l45h8z7b-perl5.34.0-Mail-IMAPClient-3.42/lib/perl5/site_perl -I/nix/st blib/script/lei q -q -t z:1..
> > nixbld1 3145724 0.0 0.5 50504 42160 ? S 01:09 0:00 \_ lei-daemon /build/pi-lei-sigpipe-7522-WoHO/lei-daemon/xdg_run/lei/5.seq.sock
> > nixbld1 3145730 0.0 0.4 50092 39736 ? S 01:09 0:00 \_ lei/store /build/pi-lei-sigpipe-7522-WoHO/lei-daemon/.local/share
>
> t/lei-sigpipe.t is on:
> > 01:30:29.508334 clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=1, tv_nsec=0}, 0x7fffffff5090) = 0 <1.000360>
> > 01:30:30.509837 clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=1, tv_nsec=0}, 0x7fffffff5090) = 0 <1.000186>
> > 01:30:31.510259 clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=1, tv_nsec=0}, 0x7fffffff5090) = 0 <1.000175>
> > [...]
Huh? but it's stuck in sysread() as noted below...
I'm not sure where clock_nanosleep(2) calls are coming from with
a 1s interval. I doubt I'd ever loop clock_nanosleep(CLOCK_REALTIME
no matter how messed up my brain gets :P
I do use nanosleep(2) with a 100ms interval on sendmsg
ENOBUFS|ENOMEM|ETOOMANYREFS in Spawn.pm and that's a different
syscall. From what I can tell, Mail::IMAPClient doesn't
clock_nanosleep, nor does Perl itself...
Sleeping on CLOCK_REALTIME seems wrong outside of a cron-like
scheduler...
> lei-daemon store is on:
> > 01:31:03.834708 epoll_wait(5,
OK, that's normal.
> Looks like it is waiting on sysread() in
> https://public-inbox.org/public-inbox.git/tree/t/lei-sigpipe.t#n35
OK, that makes more sense...
> This test does succeed outside Nix's sandbox:
> $ (cd public-inbox-1.7.0; export PERL_INLINE_DIRECTORY=$PWD/inline-c; rm -rf $PERL_INLINE_DIRECTORY; mkdir $PERL_INLINE_DIRECTORY; prove -bvw t/lei-sigpipe.t )
OK, <snip>
> More surprisingly, it even succeeds when run manually
> inside the hanging Nix sandbox:
OK, <snip>
> Even more strange, Dominique was able to reproduce
> the hang this morning, but no longer tonight..
It's been a while, and I can't reproduce it. Maybe this patch will help
with diagnosis:
------------8<---------
Subject: [PATCH] t/lei-sigpipe: attempt to improve diagnostics for stuck test
This may help diagnose a difficult-to-reproduce test failure on NixOS.
Link: https://public-inbox/meta/20211209013743.okzgim7bbrpahks7@sourcephile.fr/
---
t/lei-sigpipe.t | 8 +++++++-
1 file changed, 7 insertions(+), 1 deletion(-)
diff --git a/t/lei-sigpipe.t b/t/lei-sigpipe.t
index d9738b07..6b2772a6 100644
--- a/t/lei-sigpipe.t
+++ b/t/lei-sigpipe.t
@@ -27,11 +27,17 @@ EOM
}
lei_ok(qw(import), $f) if $imported++ == 0;
- open my $errfh, '+>', "$ENV{HOME}/stderr.log" or xbail $!;
+ open my $errfh, '+>>', "$ENV{HOME}/stderr.log" or xbail $!;
my $opt = { run_mode => 0, 2 => $errfh, 1 => $w };
my $cmd = [qw(lei q -q -t), @$out, 'z:1..'];
my $tp = start_script($cmd, undef, $opt);
close $w;
+ vec(my $rvec = '', fileno($r), 1) = 1;
+ if (!select($rvec, undef, undef, 30)) {
+ seek($errfh, 0, 0) or xbail $!;
+ my $s = do { local $/; <$errfh> };
+ xbail "lei q had no output after 30s, stderr=$s";
+ }
is(sysread($r, my $buf, 1), 1, 'read one byte');
close $r; # trigger SIGPIPE
$tp->join;
^ permalink raw reply related [relevance 63%]
* Re: Using public-inbox+lei+Emacs+mu+mu4e (was: Large delays in mailing list delivery?)
2022-02-02 9:34 48% ` Using public-inbox+lei+Emacs+mu+mu4e (was: Large delays in mailing list delivery?) Ævar Arnfjörð Bjarmason
@ 2022-02-07 21:27 69% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2022-02-07 21:27 UTC (permalink / raw)
To: Ævar Arnfjörð Bjarmason
Cc: Konstantin Ryabitsev, Elijah Newren, git, Derrick Stolee, meta
Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:
> On Mon, Dec 06 2021, Eric Wong wrote:
> > Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:
> >> On Fri, Dec 03 2021, Konstantin Ryabitsev wrote:
> >>
> >> > On Fri, Dec 03, 2021 at 09:02:48PM +0100, Ævar Arnfjörð Bjarmason wrote:
> >> >> When I've experienced delays (sometimes of half a day or more) both
> >> >> https://public-inbox.org/git/ and https://lore.kernel.org/git/ have been
> >> >> updated.
> >> >
> >> > Btw, you can source lore.kernel.org straight into your gmail inbox. :)
> >> >
> >> > https://people.kernel.org/monsieuricon/lore-lei-part-1-getting-started
> >> > https://people.kernel.org/monsieuricon/lore-lei-part-2-now-with-imap
> >> >
> >> > Or, you can read it via nntp://nntp.lore.kernel.org/.
> >>
> >> [CC'd meta@public-inbox.org, probably best to move this thread over
> >> there sooner than later, but CC'ing git@ still in case this is
> >> interesting to others]
> >>
> >> I poked a bit at setting this up but couldn't find from building
> >> public-inbox.org & trying to page through the docs how I'd get from an
> >> existing public-inbox.org/git/ checkout to a local Maildir.
> >
> > Existing, public-inboxes can be set as "externals" and managed
> > via {add,forget,ls}-external sub-commands:
> >
> > # for locally-cloned inboxes:
> > public-inbox-index /path/to/existing/inbox
> > lei add-external /path/to/existing/inbox
> >
> > # relies on curl, memoizes data downloaded for each search:
> > lei add-external https://yhbt.net/lore/git
> >
> > Local externals will be included by every "lei q" invocation;
> > HTTP(S) ones require "lei q --remote"
> >
> > If you only want to use an external as a one-off without adding
> > it, the -I/--include and -O/--only flags are useful:
> >
> > lei q -O https://yhbt.net/lore/git -o /tmp/results SEARCH_TERMS
> >
> >> If you could share some recipe or a pointer to the right docs for that
> >> that would be much appreciated. Thanks!
> >
> > lei-overview(7) manpage documents some things, at least:
> > https://public-inbox.org/lei-overview.html Patches welcome :>
> >
> > IMHO lei still kinda sucks, and I probably won't have time to
> > work on it for a bit :<
>
> Thanks. I finally got around to setting this up.
>
> The above instructions didn't quite work for me, but here's what I did
> (in the form of a script lifted from a screen(1) config I've
> got). Indented with un-indented comments. The "stuff" is screen's way of
> "run this command" (or well, input these characters):
>
> I had a ~/g/git-ml clone already, but this makes one:
>
> stuff "cd ~/g/git-ml || git clone https://public-inbox.org/git ~/g/git-ml^M"
Wait, lack of --mirror or --bare for git-clone on any
public-inbox is a big mistake in terms of inode+disk use
> The initial index:
>
> ## This will create a .git/publici-inbox in ~/g/git-ml. Takes a while
> ## the first time.
> stuff "time public-inbox-index -v \$PWD^M"
>
> I fiddled with this for a bit because it refused to work, turns out it
> was missing the .git at the end, i.e. it expected a bare repo[1]:
>
> ## When we add the lei external it *must* have the ".git" part,
> ## because it'll try to find the "public-inbox" folder at wherever we
> ## point it.
> stuff "test -d ~/.config/lei || lei add-external ~/g/git-ml/.git^M"
Yeah, public-inbox has never supported non-bare usage.
> A bit of a UX wart not to be able to specify no --limit, or maybe I'm
> missing a way:
>
> ## The one-off massive import of the Git ML. TODO: No way to specify
> ## an infinite limit? Not --no-limit or --limit=0.
> stuff "test -d ~/Maildir/lei-q-git-ml || time lei q --limit=999999999 -v -o ~/Maildir/lei-q-git-ml l:git.vger.kernel.org^M"
Oops. --no-limit or --limit=0 or --limit=-1 support will be
added at some point...
> The initial indexing:
>
> ## After the one-off import this will take forever *the first time*
> ## (or around 20m), but subsequent invocations will be fast:
> stuff "time lei up ~/Maildir/lei-q-git-ml^M"
>
> Runs an ad-hoc script to keep it up-to-date, which is quoted below:
>
> ## Run it in a loop
> stuff "public-inbox-lei-pull-index^M"
>
> That script (which I whipped up just now. Is there a better/more
> standard way? to keep a public-inbox+lei pair up-to-date with
> sleep/backoff etc?
Not yet, unfortunately. I would like to have some sort of
long-polling support (similar to IDLE in IMAP) because I hate
sleep/backoff. But I don't think I'm physically nor mentally
capable of doing that or much of anything, anymore.
> #!/bin/sh
> set -xe
>
> repo=~/g/git-ml
> while true
> do
> oid=$(git -C $repo rev-parse HEAD)
> git -C $repo pull
> noid=$(git -C $repo rev-parse HEAD)
> if test "$oid" = "$noid"
> then
> echo Nothing to update
> sleep 60
> continue
> fi
> (
> cd $repo &&
> public-inbox-index -v "$PWD"
> )
> lei up ~/Maildir/lei-q-git-ml
> sleep 1
> done
>
> I use Emacs+mu4e for my E-Mail. And since I index ~/Maildir having these
> files dropped in there will be added to its index. Then I just changed
> my saved search to also look through that maildir (I guess the entire
> first condition could be dropped, but whatever):
>
> "(maildir:/personal-gmail/* OR maildir:/lei-q-git-ml/*) AND list:git.vger.kernel.org OR recip:git@vger.kernel.org OR recip:git-packagers@googlegroups.com"
>
> Because "mu" is generally good about de-duplicating stuff I've now got
> an inbox with mixed messages I can sync from GMail (including my "Sent"
> folder), and I get up-to-the-minute ML traffic now (it's bee 10hrs-4day
> delayed for 3-4 months at least).
>
> So new messages are generally from the "lei" directory, but when I send
> one it'll be dropped in the personal-gmail.
>
> I still need to check if it's doing the wrong thing with e.g. "read"
> flags if I read a mail synced via lei that later arrives in GMail. But I
> mostly don't use "read" statuses anyway...
lei can export flags from its internal DBs to IMAP via "lei export-kw"
see lei-mail-sync-overview(7) for some details, but usability is still
rough...
> 1. Maybe this "I only tested if it complied" patch would make sense to catch that?
>
> diff --git a/lib/PublicInbox/LeiXSearch.pm b/lib/PublicInbox/LeiXSearch.pm
> index 2958d3f9..be49621f 100644
> --- a/lib/PublicInbox/LeiXSearch.pm
> +++ b/lib/PublicInbox/LeiXSearch.pm
> @@ -613,7 +613,7 @@ sub add_uri {
> }
> }
>
> -sub prepare_external {
> +sub _prepare_external {
> my ($self, $loc, $boost) = @_; # n.b. already ordered by boost
> if (ref $loc) { # already a URI, or PublicInbox::Inbox-like object
> return add_uri($self, $loc) if $loc->can('scheme');
> @@ -638,6 +638,14 @@ sub prepare_external {
> push @{$self->{locals}}, $loc;
> }
>
> +sub prepare_external {
> + my ($self, $loc, $boost) = @_;
> + my $ret = _prepare_external($self, $loc, $boost);
> + warn "W: we got nothing from $loc, did you mean $loc/.git?"
> + if !$ret && -e "$loc/.git";
> + return $ret;
> +}
Probably add a note saying non-bare repos are a terrible idea,
anyways, especially for v1 public-inboxes like public-inbox.org/git
^ permalink raw reply [relevance 69%]
* Using public-inbox+lei+Emacs+mu+mu4e (was: Large delays in mailing list delivery?)
@ 2022-02-02 9:34 48% ` Ævar Arnfjörð Bjarmason
2022-02-07 21:27 69% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Ævar Arnfjörð Bjarmason @ 2022-02-02 9:34 UTC (permalink / raw)
To: Eric Wong; +Cc: Konstantin Ryabitsev, Elijah Newren, git, Derrick Stolee, meta
On Mon, Dec 06 2021, Eric Wong wrote:
> Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:
>> On Fri, Dec 03 2021, Konstantin Ryabitsev wrote:
>>
>> > On Fri, Dec 03, 2021 at 09:02:48PM +0100, Ævar Arnfjörð Bjarmason wrote:
>> >> When I've experienced delays (sometimes of half a day or more) both
>> >> https://public-inbox.org/git/ and https://lore.kernel.org/git/ have been
>> >> updated.
>> >
>> > Btw, you can source lore.kernel.org straight into your gmail inbox. :)
>> >
>> > https://people.kernel.org/monsieuricon/lore-lei-part-1-getting-started
>> > https://people.kernel.org/monsieuricon/lore-lei-part-2-now-with-imap
>> >
>> > Or, you can read it via nntp://nntp.lore.kernel.org/.
>>
>> [CC'd meta@public-inbox.org, probably best to move this thread over
>> there sooner than later, but CC'ing git@ still in case this is
>> interesting to others]
>>
>> I poked a bit at setting this up but couldn't find from building
>> public-inbox.org & trying to page through the docs how I'd get from an
>> existing public-inbox.org/git/ checkout to a local Maildir.
>
> Existing, public-inboxes can be set as "externals" and managed
> via {add,forget,ls}-external sub-commands:
>
> # for locally-cloned inboxes:
> public-inbox-index /path/to/existing/inbox
> lei add-external /path/to/existing/inbox
>
> # relies on curl, memoizes data downloaded for each search:
> lei add-external https://yhbt.net/lore/git
>
> Local externals will be included by every "lei q" invocation;
> HTTP(S) ones require "lei q --remote"
>
> If you only want to use an external as a one-off without adding
> it, the -I/--include and -O/--only flags are useful:
>
> lei q -O https://yhbt.net/lore/git -o /tmp/results SEARCH_TERMS
>
>> If you could share some recipe or a pointer to the right docs for that
>> that would be much appreciated. Thanks!
>
> lei-overview(7) manpage documents some things, at least:
> https://public-inbox.org/lei-overview.html Patches welcome :>
>
> IMHO lei still kinda sucks, and I probably won't have time to
> work on it for a bit :<
Thanks. I finally got around to setting this up.
The above instructions didn't quite work for me, but here's what I did
(in the form of a script lifted from a screen(1) config I've
got). Indented with un-indented comments. The "stuff" is screen's way of
"run this command" (or well, input these characters):
I had a ~/g/git-ml clone already, but this makes one:
stuff "cd ~/g/git-ml || git clone https://public-inbox.org/git ~/g/git-ml^M"
The initial index:
## This will create a .git/publici-inbox in ~/g/git-ml. Takes a while
## the first time.
stuff "time public-inbox-index -v \$PWD^M"
I fiddled with this for a bit because it refused to work, turns out it
was missing the .git at the end, i.e. it expected a bare repo[1]:
## When we add the lei external it *must* have the ".git" part,
## because it'll try to find the "public-inbox" folder at wherever we
## point it.
stuff "test -d ~/.config/lei || lei add-external ~/g/git-ml/.git^M"
A bit of a UX wart not to be able to specify no --limit, or maybe I'm
missing a way:
## The one-off massive import of the Git ML. TODO: No way to specify
## an infinite limit? Not --no-limit or --limit=0.
stuff "test -d ~/Maildir/lei-q-git-ml || time lei q --limit=999999999 -v -o ~/Maildir/lei-q-git-ml l:git.vger.kernel.org^M"
The initial indexing:
## After the one-off import this will take forever *the first time*
## (or around 20m), but subsequent invocations will be fast:
stuff "time lei up ~/Maildir/lei-q-git-ml^M"
Runs an ad-hoc script to keep it up-to-date, which is quoted below:
## Run it in a loop
stuff "public-inbox-lei-pull-index^M"
That script (which I whipped up just now. Is there a better/more
standard way? to keep a public-inbox+lei pair up-to-date with
sleep/backoff etc?
#!/bin/sh
set -xe
repo=~/g/git-ml
while true
do
oid=$(git -C $repo rev-parse HEAD)
git -C $repo pull
noid=$(git -C $repo rev-parse HEAD)
if test "$oid" = "$noid"
then
echo Nothing to update
sleep 60
continue
fi
(
cd $repo &&
public-inbox-index -v "$PWD"
)
lei up ~/Maildir/lei-q-git-ml
sleep 1
done
I use Emacs+mu4e for my E-Mail. And since I index ~/Maildir having these
files dropped in there will be added to its index. Then I just changed
my saved search to also look through that maildir (I guess the entire
first condition could be dropped, but whatever):
"(maildir:/personal-gmail/* OR maildir:/lei-q-git-ml/*) AND list:git.vger.kernel.org OR recip:git@vger.kernel.org OR recip:git-packagers@googlegroups.com"
Because "mu" is generally good about de-duplicating stuff I've now got
an inbox with mixed messages I can sync from GMail (including my "Sent"
folder), and I get up-to-the-minute ML traffic now (it's bee 10hrs-4day
delayed for 3-4 months at least).
So new messages are generally from the "lei" directory, but when I send
one it'll be dropped in the personal-gmail.
I still need to check if it's doing the wrong thing with e.g. "read"
flags if I read a mail synced via lei that later arrives in GMail. But I
mostly don't use "read" statuses anyway...
1. Maybe this "I only tested if it complied" patch would make sense to catch that?
diff --git a/lib/PublicInbox/LeiXSearch.pm b/lib/PublicInbox/LeiXSearch.pm
index 2958d3f9..be49621f 100644
--- a/lib/PublicInbox/LeiXSearch.pm
+++ b/lib/PublicInbox/LeiXSearch.pm
@@ -613,7 +613,7 @@ sub add_uri {
}
}
-sub prepare_external {
+sub _prepare_external {
my ($self, $loc, $boost) = @_; # n.b. already ordered by boost
if (ref $loc) { # already a URI, or PublicInbox::Inbox-like object
return add_uri($self, $loc) if $loc->can('scheme');
@@ -638,6 +638,14 @@ sub prepare_external {
push @{$self->{locals}}, $loc;
}
+sub prepare_external {
+ my ($self, $loc, $boost) = @_;
+ my $ret = _prepare_external($self, $loc, $boost);
+ warn "W: we got nothing from $loc, did you mean $loc/.git?"
+ if !$ret && -e "$loc/.git";
+ return $ret;
+}
+
sub _lcat_i { # LeiMailSync->each_src iterator callback
my ($oidbin, $id, $each_smsg) = @_;
$each_smsg->({blob => unpack('H*', $oidbin), pct => 100});
^ permalink raw reply related [relevance 48%]
* lei q: importing messages when specifying '-f *json*'?
@ 2021-12-30 23:04 60% Kyle Meyer
0 siblings, 0 replies; 200+ results
From: Kyle Meyer @ 2021-12-30 23:04 UTC (permalink / raw)
To: meta; +Cc: piem
The lei-q interface I'm working on for piem [1] consumes JSON output to
display search results. From there, an individual message can be shown.
If the query is against a remote external, that involves two curl calls
to the remote, one for the search and one for the display.
I'd like to import hits from the first step into the local store so that
I can avoid the second curl call. However, I haven't been able to
figure out a way to do this when requesting JSON output.
For example [2], say I don't have meta's 20211124154539.350522-1-e@80x24.org
locally:
$ lei daemon-kill
$ export HOME=$(mktemp -d "${TMPDIR:-/tmp}"/pi-testing-XXXXXXX)
$ lei q m:20211124154539.350522-1-e@80x24.org
# /tmp/pi-testing-3JaSz2K/.config/lei/config created
[null]
If I search for that message against https://public-inbox.org/meta/ and
request JSON output
$ lei q -I https://public-inbox.org/meta/ -f ldjson \
m:20211124154539.350522-1-e@80x24.org
# /usr/bin/curl -Sf -s -d '' https://public-inbox.org/meta/?x=m&q=m%3A20211124154539.350522-1-e%4080x24.org
{"blob":"a8754283bd9e985d6e1156215071be59aa2b5a53",...}
then no message ends up in the local store:
$ lei q m:20211124154539.350522-1-e@80x24.org
[null]
In contrast, if I request mboxrd output
$ lei q -I https://public-inbox.org/meta/ -f mboxrd \
m:20211124154539.350522-1-e@80x24.org >/dev/null
# /usr/bin/curl -Sf -s -d '' https://public-inbox.org/meta/?x=m&q=m%3A20211124154539.350522-1-e%4080x24.org
the message is imported to the local store:
$ lei q -f ldjson m:20211124154539.350522-1-e@80x24.org
{"blob":"a8754283bd9e985d6e1156215071be59aa2b5a53",...}
$ git -C $HOME/.local/share/lei/store/local/0.git/ log --oneline
ca12a1b (HEAD -> master) [PATCH] eliminate some unused subs
I was hoping that --import-remote might do the trick, but that doesn't
seem to be the case:
$ lei daemon-kill
$ export HOME=$(mktemp -d "${TMPDIR:-/tmp}"/pi-testing-XXXXXXX)
$ lei q --import-remote -I https://public-inbox.org/meta/ \
-f ldjson m:20211124154539.350522-1-e@80x24.org
# /tmp/pi-testing-Ny7KDcB/.config/lei/config created
# /usr/bin/curl -Sf -s -d '' https://public-inbox.org/meta/?x=m&q=m%3A20211124154539.350522-1-e%4080x24.org
{"blob":"a8754283bd9e985d6e1156215071be59aa2b5a53",...}
$ lei q m:20211124154539.350522-1-e@80x24.org
[null]
Should --import-remote trigger an import in the case above?
[1] https://git.kyleam.com/piem/tree/piem-lei.el
[2] These are with public-inbox's current master (07cd8973baf).
^ permalink raw reply [relevance 60%]
* [PATCH] lei: always use 3-arg open perlop
@ 2021-11-22 18:38 71% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-11-22 18:38 UTC (permalink / raw)
To: meta
Future-proofing in case future versions of Perl warn on this, since
2-arg forms of open may be subject to injection vulnerabilities
with non-literal args.
---
lib/PublicInbox/LEI.pm | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 192f267ca1dd..4e0295fa4e8a 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -818,7 +818,8 @@ sub dispatch {
next if $d eq ''; # same as git(1)
chdir $d or return fail($self, "cd $d: $!");
}
- open $self->{3}, '.' or return fail($self, "open . $!");
+ open $self->{3}, '<', '.' or
+ return fail($self, "open . $!");
}
$cb->($self, @argv);
} elsif (grep(/\A-/, $cmd, @argv)) { # --help or -h only
^ permalink raw reply related [relevance 71%]
* Re: [PATCH] t/lei-mirror: skip lei comparisons if lei missing
2021-11-22 13:47 99% ` Jörg Rödel
@ 2021-11-22 17:24 99% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-11-22 17:24 UTC (permalink / raw)
To: Jörg Rödel; +Cc: meta
Jörg Rödel <joro@8bytes.org> wrote:
> This seems to cause warnings:
>
> [ 56s] t/lei-lcat.t ................. ok
> [ 57s] Useless use of sort in scalar context at t/lei-mirror.t line 175.
> [ 59s] Use of uninitialized value in join or string at t/lei-mirror.t line 175.
> [ 59s] t/lei-mirror.t ............... ok
Oops :x I'll squash this in before pushing:
diff --git a/t/lei-mirror.t b/t/lei-mirror.t
index d6fa6db5e3cb..32a5b0390714 100644
--- a/t/lei-mirror.t
+++ b/t/lei-mirror.t
@@ -172,7 +172,8 @@ SKIP: {
ok(run_script([qw(-index -Lbasic), "$d/t2"]), 'index v2');
SKIP: {
- skip "lei didn't run", 2 if join(sort keys %created) ne 'v1v2';
+ join('', sort(keys %created)) eq 'v1v2' or
+ skip "lei didn't run", 2;
my $f = "$d/t1/public-inbox/msgmap.sqlite3";
my $ca = PublicInbox::Msgmap->new_file($f)->created_at;
is($ca, $created{v1}, 'clone + index v1 synced ->created_at');
^ permalink raw reply related [relevance 99%]
* Re: [PATCH] t/lei-mirror: skip lei comparisons if lei missing
2021-11-22 7:42 96% ` [PATCH] t/lei-mirror: skip lei comparisons if lei missing Eric Wong
@ 2021-11-22 13:47 99% ` Jörg Rödel
2021-11-22 17:24 99% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Jörg Rödel @ 2021-11-22 13:47 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
On Mon, Nov 22, 2021 at 07:42:41AM +0000, Eric Wong wrote:
> t/lei-mirror.t | 17 +++++++++++------
> 1 file changed, 11 insertions(+), 6 deletions(-)
This seems to cause warnings:
[ 56s] t/lei-lcat.t ................. ok
[ 57s] Useless use of sort in scalar context at t/lei-mirror.t line 175.
[ 59s] Use of uninitialized value in join or string at t/lei-mirror.t line 175.
[ 59s] t/lei-mirror.t ............... ok
Regards,
Joerg
^ permalink raw reply [relevance 99%]
* [PATCH] t/lei-mirror: skip lei comparisons if lei missing
@ 2021-11-22 7:42 96% ` Eric Wong
2021-11-22 13:47 99% ` Jörg Rödel
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2021-11-22 7:42 UTC (permalink / raw)
To: Jörg Rödel; +Cc: meta
Eric Wong <e@80x24.org> wrote:
> Will work on a patch to ensure lei ran successfully.
---------8<--------
Subject: [PATCH] t/lei-mirror: skip lei comparisons if lei missing
We can't compare created_at times with lei if lei tests are
skipped due to Inline::C or Socket::MsgHdr unavailability.
Reported-by: Jörg Rödel <joro@8bytes.org>
Link: https://public-inbox.org/meta/YZebmAxlFJy4lqAw@8bytes.org/
---
t/lei-mirror.t | 17 +++++++++++------
1 file changed, 11 insertions(+), 6 deletions(-)
diff --git a/t/lei-mirror.t b/t/lei-mirror.t
index 646ff2b19f98..d6fa6db5e3cb 100644
--- a/t/lei-mirror.t
+++ b/t/lei-mirror.t
@@ -170,12 +170,17 @@ SKIP: {
local $ENV{HOME} = $tmpdir;
ok(run_script([qw(-index -Lbasic), "$d/t1"]), 'index v1');
ok(run_script([qw(-index -Lbasic), "$d/t2"]), 'index v2');
- my $f = "$d/t1/public-inbox/msgmap.sqlite3";
- my $ca = PublicInbox::Msgmap->new_file($f)->created_at;
- is($ca, $created{v1}, 'clone + index v1 synced ->created_at');
- $f = "$d/t2/msgmap.sqlite3";
- $ca = PublicInbox::Msgmap->new_file($f)->created_at;
- is($ca, $created{v2}, 'clone + index v1 synced ->created_at');
+
+ SKIP: {
+ skip "lei didn't run", 2 if join(sort keys %created) ne 'v1v2';
+ my $f = "$d/t1/public-inbox/msgmap.sqlite3";
+ my $ca = PublicInbox::Msgmap->new_file($f)->created_at;
+ is($ca, $created{v1}, 'clone + index v1 synced ->created_at');
+
+ $f = "$d/t2/msgmap.sqlite3";
+ $ca = PublicInbox::Msgmap->new_file($f)->created_at;
+ is($ca, $created{v2}, 'clone + index v2 synced ->created_at');
+ }
test_lei(sub {
lei_ok qw(inspect num:1 --dir), "$d/t1";
ok(ref(json_utf8->decode($lei_out)), 'inspect num: on v1');
^ permalink raw reply related [relevance 96%]
* Re: RFC: should lei inject its own "Received:" header?
2021-11-19 20:49 71% RFC: should lei inject its own "Received:" header? Konstantin Ryabitsev
@ 2021-11-21 10:13 71% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-11-21 10:13 UTC (permalink / raw)
To: Konstantin Ryabitsev; +Cc: meta
Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> Hello:
>
> I wonder if lei should inject its own "received"-like header on writing to a
> maildir/imap target -- to indicate where the copy of the email came from. I
> don't think it should use the actual Received: header, as this may cause some
> weird SPF/DMARC issues, but perhaps something like:
>
> X-Lei-Received: from https://lore.kernel.org/all/; Fri, 19 Nov 2021 14:32:44 -0500
>
> Just a random thought.
Probably not in the blob itself. Addition of extra headers
means slower and less-effective dedupe (not just for lei, but
also for deduplicating FSes (which operate at block-layer) and
potential FUSE implementation).
Perhaps the mail_sync.sqlite3 stuff could be taught to track the
origin of a blob, though...
^ permalink raw reply [relevance 71%]
* lei q error after deleting cache (can't find mail_sync.sqlite3)
@ 2021-11-20 16:36 71% ` Johannes Altmanninger
0 siblings, 0 replies; 200+ results
From: Johannes Altmanninger @ 2021-11-20 16:36 UTC (permalink / raw)
To: meta
On Sat, Nov 20, 2021 at 04:30:57PM +0100, Johannes Altmanninger wrote:
> I've been using lei from the 1.7.0 release, which works great.
>
> I had some unrelated problems with one of the created maildirs so I deleted
> said maildir. Now I can't recreate the maildir. I tried deleting all
> persistent data but got an error:
>
> $ rm -rf the-maildir ~/.local/share/lei ~/.cache/{public-inbox,lei}
> $ lei q -I https://lore.kernel.org/git/ -o the-maildir rt:1.week.ago..
> failed to open /home/johannes/.cache/lei/all_locals_ever.git/lei_ale.state: No such file or directory
> at /usr/share/perl5/vendor_perl/PublicInbox/LeiQuery.pm line 117.
>
> A quick look at strace doesn't show access to any other files in $HOME
This is pretty weird: it works in a temporary home, but only for the first time.
rm -rf the-maildir someuniquename; HOME=$PWD/someuniquename lei q -I https://public-inbox.org/meta/ -o the-maildir rt:1.day.ago..
When I re-run this command, I get the same error
There seems to be no mention of someuniquename anywhere on my disk (except
in my shell history).
^ permalink raw reply [relevance 71%]
* RFC: should lei inject its own "Received:" header?
@ 2021-11-19 20:49 71% Konstantin Ryabitsev
2021-11-21 10:13 71% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Konstantin Ryabitsev @ 2021-11-19 20:49 UTC (permalink / raw)
To: meta
Hello:
I wonder if lei should inject its own "received"-like header on writing to a
maildir/imap target -- to indicate where the copy of the email came from. I
don't think it should use the actual Received: header, as this may cause some
weird SPF/DMARC issues, but perhaps something like:
X-Lei-Received: from https://lore.kernel.org/all/; Fri, 19 Nov 2021 14:32:44 -0500
Just a random thought.
-K
^ permalink raw reply [relevance 71%]
* Re: lei spawns mua before results are written
2021-11-14 20:41 71% lei spawns mua before results are written Leah Neukirchen
@ 2021-11-15 0:31 71% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-11-15 0:31 UTC (permalink / raw)
To: Leah Neukirchen; +Cc: meta
Leah Neukirchen <leah@vuxu.org> wrote:
> I tried building some tooling to use lei with mblaze and stumbled upon
> the fact that lei seems to spawn the mua before all mails are written
> into the -o maildir; at least, adding a "sleep 1" in front of my
> script fixes it. Any way to wait for it? I guess I could just run lei
> and my mua scripts afterwards.
Yes, it's intentional to allow Maildir and IMAP to allow MUAs to
work progressively while loading similar to how less(1) and
typical web browsers work. It's not really practical to do for
mbox* because of locking and potential incompatible locking
protocols.
You should be able to use --alert=COMMAND to run any arbitrary
command(s) (including an MUA, as an alternative to --mua).
I'm not sure if adding a --no-early-mua switch is needed. My
gut reaction is having yet another switch to document increases
the learning curve and support overhead... *shrug*
^ permalink raw reply [relevance 71%]
* lei spawns mua before results are written
@ 2021-11-14 20:41 71% Leah Neukirchen
2021-11-15 0:31 71% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Leah Neukirchen @ 2021-11-14 20:41 UTC (permalink / raw)
To: meta
Hi,
I tried building some tooling to use lei with mblaze and stumbled upon
the fact that lei seems to spawn the mua before all mails are written
into the -o maildir; at least, adding a "sleep 1" in front of my
script fixes it. Any way to wait for it? I guess I could just run lei
and my mua scripts afterwards.
Thanks,
--
Leah Neukirchen <leah@vuxu.org> https://leahneukirchen.org/
^ permalink raw reply [relevance 71%]
* [PATCH] lei forget-search: add help for --prune
@ 2021-11-12 11:08 71% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-11-12 11:08 UTC (permalink / raw)
To: meta
This enables tab-completion, since I'm using --prune quite a bit
and my fingers are about to fall off :<
---
lib/PublicInbox/LEI.pm | 3 +++
1 file changed, 3 insertions(+)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 887025de5ded..192f267ca1dd 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -411,6 +411,9 @@ my %OPTDESC = (
'url ls-mail-source' => 'show full URL of newsgroup or IMAP folder',
'format|f=s ls-external' => $ls_format,
+'prune:s forget-search' =>
+ ['TYPE|local|remote', 'prune all, remote or local folders' ],
+
'limit|n=i@' => ['NUM', 'limit on number of matches (default: 10000)' ],
'offset=i' => ['OFF', 'search result offset (default: 0)'],
^ permalink raw reply related [relevance 71%]
* [PATCH] t/lei-watch: test with with higher sleep
@ 2021-11-10 10:33 71% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-11-10 10:33 UTC (permalink / raw)
To: meta
0.1s may not be enough for a task switch and inotify wakeup,
so try doubling it and see if it fixes test reliability, for
now. A future change may be to implement a watcher/tracer
for inotify -> lei/store events.
Link: https://public-inbox.org/meta/20211104134327.zrf5jijfz7dsvb7l@meerkat.local/
---
t/lei-watch.t | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/t/lei-watch.t b/t/lei-watch.t
index 9158571c..e6066033 100644
--- a/t/lei-watch.t
+++ b/t/lei-watch.t
@@ -52,7 +52,7 @@ test_lei(sub {
my @f = glob("$md/cur/*:2,");
is(scalar(@f), 1, 'got populated maildir with one result');
rename($f[0], "$f[0]S") or xbail "rename $!"; # set (S)een
- tick($have_fast_inotify ? 0.1 : 2.1); # always needed for 1 CPU systems
+ tick($have_fast_inotify ? 0.2 : 2.2); # always needed for 1 CPU systems
lei_ok qw(note-event done); # flushes immediately (instead of 5s)
lei_ok qw(q mid:testmessage@example.com -o), $md2, '-I', "$ro_home/t1";
^ permalink raw reply related [relevance 71%]
* [PATCH 4/4] lei q: make HTTP(S) query strings even less ugly
2021-11-10 10:25 71% [PATCH 0/4] "lei q" rawstr fallout + fixes Eric Wong
2021-11-10 10:28 70% ` [PATCH 3/4] lei q: disallow "\n" in argv[] elements Eric Wong
2021-11-10 10:28 68% ` [PATCH 2/4] lei up: infer rawstr from old searches via trailing "\n" Eric Wong
@ 2021-11-10 10:28 71% ` Eric Wong
2 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-11-10 10:28 UTC (permalink / raw)
To: meta
Following commit 57fed2e4b78ed394 (lei: normalize whitespace in
remote queries, 2021-09-11), leaving the trailing `\n' from
stdin queries to be normalized to ` ' (SP) causes it to appear
as `+' in URLs, which Xapian ignores.
---
lib/PublicInbox/LeiXSearch.pm | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/lib/PublicInbox/LeiXSearch.pm b/lib/PublicInbox/LeiXSearch.pm
index 29df07e0c8a8..2958d3f910b0 100644
--- a/lib/PublicInbox/LeiXSearch.pm
+++ b/lib/PublicInbox/LeiXSearch.pm
@@ -342,7 +342,7 @@ sub query_remote_mboxrd {
local $SIG{TERM} = sub { exit(0) }; # for DESTROY (File::Temp, $reap)
my $lei = $self->{lei};
my $opt = $lei->{opt};
- my $qstr = $lei->{mset_opt}->{qstr};
+ chomp(my $qstr = $lei->{mset_opt}->{qstr});
$qstr =~ s/[ \n\t]+/ /sg; # make URLs less ugly
my @qform = (x => 'm');
push(@qform, t => 1) if $opt->{threads};
^ permalink raw reply related [relevance 71%]
* [PATCH 3/4] lei q: disallow "\n" in argv[] elements
2021-11-10 10:25 71% [PATCH 0/4] "lei q" rawstr fallout + fixes Eric Wong
@ 2021-11-10 10:28 70% ` Eric Wong
2021-11-10 10:28 68% ` [PATCH 2/4] lei up: infer rawstr from old searches via trailing "\n" Eric Wong
2021-11-10 10:28 71% ` [PATCH 4/4] lei q: make HTTP(S) query strings even less ugly Eric Wong
2 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-11-10 10:28 UTC (permalink / raw)
To: meta
I don't expect this to be hit in real-world use via normal
interactive shells. However, somebody could accidentally add
"\n" in languages (e.g. Perl, C) where it's easy to pass "\n"
in argv[].
---
lib/PublicInbox/LeiQuery.pm | 1 +
t/lei.t | 3 +++
2 files changed, 4 insertions(+)
diff --git a/lib/PublicInbox/LeiQuery.pm b/lib/PublicInbox/LeiQuery.pm
index 352ee60131aa..51ee3d9c83e4 100644
--- a/lib/PublicInbox/LeiQuery.pm
+++ b/lib/PublicInbox/LeiQuery.pm
@@ -141,6 +141,7 @@ no query allowed on command-line with --stdin
PublicInbox::InputPipe::consume($self->{0}, \&qstr_add, $self);
return;
}
+ chomp(@argv) and $self->qerr("# trailing `\\n' removed");
$mset_opt{q_raw} = [ @argv ]; # copy
$mset_opt{qstr} =
$self->{lse}->query_argv_to_string($self->{lse}->git, \@argv);
diff --git a/t/lei.t b/t/lei.t
index f7de1b711a83..b10c9b59c72b 100644
--- a/t/lei.t
+++ b/t/lei.t
@@ -143,6 +143,9 @@ my $test_fail = sub {
lei('-C', '/dev/null', 'q', 'whatever');
is($? >> 8, 1, 'chdir at beginning fails to /dev/null');
+ lei_ok('q', "foo\n");
+ like($lei_err, qr/trailing `\\n' removed/s, "noted `\\n' removal");
+
for my $lk (qw(ei inbox)) {
my $d = "$home/newline\n$lk";
mkdir $d;
^ permalink raw reply related [relevance 70%]
* [PATCH 2/4] lei up: infer rawstr from old searches via trailing "\n"
2021-11-10 10:25 71% [PATCH 0/4] "lei q" rawstr fallout + fixes Eric Wong
2021-11-10 10:28 70% ` [PATCH 3/4] lei q: disallow "\n" in argv[] elements Eric Wong
@ 2021-11-10 10:28 68% ` Eric Wong
2021-11-10 10:28 71% ` [PATCH 4/4] lei q: make HTTP(S) query strings even less ugly Eric Wong
2 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-11-10 10:28 UTC (permalink / raw)
To: meta
For --stdin searches created prior to commit 666dde69a3f6 (lei
q|up: fix saved searches for single-phrase search, 2021-11-08)
we still want to be able to run "lei up" on them without
regressions. So assume nobody manages to enter "\n" as an
argv[] element and consider the presence of "\n" as a previous
--stdin use.
This fixes errors from "lei up" such as:
lei_xsearch 2 wq_worker: Exception: Key too long: length was 840 bytes,
maximum length of a key is 255 bytes at ../PublicInbox/IPC.pm line 250.
Fixes: 666dde69a3f6 ("lei q|up: fix saved searches for single-phrase search")
---
lib/PublicInbox/LeiUp.pm | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/lib/PublicInbox/LeiUp.pm b/lib/PublicInbox/LeiUp.pm
index d7873a3f3469..b8a9836075ba 100644
--- a/lib/PublicInbox/LeiUp.pm
+++ b/lib/PublicInbox/LeiUp.pm
@@ -29,7 +29,9 @@ sub up1 ($$) {
my $q = $lss->{-cfg}->get_all('lei.q') //
die("lei.q unset in $f (out=$out)\n");
my $lse = $lei->{lse} // die 'BUG: {lse} missing';
- if ($lss->{-cfg}->{'lei.internal.rawstr'}) {
+ my $rawstr = $lss->{-cfg}->{'lei.internal.rawstr'} //
+ (scalar(@$q) == 1 && substr($q->[0], -1) eq "\n");
+ if ($rawstr) {
scalar(@$q) > 1 and
die "$f: lei.q has multiple values (@$q) (out=$out)\n";
$lse->query_approxidate($lse->git, $mset_opt->{qstr} = $q->[0]);
^ permalink raw reply related [relevance 68%]
* [PATCH 0/4] "lei q" rawstr fallout + fixes
@ 2021-11-10 10:25 71% Eric Wong
2021-11-10 10:28 70% ` [PATCH 3/4] lei q: disallow "\n" in argv[] elements Eric Wong
` (2 more replies)
0 siblings, 3 replies; 200+ results
From: Eric Wong @ 2021-11-10 10:25 UTC (permalink / raw)
To: meta
The recent rawstr fix caused some errors for existing
--stdin searches, these patches fix them and make some
minor improvements along the way.
Eric Wong (4):
ipc: note failing sub name
lei up: infer rawstr from old searches via trailing "\n"
lei q: disallow "\n" in argv[] elements
lei q: make HTTP(S) query strings even less ugly
lib/PublicInbox/IPC.pm | 2 +-
lib/PublicInbox/LeiQuery.pm | 1 +
lib/PublicInbox/LeiUp.pm | 4 +++-
lib/PublicInbox/LeiXSearch.pm | 2 +-
t/lei.t | 3 +++
5 files changed, 9 insertions(+), 3 deletions(-)
^ permalink raw reply [relevance 71%]
* [PATCH] lei q|up: fix saved searches for single-phrase search
@ 2021-11-08 23:39 53% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-11-08 23:39 UTC (permalink / raw)
To: meta
`"' (double-quote) needs to be quoted for stdin searches.
We also need to differentiate between "lei q --stdin" usage
when calling "lei up", do it by setting an internal "rawstr"
knob to ensure we can parse the config properly regardless
of whether the initial search used --stdin or not.
---
lib/PublicInbox/LeiSavedSearch.pm | 5 ++++
lib/PublicInbox/LeiUp.pm | 10 +++++---
t/lei-q-save.t | 41 +++++++++++++++++++++++++++++++
3 files changed, 52 insertions(+), 4 deletions(-)
diff --git a/lib/PublicInbox/LeiSavedSearch.pm b/lib/PublicInbox/LeiSavedSearch.pm
index b2f1ad10..1d13aef6 100644
--- a/lib/PublicInbox/LeiSavedSearch.pm
+++ b/lib/PublicInbox/LeiSavedSearch.pm
@@ -21,6 +21,7 @@ my %cquote = ("\n" => '\\n', "\t" => '\\t', "\b" => '\\b');
sub cquote_val ($) { # cf. git-config(1)
my ($val) = @_;
$val =~ s/([\n\t\b])/$cquote{$1}/g;
+ $val =~ s/\"/\\\"/g;
$val;
}
@@ -162,6 +163,10 @@ EOM
my $val = $lei->{opt}->{$k} // next;
print $fh "\t$k = $val\n";
}
+ $lei->{opt}->{stdin} and print $fh <<EOM;
+[lei "internal"]
+ rawstr = 1 # stdin was used initially
+EOM
close($fh) or return $lei->fail("close $f: $!");
$self->{lock_path} = "$self->{-f}.flock";
$self->{-ovf} = "$dir/over.sqlite3";
diff --git a/lib/PublicInbox/LeiUp.pm b/lib/PublicInbox/LeiUp.pm
index 66d950b2..d7873a3f 100644
--- a/lib/PublicInbox/LeiUp.pm
+++ b/lib/PublicInbox/LeiUp.pm
@@ -26,13 +26,15 @@ sub up1 ($$) {
my $lss = PublicInbox::LeiSavedSearch->up($lei, $out) or return;
my $f = $lss->{'-f'};
my $mset_opt = $lei->{mset_opt} = { relevance => -2 };
- my $q = $mset_opt->{q_raw} = $lss->{-cfg}->{'lei.q'} //
+ my $q = $lss->{-cfg}->get_all('lei.q') //
die("lei.q unset in $f (out=$out)\n");
my $lse = $lei->{lse} // die 'BUG: {lse} missing';
- if (ref($q)) {
- $mset_opt->{qstr} = $lse->query_argv_to_string($lse->git, $q);
+ if ($lss->{-cfg}->{'lei.internal.rawstr'}) {
+ scalar(@$q) > 1 and
+ die "$f: lei.q has multiple values (@$q) (out=$out)\n";
+ $lse->query_approxidate($lse->git, $mset_opt->{qstr} = $q->[0]);
} else {
- $lse->query_approxidate($lse->git, $mset_opt->{qstr} = $q);
+ $mset_opt->{qstr} = $lse->query_argv_to_string($lse->git, $q);
}
# n.b. only a few CLI args are accepted for "up", so //= usually sets
for my $k ($lss->ARRAY_FIELDS) {
diff --git a/t/lei-q-save.t b/t/lei-q-save.t
index cd35461c..3d09fe37 100644
--- a/t/lei-q-save.t
+++ b/t/lei-q-save.t
@@ -240,5 +240,46 @@ test_lei(sub {
lei_ok qw(forget-search --prune);
lei_ok qw(ls-search);
unlike($lei_out, qr!\Q$home/after\E!, "`after' pruned");
+
+ my $d = "$home/d";
+ lei_ok [qw(import -q -F eml)], undef,
+ {0 => \"Subject: do not call\n\n"};
+ lei_ok qw(q -o), $d, 's:do not call';
+
+ my @orig = glob("$d/*/*");
+ is(scalar(@orig), 1, 'got one message via argv');
+ lei_ok [qw(import -q -Feml)], undef,
+ {0 => \"Subject: do not ever call\n\n"};
+ lei_ok 'up', $d;
+ is_deeply([glob("$d/*/*")], \@orig, 'nothing written');
+ lei_ok [qw(import -q -Feml)], undef,
+ {0 => \"Subject: do not call, ever\n\n"};
+ lei_ok 'up', $d;
+ @after = glob("$d/*/*");
+ is(scalar(@after), 2, '2 total, messages, now');
+ is_deeply([glob("$d/cur/*")], \@orig, 'cur untouched');
+ my @new = glob("$d/new/*");
+ is(scalar(@new), 1, "new message written to `new'");
+ is(eml_load($new[0])->header('Subject'), 'do not call, ever',
+ 'up retrieved correct message');
+
+ $d = "$home/d-stdin";
+ lei_ok [ qw(q -q -o), $d ], undef, { 0 => \'s:"do not ever call"' };
+ @orig = glob("$d/*/*");
+ is(scalar(@orig), 1, 'got one message via stdin');
+
+ lei_ok [qw(import -q -Feml)], undef,
+ {0 => \"Subject: do not fall or ever call\n\n"};
+ lei_ok [qw(import -q -Feml)], undef,
+ {0 => \"Subject: do not ever call, again\n\n"};
+ lei_ok 'up', $d;
+ @new = glob("$d/new/*");
+ is(scalar(@new), 1, "new message written to `new'") or do {
+ for (@new) { diag "$_ ".eml_load($_)->header('Subject') }
+ };
+ is_deeply([glob("$d/cur/*")], \@orig, 'cur untouched');
+ is(eml_load($new[0])->header('Subject'), 'do not ever call, again',
+ 'up retrieved correct message');
+
});
done_testing;
^ permalink raw reply related [relevance 53%]
* Re: lei: incorrect quoting on saved searches (was Re: lore+lei: getting started)
2021-11-08 22:36 88% ` Konstantin Ryabitsev
@ 2021-11-08 22:57 90% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-11-08 22:57 UTC (permalink / raw)
To: Konstantin Ryabitsev; +Cc: Rob Herring, meta, workflows
Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> On Mon, Nov 08, 2021 at 09:48:36PM +0000, Eric Wong wrote:
> > > Hmm... I noticed that when I `lei edit-search` the initial query that was
> > > causing quoting issues, I get the following:
> > >
> > > [lei]
> > > q = (dfn:drivers OR dfn:arch OR dfn:Documentation OR dfn:include OR dfn:scripts) AND f:robh@kernel.org
> > >
> > > So, the extra quotes didn't get added to the config file. Running `lei up` on
> > > that saved search seems to do the right thing, so the erroneous quotes are
> > > only added during the initial `lei q` call.
> >
> > Right, each entry in lei.q is actually an entry in argv[].
> > So the correct query should look something like:
>
> So, to be clear here... the following doesn't work because instead of multiple
> query parameters to 'lei q' the single-quoted string becomes a single
> parameter?
>
> lei q -I https://lore.kernel.org/all/ -o ~/work/temp/lei/robh-patches \
> --threads --dedupe=mid \
> '(dfn:drivers OR dfn:arch OR dfn:Documentation OR dfn:include OR dfn:scripts) AND f:robh@kernel.org'
>
> Any way to make this work? I find that it's more easily readable than the
> "echo | lei q" version.
I can't think of a way to make it work w/o breaking phrase searches
(or asking users to use both single and double-quotes):
lei q 's:"a quick brown fox"' # yuck
> For bash users, the following should work as well:
>
> lei q -I https://lore.kernel.org/all/ -o ~/work/temp/lei/robh-patches \
> --threads --dedupe=mid <<< \
> '(dfn:drivers OR dfn:arch OR dfn:Documentation OR dfn:include OR dfn:scripts) AND f:robh@kernel.org'
Oh, not sure about bash and <<<; but this heredoc should work with any
POSIX sh:
lei q -I https://lore.kernel.org/all/ \
-o ~/work/temp/lei/robh-patches \
--threads --dedupe=mid <<'EOM'
(dfn:drivers OR dfn:arch OR dfn:Documentation OR dfn:include OR dfn:scripts)
AND f:robh@kernel.org
EOM
> Suggestion, can -I accept the URL containing the query, so that the command
> becomes:
>
> lei q -o ~/mail/foo --threads --dedupe=mid -I \
> https://lore.kernel.org/all/?q=f%3Atorvalds+AND+nq%3Agarbage
>
> This way we pass both the location of the extindex to query AND the parameters
> we should use, avoiding shell quoting problems?
Maybe, but URI escaping bothers the heck out of me, too.
I guess the heredoc example is actually good...
^ permalink raw reply [relevance 90%]
* Re: lei: incorrect quoting on saved searches (was Re: lore+lei: getting started)
2021-11-08 21:48 90% ` Eric Wong
@ 2021-11-08 22:36 88% ` Konstantin Ryabitsev
2021-11-08 22:57 90% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Konstantin Ryabitsev @ 2021-11-08 22:36 UTC (permalink / raw)
To: Eric Wong; +Cc: Rob Herring, meta, workflows
On Mon, Nov 08, 2021 at 09:48:36PM +0000, Eric Wong wrote:
> > Hmm... I noticed that when I `lei edit-search` the initial query that was
> > causing quoting issues, I get the following:
> >
> > [lei]
> > q = (dfn:drivers OR dfn:arch OR dfn:Documentation OR dfn:include OR dfn:scripts) AND f:robh@kernel.org
> >
> > So, the extra quotes didn't get added to the config file. Running `lei up` on
> > that saved search seems to do the right thing, so the erroneous quotes are
> > only added during the initial `lei q` call.
>
> Right, each entry in lei.q is actually an entry in argv[].
> So the correct query should look something like:
So, to be clear here... the following doesn't work because instead of multiple
query parameters to 'lei q' the single-quoted string becomes a single
parameter?
lei q -I https://lore.kernel.org/all/ -o ~/work/temp/lei/robh-patches \
--threads --dedupe=mid \
'(dfn:drivers OR dfn:arch OR dfn:Documentation OR dfn:include OR dfn:scripts) AND f:robh@kernel.org'
Any way to make this work? I find that it's more easily readable than the
"echo | lei q" version.
For bash users, the following should work as well:
lei q -I https://lore.kernel.org/all/ -o ~/work/temp/lei/robh-patches \
--threads --dedupe=mid <<< \
'(dfn:drivers OR dfn:arch OR dfn:Documentation OR dfn:include OR dfn:scripts) AND f:robh@kernel.org'
Suggestion, can -I accept the URL containing the query, so that the command
becomes:
lei q -o ~/mail/foo --threads --dedupe=mid -I \
https://lore.kernel.org/all/?q=f%3Atorvalds+AND+nq%3Agarbage
This way we pass both the location of the extindex to query AND the parameters
we should use, avoiding shell quoting problems?
-K
^ permalink raw reply [relevance 88%]
* Re: lei: incorrect quoting on saved searches (was Re: lore+lei: getting started)
2021-11-08 21:36 90% ` Konstantin Ryabitsev
@ 2021-11-08 21:48 90% ` Eric Wong
2021-11-08 22:36 88% ` Konstantin Ryabitsev
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2021-11-08 21:48 UTC (permalink / raw)
To: Konstantin Ryabitsev; +Cc: Rob Herring, meta, workflows
Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> On Mon, Nov 08, 2021 at 08:49:23PM +0000, Eric Wong wrote:
> > Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> > > On Mon, Nov 08, 2021 at 01:49:07PM -0600, Rob Herring wrote:
> >
> > > > > lei q -I https://lore.kernel.org/all/ -o ~/Mail/floppy \
> > > > > --threads --dedupe=mid \
> > > > > '(dfn:drivers/block/floppy.c OR dfhh:floppy_* OR s:floppy \
> > > > > OR ((nq:bug OR nq:regression) AND nq:floppy)) \
> > > > > AND rt:1.month.ago..'
> > > >
> > > > I tried a similar one which I had working as a bookmark:
> >
> > That's actually treating the entire single-quoted section as
> > a phrase search for Xapian.
>
> Hmm... I noticed that when I `lei edit-search` the initial query that was
> causing quoting issues, I get the following:
>
> [lei]
> q = (dfn:drivers OR dfn:arch OR dfn:Documentation OR dfn:include OR dfn:scripts) AND f:robh@kernel.org
>
> So, the extra quotes didn't get added to the config file. Running `lei up` on
> that saved search seems to do the right thing, so the erroneous quotes are
> only added during the initial `lei q` call.
Right, each entry in lei.q is actually an entry in argv[].
So the correct query should look something like:
[lei]
q = (
q = dfn:drivers/block/floppy.c
q = OR
...
> > The correct way to use '(', ')', and '*' on the command-line for
> > Xapian is to shell escape them:
>
> But putting them into single quotes should accomplish the same result, no? At
> least, that's how I've always understood shell escaping.
Yeah, that works, too. As long as spaces/tabs don't show up
within each argv[] element, it won't be interpreted as a phrase.
I really wanted: lei q s:"a quick brown fox"
to work from a shell like it would in the WWW UI;
and thus deprioritized '(' and ')' working properly :x
^ permalink raw reply [relevance 90%]
* Re: lei: incorrect quoting on saved searches (was Re: lore+lei: getting started)
2021-11-08 20:49 90% ` Eric Wong
@ 2021-11-08 21:36 90% ` Konstantin Ryabitsev
2021-11-08 21:48 90% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Konstantin Ryabitsev @ 2021-11-08 21:36 UTC (permalink / raw)
To: Eric Wong; +Cc: Rob Herring, meta, workflows
On Mon, Nov 08, 2021 at 08:49:23PM +0000, Eric Wong wrote:
> Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> > On Mon, Nov 08, 2021 at 01:49:07PM -0600, Rob Herring wrote:
> >
> > Moving this to meta.
>
> I don't think workflows should've been dropped, though.
>
> > > > lei q -I https://lore.kernel.org/all/ -o ~/Mail/floppy \
> > > > --threads --dedupe=mid \
> > > > '(dfn:drivers/block/floppy.c OR dfhh:floppy_* OR s:floppy \
> > > > OR ((nq:bug OR nq:regression) AND nq:floppy)) \
> > > > AND rt:1.month.ago..'
> > >
> > > I tried a similar one which I had working as a bookmark:
>
> That's actually treating the entire single-quoted section as
> a phrase search for Xapian.
Hmm... I noticed that when I `lei edit-search` the initial query that was
causing quoting issues, I get the following:
[lei]
q = (dfn:drivers OR dfn:arch OR dfn:Documentation OR dfn:include OR dfn:scripts) AND f:robh@kernel.org
So, the extra quotes didn't get added to the config file. Running `lei up` on
that saved search seems to do the right thing, so the erroneous quotes are
only added during the initial `lei q` call.
> The correct way to use '(', ')', and '*' on the command-line for
> Xapian is to shell escape them:
But putting them into single quotes should accomplish the same result, no? At
least, that's how I've always understood shell escaping.
-K
^ permalink raw reply [relevance 90%]
* Re: lei: incorrect quoting on saved searches (was Re: lore+lei: getting started)
2021-11-08 20:22 86% ` lei: incorrect quoting on saved searches (was Re: lore+lei: getting started) Konstantin Ryabitsev
2021-11-08 20:49 90% ` Eric Wong
@ 2021-11-08 20:53 90% ` Rob Herring
1 sibling, 0 replies; 200+ results
From: Rob Herring @ 2021-11-08 20:53 UTC (permalink / raw)
To: Konstantin Ryabitsev; +Cc: meta
On Mon, Nov 8, 2021 at 2:22 PM Konstantin Ryabitsev
<konstantin@linuxfoundation.org> wrote:
>
> On Mon, Nov 08, 2021 at 01:49:07PM -0600, Rob Herring wrote:
>
> Moving this to meta.
>
> > > lei q -I https://lore.kernel.org/all/ -o ~/Mail/floppy \
> > > --threads --dedupe=mid \
> > > '(dfn:drivers/block/floppy.c OR dfhh:floppy_* OR s:floppy \
> > > OR ((nq:bug OR nq:regression) AND nq:floppy)) \
> > > AND rt:1.month.ago..'
> >
> > I tried a similar one which I had working as a bookmark:
> >
> > $ lei q -I https://lore.kernel.org/all/ -o ~/Mail/my-patches
> > --threads --dedupe=mid '(dfn:drivers OR dfn:arch OR dfn:Documentation
> > OR dfn:include OR dfn:scripts) AND f:robh@kernel.org'
> > # /home/rob/.local/share/lei/store 0/0
> > # /usr/bin/curl -Sf -s -d ''
> > https://lore.kernel.org/all/?x=m&t=1&q=(dfn%3A%22drivers+OR+dfn%3Aarch+OR+dfn%3ADocumentation+OR+dfn%3Ainclude+OR+dfn%3Ascripts)+AND+f%3Arobh%40kernel.org%22
> > # 0 written to /home/rob/Mail/my-patches/ (0 matches)
>
> It's true, I get the same thing if I omit "AND rt:" at the end.
>
> $ lei q -I https://lore.kernel.org/all/ -o ~/work/temp/lei/robh-patches --threads --dedupe=mid '(dfn:drivers OR dfn:arch OR dfn:Documentation OR dfn:include OR dfn:scripts) AND f:robh@kernel.org'
> # /home/user/.local/share/lei/store 0/0
> # /usr/bin/curl -Sf -s -d '' https://lore.kernel.org/all/?x=m&t=1&q=(dfn%3A%22drivers+OR+dfn%3Aarch+OR+dfn%3ADocumentation+OR+dfn%3Ainclude+OR+dfn%3Ascripts)+AND+f%3Arobh%40kernel.org%22
> # 0 written to /home/user/work/temp/lei/robh-patches/ (0 matches)
> $ lei forget-search ~/work/temp/lei/robh-patches
> $ lei q -I https://lore.kernel.org/all/ -o ~/work/temp/lei/robh-patches --threads --dedupe=mid '(dfn:drivers OR dfn:arch OR dfn:Documentation OR dfn:include OR dfn:scripts) AND f:robh@kernel.org AND rt:1.month.ago..'
> # /usr/bin/curl -Sf -s -d '' https://lore.kernel.org/all/?x=m&t=1&q=(dfn%3Adrivers+OR+dfn%3Aarch+OR+dfn%3ADocumentation+OR+dfn%3Ainclude+OR+dfn%3Ascripts)+AND+f%3Arobh%40kernel.org+AND+rt%3A1633724105..
> # /home/user/.local/share/lei/store 13/13
> # https://lore.kernel.org/all/ 65/?
> # https://lore.kernel.org/all/ 75/75
> # 45 written to /home/user/work/temp/lei/robh-patches/ (88 matches)
>
> > It seems there is some problem in quoting. Notice the '%22' that's
> > inserted in the url.
>
> Deferring to Eric here.
>
> > Also, the above query is a bit of a work-around as what I really want
> > is just all patches from me. I haven't been able to get something to
> > work. I've tried things like 'dfn:*' or 'dfn:/' or 'dfn:b/'.
>
> I think 's:patch AND nq:diff' is a good option here.
Not even close really. That mainly finds my replies with 'diff' in
them. I'm not sure why, but it misses most actual patches:
https://lore.kernel.org/all/?q=s%3Apatch+nq%3Adiff+f%3Arobh%40kernel.org
Rob
^ permalink raw reply [relevance 90%]
* Re: lei: incorrect quoting on saved searches (was Re: lore+lei: getting started)
2021-11-08 20:22 86% ` lei: incorrect quoting on saved searches (was Re: lore+lei: getting started) Konstantin Ryabitsev
@ 2021-11-08 20:49 90% ` Eric Wong
2021-11-08 21:36 90% ` Konstantin Ryabitsev
2021-11-08 20:53 90% ` Rob Herring
1 sibling, 1 reply; 200+ results
From: Eric Wong @ 2021-11-08 20:49 UTC (permalink / raw)
To: Rob Herring; +Cc: Konstantin Ryabitsev, meta, workflows
Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> On Mon, Nov 08, 2021 at 01:49:07PM -0600, Rob Herring wrote:
>
> Moving this to meta.
I don't think workflows should've been dropped, though.
> > > lei q -I https://lore.kernel.org/all/ -o ~/Mail/floppy \
> > > --threads --dedupe=mid \
> > > '(dfn:drivers/block/floppy.c OR dfhh:floppy_* OR s:floppy \
> > > OR ((nq:bug OR nq:regression) AND nq:floppy)) \
> > > AND rt:1.month.ago..'
> >
> > I tried a similar one which I had working as a bookmark:
That's actually treating the entire single-quoted section as
a phrase search for Xapian.
The correct way to use '(', ')', and '*' on the command-line for
Xapian is to shell escape them:
lei q -I https://lore.kernel.org/all/ -o ~/Mail/floppy \
--threads --dedupe=mid \
\( dfn:drivers/block/floppy.c OR dfhh:floppy_\* OR s:floppy \
OR \(\(nq:bug OR nq:regression\) AND nq:floppy\)\) \
AND rt:1.month.ago...
Since shell escaping so many metacharacters is annoying,
stdin is supported (and implicit iff file|pipe):
echo '(dfn:drivers/block/floppy.c OR dfhh:floppy_* OR s:floppy \
OR ((nq:bug OR nq:regression) AND nq:floppy)) \
AND rt:1.month.ago..' | \
lei q -I https://lore.kernel.org/all/ -o ~/Mail/floppy \
--threads --dedupe=mid
^ permalink raw reply [relevance 90%]
* lei: incorrect quoting on saved searches (was Re: lore+lei: getting started)
[not found] ` <CAL_JsqJBh1O3H2-P07AHzVq0x89BoP_N6P=rT5up6=3QyF_B0Q@mail.gmail.com>
@ 2021-11-08 20:22 86% ` Konstantin Ryabitsev
2021-11-08 20:49 90% ` Eric Wong
2021-11-08 20:53 90% ` Rob Herring
0 siblings, 2 replies; 200+ results
From: Konstantin Ryabitsev @ 2021-11-08 20:22 UTC (permalink / raw)
To: Rob Herring; +Cc: meta
On Mon, Nov 08, 2021 at 01:49:07PM -0600, Rob Herring wrote:
Moving this to meta.
> > lei q -I https://lore.kernel.org/all/ -o ~/Mail/floppy \
> > --threads --dedupe=mid \
> > '(dfn:drivers/block/floppy.c OR dfhh:floppy_* OR s:floppy \
> > OR ((nq:bug OR nq:regression) AND nq:floppy)) \
> > AND rt:1.month.ago..'
>
> I tried a similar one which I had working as a bookmark:
>
> $ lei q -I https://lore.kernel.org/all/ -o ~/Mail/my-patches
> --threads --dedupe=mid '(dfn:drivers OR dfn:arch OR dfn:Documentation
> OR dfn:include OR dfn:scripts) AND f:robh@kernel.org'
> # /home/rob/.local/share/lei/store 0/0
> # /usr/bin/curl -Sf -s -d ''
> https://lore.kernel.org/all/?x=m&t=1&q=(dfn%3A%22drivers+OR+dfn%3Aarch+OR+dfn%3ADocumentation+OR+dfn%3Ainclude+OR+dfn%3Ascripts)+AND+f%3Arobh%40kernel.org%22
> # 0 written to /home/rob/Mail/my-patches/ (0 matches)
It's true, I get the same thing if I omit "AND rt:" at the end.
$ lei q -I https://lore.kernel.org/all/ -o ~/work/temp/lei/robh-patches --threads --dedupe=mid '(dfn:drivers OR dfn:arch OR dfn:Documentation OR dfn:include OR dfn:scripts) AND f:robh@kernel.org'
# /home/user/.local/share/lei/store 0/0
# /usr/bin/curl -Sf -s -d '' https://lore.kernel.org/all/?x=m&t=1&q=(dfn%3A%22drivers+OR+dfn%3Aarch+OR+dfn%3ADocumentation+OR+dfn%3Ainclude+OR+dfn%3Ascripts)+AND+f%3Arobh%40kernel.org%22
# 0 written to /home/user/work/temp/lei/robh-patches/ (0 matches)
$ lei forget-search ~/work/temp/lei/robh-patches
$ lei q -I https://lore.kernel.org/all/ -o ~/work/temp/lei/robh-patches --threads --dedupe=mid '(dfn:drivers OR dfn:arch OR dfn:Documentation OR dfn:include OR dfn:scripts) AND f:robh@kernel.org AND rt:1.month.ago..'
# /usr/bin/curl -Sf -s -d '' https://lore.kernel.org/all/?x=m&t=1&q=(dfn%3Adrivers+OR+dfn%3Aarch+OR+dfn%3ADocumentation+OR+dfn%3Ainclude+OR+dfn%3Ascripts)+AND+f%3Arobh%40kernel.org+AND+rt%3A1633724105..
# /home/user/.local/share/lei/store 13/13
# https://lore.kernel.org/all/ 65/?
# https://lore.kernel.org/all/ 75/75
# 45 written to /home/user/work/temp/lei/robh-patches/ (88 matches)
> It seems there is some problem in quoting. Notice the '%22' that's
> inserted in the url.
Deferring to Eric here.
> Also, the above query is a bit of a work-around as what I really want
> is just all patches from me. I haven't been able to get something to
> work. I've tried things like 'dfn:*' or 'dfn:/' or 'dfn:b/'.
I think 's:patch AND nq:diff' is a good option here.
-K
^ permalink raw reply [relevance 86%]
* [PATCH 2/3] doc: -clone|lei add-external: add bit about the Makefile
@ 2021-11-03 8:34 68% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-11-03 8:34 UTC (permalink / raw)
To: meta
It's pretty useful, I think.
---
Documentation/lei-add-external.pod | 2 ++
Documentation/public-inbox-clone.pod | 5 +++++
2 files changed, 7 insertions(+)
diff --git a/Documentation/lei-add-external.pod b/Documentation/lei-add-external.pod
index 1ab65a1650a6..1761eed1ed78 100644
--- a/Documentation/lei-add-external.pod
+++ b/Documentation/lei-add-external.pod
@@ -29,6 +29,8 @@ Default: 0
=item --mirror=URL
Create C<LOCATION> by mirroring the public-inbox at C<URL>.
+C<LOCATION> will have a Makefile with a C<make update>
+target to update the external.
=item --epoch=RANGE
diff --git a/Documentation/public-inbox-clone.pod b/Documentation/public-inbox-clone.pod
index efee01eec592..103260913d92 100644
--- a/Documentation/public-inbox-clone.pod
+++ b/Documentation/public-inbox-clone.pod
@@ -18,6 +18,11 @@ L<public-inbox-index(1)>. Those commands must be run separately
if serving/searching the mirror is required. As-is,
public-inbox-clone is suitable for creating a git-only backup.
+public-inbox-clone creates a Makefile with handy targets to update the
+inbox once indexed. This Makefile may be edited by the user; it will
+not be rewritten by L<public-inbox-fetch(1)> unless it is removed
+completely.
+
public-inbox-clone does not use nor require any extra
configuration files (not even C<~/.public-inbox/config>).
^ permalink raw reply related [relevance 68%]
* Initial Fedora packaging for lei
@ 2021-11-03 1:28 71% Konstantin Ryabitsev
0 siblings, 0 replies; 200+ results
From: Konstantin Ryabitsev @ 2021-11-03 1:28 UTC (permalink / raw)
To: meta; +Cc: tools
[-- Attachment #1: Type: text/plain, Size: 622 bytes --]
Hi, all:
I did some initial work to package lei for Fedora 34 and 35 (out today). The
lei parts should be ready to use, though I'll continue to work on the server
parts (only needed if you're running httpd/nntpd/imapd daemons).
For now, you'll need to enable my copr repository to use it:
sudo dnf copr enable icon/b4
sudo dnf install lei
(you can also install python3-b4 from there if you're using b4)
I'm still working on some introductory docs, but you can use the following
resource to get yourself going with search-based lei goodness:
https://josefbacik.github.io/kernel/2021/10/18/lei-and-b4.html
-K
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]
^ permalink raw reply [relevance 71%]
* Re: [PATCH] doc: lei-q: document SEARCH TERMS prefixes
2021-11-02 23:55 43% [PATCH] doc: lei-q: document SEARCH TERMS prefixes Eric Wong
@ 2021-11-03 0:28 71% ` Kyle Meyer
0 siblings, 0 replies; 200+ results
From: Kyle Meyer @ 2021-11-03 0:28 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
Eric Wong writes:
> +=for comment
> +AUTO-GENERATED-SEARCH-TERMS-BEGIN
Looks great, thanks. By the way, this takes care of a to-do in this
file:
=for comment
TODO: Give common prefixes, or at least a description/reference.
^ permalink raw reply [relevance 71%]
* [PATCH] doc: lei-q: document SEARCH TERMS prefixes
@ 2021-11-02 23:55 43% Eric Wong
2021-11-03 0:28 71% ` Kyle Meyer
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2021-11-02 23:55 UTC (permalink / raw)
To: meta
The new Documentation/common.perl file will be used for
all manpages in the future.
---
Documentation/common.perl | 65 +++++++++++++++++++++++++++++++++++++++
Documentation/include.mk | 3 ++
Documentation/lei-q.pod | 51 +++++++++++++++++++++++++++++-
MANIFEST | 1 +
lib/PublicInbox/Search.pm | 8 ++---
5 files changed, 123 insertions(+), 5 deletions(-)
create mode 100755 Documentation/common.perl
diff --git a/Documentation/common.perl b/Documentation/common.perl
new file mode 100755
index 000000000000..5fdbe8da055e
--- /dev/null
+++ b/Documentation/common.perl
@@ -0,0 +1,65 @@
+#!perl -w
+# Copyright (C) all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+use strict;
+use Fcntl qw(SEEK_SET);
+my $have_search = eval { require PublicInbox::Search; 1 };
+my $addr = 'meta@public-inbox.org';
+for my $pod (@ARGV) {
+ open my $fh, '+<', $pod or die "open($pod): $!";
+ my $s = do { local $/; <$fh> } // die "read $!";
+ my $orig = $s;
+ $s =~ s!^=head1 COPYRIGHT\n.+?^=head1([^\n]+)\n!=head1 COPYRIGHT
+
+Copyright all contributors L<mailto:$addr>
+
+License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>
+
+=head1$1
+ !ms;
+
+ $s =~ s!^=head1 CONTACT\n.+?^=head1([^\n]+)\n!=head1 CONTACT
+
+Feedback welcome via plain-text mail to L<mailto:$addr>
+
+The mail archives are hosted at L<https://public-inbox.org/meta/> and
+L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/>
+
+=head1$1
+ !ms;
+ $have_search and $s =~ s!^=for\scomment\n
+ ^AUTO-GENERATED-SEARCH-TERMS-BEGIN\n
+ .+?
+ ^=for\scomment\n
+ ^AUTO-GENERATED-SEARCH-TERMS-END\n
+ !search_terms()!emsx;
+ $s =~ s/[ \t]+$//sgm;
+ next if $s eq $orig;
+ seek($fh, 0, SEEK_SET) or die "seek: $!";
+ truncate($fh, 0) or die "truncate: $!";
+ print $fh $s or die "print: $!";
+ close $fh or die "close: $!";
+}
+
+sub search_terms {
+ my $help = eval('\@PublicInbox::Search::HELP');
+ my $s = '';
+ my $pad = 0;
+ my $i;
+ for ($i = 0; $i < @$help; $i += 2) {
+ my $pfx = $help->[$i];
+ my $n = length($pfx);
+ $pad = $n if $n > $pad;
+ $s .= $pfx . "\0";
+ $s .= $help->[$i + 1];
+ $s .= "\f\n";
+ }
+ $pad += 2;
+ my $padding = ' ' x ($pad + 4);
+ $s =~ s/^/$padding/gms;
+ $s =~ s/^$padding(\S+)\0/" $1".(' ' x ($pad - length($1)))/egms;
+ $s =~ s/\f\n/\n/gs;
+ $s =~ s/^ //gms;
+ substr($s, 0, 0, "=for comment\nAUTO-GENERATED-SEARCH-TERMS-BEGIN\n\n");
+ $s .= "\n=for comment\nAUTO-GENERATED-SEARCH-TERMS-END\n";
+}
diff --git a/Documentation/include.mk b/Documentation/include.mk
index 5f3ffcc56937..149f9e3ce42e 100644
--- a/Documentation/include.mk
+++ b/Documentation/include.mk
@@ -80,6 +80,9 @@ Documentation/flow.txt : Documentation/flow.ge
touch -r Documentation/flow.ge $@+
mv $@+ $@
+Documentation/lei-q.pod : lib/PublicInbox/Search.pm Documentation/common.perl
+ $(PERL) -I lib -w Documentation/common.perl $@
+
NEWS NEWS.atom NEWS.html : $(news_deps)
$(PERL) -I lib -w Documentation/mknews.perl $@ $(RELEASES)
diff --git a/Documentation/lei-q.pod b/Documentation/lei-q.pod
index 574c12eb9d3a..fe281d7b0121 100644
--- a/Documentation/lei-q.pod
+++ b/Documentation/lei-q.pod
@@ -245,6 +245,55 @@ Default: C<auto>
=back
+=head1 SEARCH TERMS
+
+C<lei q> supports the same search prefixes used by HTTP(S) public-inbox
+instances:
+
+=for comment
+AUTO-GENERATED-SEARCH-TERMS-BEGIN
+
+ s: match within Subject e.g. s:"a quick brown fox"
+ d: match date-time range, git "approxidate" formats supported
+ Open-ended ranges such as `d:last.week..' and
+ `d:..2.days.ago' are supported
+ b: match within message body, including text attachments
+ nq: match non-quoted text within message body
+ q: match quoted text within message body
+ n: match filename of attachment(s)
+ t: match within the To header
+ c: match within the Cc header
+ f: match within the From header
+ a: match within the To, Cc, and From headers
+ tc: match within the To and Cc headers
+ l: match contents of the List-Id header
+ bs: match within the Subject and body
+ dfn: match filename from diff
+ dfa: match diff removed (-) lines
+ dfb: match diff added (+) lines
+ dfhh: match diff hunk header context (usually a function name)
+ dfctx: match diff context lines
+ dfpre: match pre-image git blob ID
+ dfpost: match post-image git blob ID
+ dfblob: match either pre or post-image git blob ID
+ rt: match received time, like `d:' if sender's clock was correct
+
+=for comment
+AUTO-GENERATED-SEARCH-TERMS-END
+
+Additional search prefixes which only affect the local lei/store:
+
+ L: match the given label
+ kw: match the given keywords
+
+See L<lei-tag(1)> for more info on labels and keywords.
+
+Most prefixes are probabilistic, meaning they support stemming
+and wildcards (C<*>). Ranges (such as C<d:>) and boolean prefixes
+do not support stemming or wildcards.
+The upstream Xapian query parser documentation fully explains
+the query syntax: L<https://xapian.org/docs/queryparser.html>
+
=head1 TIPS
C<-f reply> is intended to aid in turning a cover letter
@@ -261,7 +310,7 @@ L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/>
=head1 COPYRIGHT
-Copyright 2021 all contributors L<mailto:meta@public-inbox.org>
+Copyright all contributors L<mailto:meta@public-inbox.org>
License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>
diff --git a/MANIFEST b/MANIFEST
index 1e8f60fb41da..47eadad2e403 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -13,6 +13,7 @@ Documentation/RelNotes/v1.6.0.eml
Documentation/RelNotes/v1.6.1.eml
Documentation/RelNotes/v1.7.0.wip
Documentation/clients.txt
+Documentation/common.perl
Documentation/dc-dlvr-spam-flow.txt
Documentation/design_notes.txt
Documentation/design_www.txt
diff --git a/lib/PublicInbox/Search.pm b/lib/PublicInbox/Search.pm
index 600e6400d4b6..523003b3c269 100644
--- a/lib/PublicInbox/Search.pm
+++ b/lib/PublicInbox/Search.pm
@@ -155,9 +155,9 @@ my %prob_prefix = (
our @HELP = (
's:' => 'match within Subject e.g. s:"a quick brown fox"',
'd:' => <<EOF,
-match date range, git "approxidate" formats supported
-Open-ended ranges such as `d:last.week..' and `d:..2.days.ago'
-are supported
+match date-time range, git "approxidate" formats supported
+Open-ended ranges such as `d:last.week..' and
+`d:..2.days.ago' are supported
EOF
'b:' => 'match within message body, including text attachments',
'nq:' => 'match non-quoted text within message body',
@@ -179,7 +179,7 @@ EOF
'dfpost:' => 'match post-image git blob ID',
'dfblob:' => 'match either pre or post-image git blob ID',
'rt:' => <<EOF,
-match received time, like `d:' unless sender's clock was broken
+match received time, like `d:' if sender's clock was correct
EOF
);
chomp @HELP;
^ permalink raw reply related [relevance 43%]
* Re: [PATCH 2/3] lei: simplify common LeiInput users with ->wq1_start
2021-11-02 18:14 41% ` [PATCH 2/3] lei: simplify common LeiInput users with ->wq1_start Eric Wong
@ 2021-11-02 18:47 71% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-11-02 18:47 UTC (permalink / raw)
To: meta
Eric Wong <e@80x24.org> wrote:
> This method replaces a common worker of starting workers,
s/common worker/common pattern/ :x
^ permalink raw reply [relevance 71%]
* [PATCH 1/3] lei mail-diff: do not default to 'eml'
2021-11-02 18:14 71% [PATCH 0/3] lei: more coherent input handling Eric Wong
@ 2021-11-02 18:14 71% ` Eric Wong
2021-11-02 18:14 41% ` [PATCH 2/3] lei: simplify common LeiInput users with ->wq1_start Eric Wong
2021-11-02 18:14 65% ` [PATCH 3/3] lei <rediff|rm|tag>: stdin implies `-F eml' Eric Wong
2 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-11-02 18:14 UTC (permalink / raw)
To: meta
In retrospect, this doesn't make sense, since it needs at least
two messages to diff. So go about "normal" input rules and
require users to specify the format.
---
lib/PublicInbox/LeiMailDiff.pm | 1 -
1 file changed, 1 deletion(-)
diff --git a/lib/PublicInbox/LeiMailDiff.pm b/lib/PublicInbox/LeiMailDiff.pm
index b21a0c36..48ba74cf 100644
--- a/lib/PublicInbox/LeiMailDiff.pm
+++ b/lib/PublicInbox/LeiMailDiff.pm
@@ -76,7 +76,6 @@ sub input_eml_cb { # used by PublicInbox::LeiInput::input_fh
sub lei_mail_diff {
my ($lei, @argv) = @_;
- $lei->{opt}->{'in-format'} //= 'eml' if !grep(/\A[a-z0-9]+:/i, @argv);
my $self = bless {}, __PACKAGE__;
$self->prepare_inputs($lei, \@argv) or return;
my $isatty = -t $lei->{1};
^ permalink raw reply related [relevance 71%]
* [PATCH 0/3] lei: more coherent input handling
@ 2021-11-02 18:14 71% Eric Wong
2021-11-02 18:14 71% ` [PATCH 1/3] lei mail-diff: do not default to 'eml' Eric Wong
` (2 more replies)
0 siblings, 3 replies; 200+ results
From: Eric Wong @ 2021-11-02 18:14 UTC (permalink / raw)
To: meta
Hopefully everything makes more sense to new users.
Eric Wong (3):
lei mail-diff: do not default to 'eml'
lei: simplify common LeiInput users with ->wq1_start
lei <rediff|rm|tag>: stdin implies `-F eml'
lib/PublicInbox/LEI.pm | 15 ++++++++++++++-
lib/PublicInbox/LeiExportKw.pm | 7 +------
lib/PublicInbox/LeiForgetSearch.pm | 7 +------
lib/PublicInbox/LeiImport.pm | 7 +------
lib/PublicInbox/LeiLsMailSource.pm | 7 +------
lib/PublicInbox/LeiMailDiff.pm | 8 +-------
lib/PublicInbox/LeiP2q.pm | 7 +------
lib/PublicInbox/LeiRediff.pm | 7 ++-----
lib/PublicInbox/LeiRefreshMailSync.pm | 7 +------
lib/PublicInbox/LeiRm.pm | 7 ++-----
lib/PublicInbox/LeiTag.pm | 11 +++--------
lib/PublicInbox/LeiUp.pm | 5 +----
12 files changed, 29 insertions(+), 66 deletions(-)
^ permalink raw reply [relevance 71%]
* [PATCH 3/3] lei <rediff|rm|tag>: stdin implies `-F eml'
2021-11-02 18:14 71% [PATCH 0/3] lei: more coherent input handling Eric Wong
2021-11-02 18:14 71% ` [PATCH 1/3] lei mail-diff: do not default to 'eml' Eric Wong
2021-11-02 18:14 41% ` [PATCH 2/3] lei: simplify common LeiInput users with ->wq1_start Eric Wong
@ 2021-11-02 18:14 65% ` Eric Wong
2 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-11-02 18:14 UTC (permalink / raw)
To: meta
These commands are usually run on a single message, so saving
the user the trouble of typing `-F eml' on the command-line
seems reasonable. I don't think commands like "index" and
"import" will be too useful for single messages, though.
---
lib/PublicInbox/LeiRediff.pm | 2 +-
lib/PublicInbox/LeiRm.pm | 2 +-
lib/PublicInbox/LeiTag.pm | 4 ++--
3 files changed, 4 insertions(+), 4 deletions(-)
diff --git a/lib/PublicInbox/LeiRediff.pm b/lib/PublicInbox/LeiRediff.pm
index f0521bcc..c312d90f 100644
--- a/lib/PublicInbox/LeiRediff.pm
+++ b/lib/PublicInbox/LeiRediff.pm
@@ -256,7 +256,7 @@ sub lei_rediff {
($lei->{opt}->{drq} && !$lei->{opt}->{verbose}) and
$lei->{opt}->{quiet} //= 1;
$lei->_lei_store(1)->write_prepare($lei);
- $lei->{opt}->{'in-format'} //= 'eml';
+ $lei->{opt}->{'in-format'} //= 'eml' if $lei->{opt}->{stdin};
# maybe it's a non-email (code) blob from a coderepo
my $git_dirs = $lei->{opt}->{'git-dir'} //= [];
if ($lei->{opt}->{cwd} // 1) {
diff --git a/lib/PublicInbox/LeiRm.pm b/lib/PublicInbox/LeiRm.pm
index 62423ac9..00b12485 100644
--- a/lib/PublicInbox/LeiRm.pm
+++ b/lib/PublicInbox/LeiRm.pm
@@ -16,7 +16,7 @@ sub input_eml_cb { # used by PublicInbox::LeiInput::input_fh
sub lei_rm {
my ($lei, @inputs) = @_;
$lei->_lei_store(1)->write_prepare($lei);
- $lei->{opt}->{'in-format'} //= 'eml';
+ $lei->{opt}->{'in-format'} //= 'eml' if $lei->{opt}->{stdin};
my $self = bless {}, __PACKAGE__;
$self->prepare_inputs($lei, \@inputs) or return;
$lei->{-err_type} = 'non-fatal';
diff --git a/lib/PublicInbox/LeiTag.pm b/lib/PublicInbox/LeiTag.pm
index 2dc59f70..8ce96a10 100644
--- a/lib/PublicInbox/LeiTag.pm
+++ b/lib/PublicInbox/LeiTag.pm
@@ -27,8 +27,8 @@ sub pmdir_cb { # called via wq_io_do from LeiPmdir->each_mdir_fn
sub lei_tag { # the "lei tag" method
my ($lei, @argv) = @_;
- my $sto = $lei->_lei_store(1);
- $sto->write_prepare($lei);
+ $lei->{opt}->{'in-format'} //= 'eml' if $lei->{opt}->{stdin};
+ my $sto = $lei->_lei_store(1)->write_prepare($lei);
my $self = bless {}, __PACKAGE__;
$lei->ale; # refresh and prepare
my $vmd_mod = $self->vmd_mod_extract(\@argv);
^ permalink raw reply related [relevance 65%]
* [PATCH 2/3] lei: simplify common LeiInput users with ->wq1_start
2021-11-02 18:14 71% [PATCH 0/3] lei: more coherent input handling Eric Wong
2021-11-02 18:14 71% ` [PATCH 1/3] lei mail-diff: do not default to 'eml' Eric Wong
@ 2021-11-02 18:14 41% ` Eric Wong
2021-11-02 18:47 71% ` Eric Wong
2021-11-02 18:14 65% ` [PATCH 3/3] lei <rediff|rm|tag>: stdin implies `-F eml' Eric Wong
2 siblings, 1 reply; 200+ results
From: Eric Wong @ 2021-11-02 18:14 UTC (permalink / raw)
To: meta
This method replaces a common worker of starting workers,
preparing internal auth ops, and asynchronous waiting of
command completion.
It also adds missing LeiAuth support for rediff and rm
which rarely need auth.
---
lib/PublicInbox/LEI.pm | 15 ++++++++++++++-
lib/PublicInbox/LeiExportKw.pm | 7 +------
lib/PublicInbox/LeiForgetSearch.pm | 7 +------
lib/PublicInbox/LeiImport.pm | 7 +------
lib/PublicInbox/LeiLsMailSource.pm | 7 +------
lib/PublicInbox/LeiMailDiff.pm | 7 +------
lib/PublicInbox/LeiP2q.pm | 7 +------
lib/PublicInbox/LeiRediff.pm | 5 +----
lib/PublicInbox/LeiRefreshMailSync.pm | 7 +------
lib/PublicInbox/LeiRm.pm | 5 +----
lib/PublicInbox/LeiTag.pm | 7 +------
lib/PublicInbox/LeiUp.pm | 5 +----
12 files changed, 25 insertions(+), 61 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 3e1706a0..887025de 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -631,7 +631,10 @@ sub pkt_ops {
sub workers_start {
my ($lei, $wq, $jobs, $ops, $flds) = @_;
- $ops = pkt_ops($lei, { ($ops ? %$ops : ()) });
+ $ops //= {};
+ ($wq->can('net_merge_all_done') && $lei->{auth}) and
+ $lei->{auth}->op_merge($ops, $wq, $lei);
+ pkt_ops($lei, $ops);
$ops->{''} //= [ $wq->can('_lei_wq_eof') || \&wq_eof, $lei ];
my $end = $lei->pkt_op_pair;
my $ident = $wq->{-wq_ident} // "lei-$lei->{cmd} worker";
@@ -648,12 +651,22 @@ sub workers_start {
# call this when we're ready to wait on events and yield to other clients
sub wait_wq_events {
my ($lei, $op_c, $ops) = @_;
+ my $wq1 = $lei->{wq1};
+ ($wq1 && $wq1->can('net_merge_all_done') && !$lei->{auth}) and
+ $wq1->net_merge_all_done;
for my $wq (grep(defined, @$lei{qw(ikw pmd)})) { # auxiliary WQs
$wq->wq_close;
}
$op_c->{ops} = $ops;
}
+sub wq1_start {
+ my ($lei, $wq, $jobs) = @_;
+ my ($op_c, $ops) = workers_start($lei, $wq, $jobs // 1);
+ $lei->{wq1} = $wq;
+ wait_wq_events($lei, $op_c, $ops); # net_merge_all_done if !{auth}
+}
+
sub _help {
require PublicInbox::LeiHelp;
PublicInbox::LeiHelp::call($_[0], $_[1], \%CMD, \%OPTDESC);
diff --git a/lib/PublicInbox/LeiExportKw.pm b/lib/PublicInbox/LeiExportKw.pm
index 0ecfb782..d2396fa7 100644
--- a/lib/PublicInbox/LeiExportKw.pm
+++ b/lib/PublicInbox/LeiExportKw.pm
@@ -124,13 +124,8 @@ EOM
'imap_add_kw' : 'imap_set_kw');
$self->{nwr}->{-skip_creat} = 1;
}
- my $ops = {};
- $lei->{auth}->op_merge($ops, $self, $lei) if $lei->{auth};
- (my $op_c, $ops) = $lei->workers_start($self, 1, $ops);
- $lei->{wq1} = $self;
$lei->{-err_type} = 'non-fatal';
- net_merge_all_done($self) unless $lei->{auth};
- $lei->wait_wq_events($op_c, $ops); # net_merge_all_done if !{auth}
+ $lei->wq1_start($self);
}
sub _complete_export_kw {
diff --git a/lib/PublicInbox/LeiForgetSearch.pm b/lib/PublicInbox/LeiForgetSearch.pm
index dfeb0293..dd358ae1 100644
--- a/lib/PublicInbox/LeiForgetSearch.pm
+++ b/lib/PublicInbox/LeiForgetSearch.pm
@@ -45,12 +45,7 @@ sub lei_forget_search {
if ($self->{o_remote}) { # setup lei->{auth}
$self->prepare_inputs($lei, $self->{o_remote}) or return;
}
- my $ops = {};
- $lei->{auth}->op_merge($ops, $self, $lei) if $lei->{auth};
- (my $op_c, $ops) = $lei->workers_start($self, 1, $ops);
- $lei->{wq1} = $self;
- net_merge_all_done($self) unless $lei->{auth};
- $lei->wait_wq_events($op_c, $ops);
+ $lei->wq1_start($self);
}
sub do_prune {
diff --git a/lib/PublicInbox/LeiImport.pm b/lib/PublicInbox/LeiImport.pm
index d8f39fdf..bbc0634e 100644
--- a/lib/PublicInbox/LeiImport.pm
+++ b/lib/PublicInbox/LeiImport.pm
@@ -102,14 +102,9 @@ sub do_import_index ($$@) {
}
($lei->{opt}->{'new-only'} && (!$net || !$net->{imap_order})) and
warn "# --new-only is only for IMAP\n";
- my $ops = {};
- $lei->{auth}->op_merge($ops, $self, $lei) if $lei->{auth};
$lei->{-eml_noisy} = 1;
- (my $op_c, $ops) = $lei->workers_start($self, $j, $ops);
- $lei->{wq1} = $self;
$lei->{-err_type} = 'non-fatal';
- net_merge_all_done($self) unless $lei->{auth};
- $lei->wait_wq_events($op_c, $ops);
+ $lei->wq1_start($self, $j);
}
sub lei_import { # the main "lei import" method
diff --git a/lib/PublicInbox/LeiLsMailSource.pm b/lib/PublicInbox/LeiLsMailSource.pm
index 5eb7032d..50799270 100644
--- a/lib/PublicInbox/LeiLsMailSource.pm
+++ b/lib/PublicInbox/LeiLsMailSource.pm
@@ -95,13 +95,8 @@ sub lei_ls_mail_source {
$json->pretty(1)->indent(2) if $isatty || $lei->{opt}->{pretty};
}
$lei->start_pager if $isatty;
- my $ops = {};
- $lei->{auth}->op_merge($ops, $self, $lei);
- (my $op_c, $ops) = $lei->workers_start($self, 1, $ops);
- $lei->{wq1} = $self;
$lei->{-err_type} = 'non-fatal';
- net_merge_all_done($self) unless $lei->{auth};
- $lei->wait_wq_events($op_c, $ops); # net_merge_all_done if !{auth}
+ $lei->wq1_start($self);
}
sub _complete_ls_mail_source {
diff --git a/lib/PublicInbox/LeiMailDiff.pm b/lib/PublicInbox/LeiMailDiff.pm
index 48ba74cf..2b4cfd9e 100644
--- a/lib/PublicInbox/LeiMailDiff.pm
+++ b/lib/PublicInbox/LeiMailDiff.pm
@@ -81,13 +81,8 @@ sub lei_mail_diff {
my $isatty = -t $lei->{1};
$lei->{opt}->{color} //= $isatty;
$lei->start_pager if $isatty;
- my $ops = {};
- $lei->{auth}->op_merge($ops, $self, $lei) if $lei->{auth};
- (my $op_c, $ops) = $lei->workers_start($self, 1, $ops);
- $lei->{wq1} = $self;
$lei->{-err_type} = 'non-fatal';
- net_merge_all_done($self) unless $lei->{auth};
- $lei->wait_wq_events($op_c, $ops);
+ $lei->wq1_start($self);
}
no warnings 'once';
diff --git a/lib/PublicInbox/LeiP2q.pm b/lib/PublicInbox/LeiP2q.pm
index 09ec0a07..610adb78 100644
--- a/lib/PublicInbox/LeiP2q.pm
+++ b/lib/PublicInbox/LeiP2q.pm
@@ -183,12 +183,7 @@ sub lei_p2q { # the "lei patch-to-query" entry point
$lei->{opt}->{'in-format'} //= 'eml' if $lei->{opt}->{stdin};
my $self = bless { missing_ok => 1 }, __PACKAGE__;
$self->prepare_inputs($lei, \@inputs) or return;
- my $ops = {};
- $lei->{auth}->op_merge($ops, $self, $lei) if $lei->{auth};
- (my $op_c, $ops) = $lei->workers_start($self, 1, $ops);
- $lei->{wq1} = $self;
- net_merge_all_done($self) unless $lei->{auth};
- $lei->wait_wq_events($op_c, $ops);
+ $lei->wq1_start($self);
}
sub ipc_atfork_child {
diff --git a/lib/PublicInbox/LeiRediff.pm b/lib/PublicInbox/LeiRediff.pm
index 56c457fc..f0521bcc 100644
--- a/lib/PublicInbox/LeiRediff.pm
+++ b/lib/PublicInbox/LeiRediff.pm
@@ -279,10 +279,7 @@ sub lei_rediff {
my $isatty = -t $lei->{1};
$lei->{opt}->{color} //= $isatty;
$lei->start_pager if $isatty;
- my ($op_c, $ops) = $lei->workers_start($self, 1);
- $lei->{wq1} = $self;
- net_merge_all_done($self) unless $lei->{auth};
- $lei->wait_wq_events($op_c, $ops);
+ $lei->wq1_start($self);
}
sub ipc_atfork_child {
diff --git a/lib/PublicInbox/LeiRefreshMailSync.pm b/lib/PublicInbox/LeiRefreshMailSync.pm
index f516f572..7821008f 100644
--- a/lib/PublicInbox/LeiRefreshMailSync.pm
+++ b/lib/PublicInbox/LeiRefreshMailSync.pm
@@ -81,13 +81,8 @@ EOM
my $self = bless { missing_ok => 1, lms => $lms }, __PACKAGE__;
$lei->{opt}->{'mail-sync'} = 1; # for prepare_inputs
$self->prepare_inputs($lei, \@folders) or return;
- my $ops = {};
- $lei->{auth}->op_merge($ops, $self, $lei) if $lei->{auth};
- (my $op_c, $ops) = $lei->workers_start($self, 1, $ops);
- $lei->{wq1} = $self;
$lei->{-err_type} = 'non-fatal';
- net_merge_all_done($self) unless $lei->{auth};
- $lei->wait_wq_events($op_c, $ops); # net_merge_all_done if !{auth}
+ $lei->wq1_start($self);
}
sub ipc_atfork_child { # needed for PublicInbox::LeiPmdir
diff --git a/lib/PublicInbox/LeiRm.pm b/lib/PublicInbox/LeiRm.pm
index cc1abbff..62423ac9 100644
--- a/lib/PublicInbox/LeiRm.pm
+++ b/lib/PublicInbox/LeiRm.pm
@@ -19,11 +19,8 @@ sub lei_rm {
$lei->{opt}->{'in-format'} //= 'eml';
my $self = bless {}, __PACKAGE__;
$self->prepare_inputs($lei, \@inputs) or return;
- my ($op_c, $ops) = $lei->workers_start($self, 1);
- $lei->{wq1} = $self;
$lei->{-err_type} = 'non-fatal';
- net_merge_all_done($self) unless $lei->{auth};
- $lei->wait_wq_events($op_c, $ops);
+ $lei->wq1_start($self);
}
no warnings 'once';
diff --git a/lib/PublicInbox/LeiTag.pm b/lib/PublicInbox/LeiTag.pm
index d64a9f86..2dc59f70 100644
--- a/lib/PublicInbox/LeiTag.pm
+++ b/lib/PublicInbox/LeiTag.pm
@@ -37,13 +37,8 @@ sub lei_tag { # the "lei tag" method
$self->prepare_inputs($lei, \@argv) or return;
grep(defined, @$vmd_mod{qw(+kw +L -L -kw)}) or
return $lei->fail('no keywords or labels specified');
- my $ops = {};
- $lei->{auth}->op_merge($ops, $self, $lei) if $lei->{auth};
- (my $op_c, $ops) = $lei->workers_start($self, 1, $ops);
- $lei->{wq1} = $self;
$lei->{-err_type} = 'non-fatal';
- net_merge_all_done($self) unless $lei->{auth};
- $lei->wait_wq_events($op_c, $ops);
+ $lei->wq1_start($self);
}
sub note_unimported {
diff --git a/lib/PublicInbox/LeiUp.pm b/lib/PublicInbox/LeiUp.pm
index 79639d5e..66d950b2 100644
--- a/lib/PublicInbox/LeiUp.pm
+++ b/lib/PublicInbox/LeiUp.pm
@@ -136,10 +136,7 @@ EOM
if ($lei->{auth}) { # start auth worker
require PublicInbox::NetWriter;
bless $lei->{net}, 'PublicInbox::NetWriter';
- $lei->{auth}->op_merge(my $ops = {}, $self, $lei);
- (my $op_c, $ops) = $lei->workers_start($self, 1, $ops);
- $lei->{wq1} = $self;
- $lei->wait_wq_events($op_c, $ops);
+ $lei->wq1_start($self);
# net_merge_all_done will fire when auth is done
} else {
redispatch_all($self, $lei); # see below
^ permalink raw reply related [relevance 41%]
* [PATCH] t/lei-refresh-mail-sync: speed up test on FreeBSD 12
@ 2021-11-02 9:24 60% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-11-02 9:24 UTC (permalink / raw)
To: meta
And improve reliability while we're at it. It seems closing a
TCP listen socket on FreeBSD 12.2 doesn't cause connect()-ing
clients to fail. This happens regardless of whether a socket is
IPv4 or IPv6
This non-failure was causing tests to timeout slowly on the
client side instead of failing immediately. We now fork a new
process which does nothing but accept() + shutdown() to emulate
a dead server.
Reliability improves on all OSes since there's never a point in
time when another process can bind the socket.
---
t/lei-refresh-mail-sync.t | 35 +++++++++++++++++++----------------
1 file changed, 19 insertions(+), 16 deletions(-)
diff --git a/t/lei-refresh-mail-sync.t b/t/lei-refresh-mail-sync.t
index 43fbc50a..ea83a513 100644
--- a/t/lei-refresh-mail-sync.t
+++ b/t/lei-refresh-mail-sync.t
@@ -88,12 +88,9 @@ SKIP: {
$sock_cls //= ref($s);
my $cmd = [ "-$x", '-W0', "--stdout=$home/$x.out",
"--stderr=$home/$x.err" ];
- my $td = start_script($cmd, $env, { 3 => $s}) or xbail("-$x");
- $srv->{$x} = {
- addr => (my $scalar = tcp_host_port($s)),
- td => $td,
- cmd => $cmd,
- };
+ my $td = start_script($cmd, $env, { 3 => $s }) or xbail("-$x");
+ my $addr = tcp_host_port($s);
+ $srv->{$x} = { addr => $addr, td => $td, cmd => $cmd, s => $s };
}
my $url = "imap://$srv->{imapd}->{addr}/t.v1.0";
lei_ok 'import', $url, '+L:v1';
@@ -123,20 +120,26 @@ SKIP: {
$before = $lei_out;
delete $srv->{imapd}->{td}; # kill + join daemon
+ my $pid = fork // xbail "fork";
+ if ($pid == 0) { # dummy server to kill new connections
+ $SIG{TERM} = sub { POSIX::_exit(0) };
+ $srv->{imapd}->{s}->blocking(1);
+ while (1) {
+ my $caddr = accept(my $c, $srv->{imapd}->{s}) // next;
+ shutdown($c, 2);
+ }
+ POSIX::_exit(0);
+ }
+ my $ar = PublicInbox::AutoReap->new($pid);
ok(!(lei 'refresh-mail-sync', $url), 'URL fails on dead -imapd');
ok(!(lei 'refresh-mail-sync', '--all'), '--all fails on dead -imapd');
+ $ar->kill for qw(avoid sig wake miss-no signalfd or EVFILT_SIG);
+ $ar->join('TERM');
- # restart server (somewhat dangerous since we released the socket)
- my $listen = $sock_cls->new(
- ReuseAddr => 1,
- Proto => 'tcp',
- Type => Socket::SOCK_STREAM(),
- Listen => 1024,
- Blocking => 0,
- LocalAddr => $srv->{imapd}->{addr},
- ) or xbail "$sock_cls->new: $!";
my $cmd = $srv->{imapd}->{cmd};
- $srv->{imapd}->{td} = start_script($cmd, $env, { 3 => $listen }) or
+ my $s = $srv->{imapd}->{s};
+ $s->blocking(0);
+ $srv->{imapd}->{td} = start_script($cmd, $env, { 3 => $s }) or
xbail "@$cmd";
lei_ok 'refresh-mail-sync', '--all';
lei_ok 'inspect', "blob:$oid";
^ permalink raw reply related [relevance 60%]
* [PATCH] doc: lei-config: fix missing =back
@ 2021-11-01 19:00 71% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-11-01 19:00 UTC (permalink / raw)
To: meta
---
Documentation/lei-config.pod | 2 ++
1 file changed, 2 insertions(+)
diff --git a/Documentation/lei-config.pod b/Documentation/lei-config.pod
index 154295ce..663404fe 100644
--- a/Documentation/lei-config.pod
+++ b/Documentation/lei-config.pod
@@ -95,6 +95,8 @@ will also be parsed for diff coloring. git diff color slots
(C<color.diff.SLOT>) supported are C<new>, C<old>, C<meta>,
C<frag>, C<func>, and C<context>.
+=back
+
=head1 CONTACT
Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org>
^ permalink raw reply related [relevance 71%]
* Re: [PATCH 1/2] doc: add lei-mail-sync-overview manpage
2021-10-31 16:32 71% ` Kyle Meyer
@ 2021-10-31 17:30 64% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-10-31 17:30 UTC (permalink / raw)
To: Kyle Meyer; +Cc: meta
Kyle Meyer <kyle@kyleam.com> wrote:
> Eric Wong writes:
> > +++ b/Documentation/lei-export-kw.pod
> > @@ -12,6 +12,9 @@ lei export-kw MFOLDER [MFOLDER...]
> >
> > C<lei export-kw> propagates keywords (e.g. C<seen>, C<answered>,
> > C<flagged>, etc.) from lei/store to IMAP folders and/or Maildirs.
> > +It only works for messages lei knows about (e.g. was used as a
> > +C<lei q --output>, or imported via L<lei-import(1)>, or indexed
> > +via L<lei-index(1)>.
>
> The closing paren is missing here.
Yup.
> > +++ b/Documentation/lei-index.pod
> > @@ -20,6 +20,9 @@ Combined with L<lei-q(1)>, C<lei index> allows Maildir users to
> > have similar functionality to L<mairix(1)> by not duplicating
> > messages into C<lei/store>.
> >
> > +Occasional invocations of C<lei-refresh-mail-sync --all=local>
> > +is recommended to keep indexed messages retrievable.
>
> "Occasional invocations ... is" -> "Occasional invocations ... are"?
>
> Or perhaps "Occasionally invoking ... is"
I think I'll go with "are"
> > +=head1 TYPICAL WORKFLOW
> > +
> > + # import mail from a user's IMAP inbox and give it the "inbox" label:
> > + lei import +L:inbox imaps://user@example.com/INBOX
> > +
> > + # dump "inbox" labeled files from the past week to a Maildir
> > + lei q L:inbox rt:last.week.. -o /tmp/results
> > +
> > + # open /tmp/results in your favorite mail agent. If inotify or kevent
> > + # works, keyword changes (e.g. marking messages as `seen').
>
> Is this part incomplete (maybe "are synchronized" is missing at the
> end)?
Yes, "are synchronized automatically". Will squash the
following in, thanks:
diff --git a/Documentation/lei-export-kw.pod b/Documentation/lei-export-kw.pod
index b6d175f1..4cb673ac 100644
--- a/Documentation/lei-export-kw.pod
+++ b/Documentation/lei-export-kw.pod
@@ -14,7 +14,7 @@ C<lei export-kw> propagates keywords (e.g. C<seen>, C<answered>,
C<flagged>, etc.) from lei/store to IMAP folders and/or Maildirs.
It only works for messages lei knows about (e.g. was used as a
C<lei q --output>, or imported via L<lei-import(1)>, or indexed
-via L<lei-index(1)>.
+via L<lei-index(1)>).
It does not delete, write, nor modify messages themselves;
it only sets metadata on Maildirs and IMAP folders.
diff --git a/Documentation/lei-index.pod b/Documentation/lei-index.pod
index de088538..f8ff6950 100644
--- a/Documentation/lei-index.pod
+++ b/Documentation/lei-index.pod
@@ -21,7 +21,7 @@ have similar functionality to L<mairix(1)> by not duplicating
messages into C<lei/store>.
Occasional invocations of C<lei-refresh-mail-sync --all=local>
-is recommended to keep indexed messages retrievable.
+are recommended to keep indexed messages retrievable.
=head1 OPTIONS
diff --git a/Documentation/lei-mail-sync-overview.pod b/Documentation/lei-mail-sync-overview.pod
index dfe1034c..e30674bb 100644
--- a/Documentation/lei-mail-sync-overview.pod
+++ b/Documentation/lei-mail-sync-overview.pod
@@ -21,7 +21,8 @@ Future work will be done to improve it and add IMAP IDLE support.
lei q L:inbox rt:last.week.. -o /tmp/results
# open /tmp/results in your favorite mail agent. If inotify or kevent
- # works, keyword changes (e.g. marking messages as `seen').
+ # works, keyword changes (e.g. marking messages as `seen') are
+ # synchronized automatically.
# If the inotify queue overflows, or if lei-daemon crashes,
# "lei index" will tell lei about keyword changes:
^ permalink raw reply related [relevance 64%]
* Re: [PATCH 1/2] doc: add lei-mail-sync-overview manpage
2021-10-31 9:10 29% ` [PATCH 1/2] doc: add lei-mail-sync-overview manpage Eric Wong
@ 2021-10-31 16:32 71% ` Kyle Meyer
2021-10-31 17:30 64% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Kyle Meyer @ 2021-10-31 16:32 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
Eric Wong writes:
> --- a/Documentation/lei-export-kw.pod
> +++ b/Documentation/lei-export-kw.pod
> @@ -12,6 +12,9 @@ lei export-kw MFOLDER [MFOLDER...]
>
> C<lei export-kw> propagates keywords (e.g. C<seen>, C<answered>,
> C<flagged>, etc.) from lei/store to IMAP folders and/or Maildirs.
> +It only works for messages lei knows about (e.g. was used as a
> +C<lei q --output>, or imported via L<lei-import(1)>, or indexed
> +via L<lei-index(1)>.
The closing paren is missing here.
> --- a/Documentation/lei-index.pod
> +++ b/Documentation/lei-index.pod
> @@ -20,6 +20,9 @@ Combined with L<lei-q(1)>, C<lei index> allows Maildir users to
> have similar functionality to L<mairix(1)> by not duplicating
> messages into C<lei/store>.
>
> +Occasional invocations of C<lei-refresh-mail-sync --all=local>
> +is recommended to keep indexed messages retrievable.
"Occasional invocations ... is" -> "Occasional invocations ... are"?
Or perhaps "Occasionally invoking ... is"
> +=head1 TYPICAL WORKFLOW
> +
> + # import mail from a user's IMAP inbox and give it the "inbox" label:
> + lei import +L:inbox imaps://user@example.com/INBOX
> +
> + # dump "inbox" labeled files from the past week to a Maildir
> + lei q L:inbox rt:last.week.. -o /tmp/results
> +
> + # open /tmp/results in your favorite mail agent. If inotify or kevent
> + # works, keyword changes (e.g. marking messages as `seen').
Is this part incomplete (maybe "are synchronized" is missing at the
end)?
^ permalink raw reply [relevance 71%]
* [PATCH 0/2] lei: mail-sync docs + compat fix
@ 2021-10-31 9:10 71% Eric Wong
2021-10-31 9:10 29% ` [PATCH 1/2] doc: add lei-mail-sync-overview manpage Eric Wong
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2021-10-31 9:10 UTC (permalink / raw)
To: meta
Eric Wong (2):
doc: add lei-mail-sync-overview manpage
lei_input: disallow uppercase characters for labels
Documentation/lei-export-kw.pod | 3 ++
Documentation/lei-forget-mail-sync.pod | 5 ++-
Documentation/lei-import.pod | 34 +++++++++++++---
Documentation/lei-index.pod | 5 ++-
Documentation/lei-ls-label.pod | 1 +
Documentation/lei-ls-mail-source.pod | 5 ++-
Documentation/lei-ls-watch.pod | 3 +-
Documentation/lei-mail-formats.pod | 5 ++-
Documentation/lei-mail-sync-overview.pod | 52 ++++++++++++++++++++++++
Documentation/lei-overview.pod | 13 +++++-
Documentation/lei-tag.pod | 21 +++++++++-
Documentation/lei.pod | 4 +-
MANIFEST | 1 +
Makefile.PL | 4 +-
lib/PublicInbox/LeiInput.pm | 2 +-
15 files changed, 137 insertions(+), 21 deletions(-)
create mode 100644 Documentation/lei-mail-sync-overview.pod
^ permalink raw reply [relevance 71%]
* [PATCH 1/2] doc: add lei-mail-sync-overview manpage
2021-10-31 9:10 71% [PATCH 0/2] lei: mail-sync docs + compat fix Eric Wong
@ 2021-10-31 9:10 29% ` Eric Wong
2021-10-31 16:32 71% ` Kyle Meyer
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2021-10-31 9:10 UTC (permalink / raw)
To: meta
Mostly illustrating how clunky the process is :p
We'll also tweak some things in existing man pages around
mail synchronization.
---
Documentation/lei-export-kw.pod | 3 ++
Documentation/lei-forget-mail-sync.pod | 5 ++-
Documentation/lei-import.pod | 34 +++++++++++++---
Documentation/lei-index.pod | 5 ++-
Documentation/lei-ls-label.pod | 1 +
Documentation/lei-ls-mail-source.pod | 5 ++-
Documentation/lei-ls-watch.pod | 3 +-
Documentation/lei-mail-formats.pod | 5 ++-
Documentation/lei-mail-sync-overview.pod | 52 ++++++++++++++++++++++++
Documentation/lei-overview.pod | 13 +++++-
Documentation/lei-tag.pod | 21 +++++++++-
Documentation/lei.pod | 4 +-
MANIFEST | 1 +
Makefile.PL | 4 +-
14 files changed, 136 insertions(+), 20 deletions(-)
create mode 100644 Documentation/lei-mail-sync-overview.pod
diff --git a/Documentation/lei-export-kw.pod b/Documentation/lei-export-kw.pod
index cf482ca0d307..b6d175f14773 100644
--- a/Documentation/lei-export-kw.pod
+++ b/Documentation/lei-export-kw.pod
@@ -12,6 +12,9 @@ lei export-kw MFOLDER [MFOLDER...]
C<lei export-kw> propagates keywords (e.g. C<seen>, C<answered>,
C<flagged>, etc.) from lei/store to IMAP folders and/or Maildirs.
+It only works for messages lei knows about (e.g. was used as a
+C<lei q --output>, or imported via L<lei-import(1)>, or indexed
+via L<lei-index(1)>.
It does not delete, write, nor modify messages themselves;
it only sets metadata on Maildirs and IMAP folders.
diff --git a/Documentation/lei-forget-mail-sync.pod b/Documentation/lei-forget-mail-sync.pod
index e70b4d33a33a..29ff56711815 100644
--- a/Documentation/lei-forget-mail-sync.pod
+++ b/Documentation/lei-forget-mail-sync.pod
@@ -9,8 +9,9 @@ lei forget-mail-sync [OPTIONS] LOCATION [LOCATION...]
=head1 DESCRIPTION
Forget synchronization information for C<LOCATION>, an IMAP or Maildir
-folder. Note that this won't delete any messages stored in Git,
-leaving C<lei-index(1)> users with dangling references.
+folder. Note that this won't delete any messages on the filesystem.
+Users using L<lei-index(1)> without L<lei-import(1)> will be left
+with dangling references in search results.
=head1 CONTACT
diff --git a/Documentation/lei-import.pod b/Documentation/lei-import.pod
index 9322dfa8d34c..25f1daf5f32f 100644
--- a/Documentation/lei-import.pod
+++ b/Documentation/lei-import.pod
@@ -4,7 +4,7 @@ lei-import - one-time import of messages into local store
=head1 SYNOPSIS
-lei import [OPTIONS] LOCATION [LOCATION...]
+lei import [OPTIONS] LOCATION [LOCATION...] [+L:LABEL]
lei import [OPTIONS] (--stdin|-)
@@ -16,8 +16,8 @@ source of messages: a directory (Maildir), a file, or a URL
authentication use L<git-credential(1)> to
fill in the username and password.
-For a regular file, the location must have a C<E<lt>formatE<gt>:>
-prefix specifying one of the following formats: C<eml>, C<mboxrd>,
+For a regular file, the C<LOCATION> must have a C<E<lt>formatE<gt>:>
+prefix specifying one of the following formats: C<mboxrd>,
C<mboxcl2>, C<mboxcl>, or C<mboxo>.
=head1 OPTIONS
@@ -42,6 +42,29 @@ C<none>.
Default: fcntl,dotlock
+=item +L:LABEL
+
+Add the given C<LABEL> to all messages imported, where C<LABEL>
+is an arbitrary user-defined value consisting of lowercase and digits.
+See L<lei-tag(1)> for more info on labels.
+
+For example, specifying C<+L:inbox> applies the C<inbox> label
+to all messages being imported.
+
+May be specified multiple times to apply multiple labels.
+
+Default: none
+
+=item +kw:KEYWORD
+
+Apply C<KEYWORD> to all messages being imported in addition
+to any per-message keywords from the store (unless C<--no-kw>
+is specified). See L<lei-tag(1)> for more info on keywords.
+
+May be specified multiple times to apply multiple keywords.
+
+Default: none
+
=item --no-kw
Don't import message keywords (or "flags" in IMAP terminology).
@@ -73,11 +96,10 @@ L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/>
=head1 COPYRIGHT
-Copyright 2021 all contributors L<mailto:meta@public-inbox.org>
+Copyright all contributors L<mailto:meta@public-inbox.org>
License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>
-
=head1 SEE ALSO
-L<lei-add-external(1)>
+L<lei-index(1)>
diff --git a/Documentation/lei-index.pod b/Documentation/lei-index.pod
index 9e72026351a1..de088538e5ca 100644
--- a/Documentation/lei-index.pod
+++ b/Documentation/lei-index.pod
@@ -20,6 +20,9 @@ Combined with L<lei-q(1)>, C<lei index> allows Maildir users to
have similar functionality to L<mairix(1)> by not duplicating
messages into C<lei/store>.
+Occasional invocations of C<lei-refresh-mail-sync --all=local>
+is recommended to keep indexed messages retrievable.
+
=head1 OPTIONS
=over
@@ -58,4 +61,4 @@ License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>
=head1 SEE ALSO
-L<lei-store-format(5)>, L<lei-import(1)>
+L<lei-refresh-mail-sync(1)>, L<lei-store-format(5)>, L<lei-import(1)>
diff --git a/Documentation/lei-ls-label.pod b/Documentation/lei-ls-label.pod
index 41aa030d9932..8342705a6cf6 100644
--- a/Documentation/lei-ls-label.pod
+++ b/Documentation/lei-ls-label.pod
@@ -9,6 +9,7 @@ lei ls-label [OPTIONS]
=head1 DESCRIPTION
List all known message labels ("mailboxes" in JMAP terminology).
+This is handy for writing L<lei-import(1)> invocations.
=head1 OPTIONS
diff --git a/Documentation/lei-ls-mail-source.pod b/Documentation/lei-ls-mail-source.pod
index 926bbe2c02f6..59d14afe6880 100644
--- a/Documentation/lei-ls-mail-source.pod
+++ b/Documentation/lei-ls-mail-source.pod
@@ -9,6 +9,8 @@ lei ls-mail-source [OPTIONS] URL
=head1 DESCRIPTION
List information about the IMAP or NNTP mail source at C<URL>.
+This command populates the cache used for Bash shell completion
+and is handy for writing L<lei-import(1)> invocations.
=head1 OPTIONS
@@ -48,11 +50,10 @@ L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/>
=head1 COPYRIGHT
-Copyright 2021 all contributors L<mailto:meta@public-inbox.org>
+Copyright all contributors L<mailto:meta@public-inbox.org>
License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>
-
=head1 SEE ALSO
L<lei-import(1)>
diff --git a/Documentation/lei-ls-watch.pod b/Documentation/lei-ls-watch.pod
index b1681ee45782..8063d34d3273 100644
--- a/Documentation/lei-ls-watch.pod
+++ b/Documentation/lei-ls-watch.pod
@@ -8,7 +8,8 @@ lei ls-watch
=head1 DESCRIPTION
-List locations that lei is configured to watch.
+List locations that lei is configured to watch. This command is
+incomplete, mail-sync locations are implicitly watched.
=head1 CONTACT
diff --git a/Documentation/lei-mail-formats.pod b/Documentation/lei-mail-formats.pod
index 3c37c88046b6..930c5d7660ea 100644
--- a/Documentation/lei-mail-formats.pod
+++ b/Documentation/lei-mail-formats.pod
@@ -96,7 +96,10 @@ it worth supporting.
Depending on the IMAP server software and configuration, IMAP
servers may use any (or combination) of the aforementioned
-formats or a non-standard database backend.
+formats or a non-standard database backend. Currently, lei
+uses L<Mail::IMAPClient> which has acceptable performance
+over low-latency links. Performance over high-latency links
+is currently poor.
=head1 eml
diff --git a/Documentation/lei-mail-sync-overview.pod b/Documentation/lei-mail-sync-overview.pod
new file mode 100644
index 000000000000..dfe1034cea7e
--- /dev/null
+++ b/Documentation/lei-mail-sync-overview.pod
@@ -0,0 +1,52 @@
+=head1 NAME
+
+lei - an overview of lei mail synchronization
+
+=head1 DESCRIPTION
+
+L<lei(1)> provides several plumbing-level commands to synchronize
+mail and keywords (flags) between lei/store and existing IMAP
+and Maildir stores. Nothing documented in this manpage is required
+for day-to-day use against externals.
+
+Mail and keyword synchronization is currently a clunky process.
+Future work will be done to improve it and add IMAP IDLE support.
+
+=head1 TYPICAL WORKFLOW
+
+ # import mail from a user's IMAP inbox and give it the "inbox" label:
+ lei import +L:inbox imaps://user@example.com/INBOX
+
+ # dump "inbox" labeled files from the past week to a Maildir
+ lei q L:inbox rt:last.week.. -o /tmp/results
+
+ # open /tmp/results in your favorite mail agent. If inotify or kevent
+ # works, keyword changes (e.g. marking messages as `seen').
+
+ # If the inotify queue overflows, or if lei-daemon crashes,
+ # "lei index" will tell lei about keyword changes:
+ lei index /tmp/results
+
+ # Optional: cleanup stale entries from mail_sync.sqlite3
+ lei refresh-mail-sync /tmp/results
+
+ # to export keyword changes back to IMAP
+ lei export-kw imaps://user@example.com/INBOX
+
+=head1 CONTACT
+
+Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org>
+
+The mail archives are hosted at L<https://public-inbox.org/meta/> and
+L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/>
+
+=head1 COPYRIGHT
+
+Copyright all contributors L<mailto:meta@public-inbox.org>
+
+License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>
+
+=head1 SEE ALSO
+
+L<lei-import(1)>, L<lei-q(1)>, L<lei-index(1)>,
+L<lei-refresh-mail-sync(1)>, L<lei-export-kw(1)>
diff --git a/Documentation/lei-overview.pod b/Documentation/lei-overview.pod
index 99fd6ef72174..7095b504cdb8 100644
--- a/Documentation/lei-overview.pod
+++ b/Documentation/lei-overview.pod
@@ -4,8 +4,8 @@ lei - an overview of lei
=head1 DESCRIPTION
-L<lei(1)> is a local email interface for public-inbox. This document
-provides some basic examples.
+L<lei(1)> is a local email interface for public-inbox and personal mail.
+This document provides some basic examples.
=head1 LEI STORE
@@ -49,6 +49,11 @@ For existing local paths, the external needs to be indexed with
L<public-inbox-index(1)> (in the case of a regular inbox) or
L<public-inbox-extindex(1)> (in the case of an external index).
+=head1 SYNCHRONIZATION
+
+lei currently has primitive mail synchronization abilities;
+see L<lei-mail-sync-overview(7)> for more details.
+
=head2 EXAMPLES
=over
@@ -154,3 +159,7 @@ L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/>
Copyright all contributors L<mailto:meta@public-inbox.org>
License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>
+
+=head1 SEE ALSO
+
+L<lei-mail-sync-overview(7)>
diff --git a/Documentation/lei-tag.pod b/Documentation/lei-tag.pod
index b2509ba4ee26..8cb9e736b6e7 100644
--- a/Documentation/lei-tag.pod
+++ b/Documentation/lei-tag.pod
@@ -43,6 +43,26 @@ Suppress feedback messages.
=back
+=head1 LABELS
+
+Labels are user-defined values analogous to IMAP/JMAP mailbox
+names. They must only contain lowercase characters, digits, and
+a limited amount of punctuation (e.g. C<.>, C<->, C<@>).
+
+Messages may have multiple labels.
+
+=head1 KEYWORDS
+
+Keywords are "flags" in Maildir and IMAP terminology.
+Common keywords include: C<seen>, C<answered>, C<flagged>, and
+C<draft>, though C<forwarded>, C<phishing>, C<junk>, and C<notjunk>
+are also supported.
+
+When writing to various mboxes, the common keywords will be
+mapped to the C<Status> and C<X-Status> headers.
+
+Messages may have multiple keywords.
+
=head1 CONTACT
Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org>
@@ -56,7 +76,6 @@ Copyright 2021 all contributors L<mailto:meta@public-inbox.org>
License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>
-
=head1 SEE ALSO
L<lei-add-external(1)>
diff --git a/Documentation/lei.pod b/Documentation/lei.pod
index 24a585daf24c..f01f506af359 100644
--- a/Documentation/lei.pod
+++ b/Documentation/lei.pod
@@ -106,7 +106,7 @@ Other subcommands include
=item * L<lei-daemon-pid(1)>
-=item * lei-forget-mail-sync(1)
+=item * L<lei-forget-mail-sync(1)>
=item * L<lei-mail-diff(1)>
@@ -145,7 +145,7 @@ L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/>
=head1 COPYRIGHT
-Copyright 2021 all contributors L<mailto:meta@public-inbox.org>
+Copyright all contributors L<mailto:meta@public-inbox.org>
License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>
diff --git a/MANIFEST b/MANIFEST
index 9fd979ef02fb..1e8f60fb41da 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -47,6 +47,7 @@ Documentation/lei-ls-search.pod
Documentation/lei-ls-watch.pod
Documentation/lei-mail-diff.pod
Documentation/lei-mail-formats.pod
+Documentation/lei-mail-sync-overview.pod
Documentation/lei-overview.pod
Documentation/lei-p2q.pod
Documentation/lei-q.pod
diff --git a/Makefile.PL b/Makefile.PL
index b3ac59be32ee..8c8c0235f35b 100644
--- a/Makefile.PL
+++ b/Makefile.PL
@@ -1,5 +1,5 @@
#!/usr/bin/perl -w
-# Copyright (C) 2013-2021 all contributors <meta@public-inbox.org>
+# Copyright (C) all contributors <meta@public-inbox.org>
# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
use strict;
use ExtUtils::MakeMaker;
@@ -57,7 +57,7 @@ $v->{-m5} = [ qw(public-inbox-config public-inbox-v1-format
public-inbox-v2-format public-inbox-extindex-format
lei-mail-formats lei-store-format
) ];
-$v->{-m7} = [ qw(lei-overview lei-security
+$v->{-m7} = [ qw(lei-mail-sync-overview lei-overview lei-security
public-inbox-overview public-inbox-tuning
public-inbox-glossary) ];
$v->{-m8} = [ qw(public-inbox-daemon lei-daemon) ];
^ permalink raw reply related [relevance 29%]
* [PATCH 1/5] lei: do not access {sock} after SIGPIPE
2021-10-30 8:11 71% [PATCH 0/5] lei: fix various SIGPIPE problems Eric Wong
@ 2021-10-30 8:11 71% ` Eric Wong
2021-10-30 8:11 71% ` [PATCH 5/5] doc: lei-security: add a note about core dumps Eric Wong
1 sibling, 0 replies; 200+ results
From: Eric Wong @ 2021-10-30 8:11 UTC (permalink / raw)
To: meta
It's possible for this to break out of the event loop if
note_sigpipe fires via PktOp in the same iteration.
---
lib/PublicInbox/LEI.pm | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 96f7c5e315a9..78b49a3bc1af 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -1127,7 +1127,7 @@ sub event_step {
local %ENV = %{$self->{env}};
local $current_lei = $self;
eval {
- my @fds = $recv_cmd->($self->{sock}, my $buf, 4096);
+ my @fds = $recv_cmd->($self->{sock} // return, my $buf, 4096);
if (scalar(@fds) == 1 && !defined($fds[0])) {
return if $! == EAGAIN;
die "recvmsg: $!" if $! != ECONNRESET;
^ permalink raw reply related [relevance 71%]
* [PATCH 0/5] lei: fix various SIGPIPE problems
@ 2021-10-30 8:11 71% Eric Wong
2021-10-30 8:11 71% ` [PATCH 1/5] lei: do not access {sock} after SIGPIPE Eric Wong
2021-10-30 8:11 71% ` [PATCH 5/5] doc: lei-security: add a note about core dumps Eric Wong
0 siblings, 2 replies; 200+ results
From: Eric Wong @ 2021-10-30 8:11 UTC (permalink / raw)
To: meta
Most worrying was the the bug fixed in 4/5; but at least there
wasn't data loss involved...
While the bug fixed in 4/5 didn't cause data loss, it was
dumping core files and filling up my disk while polluting the
kernel log buffer.
Eric Wong (5):
lei: do not access {sock} after SIGPIPE
lei_to_mail: limit workers for text, reply and v2 outputs
lei_xsearch: quiet error message on SIG{PIPE,TERM}
lei_to_mail: avoid SEGV on worker exit via SIGTERM
doc: lei-security: add a note about core dumps
Documentation/lei-security.pod | 6 ++++++
lib/PublicInbox/LEI.pm | 2 +-
lib/PublicInbox/LeiQuery.pm | 2 +-
lib/PublicInbox/LeiToMail.pm | 10 +++++++++-
lib/PublicInbox/LeiXSearch.pm | 5 ++++-
t/lei-sigpipe.t | 7 +++++--
6 files changed, 26 insertions(+), 6 deletions(-)
^ permalink raw reply [relevance 71%]
* [PATCH 5/5] doc: lei-security: add a note about core dumps
2021-10-30 8:11 71% [PATCH 0/5] lei: fix various SIGPIPE problems Eric Wong
2021-10-30 8:11 71% ` [PATCH 1/5] lei: do not access {sock} after SIGPIPE Eric Wong
@ 2021-10-30 8:11 71% ` Eric Wong
1 sibling, 0 replies; 200+ results
From: Eric Wong @ 2021-10-30 8:11 UTC (permalink / raw)
To: meta
Maybe we can avoid them if we stop having buggy code :P
---
Documentation/lei-security.pod | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/Documentation/lei-security.pod b/Documentation/lei-security.pod
index 8cbd89934568..104bfb48a26c 100644
--- a/Documentation/lei-security.pod
+++ b/Documentation/lei-security.pod
@@ -64,6 +64,12 @@ public-facing L<public-inbox-daemon(8)> processes. They may
reside on shared storage and may be made world-readable to
other users on the local system.
+=head1 CORE DUMPS
+
+In case any process crashes, a core dumps may contain passwords or
+contents of sensitive messages. Please report these so they can be
+fixed (see L</CONTACT>).
+
=head1 NETWORK ACCESS
lei currently uses the L<curl(1)> and L<git(1)> executables in
^ permalink raw reply related [relevance 71%]
* Re: [PATCH] test_common: clear XDG_CACHE_HOME before lei tests
2021-10-28 19:16 68% ` [PATCH] test_common: clear XDG_CACHE_HOME before lei tests Eric Wong
@ 2021-10-28 19:22 71% ` Thomas Weißschuh
0 siblings, 0 replies; 200+ results
From: Thomas Weißschuh @ 2021-10-28 19:22 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
On 2021-10-28 19:16+0000, Eric Wong wrote:
> Thomas Weißschuh <thomas@t-8ch.de> wrote:
> > It turned out that in my $XDG_CONFIG_HOME/lei/all_locals_ever.git/lei_ale.state
> > there were entries for the repositories in
> > $PUBLIC_INBOX_SRC/t/home2/{t1,t2}.
> >
> > I'm not sure how those entries came to be but probably because of some
> > debugging things I did before.
>
> The presence of t1 & t2 entries is harmless for normal lei
> operation; but they shouldn't be there...
>
> I wonder, do you have XDG_CACHE_HOME explicitly set in your env?
> If so, that would've caused problems (fixed below).
Yes I have it set explicitly
> > IMO it would make sense to prevent p-i to read this global state during
> > unittests.
>
> Yes, I think this was from our failure to clobber XDG_CACHE_HOME:
That works, thanks!
Tested-by: Thomas Weißschuh <thomas@t-8ch.de>
> ------------8<------------
> Subject: [PATCH] test_common: clear XDG_CACHE_HOME before lei tests
>
> We don't want to read a users'
> $XDG_CACHE_HOME/lei/all_locals_ever.git during tests.
>
> Reported-by: Thomas Weißschuh <thomas@t-8ch.de>
> Link: https://public-inbox.org/meta/f239abac-4aee-4573-a0d6-e533c7a32662@t-8ch.de/
> ---
> lib/PublicInbox/TestCommon.pm | 6 +++++-
> 1 file changed, 5 insertions(+), 1 deletion(-)
>
> diff --git a/lib/PublicInbox/TestCommon.pm b/lib/PublicInbox/TestCommon.pm
> index c3820d3a..052d6e45 100644
> --- a/lib/PublicInbox/TestCommon.pm
> +++ b/lib/PublicInbox/TestCommon.pm
> @@ -556,16 +556,20 @@ SKIP: {
> require_git(2.6, 1) or skip('git 2.6+ required for lei test', 2);
> my $mods = $test_opt->{mods} // [ 'lei' ];
> require_mods(@$mods, 2);
> +
> + # set PERL_INLINE_DIRECTORY before clobbering XDG_CACHE_HOME
> + require PublicInbox::Spawn;
> require PublicInbox::Config;
> require File::Path;
> +
> local %ENV = %ENV;
> delete $ENV{XDG_DATA_HOME};
> delete $ENV{XDG_CONFIG_HOME};
> + delete $ENV{XDG_CACHE_HOME};
> $ENV{GIT_COMMITTER_EMAIL} = 'lei@example.com';
> $ENV{GIT_COMMITTER_NAME} = 'lei user';
> my (undef, $fn, $lineno) = caller(0);
> my $t = "$fn:$lineno";
> - require PublicInbox::Spawn;
> state $lei_daemon = PublicInbox::Spawn->can('send_cmd4') ||
> eval { require Socket::MsgHdr; 1 };
> unless ($lei_daemon) {
^ permalink raw reply [relevance 71%]
* [PATCH] test_common: clear XDG_CACHE_HOME before lei tests
2021-10-28 14:03 71% ` Thomas Weißschuh
@ 2021-10-28 19:16 68% ` Eric Wong
2021-10-28 19:22 71% ` Thomas Weißschuh
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2021-10-28 19:16 UTC (permalink / raw)
To: Thomas Weißschuh; +Cc: meta
Thomas Weißschuh <thomas@t-8ch.de> wrote:
> It turned out that in my $XDG_CONFIG_HOME/lei/all_locals_ever.git/lei_ale.state
> there were entries for the repositories in
> $PUBLIC_INBOX_SRC/t/home2/{t1,t2}.
>
> I'm not sure how those entries came to be but probably because of some
> debugging things I did before.
The presence of t1 & t2 entries is harmless for normal lei
operation; but they shouldn't be there...
I wonder, do you have XDG_CACHE_HOME explicitly set in your env?
If so, that would've caused problems (fixed below).
> IMO it would make sense to prevent p-i to read this global state during
> unittests.
Yes, I think this was from our failure to clobber XDG_CACHE_HOME:
------------8<------------
Subject: [PATCH] test_common: clear XDG_CACHE_HOME before lei tests
We don't want to read a users'
$XDG_CACHE_HOME/lei/all_locals_ever.git during tests.
Reported-by: Thomas Weißschuh <thomas@t-8ch.de>
Link: https://public-inbox.org/meta/f239abac-4aee-4573-a0d6-e533c7a32662@t-8ch.de/
---
lib/PublicInbox/TestCommon.pm | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/lib/PublicInbox/TestCommon.pm b/lib/PublicInbox/TestCommon.pm
index c3820d3a..052d6e45 100644
--- a/lib/PublicInbox/TestCommon.pm
+++ b/lib/PublicInbox/TestCommon.pm
@@ -556,16 +556,20 @@ SKIP: {
require_git(2.6, 1) or skip('git 2.6+ required for lei test', 2);
my $mods = $test_opt->{mods} // [ 'lei' ];
require_mods(@$mods, 2);
+
+ # set PERL_INLINE_DIRECTORY before clobbering XDG_CACHE_HOME
+ require PublicInbox::Spawn;
require PublicInbox::Config;
require File::Path;
+
local %ENV = %ENV;
delete $ENV{XDG_DATA_HOME};
delete $ENV{XDG_CONFIG_HOME};
+ delete $ENV{XDG_CACHE_HOME};
$ENV{GIT_COMMITTER_EMAIL} = 'lei@example.com';
$ENV{GIT_COMMITTER_NAME} = 'lei user';
my (undef, $fn, $lineno) = caller(0);
my $t = "$fn:$lineno";
- require PublicInbox::Spawn;
state $lei_daemon = PublicInbox::Spawn->can('send_cmd4') ||
eval { require Socket::MsgHdr; 1 };
unless ($lei_daemon) {
^ permalink raw reply related [relevance 68%]
* Re: lei-q-remote-import failures [was: [PATCH] t/index-git-times: support non-master default branch]
2021-10-27 23:48 63% ` Eric Wong
@ 2021-10-28 14:03 71% ` Thomas Weißschuh
2021-10-28 19:16 68% ` [PATCH] test_common: clear XDG_CACHE_HOME before lei tests Eric Wong
0 siblings, 1 reply; 200+ results
From: Thomas Weißschuh @ 2021-10-28 14:03 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
On 2021-10-27 23:48+0000, Eric Wong wrote:
> Thomas Weißschuh <thomas@t-8ch.de> wrote:
> > It didn't help, neither did larger sleeps.
>
> Thanks for testing.
>
> > I'll see if I can debug it in the coming days.
>
> OK, maybe the patch below can get you started...
>
> I've also been wondering if something like GIT_TRACE is
> necessary, but it's also more work to support + document.
> And Perl code is easily modified compared to AOT languages.
It turned out that in my $XDG_CONFIG_HOME/lei/all_locals_ever.git/lei_ale.state
there were entries for the repositories in
$PUBLIC_INBOX_SRC/t/home2/{t1,t2}.
I'm not sure how those entries came to be but probably because of some
debugging things I did before.
IMO it would make sense to prevent p-i to read this global state during
unittests.
^ permalink raw reply [relevance 71%]
* [PATCH 2/8] lei convert: use "--output" in failure message
2021-10-28 11:14 71% [PATCH 0/8] lei: docs and cleanups Eric Wong
2021-10-28 11:14 68% ` [PATCH 1/8] xt/net_writer_imap: test "lei convert" w/ IMAP source Eric Wong
@ 2021-10-28 11:14 71% ` Eric Wong
2021-10-28 11:14 62% ` [PATCH 3/8] doc: lei-convert: various updates and cleanups Eric Wong
` (5 subsequent siblings)
7 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-10-28 11:14 UTC (permalink / raw)
To: meta
The extra dashes should help users find the correct option
more easily.
---
lib/PublicInbox/LeiConvert.pm | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/lib/PublicInbox/LeiConvert.pm b/lib/PublicInbox/LeiConvert.pm
index 424eab8e22e2..623599aef883 100644
--- a/lib/PublicInbox/LeiConvert.pm
+++ b/lib/PublicInbox/LeiConvert.pm
@@ -51,7 +51,7 @@ sub lei_convert { # the main "lei convert" method
my $self = bless {}, __PACKAGE__;
my $ovv = PublicInbox::LeiOverview->new($lei, 'out-format');
$lei->{l2m} or return
- $lei->fail("output not specified or is not a mail destination");
+ $lei->fail('--output unspecified or is not a mail destination');
my $devfd = $lei->path_to_fd($ovv->{dst}) // return;
$lei->{opt}->{augment} = 1 if $devfd < 0;
$self->prepare_inputs($lei, \@inputs) or return;
^ permalink raw reply related [relevance 71%]
* [PATCH 5/8] lei convert: remove redundant input_net_cb
2021-10-28 11:14 71% [PATCH 0/8] lei: docs and cleanups Eric Wong
` (3 preceding siblings ...)
2021-10-28 11:14 70% ` [PATCH 4/8] doc: lei blob: wording fixups, describe --remote Eric Wong
@ 2021-10-28 11:14 71% ` Eric Wong
2021-10-28 11:14 70% ` [PATCH 6/8] doc: lei-add-watch: add warning about unreliability Eric Wong
` (2 subsequent siblings)
7 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-10-28 11:14 UTC (permalink / raw)
To: meta
Use the one provided by the LeiInput parent class.
---
lib/PublicInbox/LeiConvert.pm | 5 -----
1 file changed, 5 deletions(-)
diff --git a/lib/PublicInbox/LeiConvert.pm b/lib/PublicInbox/LeiConvert.pm
index 623599aef883..906f30268344 100644
--- a/lib/PublicInbox/LeiConvert.pm
+++ b/lib/PublicInbox/LeiConvert.pm
@@ -23,11 +23,6 @@ sub input_eml_cb { # used by PublicInbox::LeiInput::input_fh
$self->{wcb}->(undef, {}, $eml);
}
-sub input_net_cb { # callback for ->imap_each, ->nntp_each
- my (undef, undef, $kw, $eml, $self) = @_; # @_[0,1]: url + uid ignored
- $self->{wcb}->(undef, { kw => $kw }, $eml);
-}
-
sub input_maildir_cb {
my (undef, $kw, $eml, $self) = @_; # $_[0] $filename ignored
$self->{wcb}->(undef, { kw => $kw }, $eml);
^ permalink raw reply related [relevance 71%]
* [PATCH 7/8] lei sucks: show nproc in CPU info
2021-10-28 11:14 71% [PATCH 0/8] lei: docs and cleanups Eric Wong
` (5 preceding siblings ...)
2021-10-28 11:14 70% ` [PATCH 6/8] doc: lei-add-watch: add warning about unreliability Eric Wong
@ 2021-10-28 11:15 71% ` Eric Wong
2021-10-28 11:15 63% ` [PATCH 8/8] lei rm: move generic input_maildir_cb to LeiInput parent class Eric Wong
7 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-10-28 11:15 UTC (permalink / raw)
To: meta
Some bugs are triggered with more CPUs, some with 1 CPU.
---
lib/PublicInbox/LeiSucks.pm | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/lib/PublicInbox/LeiSucks.pm b/lib/PublicInbox/LeiSucks.pm
index e832f95e0736..8e866fc96655 100644
--- a/lib/PublicInbox/LeiSucks.pm
+++ b/lib/PublicInbox/LeiSucks.pm
@@ -11,6 +11,7 @@ use Digest::SHA ();
use Config;
use POSIX ();
use PublicInbox::Config;
+use PublicInbox::IPC;
sub lei_sucks {
my ($lei, @argv) = @_;
@@ -22,9 +23,10 @@ sub lei_sucks {
}
eval { require PublicInbox };
my $pi_ver = eval('$PublicInbox::VERSION') // '(???)';
+ my $nproc = PublicInbox::IPC::detect_nproc() // '?';
my @out = ("lei $pi_ver\n",
"perl $Config{version} / $os $rel / $mac ".
- "ptrsize=$Config{ptrsize}\n");
+ "ptrsize=$Config{ptrsize} nproc=$nproc\n");
chomp(my $gv = `git --version` || "git missing");
$gv =~ s/ version / /;
my $json = ref(PublicInbox::Config->json);
^ permalink raw reply related [relevance 71%]
* [PATCH 6/8] doc: lei-add-watch: add warning about unreliability
2021-10-28 11:14 71% [PATCH 0/8] lei: docs and cleanups Eric Wong
` (4 preceding siblings ...)
2021-10-28 11:14 71% ` [PATCH 5/8] lei convert: remove redundant input_net_cb Eric Wong
@ 2021-10-28 11:14 70% ` Eric Wong
2021-10-28 11:15 71% ` [PATCH 7/8] lei sucks: show nproc in CPU info Eric Wong
2021-10-28 11:15 63% ` [PATCH 8/8] lei rm: move generic input_maildir_cb to LeiInput parent class Eric Wong
7 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-10-28 11:14 UTC (permalink / raw)
To: meta
This needs work at some point in the future.
---
Documentation/lei-add-watch.pod | 8 ++++++--
1 file changed, 6 insertions(+), 2 deletions(-)
diff --git a/Documentation/lei-add-watch.pod b/Documentation/lei-add-watch.pod
index 609846181578..1b48b43b589c 100644
--- a/Documentation/lei-add-watch.pod
+++ b/Documentation/lei-add-watch.pod
@@ -11,6 +11,11 @@ lei add-watch [OPTIONS] LOCATION [LOCATION...]
Tell lei to watch C<LOCATION> for new messages and flag changes.
Currently only Maildir locations are supported.
+WARNING: watches are not always reliable, occasional use
+of L<lei-index(1)> and L<lei-refresh-mail-sync(1)> is recommended
+if L<lei-daemon(8)> crashes or needs to be restarted. This will
+be improved in the future.
+
=for comment
TODO: Document --state? Believe valid values are pause, import-ro,
@@ -23,11 +28,10 @@ L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/>
=head1 COPYRIGHT
-Copyright 2021 all contributors L<mailto:meta@public-inbox.org>
+Copyright all contributors L<mailto:meta@public-inbox.org>
License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>
-
=head1 SEE ALSO
L<lei-ls-watch(1)>, L<lei-rm-watch(1)>
^ permalink raw reply related [relevance 70%]
* [PATCH 4/8] doc: lei blob: wording fixups, describe --remote
2021-10-28 11:14 71% [PATCH 0/8] lei: docs and cleanups Eric Wong
` (2 preceding siblings ...)
2021-10-28 11:14 62% ` [PATCH 3/8] doc: lei-convert: various updates and cleanups Eric Wong
@ 2021-10-28 11:14 70% ` Eric Wong
2021-10-28 11:14 71% ` [PATCH 5/8] lei convert: remove redundant input_net_cb Eric Wong
` (3 subsequent siblings)
7 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-10-28 11:14 UTC (permalink / raw)
To: meta
There's no current way to retrieve blobs by OID directly
from remote externals. Maybe the $INBOX_NAME/$OID/s/raw.eml
endpoint could be overloaded for that.
---
Documentation/lei-blob.pod | 12 ++++++++----
1 file changed, 8 insertions(+), 4 deletions(-)
diff --git a/Documentation/lei-blob.pod b/Documentation/lei-blob.pod
index 429d206e358e..e401bb4711a6 100644
--- a/Documentation/lei-blob.pod
+++ b/Documentation/lei-blob.pod
@@ -9,9 +9,10 @@ lei blob [OPTIONS] OID
=head1 DESCRIPTION
Display a git blob. The blob may correspond to a message from the
-local store, an existing blob in the current repository, or a
-not-yet-created blob in the current git project repository (if any)
-that can be reconstructed from a message.
+local store, any local external, or blobs associated with a
+project git repository (if run from a git (working) directory).
+For blobs which do not exist, it will attempt to recreate the blob
+using patch emails.
=head1 OPTIONS
@@ -64,7 +65,10 @@ L<lei-q(1)>.
=over
-=item --[no-]remote
+=item --remote
+
+Remote externals only get queried when the blob needs to be
+reconstructed from patch emails.
=item --no-local
^ permalink raw reply related [relevance 70%]
* [PATCH 8/8] lei rm: move generic input_maildir_cb to LeiInput parent class
2021-10-28 11:14 71% [PATCH 0/8] lei: docs and cleanups Eric Wong
` (6 preceding siblings ...)
2021-10-28 11:15 71% ` [PATCH 7/8] lei sucks: show nproc in CPU info Eric Wong
@ 2021-10-28 11:15 63% ` Eric Wong
7 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-10-28 11:15 UTC (permalink / raw)
To: meta
It's not much of a savings, right now, but maybe it can be in the
future. I wanted to eliminate the "lei convert" one, too, but
convert needs to preserve keywords which isn't possible with the
generic fallback, so new tests were written for convert, instead.
---
lib/PublicInbox/LeiInput.pm | 5 +++++
lib/PublicInbox/LeiRm.pm | 5 -----
t/lei-convert.t | 10 ++++++++++
3 files changed, 15 insertions(+), 5 deletions(-)
diff --git a/lib/PublicInbox/LeiInput.pm b/lib/PublicInbox/LeiInput.pm
index 540681e3ff6b..84fc579dc480 100644
--- a/lib/PublicInbox/LeiInput.pm
+++ b/lib/PublicInbox/LeiInput.pm
@@ -64,6 +64,11 @@ sub input_mbox_cb { # base MboxReader callback
$self->input_eml_cb($eml);
}
+sub input_maildir_cb {
+ my ($fn, $kw, $eml, $self) = @_;
+ $self->input_eml_cb($eml);
+}
+
sub input_net_cb { # imap_each, nntp_each cb
my ($url, $uid, $kw, $eml, $self) = @_;
$self->input_eml_cb($eml);
diff --git a/lib/PublicInbox/LeiRm.pm b/lib/PublicInbox/LeiRm.pm
index 524c178e3b47..cc1abbff66cb 100644
--- a/lib/PublicInbox/LeiRm.pm
+++ b/lib/PublicInbox/LeiRm.pm
@@ -13,11 +13,6 @@ sub input_eml_cb { # used by PublicInbox::LeiInput::input_fh
$self->{lei}->{sto}->wq_do('remove_eml', $eml);
}
-sub input_maildir_cb {
- my (undef, $kw, $eml, $self) = @_; # $_[0] $filename ignored
- input_eml_cb($self, $eml);
-}
-
sub lei_rm {
my ($lei, @inputs) = @_;
$lei->_lei_store(1)->write_prepare($lei);
diff --git a/t/lei-convert.t b/t/lei-convert.t
index 0ea860c82189..e1849ff796cd 100644
--- a/t/lei-convert.t
+++ b/t/lei-convert.t
@@ -115,5 +115,15 @@ test_lei({ tmpdir => $tmpdir }, sub {
@bar = ();
PublicInbox::MboxReader->mboxrd($fh, sub { push @bar, shift });
is_deeply(\@bar, [ $qp_eml ], 'readed gzipped mboxrd');
+
+ # Status => Maildir flag => Status round trip
+ $lei_out =~ s/^Status: O/Status: RO/sm or xbail "`seen' Status";
+ $rdr = { 0 => \($in = $lei_out), %$lei_opt };
+ lei_ok([qw(convert -F mboxrd -o), "$d/md2"], undef, $rdr);
+ @md = glob("$d/md2/*/*");
+ is(scalar(@md), 1, 'one message');
+ like($md[0], qr/:2,S\z/, "`seen' flag set in Maildir");
+ lei_ok(qw(convert -o mboxrd:/dev/stdout), "$d/md2");
+ like($lei_out, qr/^Status: RO/sm, "`seen' flag preserved");
});
done_testing;
^ permalink raw reply related [relevance 63%]
* [PATCH 3/8] doc: lei-convert: various updates and cleanups
2021-10-28 11:14 71% [PATCH 0/8] lei: docs and cleanups Eric Wong
2021-10-28 11:14 68% ` [PATCH 1/8] xt/net_writer_imap: test "lei convert" w/ IMAP source Eric Wong
2021-10-28 11:14 71% ` [PATCH 2/8] lei convert: use "--output" in failure message Eric Wong
@ 2021-10-28 11:14 62% ` Eric Wong
2021-10-28 11:14 70% ` [PATCH 4/8] doc: lei blob: wording fixups, describe --remote Eric Wong
` (4 subsequent siblings)
7 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-10-28 11:14 UTC (permalink / raw)
To: meta
Note that "-o OUTPUT" is required in the synopsis.
Leave out "eml:" for now since it doesn't work as an output and
I doubt anybody would use it as a prefix, and it's not really
useful.
--no-import-remote is also not accepted by convert, since it
doesn't touch lei/store at all.
---
Documentation/lei-convert.pod | 21 +++++++++------------
1 file changed, 9 insertions(+), 12 deletions(-)
diff --git a/Documentation/lei-convert.pod b/Documentation/lei-convert.pod
index 750ba54ffa1a..c113db18b1e4 100644
--- a/Documentation/lei-convert.pod
+++ b/Documentation/lei-convert.pod
@@ -1,23 +1,23 @@
=head1 NAME
-lei-convert - one-time conversion from URL or filesystem to another format
+lei-convert - one-time conversion from one mail format to another
=head1 SYNOPSIS
-lei convert [OPTIONS] LOCATION
+lei convert -o OUTPUT [OPTIONS] LOCATION
-lei convert [OPTIONS] (--stdin|-)
+lei convert -o OUTPUT [OPTIONS] (--stdin|-)
=head1 DESCRIPTION
Convert messages to another format. C<LOCATION> is a source of
-messages: a directory (Maildir), a file, or a URL (C<imap://>,
-C<imaps://>, C<nntp://>, or C<nntps://>). URLs requiring
-authentication use L<git-credential(1)> to
+messages: a directory (Maildir), a file (various mbox), or a URL
+(C<imap://>, C<imaps://>, C<nntp://>, or C<nntps://>). URLs
+requiring authentication use L<git-credential(1)> to
fill in the username and password.
For a regular file, the location must have a C<E<lt>formatE<gt>:>
-prefix specifying one of the following formats: C<eml>, C<mboxrd>,
+prefix specifying one of the following formats: C<mboxrd>,
C<mboxcl2>, C<mboxcl>, or C<mboxo>.
=head1 OPTIONS
@@ -44,8 +44,6 @@ L<lei-q(1)>.
=item --no-kw
-=item --no-import-remote
-
=item --torsocks=auto|no|yes
=item --no-torsocks
@@ -63,11 +61,10 @@ and L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta
=head1 COPYRIGHT
-Copyright 2021 all contributors L<mailto:meta@public-inbox.org>
+Copyright all contributors L<mailto:meta@public-inbox.org>
License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>
-
=head1 SEE ALSO
-L<lei-add-external(1)>
+L<lei-mail-formats(5)>
^ permalink raw reply related [relevance 62%]
* [PATCH 0/8] lei: docs and cleanups
@ 2021-10-28 11:14 71% Eric Wong
2021-10-28 11:14 68% ` [PATCH 1/8] xt/net_writer_imap: test "lei convert" w/ IMAP source Eric Wong
` (7 more replies)
0 siblings, 8 replies; 200+ results
From: Eric Wong @ 2021-10-28 11:14 UTC (permalink / raw)
To: meta
Expect more on the way. As always some bugs and fixes were
found while attempting to write docs :x
Eric Wong (8):
xt/net_writer_imap: test "lei convert" w/ IMAP source
lei convert: use "--output" in failure message
doc: lei-convert: various updates and cleanups
doc: lei blob: wording fixups, describe --remote
lei convert: remove redundant input_net_cb
doc: lei-add-watch: add warning about unreliability
lei sucks: show nproc in CPU info
lei rm: move generic input_maildir_cb to LeiInput parent class
Documentation/lei-add-watch.pod | 8 ++++++--
Documentation/lei-blob.pod | 12 ++++++++----
Documentation/lei-convert.pod | 21 +++++++++------------
lib/PublicInbox/LeiConvert.pm | 8 ++------
lib/PublicInbox/LeiInput.pm | 5 +++++
lib/PublicInbox/LeiRm.pm | 5 -----
lib/PublicInbox/LeiSucks.pm | 4 +++-
t/lei-convert.t | 10 ++++++++++
xt/net_writer-imap.t | 7 +++++++
9 files changed, 50 insertions(+), 30 deletions(-)
^ permalink raw reply [relevance 71%]
* [PATCH 1/8] xt/net_writer_imap: test "lei convert" w/ IMAP source
2021-10-28 11:14 71% [PATCH 0/8] lei: docs and cleanups Eric Wong
@ 2021-10-28 11:14 68% ` Eric Wong
2021-10-28 11:14 71% ` [PATCH 2/8] lei convert: use "--output" in failure message Eric Wong
` (6 subsequent siblings)
7 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-10-28 11:14 UTC (permalink / raw)
To: meta
I just did a double-take and nearly thought authentication
was broken while reading LeiConvert.pm. Add a comment in
LeiConvert.pm to clarify things, too.
---
lib/PublicInbox/LeiConvert.pm | 1 +
xt/net_writer-imap.t | 7 +++++++
2 files changed, 8 insertions(+)
diff --git a/lib/PublicInbox/LeiConvert.pm b/lib/PublicInbox/LeiConvert.pm
index 68fc7c0ba2b0..424eab8e22e2 100644
--- a/lib/PublicInbox/LeiConvert.pm
+++ b/lib/PublicInbox/LeiConvert.pm
@@ -55,6 +55,7 @@ sub lei_convert { # the main "lei convert" method
my $devfd = $lei->path_to_fd($ovv->{dst}) // return;
$lei->{opt}->{augment} = 1 if $devfd < 0;
$self->prepare_inputs($lei, \@inputs) or return;
+ # n.b. {net} {auth} is handled by l2m worker
my ($op_c, $ops) = $lei->workers_start($self, 1);
$lei->{wq1} = $self;
$self->wq_io_do('process_inputs', []);
diff --git a/xt/net_writer-imap.t b/xt/net_writer-imap.t
index afa4bcc3e881..333e0e3b7ee0 100644
--- a/xt/net_writer-imap.t
+++ b/xt/net_writer-imap.t
@@ -137,6 +137,13 @@ test_lei(sub {
is_deeply($res->[0]->[1], $plack_qp_eml,
'lei q wrote expected result');
+ my $mdir = "$ENV{HOME}/t.mdir";
+ lei_ok 'convert', $folder_url, '-o', $mdir;
+ my @mdfiles = glob("$mdir/*/*");
+ is(scalar(@mdfiles), 1, '1 message from IMAP => Maildir conversion');
+ is_deeply(eml_load($mdfiles[0]), $plack_qp_eml,
+ 'conversion from IMAP to Maildir');
+
lei_ok qw(q f:matz -a -o), $folder_url;
$nwr->imap_each($folder_uri, $imap_slurp_all, my $aug = []);
is(scalar(@$aug), 2, '2 results after augment') or diag explain($aug);
^ permalink raw reply related [relevance 68%]
* [PATCH] lei add-watch: ensure folders are known to mail_sync.sqlite3
@ 2021-10-28 6:17 71% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-10-28 6:17 UTC (permalink / raw)
To: meta
This prevents noisy errors in syslog when running t/lei-watch.t
---
lib/PublicInbox/LeiAddWatch.pm | 2 ++
1 file changed, 2 insertions(+)
diff --git a/lib/PublicInbox/LeiAddWatch.pm b/lib/PublicInbox/LeiAddWatch.pm
index 671d54f92467..97e7a3422897 100644
--- a/lib/PublicInbox/LeiAddWatch.pm
+++ b/lib/PublicInbox/LeiAddWatch.pm
@@ -34,6 +34,8 @@ sub lei_add_watch {
next if defined $cfg->{"watch.$w.state"};
$lei->_config("watch.$w.state", $state);
}
+ $lei->_lei_store(1); # create
+ $lei->lms(1)->lms_write_prepare->add_folders(@{$self->{inputs}});
delete $lei->{cfg}; # force reload
$lei->refresh_watches;
}
^ permalink raw reply related [relevance 71%]
* Re: lei-q-remote-import failures [was: [PATCH] t/index-git-times: support non-master default branch]
2021-10-27 21:24 39% ` Thomas Weißschuh
@ 2021-10-27 23:48 63% ` Eric Wong
2021-10-28 14:03 71% ` Thomas Weißschuh
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2021-10-27 23:48 UTC (permalink / raw)
To: Thomas Weißschuh; +Cc: meta
Thomas Weißschuh <thomas@t-8ch.de> wrote:
> It didn't help, neither did larger sleeps.
Thanks for testing.
> I'll see if I can debug it in the coming days.
OK, maybe the patch below can get you started...
I've also been wondering if something like GIT_TRACE is
necessary, but it's also more work to support + document.
And Perl code is easily modified compared to AOT languages.
diff --git a/lib/PublicInbox/LeiXSearch.pm b/lib/PublicInbox/LeiXSearch.pm
index 2a037f2bd79b..a0982138966e 100644
--- a/lib/PublicInbox/LeiXSearch.pm
+++ b/lib/PublicInbox/LeiXSearch.pm
@@ -281,13 +281,19 @@ sub each_remote_eml { # callback for MboxReader->mboxrd
my ($eml, $self, $lei, $each_smsg) = @_;
my $xoids = $lei->{ale}->xoids_for($eml, 1);
my $smsg = bless {}, 'PublicInbox::Smsg';
+ use Data::Dumper;
if ($self->{import_sto} && !$xoids) {
my ($res, $kw) = $self->{import_sto}->wq_do('add_eml', $eml);
if (ref($res) eq ref($smsg)) { # totally new message
$smsg = $res;
+ $lei->qerr("# imported <$smsg->{mid}>") if $ENV{DBG};
$self->{-imported} = 1;
+ } elsif ($ENV{DBG}) {
+ $lei->qerr("# res, kw = ".Dumper([$res, $kw]));
}
$smsg->{kw} = $kw; # short-circuit xsmsg_vmd
+ } elsif ($ENV{DBG}) {
+ $lei->qerr("# no import $self->{import_sto}".Dumper($xoids));
}
$smsg->{blob} //= $xoids ? (keys(%$xoids))[0]
: $lei->git_oid($eml)->hexdigest;
@@ -378,6 +384,9 @@ sub query_remote_mboxrd {
$lei, $each_smsg);
if ($self->{import_sto} && delete($self->{-imported})) {
my $wait = $self->{import_sto}->wq_do('done');
+ $lei->qerr("# flushed imported messages") if $ENV{DBG};
+ } elsif ($ENV{DBG}) {
+ $lei->qerr("# nothing flushed $self->{import_sto}");
}
$reap_curl->join;
if ($? == 0) {
diff --git a/t/lei-q-remote-import.t b/t/lei-q-remote-import.t
index 92d8c9b6058c..e10af2eb7849 100644
--- a/t/lei-q-remote-import.t
+++ b/t/lei-q-remote-import.t
@@ -32,7 +32,11 @@ test_lei({ tmpdir => $tmpdir }, sub {
lei_ok(@cmd);
ok(-f $o && !-s _, 'output exists but is empty');
unlink $o or BAIL_OUT $!;
- lei_ok(@cmd, '-I', $url);
+ lei_ok([@cmd, '-I', $url], { DBG => 1 });
+
+# should say 'imported <qp@example.com>' + 'flushed imported messages'
+ diag $lei_err;
+return; # drop this if the above diag looks right
is_deeply($slurp_emls->($o), $exp1, 'got results after remote search');
unlink $o or BAIL_OUT $!;
lei_ok(@cmd);
^ permalink raw reply related [relevance 63%]
* Re: [PATCH] lei q: fix remote import accounting
2021-10-27 21:09 64% [PATCH] lei q: fix remote import accounting Eric Wong
@ 2021-10-27 21:28 71% ` Thomas Weißschuh
0 siblings, 0 replies; 200+ results
From: Thomas Weißschuh @ 2021-10-27 21:28 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
On 2021-10-27 21:09+0000, Eric Wong wrote:
> We need to update the {-nr_remote_eml} counter regardless
> of progress display being enabled since it's needed for
> saved searches. We'll also split out the {-imported} flag
> separately and only call LeiStore->done if a new message
> was imported.
>
> Note: this change is NOT expected to fix errors reported by
> Thomas in <ebf92218-1470-4602-b534-6dae59639dc6@t-8ch.de>
For the record: This patch did in fact *not* fix the issue.
^ permalink raw reply [relevance 71%]
* Re: lei-q-remote-import failures [was: [PATCH] t/index-git-times: support non-master default branch]
2021-10-27 21:15 71% ` lei-q-remote-import failures [was: [PATCH] t/index-git-times: support non-master default branch] Eric Wong
@ 2021-10-27 21:24 39% ` Thomas Weißschuh
2021-10-27 23:48 63% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Thomas Weißschuh @ 2021-10-27 21:24 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
[-- Attachment #1: Type: text/plain, Size: 2131 bytes --]
On 2021-10-27 21:15+0000, Eric Wong wrote:
> Thomas Weißschuh <thomas@t-8ch.de> wrote:
> > On 2021-10-26 05:28+0000, Eric Wong wrote:
> > > Thomas Weißschuh <thomas@t-8ch.de> wrote:
> > > > The only failures I still see are in t/lei-q-remote-import.t which is unrelated
> > > > to the git branch and some more that fail only during parallel test execution.
> > > >
> > > > In t/lei-q-remote-import.t it seems the search results are not memoized to the
> > > > local store. (See attachment)
> > >
> > > Thanks, I'll check out lei-q-remote-import.t separately.
> > > That could be related to SMP or slow/fast storage.
> > > Any details you can share about CPU core count, speeds
> > > or storage speeds, or CPU scheduler, CONFIG_HZ?
> >
> > CPU:
> >
> > Model name: 11th Gen Intel(R) Core(TM) i7-1165G7 @ 2.80GHz
> > CPU family: 6
> > Model: 140
> > Thread(s) per core: 2
> > Core(s) per socket: 4
> > Socket(s): 1
> > Stepping: 1
> > CPU max MHz: 4700,0000
> > CPU min MHz: 400,0000
> >
> > Storage: Samsung EVO 970 (with plenty of memory cache)
> > CONFIG_HZ: 300
>
> I'm puzzled by this failure; I'm not sure if I've
> ever seen it. Does it fail when run standalone?
> (prove -bvw t/lei-q-remote-import.t)
It also does. The protocol is attached.
> Maybe running under TEST_LEI_ERR_LOUD=1 or checking
> syslog can reveal something.
>
> Adding a sleep shouldn't be necessary, but maybe something
> else is broken I'm not seeing...:
It didn't help, neither did larger sleeps.
> diff --git a/t/lei-q-remote-import.t b/t/lei-q-remote-import.t
> index 92d8c9b6058c..a2d643cd06c7 100644
> --- a/t/lei-q-remote-import.t
> +++ b/t/lei-q-remote-import.t
> @@ -35,6 +35,7 @@ test_lei({ tmpdir => $tmpdir }, sub {
> lei_ok(@cmd, '-I', $url);
> is_deeply($slurp_emls->($o), $exp1, 'got results after remote search');
> unlink $o or BAIL_OUT $!;
> + sleep 1;
> lei_ok(@cmd);
> ok(-f $o && -s _, 'output exists after import but is not empty') or
> diag $lei_err;
I'll see if I can debug it in the coming days.
[-- Attachment #2: output-lei-q-remote-import.t --]
[-- Type: text/troff, Size: 6315 bytes --]
$ git checkout origin/master
Note: switching to 'origin/master'.
HEAD is now at ded9dad2 test_common: key test inboxes to init.defaultBranch
$ git clean -fdx t/data-gen/
...
$ TEST_LEI_ERR_LOUD=1 prove -bvw t/lei-q-remote-import.t
# # /tmp/pi-lei-q-remote-import-5668-gmA0/lei-daemon/.local/share/lei/store 0/0
# # 0 written to /tmp/pi-lei-q-remote-import-5668-gmA0/lei-daemon/o.mboxrd (0 matches)
ok 1 - lei q -o mboxrd:/tmp/pi-lei-q-remote-import-5668-gmA0/lei-daemon/o.mboxrd m:qp@example.com
ok 2 - output exists but is empty
# lei_err=# /usr/bin/curl -Sf -s -d '' http://[::1]:46299/t2/?x=m&q=m%3Aqp%40example.com
# # /tmp/pi-lei-q-remote-import-5668-gmA0/lei-daemon/.local/share/lei/store 0/0
# # http://[::1]:46299/t2/ 1/1
# # 1 written to /tmp/pi-lei-q-remote-import-5668-gmA0/lei-daemon/o.mboxrd (1 matches)
ok 3 - lei q -o mboxrd:/tmp/pi-lei-q-remote-import-5668-gmA0/lei-daemon/o.mboxrd m:qp@example.com -I http://[::1]:46299/t2/
ok 4 - got results after remote search
ok 5 - lei q -o mboxrd:/tmp/pi-lei-q-remote-import-5668-gmA0/lei-daemon/o.mboxrd m:qp@example.com
not ok 6 - output exists after import but is not empty
# lei_err=# /tmp/pi-lei-q-remote-import-5668-gmA0/lei-daemon/.local/share/lei/store 0/0
# # 0 written to /tmp/pi-lei-q-remote-import-5668-gmA0/lei-daemon/o.mboxrd (0 matches)
# Failed test 'output exists after import but is not empty'
# at t/lei-q-remote-import.t line 39.
# # /tmp/pi-lei-q-remote-import-5668-gmA0/lei-daemon/.local/share/lei/store 0/0
# # 0 written to /tmp/pi-lei-q-remote-import-5668-gmA0/lei-daemon/o.mboxrd (0 matches)
not ok 7 - got results w/o remote search
# Failed test 'got results w/o remote search'
# at t/lei-q-remote-import.t line 41.
# Structures begin differing at:
# $got->[0] = Does not exist
# $expected->[0] = PublicInbox::Eml=HASH(0x565282aeff78)
ok 8 - lei q -o mboxrd:/tmp/pi-lei-q-remote-import-5668-gmA0/lei-daemon/o.mboxrd m:199707281508.AAA24167@hoyogw.example -I http://[::1]:46299/t2/ --no-import-remote
# lei_err=# /tmp/pi-lei-q-remote-import-5668-gmA0/lei-daemon/.local/share/lei/store 0/0
# # /usr/bin/curl -Sf -s -d '' http://[::1]:46299/t2/?x=m&q=m%3A199707281508.AAA24167%40hoyogw.example
# # http://[::1]:46299/t2/ 1/1
# # 1 written to /tmp/pi-lei-q-remote-import-5668-gmA0/lei-daemon/o.mboxrd (1 matches)
ok 9 - got another after remote search
ok 10 - lei q -o mboxrd:/tmp/pi-lei-q-remote-import-5668-gmA0/lei-daemon/o.mboxrd m:199707281508.AAA24167@hoyogw.example
ok 11 - --no-import-remote did not memoize
# lei_err=# /tmp/pi-lei-q-remote-import-5668-gmA0/lei-daemon/.local/share/lei/store 0/0
# # 0 written to /tmp/pi-lei-q-remote-import-5668-gmA0/lei-daemon/o.mboxrd (0 matches)
ok 12 - lei q -o mboxrd:/tmp/pi-lei-q-remote-import-5668-gmA0/lei-daemon/o.mboxrd m:qp@example.com --lock=none
not ok 13 - --lock=none respected
# lei_err=# /tmp/pi-lei-q-remote-import-5668-gmA0/lei-daemon/.local/share/lei/store 0/0
# # 0 written to /tmp/pi-lei-q-remote-import-5668-gmA0/lei-daemon/o.mboxrd (0 matches)
# Failed test '--lock=none respected'
# at t/lei-q-remote-import.t line 55.
# # /tmp/pi-lei-q-remote-import-5668-gmA0/lei-daemon/.local/share/lei/store 0/0
# # 0 written to /tmp/pi-lei-q-remote-import-5668-gmA0/lei-daemon/o.mboxrd (0 matches)
# lei_err=dotlock timeout /tmp/pi-lei-q-remote-import-5668-gmA0/lei-daemon/o.mboxrd.lock
ok 14 - dotlock fails
ok 15 - timeout noted
ok 16 - nothing output on lock failure
ok 17 - lei q -o mboxrd:/tmp/pi-lei-q-remote-import-5668-gmA0/lei-daemon/o.mboxrd m:qp@example.com --lock=dotlock,timeout=0.000001 (succeeds after lock removal)
# lei_err=# /tmp/pi-lei-q-remote-import-5668-gmA0/lei-daemon/.local/share/lei/store 0/0
# # 0 written to /tmp/pi-lei-q-remote-import-5668-gmA0/lei-daemon/o.mboxrd (0 matches)
ok 18 - lei add-external -q /home/t-8ch/Projekte/public-inbox.org/t/data-gen/lei-q-remote-import.local-external-master
ok 19 - lei q -q -o mboxrd:/tmp/pi-lei-q-remote-import-5668-gmA0/lei-daemon/o.mboxrd --only http://[::1]:46299/t2/ m:testmessage@example.com
ok 20 - no warnings or errors
ok 21 - got result from remote external
ok 22 - got expected result
ok 23 - lei q --no-external -o mboxrd:/dev/stdout m:testmessage@example.com
ok 24 - message not imported when in local external
ok 25 - lei q -o mboxrd:/tmp/pi-lei-q-remote-import-5668-gmA0/lei-daemon/o.mboxrd m:testmessage@example.com
# lei_err=# /tmp/pi-lei-q-remote-import-5668-gmA0/lei-daemon/.local/share/lei/store 0/0
# # /home/t-8ch/Projekte/public-inbox.org/t/data-gen/lei-q-remote-import.local-external-master 1/1
# # 1 written to /tmp/pi-lei-q-remote-import-5668-gmA0/lei-daemon/o.mboxrd (1 matches)
ok 26 - got expected result after clobber
ok 27 - lei q -o mboxrd:/dev/stdout m:never-before-seen@example.com
ok 28 - --import-before imported totally unseen message
ok 29 - lei q --save z:0.. -o /tmp/pi-lei-q-remote-import-5668-gmA0/lei-daemon/md --only http://[::1]:46299/t2/
# lei_err=# /usr/bin/curl -Sf -s -d '' http://[::1]:46299/t2/?x=m&q=z%3A0..
# # http://[::1]:46299/t2/ 15/15
# # 15 written to /tmp/pi-lei-q-remote-import-5668-gmA0/lei-daemon/md/ (15 matches)
ok 30 - lei up /tmp/pi-lei-q-remote-import-5668-gmA0/lei-daemon/md
ok 31 - lei up remote dedupe works on maildir
# lei_err=# http://[::1]:46299/t2/ limiting to 2021-10-25 23:18 +0200 and newer
# # /usr/bin/curl -Sf -s -d '' http://[::1]:46299/t2/?x=m&q=(z%3A0..)+AND+dt%3A20211025211849..
# # http://[::1]:46299/t2/ 11/11
# # 0 written to /tmp/pi-lei-q-remote-import-5668-gmA0/lei-daemon/md/ (11 matches)
ok 32 - lei edit-search /tmp/pi-lei-q-remote-import-5668-gmA0/lei-daemon/md
ok 33 - lastresult set
ok 34 - lei daemon-pid (daemon-pid after t/lei-q-remote-import.t:106)
ok 35 - daemon running after t/lei-q-remote-import.t:106
ok 36 - lei daemon-kill (daemon-kill after t/lei-q-remote-import.t:106)
ok 37 - t/lei-q-remote-import.t:106 daemon stopped
ok 38 - t/lei-q-remote-import.t:106 daemon XDG_RUNTIME_DIR/lei/errors.log empty
1..38
# Looks like you failed 3 tests of 38.
Dubious, test returned 3 (wstat 768, 0x300)
Failed 3/38 subtests
Test Summary Report
-------------------
t/lei-q-remote-import.t (Wstat: 768 Tests: 38 Failed: 3)
Failed tests: 6-7, 13
Non-zero exit status: 3
Files=1, Tests=38, 1 wallclock secs ( 0.01 usr 0.01 sys + 0.32 cusr 0.08 csys = 0.42 CPU)
Result: FAIL
^ permalink raw reply [relevance 39%]
* lei-q-remote-import failures [was: [PATCH] t/index-git-times: support non-master default branch]
@ 2021-10-27 21:15 71% ` Eric Wong
2021-10-27 21:24 39% ` Thomas Weißschuh
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2021-10-27 21:15 UTC (permalink / raw)
To: Thomas Weißschuh; +Cc: meta
Thomas Weißschuh <thomas@t-8ch.de> wrote:
> On 2021-10-26 05:28+0000, Eric Wong wrote:
> > Thomas Weißschuh <thomas@t-8ch.de> wrote:
> > > The only failures I still see are in t/lei-q-remote-import.t which is unrelated
> > > to the git branch and some more that fail only during parallel test execution.
> > >
> > > In t/lei-q-remote-import.t it seems the search results are not memoized to the
> > > local store. (See attachment)
> >
> > Thanks, I'll check out lei-q-remote-import.t separately.
> > That could be related to SMP or slow/fast storage.
> > Any details you can share about CPU core count, speeds
> > or storage speeds, or CPU scheduler, CONFIG_HZ?
>
> CPU:
>
> Model name: 11th Gen Intel(R) Core(TM) i7-1165G7 @ 2.80GHz
> CPU family: 6
> Model: 140
> Thread(s) per core: 2
> Core(s) per socket: 4
> Socket(s): 1
> Stepping: 1
> CPU max MHz: 4700,0000
> CPU min MHz: 400,0000
>
> Storage: Samsung EVO 970 (with plenty of memory cache)
> CONFIG_HZ: 300
I'm puzzled by this failure; I'm not sure if I've
ever seen it. Does it fail when run standalone?
(prove -bvw t/lei-q-remote-import.t)
Maybe running under TEST_LEI_ERR_LOUD=1 or checking
syslog can reveal something.
Adding a sleep shouldn't be necessary, but maybe something
else is broken I'm not seeing...:
diff --git a/t/lei-q-remote-import.t b/t/lei-q-remote-import.t
index 92d8c9b6058c..a2d643cd06c7 100644
--- a/t/lei-q-remote-import.t
+++ b/t/lei-q-remote-import.t
@@ -35,6 +35,7 @@ test_lei({ tmpdir => $tmpdir }, sub {
lei_ok(@cmd, '-I', $url);
is_deeply($slurp_emls->($o), $exp1, 'got results after remote search');
unlink $o or BAIL_OUT $!;
+ sleep 1;
lei_ok(@cmd);
ok(-f $o && -s _, 'output exists after import but is not empty') or
diag $lei_err;
^ permalink raw reply related [relevance 71%]
* [PATCH] lei q: fix remote import accounting
@ 2021-10-27 21:09 64% Eric Wong
2021-10-27 21:28 71% ` Thomas Weißschuh
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2021-10-27 21:09 UTC (permalink / raw)
To: meta; +Cc: Thomas Weißschuh
We need to update the {-nr_remote_eml} counter regardless
of progress display being enabled since it's needed for
saved searches. We'll also split out the {-imported} flag
separately and only call LeiStore->done if a new message
was imported.
Note: this change is NOT expected to fix errors reported by
Thomas in <ebf92218-1470-4602-b534-6dae59639dc6@t-8ch.de>
Cc: Thomas Weißschuh <thomas@t-8ch.de>
---
lib/PublicInbox/LeiXSearch.pm | 17 ++++++++++-------
1 file changed, 10 insertions(+), 7 deletions(-)
diff --git a/lib/PublicInbox/LeiXSearch.pm b/lib/PublicInbox/LeiXSearch.pm
index acc36897d4e3..2a037f2bd79b 100644
--- a/lib/PublicInbox/LeiXSearch.pm
+++ b/lib/PublicInbox/LeiXSearch.pm
@@ -283,20 +283,22 @@ sub each_remote_eml { # callback for MboxReader->mboxrd
my $smsg = bless {}, 'PublicInbox::Smsg';
if ($self->{import_sto} && !$xoids) {
my ($res, $kw) = $self->{import_sto}->wq_do('add_eml', $eml);
- $smsg = $res if ref($res) eq ref($smsg); # totally new message
+ if (ref($res) eq ref($smsg)) { # totally new message
+ $smsg = $res;
+ $self->{-imported} = 1;
+ }
$smsg->{kw} = $kw; # short-circuit xsmsg_vmd
}
$smsg->{blob} //= $xoids ? (keys(%$xoids))[0]
: $lei->git_oid($eml)->hexdigest;
_smsg_fill($smsg, $eml);
wait_startq($lei);
+ my $nr = ++$lei->{-nr_remote_eml}; # needed for lss->cfg_set
if ($lei->{-progress}) {
- ++$lei->{-nr_remote_eml};
my $now = now();
my $next = $lei->{-next_progress} //= ($now + 1);
if ($now > $next) {
$lei->{-next_progress} = $now + 1;
- my $nr = $lei->{-nr_remote_eml};
mset_progress($lei, $lei->{-current_url}, $nr, '?');
}
}
@@ -374,13 +376,14 @@ sub query_remote_mboxrd {
$fh = IO::Uncompress::Gunzip->new($fh, MultiStream => 1);
PublicInbox::MboxReader->mboxrd($fh, \&each_remote_eml, $self,
$lei, $each_smsg);
- my $nr = $lei->{-nr_remote_eml};
- my $wait = $lei->{sto}->wq_do('done') if $nr && $lei->{sto};
+ if ($self->{import_sto} && delete($self->{-imported})) {
+ my $wait = $self->{import_sto}->wq_do('done');
+ }
$reap_curl->join;
if ($? == 0) {
# don't update if no results, maybe MTA is down
- $key && $nr and
- $lei->{lss}->cfg_set($key, $start);
+ my $nr = $lei->{-nr_remote_eml};
+ $lei->{lss}->cfg_set($key, $start) if $key && $nr;
mset_progress($lei, $lei->{-current_url}, $nr, $nr);
next;
}
^ permalink raw reply related [relevance 64%]
* [PATCH] lei mail-diff: support more inputs, split newlines
@ 2021-10-26 21:18 64% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-10-26 21:18 UTC (permalink / raw)
To: meta
Support --in-format like the rest of LeiInput users, and don't
default to .eml if a per-input format was specified. In any
case, I saved a bunch of messages from mutt which uses mboxcl2.
We'll also split newlines for diff, since it's a pain to read
diffs with escaped "\n" characters in them.
---
lib/PublicInbox/LEI.pm | 2 +-
lib/PublicInbox/LeiMailDiff.pm | 6 +++---
2 files changed, 4 insertions(+), 4 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index f9d24f29c87d..96f7c5e315a9 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -202,7 +202,7 @@ our %CMD = ( # sorted in order of importance/use:
'mail-diff' => [ '--stdin|LOCATION...', 'diff the contents of emails',
'stdin|', # /|\z/ must be first for lone dash
- qw(verbose|v+ color:s no-color raw-header),
+ qw(verbose|v+ in-format|F=s color:s no-color raw-header),
@diff_opt, @net_opt, @c_opt ],
'add-external' => [ 'LOCATION',
diff --git a/lib/PublicInbox/LeiMailDiff.pm b/lib/PublicInbox/LeiMailDiff.pm
index 4f3a4608d8e0..b21a0c366fd6 100644
--- a/lib/PublicInbox/LeiMailDiff.pm
+++ b/lib/PublicInbox/LeiMailDiff.pm
@@ -76,7 +76,7 @@ sub input_eml_cb { # used by PublicInbox::LeiInput::input_fh
sub lei_mail_diff {
my ($lei, @argv) = @_;
- $lei->{opt}->{'in-format'} //= 'eml';
+ $lei->{opt}->{'in-format'} //= 'eml' if !grep(/\A[a-z0-9]+:/i, @argv);
my $self = bless {}, __PACKAGE__;
$self->prepare_inputs($lei, \@argv) or return;
my $isatty = -t $lei->{1};
@@ -94,7 +94,7 @@ sub lei_mail_diff {
no warnings 'once';
*net_merge_all_done = \&PublicInbox::LeiInput::input_only_net_merge_all_done;
-package PublicInbox::ContentDigestDbg;
+package PublicInbox::ContentDigestDbg; # cf. PublicInbox::ContentDigest
use strict;
use v5.10.1;
use Data::Dumper;
@@ -103,7 +103,7 @@ sub new { bless { dig => Digest::SHA->new(256), fh => $_[1] }, __PACKAGE__ }
sub add {
$_[0]->{dig}->add($_[1]);
- print { $_[0]->{fh} } Dumper($_[1]) or die "print $!";
+ print { $_[0]->{fh} } Dumper([split(/^/sm, $_[1])]) or die "print $!";
}
sub hexdigest { $_[0]->{dig}->hexdigest; }
^ permalink raw reply related [relevance 64%]
* [PATCH] t/lei-watch: add diagnostics for failure
@ 2021-10-26 10:47 71% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-10-26 10:47 UTC (permalink / raw)
To: meta
I just got a difficult-to-reproduce failure, here; so there's
still some issues with the up-to-dateness of the inotify watcher.
---
t/lei-watch.t | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/t/lei-watch.t b/t/lei-watch.t
index df887a03dc77..9158571c60cc 100644
--- a/t/lei-watch.t
+++ b/t/lei-watch.t
@@ -58,7 +58,7 @@ test_lei(sub {
lei_ok qw(q mid:testmessage@example.com -o), $md2, '-I', "$ro_home/t1";
my @f2 = glob("$md2/*/*");
is(scalar(@f2), 1, 'got one result');
- like($f2[0], qr/S\z/, 'seen set from rename');
+ like($f2[0], qr/S\z/, 'seen set from rename') or diag explain(\@f2);
my $e2 = eml_load($f2[0]);
my $e1 = eml_load("$f[0]S");
is_deeply($e2, $e1, 'results match');
^ permalink raw reply related [relevance 71%]
* [PATCH 4/9] lei q: enable expensive Xapian flags
2021-10-26 10:35 69% [PATCH 0/9] lei p2q: more capable than originally thought Eric Wong
2021-10-26 10:35 67% ` [PATCH 2/9] doc: lei-store-format: mail sync section, update IPC Eric Wong
@ 2021-10-26 10:35 71% ` Eric Wong
2021-10-26 10:35 71% ` [PATCH 5/9] lei inspect: fix atfork hook Eric Wong
` (3 subsequent siblings)
5 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-10-26 10:35 UTC (permalink / raw)
To: meta
FLAG_PURE_NOT is too expensive for public-facing WWW use, but
lei isn't public-facing. We'll also unconditionally enable
phrase search on old "chert" DBs since lei doesn't need to
worry about fairness across 10K users.
---
lib/PublicInbox/LeiSearch.pm | 2 ++
1 file changed, 2 insertions(+)
diff --git a/lib/PublicInbox/LeiSearch.pm b/lib/PublicInbox/LeiSearch.pm
index 1fb38da1d7aa..d0ca13f0aa11 100644
--- a/lib/PublicInbox/LeiSearch.pm
+++ b/lib/PublicInbox/LeiSearch.pm
@@ -175,6 +175,8 @@ sub all_terms {
sub qparse_new {
my ($self) = @_;
my $qp = $self->SUPER::qparse_new; # PublicInbox::Search
+ $self->{qp_flags} |= PublicInbox::Search::FLAG_PHRASE() |
+ PublicInbox::Search::FLAG_PURE_NOT();
$qp->add_boolean_prefix('kw', 'K');
$qp->add_boolean_prefix('L', 'L');
$qp
^ permalink raw reply related [relevance 71%]
* [PATCH 5/9] lei inspect: fix atfork hook
2021-10-26 10:35 69% [PATCH 0/9] lei p2q: more capable than originally thought Eric Wong
2021-10-26 10:35 67% ` [PATCH 2/9] doc: lei-store-format: mail sync section, update IPC Eric Wong
2021-10-26 10:35 71% ` [PATCH 4/9] lei q: enable expensive Xapian flags Eric Wong
@ 2021-10-26 10:35 71% ` Eric Wong
2021-10-26 10:35 66% ` [PATCH 6/9] lei: add net getopt spec to various commands Eric Wong
` (2 subsequent siblings)
5 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-10-26 10:35 UTC (permalink / raw)
To: meta
The misnamed sub wasn't firing, but was unlikely to be
noticeable given the short lifetime of the process.
Fixes: 1f887bd51d92b0d4 ("lei inspect: add atfork hook")
---
lib/PublicInbox/LeiInspect.pm | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/lib/PublicInbox/LeiInspect.pm b/lib/PublicInbox/LeiInspect.pm
index 5ea32ccb7e66..d7775d4b6162 100644
--- a/lib/PublicInbox/LeiInspect.pm
+++ b/lib/PublicInbox/LeiInspect.pm
@@ -294,7 +294,7 @@ sub _complete_inspect {
# TODO: message-ids?, blobs? could get expensive...
}
-sub input_only_atfork_child {
+sub ipc_atfork_child {
my ($self) = @_;
$self->{lei}->_lei_atfork_child;
$self->SUPER::ipc_atfork_child;
^ permalink raw reply related [relevance 71%]
* [PATCH 0/9] lei p2q: more capable than originally thought
@ 2021-10-26 10:35 69% Eric Wong
2021-10-26 10:35 67% ` [PATCH 2/9] doc: lei-store-format: mail sync section, update IPC Eric Wong
` (5 more replies)
0 siblings, 6 replies; 200+ results
From: Eric Wong @ 2021-10-26 10:35 UTC (permalink / raw)
To: meta
I started writing documentation and managed to cobble up a p2q
example in patch 7/9 to find unapplied patches when combined
with "git log". 7/9 makes p2q use less memory, now.
Fixed a bunch of small bugs along the way and killed some
redundant code, too.
Eric Wong (9):
doc: tuning: additional notes for many inboxes
doc: lei-store-format: mail sync section, update IPC
eml: keep body if no headers are found
lei q: enable expensive Xapian flags
lei inspect: fix atfork hook
lei: add net getopt spec to various commands
lei p2q: use LeiInput for multi-patch series
lei rm|tag: drop redundant mbox+net callbacks
input_pipe: account for undefined {sock}
Documentation/lei-p2q.pod | 6 ++
Documentation/lei-store-format.pod | 14 +++-
Documentation/public-inbox-tuning.pod | 11 ++-
lib/PublicInbox/Eml.pm | 7 +-
lib/PublicInbox/InputPipe.pm | 2 +-
lib/PublicInbox/LEI.pm | 11 +--
lib/PublicInbox/LeiInput.pm | 30 +++++--
lib/PublicInbox/LeiInspect.pm | 2 +-
lib/PublicInbox/LeiP2q.pm | 115 +++++++++++++-------------
lib/PublicInbox/LeiRm.pm | 10 ---
lib/PublicInbox/LeiSearch.pm | 2 +
lib/PublicInbox/LeiTag.pm | 11 ---
t/eml.t | 11 +++
t/mbox_reader.t | 6 +-
14 files changed, 134 insertions(+), 104 deletions(-)
^ permalink raw reply [relevance 69%]
* [PATCH 8/9] lei rm|tag: drop redundant mbox+net callbacks
2021-10-26 10:35 69% [PATCH 0/9] lei p2q: more capable than originally thought Eric Wong
` (4 preceding siblings ...)
2021-10-26 10:35 35% ` [PATCH 7/9] lei p2q: use LeiInput for multi-patch series Eric Wong
@ 2021-10-26 10:35 68% ` Eric Wong
5 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-10-26 10:35 UTC (permalink / raw)
To: meta
These are supplied by the base LeiInput class
---
lib/PublicInbox/LeiRm.pm | 10 ----------
lib/PublicInbox/LeiTag.pm | 11 -----------
2 files changed, 21 deletions(-)
diff --git a/lib/PublicInbox/LeiRm.pm b/lib/PublicInbox/LeiRm.pm
index 97b1c5c15dc4..524c178e3b47 100644
--- a/lib/PublicInbox/LeiRm.pm
+++ b/lib/PublicInbox/LeiRm.pm
@@ -13,16 +13,6 @@ sub input_eml_cb { # used by PublicInbox::LeiInput::input_fh
$self->{lei}->{sto}->wq_do('remove_eml', $eml);
}
-sub input_mbox_cb { # MboxReader callback
- my ($eml, $self) = @_;
- input_eml_cb($self, $eml);
-}
-
-sub input_net_cb { # callback for ->imap_each, ->nntp_each
- my (undef, undef, $kw, $eml, $self) = @_; # @_[0,1]: url + uid ignored
- input_eml_cb($self, $eml);
-}
-
sub input_maildir_cb {
my (undef, $kw, $eml, $self) = @_; # $_[0] $filename ignored
input_eml_cb($self, $eml);
diff --git a/lib/PublicInbox/LeiTag.pm b/lib/PublicInbox/LeiTag.pm
index 77654d1a2f22..d64a9f86e05a 100644
--- a/lib/PublicInbox/LeiTag.pm
+++ b/lib/PublicInbox/LeiTag.pm
@@ -19,23 +19,12 @@ sub input_eml_cb { # used by PublicInbox::LeiInput::input_fh
}
}
-sub input_mbox_cb {
- my ($eml, $self) = @_;
- $eml->header_set($_) for (qw(X-Status Status));
- input_eml_cb($self, $eml);
-}
-
sub pmdir_cb { # called via wq_io_do from LeiPmdir->each_mdir_fn
my ($self, $f) = @_;
my $eml = eml_from_path($f) or return;
input_eml_cb($self, $eml);
}
-sub input_net_cb { # imap_each, nntp_each cb
- my ($url, $uid, $kw, $eml, $self) = @_;
- input_eml_cb($self, $eml);
-}
-
sub lei_tag { # the "lei tag" method
my ($lei, @argv) = @_;
my $sto = $lei->_lei_store(1);
^ permalink raw reply related [relevance 68%]
* [PATCH 2/9] doc: lei-store-format: mail sync section, update IPC
2021-10-26 10:35 69% [PATCH 0/9] lei p2q: more capable than originally thought Eric Wong
@ 2021-10-26 10:35 67% ` Eric Wong
2021-10-26 10:35 71% ` [PATCH 4/9] lei q: enable expensive Xapian flags Eric Wong
` (4 subsequent siblings)
5 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-10-26 10:35 UTC (permalink / raw)
To: meta
mail_sync.sqlite3 needs to be documented, and brings the IPC
section up-to-date while we're in the area.
---
Documentation/lei-store-format.pod | 14 ++++++++++----
1 file changed, 10 insertions(+), 4 deletions(-)
diff --git a/Documentation/lei-store-format.pod b/Documentation/lei-store-format.pod
index 71aa72cb96e3..625c60f436a8 100644
--- a/Documentation/lei-store-format.pod
+++ b/Documentation/lei-store-format.pod
@@ -30,7 +30,6 @@ prevent them from being accidentally treated as a v2 inbox.
$SHARD - Integer starting with 0 based on parallelism
~/.local/share/lei/store
- - ipc.lock # lock file for internal lei IPC
- local/$EPOCH.git # normal bare git repositories
- mail_sync.sqlite3 # sync state IMAP, Maildir, NNTP
@@ -66,11 +65,18 @@ stored in Xapian indices, volatile metadata is associated with
the Xapian document, thus it is shared across different blobs of
the "same" message.
+=head2 mail_sync.sqlite3
+
+This SQLite database maintained for bidirectional mapping of
+git blobs to IMAP UIDs, Maildir file names, and NNTP article numbers.
+
+It is also used for retrieving messages from Maildirs indexed by
+L<lei-index(1)>.
+
=head1 IPC
-When L<lei(1)> is run in daemon mode, L<flock(2)> is used on
-C<ipc.lock> is used to serialize writes to C<lei/store> across
-multiple internal lei workers while minimizing commits.
+L<lei-daemon(8)> communicates with the C<lei/store> process using
+L<unix(7)> C<SOCK_SEQPACKET> sockets.
=head1 CAVEATS
^ permalink raw reply related [relevance 67%]
* [PATCH 6/9] lei: add net getopt spec to various commands
2021-10-26 10:35 69% [PATCH 0/9] lei p2q: more capable than originally thought Eric Wong
` (2 preceding siblings ...)
2021-10-26 10:35 71% ` [PATCH 5/9] lei inspect: fix atfork hook Eric Wong
@ 2021-10-26 10:35 66% ` Eric Wong
2021-10-26 10:35 35% ` [PATCH 7/9] lei p2q: use LeiInput for multi-patch series Eric Wong
2021-10-26 10:35 68% ` [PATCH 8/9] lei rm|tag: drop redundant mbox+net callbacks Eric Wong
5 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-10-26 10:35 UTC (permalink / raw)
To: meta
All of these commands should support --proxy, at least, if not
other curl options.
---
lib/PublicInbox/LEI.pm | 7 ++++---
1 file changed, 4 insertions(+), 3 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 32d4b9f3b427..93a7b426de3c 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -179,7 +179,7 @@ our %CMD = ( # sorted in order of importance/use:
'up' => [ 'OUTPUT...|--all', 'update saved search',
qw(jobs|j=s lock=s@ alert=s@ mua=s verbose|v+ exclude=s@
- remote-fudge-time=s all:s remote! local! external!), @c_opt ],
+ remote-fudge-time=s all:s remote! local! external!), @net_opt, @c_opt ],
'lcat' => [ '--stdin|MSGID_OR_URL...', 'display local copy of message(s)',
'stdin|', # /|\z/ must be first for lone dash
@@ -215,7 +215,7 @@ our %CMD = ( # sorted in order of importance/use:
'ls-mail-sync' => [ '[FILTER]', 'list mail sync folders',
qw(z|0 globoff|g invert-match|v local remote), @c_opt ],
'ls-mail-source' => [ 'URL', 'list IMAP or NNTP mail source folders',
- qw(z|0 ascii l pretty url), @c_opt ],
+ qw(z|0 ascii l pretty url), @net_opt, @c_opt ],
'forget-external' => [ 'LOCATION...|--prune',
'exclude further results from a publicinbox|extindex',
qw(prune), @c_opt ],
@@ -265,7 +265,8 @@ our %CMD = ( # sorted in order of importance/use:
'forget-mail-sync' => [ 'LOCATION...',
'forget sync information for a mail folder', @c_opt ],
'refresh-mail-sync' => [ 'LOCATION...|--all',
- 'prune dangling sync data for a mail folder', 'all:s', @c_opt ],
+ 'prune dangling sync data for a mail folder', 'all:s',
+ @net_opt, @c_opt ],
'export-kw' => [ 'LOCATION...|--all',
'one-time export of keywords of sync sources',
qw(all:s mode=s), @net_opt, @c_opt ],
^ permalink raw reply related [relevance 66%]
* [PATCH 7/9] lei p2q: use LeiInput for multi-patch series
2021-10-26 10:35 69% [PATCH 0/9] lei p2q: more capable than originally thought Eric Wong
` (3 preceding siblings ...)
2021-10-26 10:35 66% ` [PATCH 6/9] lei: add net getopt spec to various commands Eric Wong
@ 2021-10-26 10:35 35% ` Eric Wong
2021-10-26 10:35 68% ` [PATCH 8/9] lei rm|tag: drop redundant mbox+net callbacks Eric Wong
5 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-10-26 10:35 UTC (permalink / raw)
To: meta
The LeiInput backend now allows p2q to work like any other
command which reads .eml, .patch, mbox*, Maildir, IMAP, and NNTP
input. Running "git format-patch --stdout -1 $COMMIT" remains
supported.
This is intended to allow lower memory use while parsing
"git log --pretty=mboxrd -p" output. Previously, the entire
output of "git log" would be slurped into memory at once.
The intended use is to allow easy(-ish :P) searching for
unapplied patches as documented in the new example in the
manpage.
---
Documentation/lei-p2q.pod | 6 ++
lib/PublicInbox/LEI.pm | 4 +-
lib/PublicInbox/LeiInput.pm | 30 ++++++++--
lib/PublicInbox/LeiP2q.pm | 115 ++++++++++++++++++------------------
4 files changed, 89 insertions(+), 66 deletions(-)
diff --git a/Documentation/lei-p2q.pod b/Documentation/lei-p2q.pod
index 2e0b1ab676ed..4bc5d25f8ef0 100644
--- a/Documentation/lei-p2q.pod
+++ b/Documentation/lei-p2q.pod
@@ -77,6 +77,12 @@ Suppress feedback messages.
# to view results on a remote HTTP(S) public-inbox instance
$BROWSER https://example.com/pub-inbox/?q=$(lei p2q --uri $COMMIT_OID)
+ # to view unapplied patches for a given $FILE from the past year:
+ echo \( rt:last.year.. AND dfn:$FILE \) AND NOT \( \
+ $(git log -p --pretty=mboxrd --since=last.year $FILE |
+ lei p2q -F mboxrd )
+ \) | lei q -o /tmp/unapplied
+
=head1 CONTACT
Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org>
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 93a7b426de3c..f9d24f29c87d 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -274,9 +274,9 @@ our %CMD = ( # sorted in order of importance/use:
'one-time conversion from URL or filesystem to another format',
qw(stdin| in-format|F=s out-format|f=s output|mfolder|o=s lock=s@ kw!),
@net_opt, @c_opt ],
-'p2q' => [ 'FILE|COMMIT_OID|--stdin',
+'p2q' => [ 'LOCATION_OR_COMMIT...|--stdin',
"use a patch to generate a query for `lei q --stdin'",
- qw(stdin| want|w=s@ uri debug), @c_opt ],
+ qw(stdin| in-format|F=s want|w=s@ uri debug), @net_opt, @c_opt ],
'config' => [ '[...]', sub {
'git-config(1) wrapper for '._config_path($_[0]);
}, qw(config-file|system|global|file|f=s), # for conflict detection
diff --git a/lib/PublicInbox/LeiInput.pm b/lib/PublicInbox/LeiInput.pm
index 2621fc1f9d05..540681e3ff6b 100644
--- a/lib/PublicInbox/LeiInput.pm
+++ b/lib/PublicInbox/LeiInput.pm
@@ -64,6 +64,11 @@ sub input_mbox_cb { # base MboxReader callback
$self->input_eml_cb($eml);
}
+sub input_net_cb { # imap_each, nntp_each cb
+ my ($url, $uid, $kw, $eml, $self) = @_;
+ $self->input_eml_cb($eml);
+}
+
# import a single file handle of $name
# Subclass must define ->input_eml_cb and ->input_mbox_cb
sub input_fh {
@@ -108,10 +113,10 @@ sub handle_http_input ($$@) {
grep(/\A--compressed\z/, @$curl) or
$fh = IO::Uncompress::Gunzip->new($fh, MultiStream => 1);
eval { $self->input_fh('mboxrd', $fh, $url, @args) };
- my $err = $@;
+ my @err = ($@ ? $@ : ());
$ar->join;
- $? || $err and
- $lei->child_error($?, "@$cmd failed".$err ? " $err" : '');
+ push(@err, "\$?=$?") if $?;
+ $lei->child_error($?, "@$cmd failed: @err") if @err;
}
sub input_path_url {
@@ -184,7 +189,17 @@ EOM
$self, @args);
}
} elsif ($self->{missing_ok} && !-e $input) { # don't ->fail
- $self->folder_missing("$ifmt:$input");
+ if ($lei->{cmd} eq 'p2q') {
+ my $fp = [ qw(git format-patch --stdout -1), $input ];
+ my $rdr = { 2 => $lei->{2} };
+ my $fh = popen_rd($fp, undef, $rdr);
+ eval { $self->input_fh('eml', $fh, $input, @args) };
+ my @err = ($@ ? $@ : ());
+ close($fh) or push @err, "\$?=$?";
+ $lei->child_error($?, "@$fp failed: @err") if @err;
+ } else {
+ $self->folder_missing("$ifmt:$input");
+ }
} else {
$lei->fail("$ifmt_pfx$input unsupported (TODO)");
}
@@ -330,9 +345,12 @@ $input is `eml', not --in-format=$in_fmt
}
push @md, $input;
} elsif ($self->{missing_ok} && !-e $input) {
- # for lei rm-watch
- $may_sync and $input = 'maildir:'.
+ if ($lei->{cmd} eq 'p2q') {
+ # will run "git format-patch"
+ } elsif ($may_sync) { # for lei rm-watch
+ $input = 'maildir:'.
$lei->abs_path($input);
+ }
} else {
return $lei->fail("Unable to handle $input")
}
diff --git a/lib/PublicInbox/LeiP2q.pm b/lib/PublicInbox/LeiP2q.pm
index 08ec81c5295e..09ec0a079bb9 100644
--- a/lib/PublicInbox/LeiP2q.pm
+++ b/lib/PublicInbox/LeiP2q.pm
@@ -1,16 +1,16 @@
-# Copyright (C) 2021 all contributors <meta@public-inbox.org>
+# Copyright (C) all contributors <meta@public-inbox.org>
# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
# front-end for the "lei patch-to-query" sub-command
package PublicInbox::LeiP2q;
use strict;
use v5.10.1;
-use parent qw(PublicInbox::IPC);
+use parent qw(PublicInbox::IPC PublicInbox::LeiInput);
use PublicInbox::Eml;
use PublicInbox::Smsg;
use PublicInbox::MsgIter qw(msg_part_text);
use PublicInbox::Git qw(git_unquote);
-use PublicInbox::Spawn qw(popen_rd);
+use PublicInbox::OnDestroy;
use URI::Escape qw(uri_escape_utf8);
my $FN = qr!((?:"?[^/\n]+/[^\r\n]+)|/dev/null)!;
@@ -28,8 +28,16 @@ sub xphrase ($) {
} ($s =~ m!(\w[\|=><,\./:\\\@\-\w\s]+)!g);
}
+sub add_qterm ($$@) {
+ my ($self, $p, @v) = @_;
+ for (@v) {
+ $self->{qseen}->{"$p\0$_"} //=
+ push(@{$self->{qterms}->{$p}}, $_);
+ }
+}
+
sub extract_terms { # eml->each_part callback
- my ($p, $lei) = @_;
+ my ($p, $self) = @_;
my $part = $p->[0]; # ignore $depth and @idx;
my $ct = $part->content_type || 'text/plain';
my ($s, undef) = msg_part_text($part, $ct);
@@ -38,7 +46,7 @@ sub extract_terms { # eml->each_part callback
# TODO: b: nq: q:
for (split(/\n/, $s)) {
if ($in_diff && s/^ //) { # diff context
- push @{$lei->{qterms}->{dfctx}}, xphrase($_);
+ add_qterm($self, 'dfctx', xphrase($_));
} elsif (/^-- $/) { # email signature begins
$in_diff = undef;
} elsif (m!^diff --git $FN $FN!) {
@@ -46,21 +54,21 @@ sub extract_terms { # eml->each_part callback
$in_diff = 1;
} elsif (/^index ([a-f0-9]+)\.\.([a-f0-9]+)\b/) {
my ($oa, $ob) = ($1, $2);
- push @{$lei->{qterms}->{dfpre}}, $oa;
- push @{$lei->{qterms}->{dfpost}}, $ob;
+ add_qterm($self, 'dfpre', $oa);
+ add_qterm($self, 'dfpost', $ob);
# who uses dfblob?
} elsif (m!^(?:---|\+{3}) ($FN)!) {
next if $1 eq '/dev/null';
my $fn = (split(m!/!, git_unquote($1.''), 2))[1];
- push @{$lei->{qterms}->{dfn}}, xphrase($fn);
+ add_qterm($self, 'dfn', xphrase($fn));
} elsif ($in_diff && s/^\+//) { # diff added
- push @{$lei->{qterms}->{dfb}}, xphrase($_);
+ add_qterm($self, 'dfb', xphrase($_));
} elsif ($in_diff && s/^-//) { # diff removed
- push @{$lei->{qterms}->{dfa}}, xphrase($_);
+ add_qterm($self, 'dfa', xphrase($_));
} elsif (/^@@ (?:\S+) (?:\S+) @@\s*$/) {
# traditional diff w/o -p
} elsif (/^@@ (?:\S+) (?:\S+) @@\s*(\S+.*)/) {
- push @{$lei->{qterms}->{dfhh}}, xphrase($1);
+ add_qterm($self, 'dfhh', xphrase($1));
} elsif (/^(?:dis)similarity index/ ||
/^(?:old|new) mode/ ||
/^(?:deleted|new) file mode/ ||
@@ -92,53 +100,43 @@ my %pfx2smsg = (
rt => [ qw(ts) ], # ditto...
);
-sub do_p2q { # via wq_do
- my ($self) = @_;
- my $lei = $self->{lei};
- my $want = $lei->{opt}->{want} // [ qw(dfpost7) ];
- my @want = split(/[, ]+/, "@$want");
- for (@want) {
- /\A(?:(d|dt|rt):)?([0-9]+)(\.(?:day|weeks)s?)?\z/ or next;
- my ($pfx, $n, $unit) = ($1, $2, $3);
- $n *= 86400 * ($unit =~ /week/i ? 7 : 1);
- $_ = [ $pfx, $n ];
- }
- my $smsg = bless {}, 'PublicInbox::Smsg';
- my $in = $self->{0};
- my @cmd;
- unless ($in) {
- my $input = $self->{input};
- my $devfd = $lei->path_to_fd($input) // return;
- if ($devfd >= 0) {
- $in = $lei->{$devfd};
- } elsif (-e $input) {
- open($in, '<', $input) or
- return $lei->fail("open < $input: $!");
- } else {
- @cmd = (qw(git format-patch --stdout -1), $input);
- $in = popen_rd(\@cmd, undef, { 2 => $lei->{2} });
+sub input_eml_cb { # used by PublicInbox::LeiInput::input_fh
+ my ($self, $eml) = @_;
+ my $diff_want = $self->{diff_want} // do {
+ my $want = $self->{lei}->{opt}->{want} // [ qw(dfpost7) ];
+ my @want = split(/[, ]+/, "@$want");
+ for (@want) {
+ /\A(?:(d|dt|rt):)?([0-9]+)(\.(?:day|weeks)s?)?\z/
+ or next;
+ my ($pfx, $n, $unit) = ($1, $2, $3);
+ $n *= 86400 * ($unit =~ /week/i ? 7 : 1);
+ $_ = [ $pfx, $n ];
}
+ $self->{want_order} = \@want;
+ $self->{diff_want} = +{ map { $_ => 1 } @want };
};
- my $str = do { local $/; <$in> };
- @cmd && !close($in) and return $lei->fail("E: @cmd failed: $?");
- my $eml = PublicInbox::Eml->new(\$str);
- $lei->{diff_want} = +{ map { $_ => 1 } @want };
+ my $smsg = bless {}, 'PublicInbox::Smsg';
$smsg->populate($eml);
while (my ($pfx, $fields) = each %pfx2smsg) {
- next unless $lei->{diff_want}->{$pfx};
+ next unless $diff_want->{$pfx};
for my $f (@$fields) {
my $v = $smsg->{$f} // next;
- push @{$lei->{qterms}->{$pfx}}, xphrase($v);
+ add_qterm($self, $pfx, xphrase($v));
}
}
- $eml->each_part(\&extract_terms, $lei, 1);
+ $eml->each_part(\&extract_terms, $self, 1);
+}
+
+sub emit_query {
+ my ($self) = @_;
+ my $lei = $self->{lei};
if ($lei->{opt}->{debug}) {
my $json = ref(PublicInbox::Config->json)->new;
$json->utf8->canonical->pretty;
- print { $lei->{2} } $json->encode($lei->{qterms});
+ print { $lei->{2} } $json->encode($self->{qterms});
}
my (@q, %seen);
- for my $pfx (@want) {
+ for my $pfx (@{$self->{want_order}}) {
if (ref($pfx) eq 'ARRAY') {
my ($p, $t_range) = @$pfx; # TODO
@@ -148,7 +146,7 @@ sub do_p2q { # via wq_do
} else {
my $plusminus = ($pfx =~ s/\A([\+\-])//) ? $1 : '';
my $end = ($pfx =~ s/([0-9\*]+)\z//) ? $1 : '';
- my $x = delete($lei->{qterms}->{$pfx}) or next;
+ my $x = delete($self->{qterms}->{$pfx}) or next;
my $star = $end =~ tr/*//d ? '*' : '';
my $min_len = ($end || 0) + 0;
@@ -181,24 +179,25 @@ sub do_p2q { # via wq_do
}
sub lei_p2q { # the "lei patch-to-query" entry point
- my ($lei, $input) = @_;
- my $self = bless {}, __PACKAGE__;
- if ($lei->{opt}->{stdin}) {
- $self->{0} = delete $lei->{0}; # guard from _lei_atfork_child
- } else {
- $self->{input} = $input;
- }
- my ($op_c, $ops) = $lei->workers_start($self, 1);
+ my ($lei, @inputs) = @_;
+ $lei->{opt}->{'in-format'} //= 'eml' if $lei->{opt}->{stdin};
+ my $self = bless { missing_ok => 1 }, __PACKAGE__;
+ $self->prepare_inputs($lei, \@inputs) or return;
+ my $ops = {};
+ $lei->{auth}->op_merge($ops, $self, $lei) if $lei->{auth};
+ (my $op_c, $ops) = $lei->workers_start($self, 1, $ops);
$lei->{wq1} = $self;
- $self->wq_io_do('do_p2q', []);
- $self->wq_close;
+ net_merge_all_done($self) unless $lei->{auth};
$lei->wait_wq_events($op_c, $ops);
}
sub ipc_atfork_child {
my ($self) = @_;
- $self->{lei}->_lei_atfork_child;
- $self->SUPER::ipc_atfork_child;
+ PublicInbox::LeiInput::input_only_atfork_child($self);
+ PublicInbox::OnDestroy->new($$, \&emit_query, $self);
}
+no warnings 'once';
+*net_merge_all_done = \&PublicInbox::LeiInput::input_only_net_merge_all_done;
+
1;
^ permalink raw reply related [relevance 35%]
* Re: [PATCH] lei p2q: document --uri, add examples
2021-10-25 19:31 64% [PATCH] lei p2q: document --uri, add examples Eric Wong
@ 2021-10-25 23:39 71% ` Kyle Meyer
0 siblings, 0 replies; 200+ results
From: Kyle Meyer @ 2021-10-25 23:39 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
Eric Wong writes:
> This is useful for users lacking in local storage. Also,
> referencing lei-add-external(1) seems to make less sense
> than referencing lei-q(1).
True, not sure why I went with lei-add-external there (probably forgot
to update that section after copying over an existing lei manpage as a
template).
> We'll also stop dropping years from the copyright statement
> to reduce future churn.
s/stop/start/ ?
^ permalink raw reply [relevance 71%]
* [PATCH] lei p2q: document --uri, add examples
@ 2021-10-25 19:31 64% Eric Wong
2021-10-25 23:39 71% ` Kyle Meyer
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2021-10-25 19:31 UTC (permalink / raw)
To: meta
This is useful for users lacking in local storage. Also,
referencing lei-add-external(1) seems to make less sense
than referencing lei-q(1).
We'll also stop dropping years from the copyright statement
to reduce future churn.
---
Documentation/lei-p2q.pod | 17 ++++++++++++++---
lib/PublicInbox/LEI.pm | 1 +
2 files changed, 15 insertions(+), 3 deletions(-)
diff --git a/Documentation/lei-p2q.pod b/Documentation/lei-p2q.pod
index 1068ff0beb27..2e0b1ab676ed 100644
--- a/Documentation/lei-p2q.pod
+++ b/Documentation/lei-p2q.pod
@@ -57,6 +57,10 @@ Dump output that shows the information collected for every prefix.
This information can be useful for seeing how a patch is processed,
but the format should not be considered stable.
+=item --uri
+
+URI escape output for interacting with HTTP(S) public-inbox instances.
+
=item -q
=item --quiet
@@ -65,6 +69,14 @@ Suppress feedback messages.
=back
+=head1 EXAMPLES
+
+ # to search for all threads which touch a given thread:
+ lei p2q $COMMIT_OID | lei q -t -o /tmp/results
+
+ # to view results on a remote HTTP(S) public-inbox instance
+ $BROWSER https://example.com/pub-inbox/?q=$(lei p2q --uri $COMMIT_OID)
+
=head1 CONTACT
Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org>
@@ -74,11 +86,10 @@ L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/>
=head1 COPYRIGHT
-Copyright 2021 all contributors L<mailto:meta@public-inbox.org>
+Copyright all contributors L<mailto:meta@public-inbox.org>
License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>
-
=head1 SEE ALSO
-L<lei-add-external(1)>
+L<lei-q(1)>
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index d24b8e2d6da2..e35339c16d8d 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -355,6 +355,7 @@ my %OPTDESC = (
'want|w=s@' => [ 'PREFIX|dfpost|dfn', # common ones in help...
'search prefixes to extract (default: dfpost7)' ],
+'uri p2q' => [ 'URI escape output' ],
'alert=s@' => ['CMD,:WINCH,:bell,<any command>',
'run command(s) or perform ops when done writing to output ' .
^ permalink raw reply related [relevance 64%]
* [PATCH 1/7] lei: always pass $lei to LeiAuth->op_merge
@ 2021-10-24 0:20 75% ` Eric Wong
2021-10-24 0:20 61% ` [PATCH 2/7] lei export-kw: skip read-only IMAP folders Eric Wong
1 sibling, 0 replies; 200+ results
From: Eric Wong @ 2021-10-24 0:20 UTC (permalink / raw)
To: meta
This will make future developments easier.
---
lib/PublicInbox/LeiExportKw.pm | 2 +-
lib/PublicInbox/LeiForgetSearch.pm | 2 +-
lib/PublicInbox/LeiImport.pm | 2 +-
lib/PublicInbox/LeiLsMailSource.pm | 2 +-
lib/PublicInbox/LeiMailDiff.pm | 2 +-
lib/PublicInbox/LeiRefreshMailSync.pm | 2 +-
lib/PublicInbox/LeiTag.pm | 2 +-
lib/PublicInbox/LeiXSearch.pm | 2 +-
8 files changed, 8 insertions(+), 8 deletions(-)
diff --git a/lib/PublicInbox/LeiExportKw.pm b/lib/PublicInbox/LeiExportKw.pm
index 5be9e51f..756d0e9c 100644
--- a/lib/PublicInbox/LeiExportKw.pm
+++ b/lib/PublicInbox/LeiExportKw.pm
@@ -115,7 +115,7 @@ EOM
$self->{nwr}->{-skip_creat} = 1;
}
my $ops = {};
- $lei->{auth}->op_merge($ops, $self) if $lei->{auth};
+ $lei->{auth}->op_merge($ops, $self, $lei) if $lei->{auth};
(my $op_c, $ops) = $lei->workers_start($self, 1, $ops);
$lei->{wq1} = $self;
$lei->{-err_type} = 'non-fatal';
diff --git a/lib/PublicInbox/LeiForgetSearch.pm b/lib/PublicInbox/LeiForgetSearch.pm
index f353fe52..dfeb0293 100644
--- a/lib/PublicInbox/LeiForgetSearch.pm
+++ b/lib/PublicInbox/LeiForgetSearch.pm
@@ -46,7 +46,7 @@ sub lei_forget_search {
$self->prepare_inputs($lei, $self->{o_remote}) or return;
}
my $ops = {};
- $lei->{auth}->op_merge($ops, $self) if $lei->{auth};
+ $lei->{auth}->op_merge($ops, $self, $lei) if $lei->{auth};
(my $op_c, $ops) = $lei->workers_start($self, 1, $ops);
$lei->{wq1} = $self;
net_merge_all_done($self) unless $lei->{auth};
diff --git a/lib/PublicInbox/LeiImport.pm b/lib/PublicInbox/LeiImport.pm
index 2f8fd6c6..d8f39fdf 100644
--- a/lib/PublicInbox/LeiImport.pm
+++ b/lib/PublicInbox/LeiImport.pm
@@ -103,7 +103,7 @@ sub do_import_index ($$@) {
($lei->{opt}->{'new-only'} && (!$net || !$net->{imap_order})) and
warn "# --new-only is only for IMAP\n";
my $ops = {};
- $lei->{auth}->op_merge($ops, $self) if $lei->{auth};
+ $lei->{auth}->op_merge($ops, $self, $lei) if $lei->{auth};
$lei->{-eml_noisy} = 1;
(my $op_c, $ops) = $lei->workers_start($self, $j, $ops);
$lei->{wq1} = $self;
diff --git a/lib/PublicInbox/LeiLsMailSource.pm b/lib/PublicInbox/LeiLsMailSource.pm
index 7c3c0cda..5eb7032d 100644
--- a/lib/PublicInbox/LeiLsMailSource.pm
+++ b/lib/PublicInbox/LeiLsMailSource.pm
@@ -96,7 +96,7 @@ sub lei_ls_mail_source {
}
$lei->start_pager if $isatty;
my $ops = {};
- $lei->{auth}->op_merge($ops, $self);
+ $lei->{auth}->op_merge($ops, $self, $lei);
(my $op_c, $ops) = $lei->workers_start($self, 1, $ops);
$lei->{wq1} = $self;
$lei->{-err_type} = 'non-fatal';
diff --git a/lib/PublicInbox/LeiMailDiff.pm b/lib/PublicInbox/LeiMailDiff.pm
index a29ae225..4f3a4608 100644
--- a/lib/PublicInbox/LeiMailDiff.pm
+++ b/lib/PublicInbox/LeiMailDiff.pm
@@ -83,7 +83,7 @@ sub lei_mail_diff {
$lei->{opt}->{color} //= $isatty;
$lei->start_pager if $isatty;
my $ops = {};
- $lei->{auth}->op_merge($ops, $self) if $lei->{auth};
+ $lei->{auth}->op_merge($ops, $self, $lei) if $lei->{auth};
(my $op_c, $ops) = $lei->workers_start($self, 1, $ops);
$lei->{wq1} = $self;
$lei->{-err_type} = 'non-fatal';
diff --git a/lib/PublicInbox/LeiRefreshMailSync.pm b/lib/PublicInbox/LeiRefreshMailSync.pm
index 0cb9f3dc..f516f572 100644
--- a/lib/PublicInbox/LeiRefreshMailSync.pm
+++ b/lib/PublicInbox/LeiRefreshMailSync.pm
@@ -82,7 +82,7 @@ EOM
$lei->{opt}->{'mail-sync'} = 1; # for prepare_inputs
$self->prepare_inputs($lei, \@folders) or return;
my $ops = {};
- $lei->{auth}->op_merge($ops, $self) if $lei->{auth};
+ $lei->{auth}->op_merge($ops, $self, $lei) if $lei->{auth};
(my $op_c, $ops) = $lei->workers_start($self, 1, $ops);
$lei->{wq1} = $self;
$lei->{-err_type} = 'non-fatal';
diff --git a/lib/PublicInbox/LeiTag.pm b/lib/PublicInbox/LeiTag.pm
index 817b87f8..77654d1a 100644
--- a/lib/PublicInbox/LeiTag.pm
+++ b/lib/PublicInbox/LeiTag.pm
@@ -49,7 +49,7 @@ sub lei_tag { # the "lei tag" method
grep(defined, @$vmd_mod{qw(+kw +L -L -kw)}) or
return $lei->fail('no keywords or labels specified');
my $ops = {};
- $lei->{auth}->op_merge($ops, $self) if $lei->{auth};
+ $lei->{auth}->op_merge($ops, $self, $lei) if $lei->{auth};
(my $op_c, $ops) = $lei->workers_start($self, 1, $ops);
$lei->{wq1} = $self;
$lei->{-err_type} = 'non-fatal';
diff --git a/lib/PublicInbox/LeiXSearch.pm b/lib/PublicInbox/LeiXSearch.pm
index 119070a2..acc36897 100644
--- a/lib/PublicInbox/LeiXSearch.pm
+++ b/lib/PublicInbox/LeiXSearch.pm
@@ -548,7 +548,7 @@ sub do_query {
'child_error' => [ $lei ],
'incr_start_query' => [ $self, $lei ],
};
- $lei->{auth}->op_merge($ops, $l2m) if $l2m && $lei->{auth};
+ $lei->{auth}->op_merge($ops, $l2m, $lei) if $l2m && $lei->{auth};
my $end = $lei->pkt_op_pair;
$lei->{1}->autoflush(1);
$lei->start_pager if delete $lei->{need_pager};
^ permalink raw reply related [relevance 75%]
* [PATCH 2/7] lei export-kw: skip read-only IMAP folders
2021-10-24 0:20 75% ` [PATCH 1/7] lei: always pass $lei to LeiAuth->op_merge Eric Wong
@ 2021-10-24 0:20 61% ` Eric Wong
1 sibling, 0 replies; 200+ results
From: Eric Wong @ 2021-10-24 0:20 UTC (permalink / raw)
To: meta
Since we want to store IMAP flags asynchronously and not wait
for results, we can't check for IMAP errors this way and end up
wasting bandwidth on public-inbox-imapd. Now, we just check
PERMANENTFLAGS up front to ensure a folder can handle IMAP flag
storage before proceeding.
---
lib/PublicInbox/LeiExportKw.pm | 12 +++++++++++-
lib/PublicInbox/NetWriter.pm | 9 +++++++++
t/lei-export-kw.t | 2 +-
t/lei-import-imap.t | 3 +++
4 files changed, 24 insertions(+), 2 deletions(-)
diff --git a/lib/PublicInbox/LeiExportKw.pm b/lib/PublicInbox/LeiExportKw.pm
index 756d0e9c..0ecfb782 100644
--- a/lib/PublicInbox/LeiExportKw.pm
+++ b/lib/PublicInbox/LeiExportKw.pm
@@ -67,7 +67,17 @@ sub input_path_url {
$self->{lms}->each_src($input, \&export_kw_md, $self, $mdir);
} elsif ($input =~ m!\Aimaps?://!i) {
my $uri = PublicInbox::URIimap->new($input);
- if (my $mic = $self->{nwr}->mic_for_folder($uri)) {
+ my $mic = $self->{nwr}->mic_for_folder($uri);
+ if ($mic && !$self->{nwr}->can_store_flags($mic)) {
+ my $m = "$input does not support PERMANENTFLAGS";
+ if (defined $self->{lei}->{opt}->{all}) {
+ $self->{lei}->qerr("# $m");
+ } else { # set error code if user explicitly requested
+ $self->{lei}->child_error(0, "E: $m");
+ }
+ return;
+ }
+ if ($mic) {
$self->{lms}->each_src($$uri, \&export_kw_imap,
$self, $mic);
$mic->expunge;
diff --git a/lib/PublicInbox/NetWriter.pm b/lib/PublicInbox/NetWriter.pm
index 629a752a..4a1f34f6 100644
--- a/lib/PublicInbox/NetWriter.pm
+++ b/lib/PublicInbox/NetWriter.pm
@@ -56,4 +56,13 @@ sub imap_set_kw {
$mic; # caller must ->expunge
}
+sub can_store_flags {
+ my ($self, $mic) = @_;
+ for ($mic->Results) {
+ /^\* OK \[PERMANENTFLAGS \(([^\)]*)\)\].*/ and
+ return $self->can('perm_fl_ok')->($1);
+ }
+ undef;
+}
+
1;
diff --git a/t/lei-export-kw.t b/t/lei-export-kw.t
index 55730e87..88b2a80b 100644
--- a/t/lei-export-kw.t
+++ b/t/lei-export-kw.t
@@ -4,7 +4,7 @@
use strict; use v5.10.1; use PublicInbox::TestCommon;
use File::Copy qw(cp);
use File::Path qw(make_path);
-require_mods(qw(lei -imapd Mail::IMAPClient));
+require_mods(qw(lei)); # see lei-import-imap.t for IMAP tests
my ($tmpdir, $for_destroy) = tmpdir;
my $expect = eml_load('t/data/0001.patch');
my $do_export_kw = 1;
diff --git a/t/lei-import-imap.t b/t/lei-import-imap.t
index 315567b3..3b6cb299 100644
--- a/t/lei-import-imap.t
+++ b/t/lei-import-imap.t
@@ -110,6 +110,9 @@ test_lei({ tmpdir => $tmpdir }, sub {
is(scalar(@$out), 2, 'got JSON') or diag explain($out);
lei_ok qw(lcat), $url_orig;
is($lei_out, $orig, 'lcat w/o UID works');
+
+ ok(!lei(qw(export-kw), $url_orig), 'export-kw fails on read-only IMAP');
+ like($lei_err, qr/does not support/, 'error noted in failure');
});
done_testing;
^ permalink raw reply related [relevance 61%]
* Re: [PATCH] doc: lei-forget-search: fix option name in --prune description
2021-10-23 0:22 71% [PATCH] doc: lei-forget-search: fix option name in --prune description Kyle Meyer
@ 2021-10-23 2:54 71% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-10-23 2:54 UTC (permalink / raw)
To: Kyle Meyer; +Cc: meta
Thanks, pushed as 0e22868f2c5ad03193ae077d17fdad64b51fdac2
^ permalink raw reply [relevance 71%]
* [PATCH] doc: lei-forget-search: fix option name in --prune description
@ 2021-10-23 0:22 71% Kyle Meyer
2021-10-23 2:54 71% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Kyle Meyer @ 2021-10-23 0:22 UTC (permalink / raw)
To: meta
Fixes: 6f8e16a266b30819 ("lei forget-search: support --prune=<local|remote>")
---
Documentation/lei-forget-search.pod | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/Documentation/lei-forget-search.pod b/Documentation/lei-forget-search.pod
index 3a7f493c..adbe7638 100644
--- a/Documentation/lei-forget-search.pod
+++ b/Documentation/lei-forget-search.pod
@@ -17,8 +17,8 @@ Forget a saved search at C<OUTPUT>.
=item --prune[=<local|remote>]
C<--prune> will forget saved searches if the C<OUTPUT> no longer
-exists. C<--all=local> only prunes local mailboxes,
-C<--all=remote> only prunes remote mailboxes (currently
+exists. C<--prune=local> only prunes local mailboxes,
+C<--prune=remote> only prunes remote mailboxes (currently
C<imap://> and C<imaps://>).
=back
base-commit: 6f8e16a266b30819ff74c40bc532f8c3f4a9f4b7
--
2.33.1
^ permalink raw reply related [relevance 71%]
* [PATCH 0/3] lei: some cleanup stuff
@ 2021-10-22 8:22 71% Eric Wong
2021-10-22 8:22 60% ` [PATCH 1/3] lei export-kw: don't recreate deleted IMAP folders Eric Wong
` (2 more replies)
0 siblings, 3 replies; 200+ results
From: Eric Wong @ 2021-10-22 8:22 UTC (permalink / raw)
To: meta
Sometimes users delete stuff, and ordering shouldn't matter.
"lei fsck" should be easier to deal with if users have more
cleanup mechanisms available to them.
Eric Wong (3):
lei export-kw: don't recreate deleted IMAP folders
lei export-kw: completion returns all Maildir+IMAP
lei forget-search: support --prune=<local|remote>
Documentation/lei-forget-search.pod | 13 ++++++++
lib/PublicInbox/LEI.pm | 4 +--
lib/PublicInbox/LeiExportKw.pm | 13 +++++---
lib/PublicInbox/LeiForgetSearch.pm | 52 +++++++++++++++++++++++++++--
lib/PublicInbox/LeiUp.pm | 26 +++++++++------
lib/PublicInbox/NetReader.pm | 1 +
t/lei-q-save.t | 7 ++++
xt/net_writer-imap.t | 15 +++++++--
8 files changed, 110 insertions(+), 21 deletions(-)
^ permalink raw reply [relevance 71%]
* [PATCH 2/3] lei export-kw: completion returns all Maildir+IMAP
2021-10-22 8:22 71% [PATCH 0/3] lei: some cleanup stuff Eric Wong
2021-10-22 8:22 60% ` [PATCH 1/3] lei export-kw: don't recreate deleted IMAP folders Eric Wong
@ 2021-10-22 8:22 71% ` Eric Wong
2021-10-22 8:22 47% ` [PATCH 3/3] lei forget-search: support --prune=<local|remote> Eric Wong
2 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-10-22 8:22 UTC (permalink / raw)
To: meta
It's theoretically possible an AUTH=ANONYMOUS login could be
writable and allowed to store flags for various people (e.g.
within a private network).
---
lib/PublicInbox/LeiExportKw.pm | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/lib/PublicInbox/LeiExportKw.pm b/lib/PublicInbox/LeiExportKw.pm
index 12c8f4067e9e..5be9e51ff311 100644
--- a/lib/PublicInbox/LeiExportKw.pm
+++ b/lib/PublicInbox/LeiExportKw.pm
@@ -128,7 +128,7 @@ sub _complete_export_kw {
my $lms = $lei->lms or return ();
my $match_cb = $lei->complete_url_prepare(\@argv);
# filter-out read-only sources:
- my @k = grep(!m!(?://;AUTH=ANONYMOUS\@|\A(?:nntps?|s?news)://)!,
+ my @k = grep(m!(?:maildir|imaps?):!,
$lms->folders($argv[-1] // undef, 1));
my @m = map { $match_cb->($_) } @k;
@m ? @m : @k;
^ permalink raw reply related [relevance 71%]
* [PATCH 1/3] lei export-kw: don't recreate deleted IMAP folders
2021-10-22 8:22 71% [PATCH 0/3] lei: some cleanup stuff Eric Wong
@ 2021-10-22 8:22 60% ` Eric Wong
2021-10-22 8:22 71% ` [PATCH 2/3] lei export-kw: completion returns all Maildir+IMAP Eric Wong
2021-10-22 8:22 47% ` [PATCH 3/3] lei forget-search: support --prune=<local|remote> Eric Wong
2 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-10-22 8:22 UTC (permalink / raw)
To: meta
In case an IMAP folder is deleted, just set an error and
ignore it rather than creating an empty folder which we
attempt to export keywords to for non-existent messages.
---
lib/PublicInbox/LeiExportKw.pm | 11 ++++++++---
lib/PublicInbox/NetReader.pm | 1 +
xt/net_writer-imap.t | 15 +++++++++++++--
3 files changed, 22 insertions(+), 5 deletions(-)
diff --git a/lib/PublicInbox/LeiExportKw.pm b/lib/PublicInbox/LeiExportKw.pm
index ceeef7f21d54..12c8f4067e9e 100644
--- a/lib/PublicInbox/LeiExportKw.pm
+++ b/lib/PublicInbox/LeiExportKw.pm
@@ -67,9 +67,13 @@ sub input_path_url {
$self->{lms}->each_src($input, \&export_kw_md, $self, $mdir);
} elsif ($input =~ m!\Aimaps?://!i) {
my $uri = PublicInbox::URIimap->new($input);
- my $mic = $self->{nwr}->mic_for_folder($uri);
- $self->{lms}->each_src($$uri, \&export_kw_imap, $self, $mic);
- $mic->expunge;
+ if (my $mic = $self->{nwr}->mic_for_folder($uri)) {
+ $self->{lms}->each_src($$uri, \&export_kw_imap,
+ $self, $mic);
+ $mic->expunge;
+ } else {
+ $self->{lei}->child_error(0, "$input unavailable: $@");
+ }
} else { die "BUG: $input not supported" }
}
@@ -108,6 +112,7 @@ EOM
$self->{nwr} = bless $net, 'PublicInbox::NetWriter';
$self->{imap_mod_kw} = $net->can($self->{-merge_kw} ?
'imap_add_kw' : 'imap_set_kw');
+ $self->{nwr}->{-skip_creat} = 1;
}
my $ops = {};
$lei->{auth}->op_merge($ops, $self) if $lei->{auth};
diff --git a/lib/PublicInbox/NetReader.pm b/lib/PublicInbox/NetReader.pm
index 4da19ab969b5..032b4fda557c 100644
--- a/lib/PublicInbox/NetReader.pm
+++ b/lib/PublicInbox/NetReader.pm
@@ -359,6 +359,7 @@ sub imap_common_init ($;$) {
mic_for($self, $uri, $mic_common, $lei) //
die "Unable to continue\n";
next unless $self->isa('PublicInbox::NetWriter');
+ next if $self->{-skip_creat};
my $dst = $orig_uri->mailbox // next;
next if $mic->exists($dst); # already exists
$mic->create($dst) or die "CREATE $dst failed <$orig_uri>: $@";
diff --git a/xt/net_writer-imap.t b/xt/net_writer-imap.t
index cb2ea61ff8e3..afa4bcc3e881 100644
--- a/xt/net_writer-imap.t
+++ b/xt/net_writer-imap.t
@@ -83,8 +83,11 @@ my $mics = do {
};
my $mic = (values %$mics)[0];
my $cleanup = PublicInbox::OnDestroy->new($$, sub {
- my $mic = $nwr->mic_get($uri);
- $mic->delete($folder) or fail "delete $folder <$folder_uri>: $@";
+ if (defined($folder)) {
+ my $mic = $nwr->mic_get($uri);
+ $mic->delete($folder) or
+ fail "delete $folder <$folder_uri>: $@";
+ }
if ($tmpdir && -f "$tmpdir/.gitconfig") {
local $ENV{HOME} = $tmpdir;
system(qw(git credential-cache exit));
@@ -250,6 +253,14 @@ EOM
lei_ok qw(q m:testmessage --no-external -o), $folder_url;
lei_ok qw(up), $folder_url;
lei_ok qw(up --all=remote);
+ $mic = $nwr->mic_get($uri);
+ $mic->delete($folder) or fail "delete $folder <$folder_uri>: $@";
+ $mic->expunge;
+ undef $mic;
+ undef $folder;
+ ok(!lei(qw(export-kw), $folder_url),
+ 'export-kw fails w/ non-existent folder');
+
});
undef $cleanup; # remove temporary folder
^ permalink raw reply related [relevance 60%]
* [PATCH 3/3] lei forget-search: support --prune=<local|remote>
2021-10-22 8:22 71% [PATCH 0/3] lei: some cleanup stuff Eric Wong
2021-10-22 8:22 60% ` [PATCH 1/3] lei export-kw: don't recreate deleted IMAP folders Eric Wong
2021-10-22 8:22 71% ` [PATCH 2/3] lei export-kw: completion returns all Maildir+IMAP Eric Wong
@ 2021-10-22 8:22 47% ` Eric Wong
2 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-10-22 8:22 UTC (permalink / raw)
To: meta
Instead of:
lei forget-search $OUTPUT && rm -r $OUTPUT
we'll also allow a user to do:
rm -r $OUTPUT && lei forget-search --prune
This gives users flexibility to choose whatever flow
is most natural to them.
---
Documentation/lei-forget-search.pod | 13 ++++++++
lib/PublicInbox/LEI.pm | 4 +--
lib/PublicInbox/LeiForgetSearch.pm | 52 +++++++++++++++++++++++++++--
lib/PublicInbox/LeiUp.pm | 26 +++++++++------
t/lei-q-save.t | 7 ++++
5 files changed, 87 insertions(+), 15 deletions(-)
diff --git a/Documentation/lei-forget-search.pod b/Documentation/lei-forget-search.pod
index f3f043f93252..3a7f493cee8e 100644
--- a/Documentation/lei-forget-search.pod
+++ b/Documentation/lei-forget-search.pod
@@ -10,6 +10,19 @@ lei forget-search [OPTIONS] OUTPUT
Forget a saved search at C<OUTPUT>.
+=head1 OPTIONS
+
+=over
+
+=item --prune[=<local|remote>]
+
+C<--prune> will forget saved searches if the C<OUTPUT> no longer
+exists. C<--all=local> only prunes local mailboxes,
+C<--all=remote> only prunes remote mailboxes (currently
+C<imap://> and C<imaps://>).
+
+=back
+
=head1 CONTACT
Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org>
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 43baeeb3d51c..cb1e5433cc2d 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -222,8 +222,8 @@ our %CMD = ( # sorted in order of importance/use:
'ls-search' => [ '[PREFIX]', 'list saved search queries',
qw(format|f=s pretty l ascii z|0), @c_opt ],
-'forget-search' => [ 'OUTPUT...', 'forget a saved search',
- qw(verbose|v+), @c_opt ],
+'forget-search' => [ 'OUTPUT...|--prune', 'forget a saved search',
+ qw(verbose|v+ prune:s), @c_opt ],
'edit-search' => [ 'OUTPUT', "edit saved search via `git config --edit'",
@c_opt ],
'rm' => [ '--stdin|LOCATION...',
diff --git a/lib/PublicInbox/LeiForgetSearch.pm b/lib/PublicInbox/LeiForgetSearch.pm
index 0db9c75b8be3..f353fe52a958 100644
--- a/lib/PublicInbox/LeiForgetSearch.pm
+++ b/lib/PublicInbox/LeiForgetSearch.pm
@@ -5,12 +5,12 @@
package PublicInbox::LeiForgetSearch;
use strict;
use v5.10.1;
+use parent qw(PublicInbox::LeiUp);
use PublicInbox::LeiSavedSearch;
-use PublicInbox::LeiUp;
use File::Path ();
use SelectSaver;
-sub lei_forget_search {
+sub do_forget_search {
my ($lei, @outs) = @_;
my @dirs; # paths in ~/.local/share/lei/saved-search/
my $cwd;
@@ -30,9 +30,55 @@ sub lei_forget_search {
$save = SelectSaver->new($lei->{2});
}
File::Path::remove_tree(@dirs, $opt);
- $lei->fail if defined $cwd;
+ $lei->child_error(0) if defined $cwd;
+}
+
+sub lei_forget_search {
+ my ($lei, @outs) = @_;
+ my $prune = $lei->{opt}->{prune};
+ $prune // return do_forget_search($lei, @outs);
+ return $lei->fail("--prune and @outs incompatible") if @outs;
+ my @tmp = PublicInbox::LeiSavedSearch::list($lei);
+ my $self = bless { -mail_sync => 1 }, __PACKAGE__;
+ $self->filter_lss($lei, $prune) // return
+ $lei->fail("only --prune=$prune not understood");
+ if ($self->{o_remote}) { # setup lei->{auth}
+ $self->prepare_inputs($lei, $self->{o_remote}) or return;
+ }
+ my $ops = {};
+ $lei->{auth}->op_merge($ops, $self) if $lei->{auth};
+ (my $op_c, $ops) = $lei->workers_start($self, 1, $ops);
+ $lei->{wq1} = $self;
+ net_merge_all_done($self) unless $lei->{auth};
+ $lei->wait_wq_events($op_c, $ops);
+}
+
+sub do_prune {
+ my ($self) = @_;
+ my $lei = $self->{lei};
+ for my $o (@{$self->{o_local} // []}) {
+ next if -e $o;
+ $lei->qerr("# pruning $o");
+ eval { do_forget_search($lei, $o) };
+ $lei->child_error(0, "E: $@") if $@;
+ }
+ for my $o (@{$self->{o_remote} // []}) {
+ my $uri = PublicInbox::URIimap->new($o);
+ next if $lei->{net}->mic_for_folder($uri);
+ $lei->qerr("# pruning $uri");
+ eval { do_forget_search($lei, $o) };
+ $lei->child_error(0, "E: $@") if $@;
+ }
+}
+
+# called in top-level lei-daemon when LeiAuth is done
+sub net_merge_all_done {
+ my ($self) = @_;
+ $self->wq_do('do_prune');
+ $self->wq_close;
}
+*_wq_done_wait = \&PublicInbox::LEI::wq_done_wait;
*_complete_forget_search = \&PublicInbox::LeiUp::_complete_up;
1;
diff --git a/lib/PublicInbox/LeiUp.pm b/lib/PublicInbox/LeiUp.pm
index dac0fc287885..79639d5e62a4 100644
--- a/lib/PublicInbox/LeiUp.pm
+++ b/lib/PublicInbox/LeiUp.pm
@@ -93,6 +93,21 @@ sub redispatch_all ($$) {
}
}
+sub filter_lss {
+ my ($self, $lei, $all) = @_;
+ my @outs = PublicInbox::LeiSavedSearch::list($lei);
+ if ($all eq 'local') {
+ $self->{o_local} = [ grep(!/$REMOTE_RE/, @outs) ];
+ } elsif ($all eq 'remote') {
+ $self->{o_remote} = [ grep(/$REMOTE_RE/, @outs) ];
+ } elsif ($all eq '') {
+ $self->{o_remote} = [ grep(/$REMOTE_RE/, @outs) ];
+ $self->{o_local} = [ grep(!/$REMOTE_RE/, @outs) ];
+ } else {
+ undef;
+ }
+}
+
sub lei_up {
my ($lei, @outs) = @_;
my $opt = $lei->{opt};
@@ -101,17 +116,8 @@ sub lei_up {
return $lei->fail("--all and @outs incompatible") if @outs;
defined($opt->{mua}) and return
$lei->fail('--all and --mua= are incompatible');
- @outs = PublicInbox::LeiSavedSearch::list($lei);
- if ($all eq 'local') {
- $self->{o_local} = [ grep(!/$REMOTE_RE/, @outs) ];
- } elsif ($all eq 'remote') {
- $self->{o_remote} = [ grep(/$REMOTE_RE/, @outs) ];
- } elsif ($all eq '') {
- $self->{o_remote} = [ grep(/$REMOTE_RE/, @outs) ];
- $self->{o_local} = [ grep(!/$REMOTE_RE/, @outs) ];
- } else {
+ filter_lss($self, $lei, $all) // return
$lei->fail("only --all=$all not understood");
- }
} elsif ($lei->{lse}) { # redispatched
scalar(@outs) == 1 or die "BUG: lse set w/ >1 out[@outs]";
return up1($lei, $outs[0]);
diff --git a/t/lei-q-save.t b/t/lei-q-save.t
index 05d5d9f4436c..cd35461ce1b7 100644
--- a/t/lei-q-save.t
+++ b/t/lei-q-save.t
@@ -4,6 +4,7 @@
use strict; use v5.10.1; use PublicInbox::TestCommon;
use PublicInbox::Smsg;
use List::Util qw(sum);
+use File::Path qw(remove_tree);
my $doc1 = eml_load('t/plack-qp.eml');
$doc1->header_set('Date', PublicInbox::Smsg::date({ds => time - (86400 * 5)}));
@@ -233,5 +234,11 @@ test_lei(sub {
and xbail "-ipe $lss[0]: $?";
lei_ok qw(ls-search);
is($lei_err, '', 'no errors w/ fixed config');
+
+ like($lei_out, qr!\Q$home/after\E!, "`after' in ls-search");
+ remove_tree("$home/after");
+ lei_ok qw(forget-search --prune);
+ lei_ok qw(ls-search);
+ unlike($lei_out, qr!\Q$home/after\E!, "`after' pruned");
});
done_testing;
^ permalink raw reply related [relevance 47%]
* [PATCH 03/15] t/lei-p2q: extra diagnostics
2021-10-21 21:10 57% ` [PATCH 01/15] t/lei-{auto-watch,export-kw}: extra diagnostics on failure Eric Wong
2021-10-21 21:10 71% ` [PATCH 02/15] t/lei-import-maildir: rename fix (SR -> RS) Eric Wong
@ 2021-10-21 21:10 71% ` Eric Wong
2021-10-21 21:10 71% ` [PATCH 04/15] lei/store: check for any unexpected process death Eric Wong
` (7 subsequent siblings)
10 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-10-21 21:10 UTC (permalink / raw)
To: meta
I got one mysterious test failure here, once, and can't seem
to reproduce it...
---
t/lei-p2q.t | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/t/lei-p2q.t b/t/lei-p2q.t
index 495d81de78a5..bf40a43be466 100644
--- a/t/lei-p2q.t
+++ b/t/lei-p2q.t
@@ -7,7 +7,7 @@ require_mods(qw(json DBD::SQLite Search::Xapian));
test_lei(sub {
ok(!lei(qw(p2q this-better-cause-format-patch-to-fail)),
- 'p2q fails on bogus arg');
+ 'p2q fails on bogus arg') or diag $lei_err;
like($lei_err, qr/format-patch.*failed/, 'notes format-patch failure');
lei_ok(qw(p2q -w dfpost t/data/0001.patch));
is($lei_out, "dfpost:6e006fd73b1d\n", 'pathname') or diag $lei_err;
^ permalink raw reply related [relevance 71%]
* [PATCH 04/15] lei/store: check for any unexpected process death
` (2 preceding siblings ...)
2021-10-21 21:10 71% ` [PATCH 03/15] t/lei-p2q: extra diagnostics Eric Wong
@ 2021-10-21 21:10 71% ` Eric Wong
2021-10-21 21:10 71% ` [PATCH 05/15] lei note-event: drop unnecessary eval guard Eric Wong
` (6 subsequent siblings)
10 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-10-21 21:10 UTC (permalink / raw)
To: meta
The lei/store process should only exit from EOF on the
socket, so make sure we note any unintended signals
---
lib/PublicInbox/LeiStore.pm | 8 +++++++-
1 file changed, 7 insertions(+), 1 deletion(-)
diff --git a/lib/PublicInbox/LeiStore.pm b/lib/PublicInbox/LeiStore.pm
index 821045701dfe..16e7d302dc2f 100644
--- a/lib/PublicInbox/LeiStore.pm
+++ b/lib/PublicInbox/LeiStore.pm
@@ -571,6 +571,12 @@ sub recv_and_run {
$self->SUPER::recv_and_run(@args);
}
+sub _sto_atexit { # dwaitpid callback
+ my ($args, $pid) = @_;
+ my $self = $args->[0];
+ warn "lei/store PID:$pid died \$?=$?\n" if $?;
+}
+
sub write_prepare {
my ($self, $lei) = @_;
$lei // die 'BUG: $lei not passed';
@@ -587,7 +593,7 @@ sub write_prepare {
-err_wr => $w,
to_close => [ $r ],
});
- $self->wq_wait_async; # outlives $lei
+ $self->wq_wait_async(\&_sto_atexit); # outlives $lei
require PublicInbox::LeiStoreErr;
PublicInbox::LeiStoreErr->new($r, $lei);
}
^ permalink raw reply related [relevance 71%]
* [PATCH 05/15] lei note-event: drop unnecessary eval guard
` (3 preceding siblings ...)
2021-10-21 21:10 71% ` [PATCH 04/15] lei/store: check for any unexpected process death Eric Wong
@ 2021-10-21 21:10 71% ` Eric Wong
2021-10-21 21:10 71% ` [PATCH 06/15] lei note-event: wq_io_do => wq_do Eric Wong
` (5 subsequent siblings)
10 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-10-21 21:10 UTC (permalink / raw)
To: meta
We don't want to lose the failure message in case note-event
fails.
---
lib/PublicInbox/LeiNoteEvent.pm | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/lib/PublicInbox/LeiNoteEvent.pm b/lib/PublicInbox/LeiNoteEvent.pm
index 1749c98f8312..0709754f6d2a 100644
--- a/lib/PublicInbox/LeiNoteEvent.pm
+++ b/lib/PublicInbox/LeiNoteEvent.pm
@@ -72,8 +72,7 @@ sub lei_note_event {
my $lms = $lei->lms or return;
$lms->lms_write_prepare if $new_cur eq ''; # for ->clear_src below
$lei->{opt}->{quiet} = 1;
- eval { $lms->arg2folder($lei, [ $folder ]) };
- return if $@;
+ $lms->arg2folder($lei, [ $folder ]);
my $state = $cfg->get_1("watch.$folder.state") // 'tag-rw';
return if $state eq 'pause';
return $lms->clear_src($folder, \$bn) if $new_cur eq '';
^ permalink raw reply related [relevance 71%]
* [PATCH 06/15] lei note-event: wq_io_do => wq_do
` (4 preceding siblings ...)
2021-10-21 21:10 71% ` [PATCH 05/15] lei note-event: drop unnecessary eval guard Eric Wong
@ 2021-10-21 21:10 71% ` Eric Wong
2021-10-21 21:10 69% ` [PATCH 07/15] lei_search: try harder to associate "lei index"-ed messages Eric Wong
` (4 subsequent siblings)
10 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-10-21 21:10 UTC (permalink / raw)
To: meta
No need to pass extra arrayref args, here.
---
lib/PublicInbox/LeiNoteEvent.pm | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/lib/PublicInbox/LeiNoteEvent.pm b/lib/PublicInbox/LeiNoteEvent.pm
index 0709754f6d2a..3472e73070d6 100644
--- a/lib/PublicInbox/LeiNoteEvent.pm
+++ b/lib/PublicInbox/LeiNoteEvent.pm
@@ -97,7 +97,7 @@ sub lei_note_event {
return if index($fl, 'T') >= 0;
my $kw = PublicInbox::MdirReader::flags2kw($fl);
my $vmd = { kw => $kw, sync_info => [ $folder, \$bn ] };
- $self->wq_io_do('maildir_event', [], $fn, $vmd, $state);
+ $self->wq_do('maildir_event', $fn, $vmd, $state);
} # else: TODO: imap
}
^ permalink raw reply related [relevance 71%]
* [PATCH 10/15] doc: lei-overview: add CAVEATS section
` (6 preceding siblings ...)
2021-10-21 21:10 69% ` [PATCH 07/15] lei_search: try harder to associate "lei index"-ed messages Eric Wong
@ 2021-10-21 21:10 71% ` Eric Wong
2021-10-21 21:10 71% ` [PATCH 11/15] lei note-event: clear_src on ENOENT Eric Wong
` (2 subsequent siblings)
10 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-10-21 21:10 UTC (permalink / raw)
To: meta
IMAP and NNTP client performance absolutely sucks compared to what
the read-only daemons are capable of...
---
Documentation/lei-overview.pod | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/Documentation/lei-overview.pod b/Documentation/lei-overview.pod
index bb2fe50f7cd9..99fd6ef72174 100644
--- a/Documentation/lei-overview.pod
+++ b/Documentation/lei-overview.pod
@@ -137,6 +137,11 @@ Since lei runs as a daemon, L<lei-daemon-kill(1)> is required to kill
the daemon so it can load new code. It will be restarted with the
next invocation of any lei command.
+=head1 CAVEATS
+
+IMAP and NNTP client performance is poor on high-latency connections.
+It will hopefully be fixed in 2022.
+
=head1 CONTACT
Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org>
^ permalink raw reply related [relevance 71%]
* [PATCH 11/15] lei note-event: clear_src on ENOENT
` (7 preceding siblings ...)
2021-10-21 21:10 71% ` [PATCH 10/15] doc: lei-overview: add CAVEATS section Eric Wong
@ 2021-10-21 21:10 71% ` Eric Wong
2021-10-21 21:10 71% ` [PATCH 13/15] lei: no Perl FileHandle for `undef' w/ ECONNRESET Eric Wong
2021-10-21 21:10 37% ` [PATCH 15/15] lei: use RENAME_NOREPLACE on Linux 3.15+ Eric Wong
10 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-10-21 21:10 UTC (permalink / raw)
To: meta
When a file goes away, try to make sure we don't waste
time trying to access or store it.
---
lib/PublicInbox/LeiNoteEvent.pm | 8 ++++++--
1 file changed, 6 insertions(+), 2 deletions(-)
diff --git a/lib/PublicInbox/LeiNoteEvent.pm b/lib/PublicInbox/LeiNoteEvent.pm
index 3472e73070d6..22d6ffac9feb 100644
--- a/lib/PublicInbox/LeiNoteEvent.pm
+++ b/lib/PublicInbox/LeiNoteEvent.pm
@@ -8,6 +8,7 @@ use strict;
use v5.10.1;
use parent qw(PublicInbox::IPC);
use PublicInbox::DS;
+use Errno qw(ENOENT);
our $to_flush; # { cfgpath => $lei }
@@ -59,8 +60,11 @@ sub eml_event ($$$$) {
sub maildir_event { # via wq_io_do
my ($self, $fn, $vmd, $state) = @_;
- my $eml = PublicInbox::InboxWritable::eml_from_path($fn) // return;
- eml_event($self, $eml, $vmd, $state);
+ if (my $eml = PublicInbox::InboxWritable::eml_from_path($fn)) {
+ eml_event($self, $eml, $vmd, $state);
+ } elsif ($! == ENOENT) {
+ $self->{lms}->clear_src(@{$vmd->{sync_info}});
+ } # else: eml_from_path already warns
}
sub lei_note_event {
^ permalink raw reply related [relevance 71%]
* [PATCH 13/15] lei: no Perl FileHandle for `undef' w/ ECONNRESET
` (8 preceding siblings ...)
2021-10-21 21:10 71% ` [PATCH 11/15] lei note-event: clear_src on ENOENT Eric Wong
@ 2021-10-21 21:10 71% ` Eric Wong
2021-10-21 21:10 37% ` [PATCH 15/15] lei: use RENAME_NOREPLACE on Linux 3.15+ Eric Wong
10 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-10-21 21:10 UTC (permalink / raw)
To: meta
Error reporting for recv_cmd4 methods is a bit wonky.
---
lib/PublicInbox/LEI.pm | 1 +
1 file changed, 1 insertion(+)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index b68e526bf365..43baeeb3d51c 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -1129,6 +1129,7 @@ sub event_step {
if (scalar(@fds) == 1 && !defined($fds[0])) {
return if $! == EAGAIN;
die "recvmsg: $!" if $! != ECONNRESET;
+ @fds = (); # for open loop below:
}
for (@fds) { open my $rfh, '+<&=', $_ }
if ($buf eq '') {
^ permalink raw reply related [relevance 71%]
* [PATCH 07/15] lei_search: try harder to associate "lei index"-ed messages
` (5 preceding siblings ...)
2021-10-21 21:10 71% ` [PATCH 06/15] lei note-event: wq_io_do => wq_do Eric Wong
@ 2021-10-21 21:10 69% ` Eric Wong
2021-10-21 21:10 71% ` [PATCH 10/15] doc: lei-overview: add CAVEATS section Eric Wong
` (3 subsequent siblings)
10 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-10-21 21:10 UTC (permalink / raw)
To: meta
Allow checking for keyword changes if we have an known OID,
even if the blob isn't currently reachable.
---
lib/PublicInbox/LeiSearch.pm | 13 ++++++++++++-
1 file changed, 12 insertions(+), 1 deletion(-)
diff --git a/lib/PublicInbox/LeiSearch.pm b/lib/PublicInbox/LeiSearch.pm
index 3e046b2146c6..1fb38da1d7aa 100644
--- a/lib/PublicInbox/LeiSearch.pm
+++ b/lib/PublicInbox/LeiSearch.pm
@@ -7,7 +7,7 @@ use strict;
use v5.10.1;
use parent qw(PublicInbox::ExtSearch); # PublicInbox::Search->reopen
use PublicInbox::Search qw(xap_terms);
-use PublicInbox::ContentHash qw(content_digest content_hash);
+use PublicInbox::ContentHash qw(content_digest content_hash git_sha);
use PublicInbox::MID qw(mids mids_for_index);
use Carp qw(croak);
@@ -118,6 +118,13 @@ sub xoids_for {
}
}
$git->async_wait_all;
+
+ # it could be an 'lei index'-ed file that just got renamed
+ if (scalar(keys %$xoids) < ($min // 1) && defined($self->{topdir})) {
+ my $hex = git_sha(1, $eml)->hexdigest;
+ my @n = $overs[0]->blob_exists($hex);
+ for (@n) { $xoids->{$hex} //= $_ }
+ }
scalar(keys %$xoids) ? $xoids : undef;
}
@@ -129,6 +136,10 @@ sub kw_changed {
my $xoids = xoids_for($self, $eml) // return;
$docids //= [];
@$docids = sort { $a <=> $b } values %$xoids;
+ if (!@$docids && $self->over) {
+ my $bin = git_sha(1, $eml)->digest;
+ @$docids = $self->over->oidbin_exists($bin);
+ }
}
for my $id (@$docids) {
$cur_kw = eval { msg_keywords($self, $id) } and last;
^ permalink raw reply related [relevance 69%]
* [PATCH 15/15] lei: use RENAME_NOREPLACE on Linux 3.15+
` (9 preceding siblings ...)
2021-10-21 21:10 71% ` [PATCH 13/15] lei: no Perl FileHandle for `undef' w/ ECONNRESET Eric Wong
@ 2021-10-21 21:10 37% ` Eric Wong
10 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-10-21 21:10 UTC (permalink / raw)
To: meta
One syscall is better than two for atomicity in Maildirs. This
means there's no window where another process can see both the
old and new file at the same time (link && unlink), nor a window
where we might inadvertantly clobber an existing file if we were
to do `stat && rename'.
---
MANIFEST | 1 +
devel/syscall-list | 8 +++++-
lib/PublicInbox/LeiExportKw.pm | 19 +++++--------
lib/PublicInbox/LeiStore.pm | 8 +++---
lib/PublicInbox/LeiToMail.pm | 7 +++--
lib/PublicInbox/Syscall.pm | 49 +++++++++++++++++++++++++++++++---
t/rename_noreplace.t | 26 ++++++++++++++++++
7 files changed, 92 insertions(+), 26 deletions(-)
create mode 100644 t/rename_noreplace.t
diff --git a/MANIFEST b/MANIFEST
index af1522d71bd1..9fd979ef02fb 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -528,6 +528,7 @@ t/psgi_v2.t
t/purge.t
t/qspawn.t
t/reindex-time-range.t
+t/rename_noreplace.t
t/replace.t
t/reply.t
t/run.perl
diff --git a/devel/syscall-list b/devel/syscall-list
index b33401d98ce4..3d55df1fc1d7 100755
--- a/devel/syscall-list
+++ b/devel/syscall-list
@@ -1,4 +1,4 @@
-# Copyright 2021 all contributors <meta@public-inbox.org>
+# Copyright all contributors <meta@public-inbox.org>
# License: AGPL-3.0+ <http://www.gnu.org/licenses/agpl-3.0.txt>
# Dump syscall numbers under Linux and any other kernel which
# promises stable syscall numbers. This is to maintain
@@ -9,7 +9,10 @@
eval 'exec perl -S $0 ${1+"$@"}' # no shebang
if 0; # running under some shell
use strict;
+use v5.10.1;
use File::Temp 0.19;
+use POSIX qw(uname);
+say '$machine='.(POSIX::uname())[-1];
my $cc = $ENV{CC} // 'cc';
my @cflags = split(/\s+/, $ENV{CFLAGS} // '-Wall');
my $str = do { local $/; <DATA> };
@@ -43,6 +46,9 @@ int main(void)
D(SYS_inotify_add_watch);
D(SYS_inotify_rm_watch);
D(SYS_prctl);
+#ifdef SYS_renameat2
+ D(SYS_renameat2);
+#endif
#endif /* Linux, any other OSes with stable syscalls? */
printf("size_t=%zu off_t=%zu\n", sizeof(size_t), sizeof(off_t));
return 0;
diff --git a/lib/PublicInbox/LeiExportKw.pm b/lib/PublicInbox/LeiExportKw.pm
index 0b65c2762633..ceeef7f21d54 100644
--- a/lib/PublicInbox/LeiExportKw.pm
+++ b/lib/PublicInbox/LeiExportKw.pm
@@ -7,6 +7,7 @@ use strict;
use v5.10.1;
use parent qw(PublicInbox::IPC PublicInbox::LeiInput);
use Errno qw(EEXIST ENOENT);
+use PublicInbox::Syscall qw(rename_noreplace);
sub export_kw_md { # LeiMailSync->each_src callback
my ($oidbin, $id, $self, $mdir) = @_;
@@ -30,30 +31,22 @@ sub export_kw_md { # LeiMailSync->each_src callback
my $lei = $self->{lei};
for my $d (@try) {
my $src = "$mdir/$d/$$id";
-
- # we use link(2) + unlink(2) since rename(2) may
- # inadvertently clobber if the "uniquefilename" part wasn't
- # actually unique.
- if (link($src, $dst)) { # success
- # unlink(2) may ENOENT from parallel invocation,
- # ignore it, but not other serious errors
- if (!unlink($src) and $! != ENOENT) {
- $lei->child_error(1, "E: unlink($src): $!");
- }
+ if (rename_noreplace($src, $dst)) { # success
$self->{lms}->mv_src("maildir:$mdir",
$oidbin, $id, $bn);
- return; # success anyways if link(2) worked
+ return; # success
} elsif ($! == EEXIST) { # lost race with lei/store?
return;
} elsif ($! != ENOENT) {
- $lei->child_error(1, "E: link($src -> $dst): $!");
+ $lei->child_error(1,
+ "E: rename_noreplace($src -> $dst): $!");
} # else loop @try
}
my $e = $!;
# both tries failed
my $oidhex = unpack('H*', $oidbin);
my $src = "$mdir/{".join(',', @try)."}/$$id";
- $lei->child_error(1, "link($src -> $dst) ($oidhex): $e");
+ $lei->child_error(1, "rename_noreplace($src -> $dst) ($oidhex): $e");
for (@try) { return if -e "$mdir/$_/$$id" }
$self->{lms}->clear_src("maildir:$mdir", $id);
}
diff --git a/lib/PublicInbox/LeiStore.pm b/lib/PublicInbox/LeiStore.pm
index 16e7d302dc2f..f1316229bb32 100644
--- a/lib/PublicInbox/LeiStore.pm
+++ b/lib/PublicInbox/LeiStore.pm
@@ -32,6 +32,7 @@ use POSIX ();
use IO::Handle (); # ->autoflush
use Sys::Syslog qw(syslog openlog);
use Errno qw(EEXIST ENOENT);
+use PublicInbox::Syscall qw(rename_noreplace);
sub new {
my (undef, $dir, $opt) = @_;
@@ -185,10 +186,7 @@ sub export1_kw_md ($$$$$) {
my $dst = "$mdir/cur/$bn";
for my $d (@try) {
my $src = "$mdir/$d/$orig";
- if (link($src, $dst)) {
- if (!unlink($src) and $! != ENOENT) {
- syslog('warning', "unlink($src): $!");
- }
+ if (rename_noreplace($src, $dst)) {
# TODO: verify oidbin?
$self->{lms}->mv_src("maildir:$mdir",
$oidbin, \$orig, $bn);
@@ -196,7 +194,7 @@ sub export1_kw_md ($$$$$) {
} elsif ($! == EEXIST) { # lost race with "lei export-kw"?
return;
} elsif ($! != ENOENT) {
- syslog('warning', "link($src -> $dst): $!");
+ syslog('warning', "rename_noreplace($src -> $dst): $!");
}
}
for (@try) { return if -e "$mdir/$_/$orig" };
diff --git a/lib/PublicInbox/LeiToMail.pm b/lib/PublicInbox/LeiToMail.pm
index ca4e92de48b7..d33d27aec006 100644
--- a/lib/PublicInbox/LeiToMail.pm
+++ b/lib/PublicInbox/LeiToMail.pm
@@ -12,6 +12,7 @@ use PublicInbox::Spawn qw(spawn);
use Symbol qw(gensym);
use IO::Handle; # ->autoflush
use Fcntl qw(SEEK_SET SEEK_END O_CREAT O_EXCL O_WRONLY);
+use PublicInbox::Syscall qw(rename_noreplace);
my %kw2char = ( # Maildir characters
draft => 'D',
@@ -262,10 +263,8 @@ sub _buf2maildir ($$$$) {
$rand = '';
do {
$base = $rand.$common.':2,'.kw2suffix($kw);
- } while (!($ok = link($tmp, $dst.$base)) && $!{EEXIST} &&
- ($rand = _rand.','));
- die "link($tmp, $dst$base): $!" unless $ok;
- unlink($tmp) or warn "W: failed to unlink $tmp: $!\n";
+ } while (!($ok = rename_noreplace($tmp, $dst.$base)) &&
+ $!{EEXIST} && ($rand = _rand.','));
\$base;
} else {
my $err = "Error writing $smsg->{blob} to $dst: $!\n";
diff --git a/lib/PublicInbox/Syscall.pm b/lib/PublicInbox/Syscall.pm
index 7ab4291119ea..c00385b94db8 100644
--- a/lib/PublicInbox/Syscall.pm
+++ b/lib/PublicInbox/Syscall.pm
@@ -13,8 +13,9 @@
# License or the Artistic License, as specified in the Perl README file.
package PublicInbox::Syscall;
use strict;
+use v5.10.1;
use parent qw(Exporter);
-use POSIX qw(ENOSYS O_NONBLOCK);
+use POSIX qw(ENOENT EEXIST ENOSYS O_NONBLOCK);
use Config;
# $VERSION = '0.25'; # Sys::Syscall version
@@ -22,7 +23,7 @@ our @EXPORT_OK = qw(epoll_ctl epoll_create epoll_wait
EPOLLIN EPOLLOUT EPOLLET
EPOLL_CTL_ADD EPOLL_CTL_DEL EPOLL_CTL_MOD
EPOLLONESHOT EPOLLEXCLUSIVE
- signalfd);
+ signalfd rename_noreplace);
our %EXPORT_TAGS = (epoll => [qw(epoll_ctl epoll_create epoll_wait
EPOLLIN EPOLLOUT
EPOLL_CTL_ADD EPOLL_CTL_DEL EPOLL_CTL_MOD
@@ -64,13 +65,16 @@ our (
$SYS_epoll_ctl,
$SYS_epoll_wait,
$SYS_signalfd4,
+ $SYS_renameat2,
);
my $SFD_CLOEXEC = 02000000; # Perl does not expose O_CLOEXEC
our $no_deprecated = 0;
if ($^O eq "linux") {
- my $machine = (POSIX::uname())[-1];
+ my (undef, undef, $release, undef, $machine) = POSIX::uname();
+ my ($maj, $min) = ($release =~ /\A([0-9]+)\.([0-9]+)/);
+ $SYS_renameat2 = 0 if "$maj.$min" < 3.15;
# whether the machine requires 64-bit numbers to be on 8-byte
# boundaries.
my $u64_mod_8 = 0;
@@ -91,22 +95,26 @@ if ($^O eq "linux") {
$SYS_epoll_ctl = 255;
$SYS_epoll_wait = 256;
$SYS_signalfd4 = 327;
+ $SYS_renameat2 //= 353;
} elsif ($machine eq "x86_64") {
$SYS_epoll_create = 213;
$SYS_epoll_ctl = 233;
$SYS_epoll_wait = 232;
$SYS_signalfd4 = 289;
+ $SYS_renameat2 //= 316;
} elsif ($machine eq 'x32') {
$SYS_epoll_create = 1073742037;
$SYS_epoll_ctl = 1073742057;
$SYS_epoll_wait = 1073742056;
$SYS_signalfd4 = 1073742113;
+ $SYS_renameat2 //= 0x40000000 + 316;
} elsif ($machine eq 'sparc64') {
$SYS_epoll_create = 193;
$SYS_epoll_ctl = 194;
$SYS_epoll_wait = 195;
$u64_mod_8 = 1;
$SYS_signalfd4 = 317;
+ $SYS_renameat2 //= 345;
$SFD_CLOEXEC = 020000000;
} elsif ($machine =~ m/^parisc/) {
$SYS_epoll_create = 224;
@@ -120,18 +128,21 @@ if ($^O eq "linux") {
$SYS_epoll_wait = 238;
$u64_mod_8 = 1;
$SYS_signalfd4 = 313;
+ $SYS_renameat2 //= 357;
} elsif ($machine eq "ppc") {
$SYS_epoll_create = 236;
$SYS_epoll_ctl = 237;
$SYS_epoll_wait = 238;
$u64_mod_8 = 1;
$SYS_signalfd4 = 313;
+ $SYS_renameat2 //= 357;
} elsif ($machine =~ m/^s390/) {
$SYS_epoll_create = 249;
$SYS_epoll_ctl = 250;
$SYS_epoll_wait = 251;
$u64_mod_8 = 1;
$SYS_signalfd4 = 322;
+ $SYS_renameat2 //= 347;
} elsif ($machine eq "ia64") {
$SYS_epoll_create = 1243;
$SYS_epoll_ctl = 1244;
@@ -153,6 +164,7 @@ if ($^O eq "linux") {
$u64_mod_8 = 1;
$no_deprecated = 1;
$SYS_signalfd4 = 74;
+ $SYS_renameat2 //= 276;
} elsif ($machine =~ m/arm(v\d+)?.*l/) {
# ARM OABI
$SYS_epoll_create = 250;
@@ -160,18 +172,21 @@ if ($^O eq "linux") {
$SYS_epoll_wait = 252;
$u64_mod_8 = 1;
$SYS_signalfd4 = 355;
+ $SYS_renameat2 //= 382;
} elsif ($machine =~ m/^mips64/) {
$SYS_epoll_create = 5207;
$SYS_epoll_ctl = 5208;
$SYS_epoll_wait = 5209;
$u64_mod_8 = 1;
$SYS_signalfd4 = 5283;
+ $SYS_renameat2 //= 5311;
} elsif ($machine =~ m/^mips/) {
$SYS_epoll_create = 4248;
$SYS_epoll_ctl = 4249;
$SYS_epoll_wait = 4250;
$u64_mod_8 = 1;
$SYS_signalfd4 = 4324;
+ $SYS_renameat2 //= 4351;
} else {
# as a last resort, try using the *.ph files which may not
# exist or may be wrong
@@ -280,6 +295,34 @@ sub signalfd ($$) {
}
}
+sub _rename_noreplace_racy ($$) {
+ my ($old, $new) = @_;
+ if (link($old, $new)) {
+ warn "unlink $old: $!\n" if !unlink($old) && $! != ENOENT;
+ 1
+ } else {
+ undef;
+ }
+}
+
+# TODO: support FD args?
+sub rename_noreplace ($$) {
+ my ($old, $new) = @_;
+ if ($SYS_renameat2) { # RENAME_NOREPLACE = 1, AT_FDCWD = -100
+ my $ret = syscall($SYS_renameat2, -100, $old, -100, $new, 1);
+ if ($ret == 0) {
+ 1; # like rename() perlop
+ } elsif ($! == ENOSYS) {
+ undef $SYS_renameat2;
+ _rename_noreplace_racy($old, $new);
+ } else {
+ undef
+ }
+ } else {
+ _rename_noreplace_racy($old, $new);
+ }
+}
+
1;
=head1 WARRANTY
diff --git a/t/rename_noreplace.t b/t/rename_noreplace.t
new file mode 100644
index 000000000000..bd1c4e9236a7
--- /dev/null
+++ b/t/rename_noreplace.t
@@ -0,0 +1,26 @@
+#!perl -w
+# Copyright (C) all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+use strict;
+use v5.10.1;
+use PublicInbox::TestCommon;
+use_ok 'PublicInbox::Syscall', 'rename_noreplace';
+my ($tmpdir, $for_destroy) = tmpdir;
+
+open my $fh, '>', "$tmpdir/a" or xbail $!;
+my @sa = stat($fh);
+is(rename_noreplace("$tmpdir/a", "$tmpdir/b"), 1, 'rename_noreplace');
+my @sb = stat("$tmpdir/b");
+ok(scalar(@sb), 'new file exists');
+ok(!-e "$tmpdir/a", 'original gone');
+is("@sa[0,1]", "@sb[0,1]", 'same st_dev + st_ino');
+
+is(rename_noreplace("$tmpdir/a", "$tmpdir/c"), undef, 'undef on ENOENT');
+ok($!{ENOENT}, 'ENOENT set when missing');
+
+open $fh, '>', "$tmpdir/a" or xbail $!;
+is(rename_noreplace("$tmpdir/a", "$tmpdir/b"), undef, 'undef on EEXIST');
+ok($!{EEXIST}, 'EEXIST set when missing');
+is_deeply([stat("$tmpdir/b")], \@sb, 'target unchanged on EEXIST');
+
+done_testing;
^ permalink raw reply related [relevance 37%]
* [PATCH 02/15] t/lei-import-maildir: rename fix (SR -> RS)
2021-10-21 21:10 57% ` [PATCH 01/15] t/lei-{auto-watch,export-kw}: extra diagnostics on failure Eric Wong
@ 2021-10-21 21:10 71% ` Eric Wong
2021-10-21 21:10 71% ` [PATCH 03/15] t/lei-p2q: extra diagnostics Eric Wong
` (8 subsequent siblings)
10 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-10-21 21:10 UTC (permalink / raw)
To: meta
While it doesn't matter to us, the Maildir spec specifies
characters are to be sorted in alphabetical order.
---
t/lei-import-maildir.t | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/t/lei-import-maildir.t b/t/lei-import-maildir.t
index c81e7805fe7f..1e7eddd571d8 100644
--- a/t/lei-import-maildir.t
+++ b/t/lei-import-maildir.t
@@ -52,7 +52,7 @@ test_lei(sub {
my $r2 = json_utf8->decode($lei_out);
is_deeply($r2, $res, 'idempotent import')
or diag explain($imp_err, $res);
- rename("$md/cur/x:2,S", "$md/cur/x:2,SR") or BAIL_OUT "rename: $!";
+ rename("$md/cur/x:2,S", "$md/cur/x:2,RS") or BAIL_OUT "rename: $!";
lei_ok('import', "maildir:$md", \'import Maildir after +answered');
lei_ok(qw(q -d none s:boolean), \'lei q after +answered');
$res = json_utf8->decode($lei_out);
^ permalink raw reply related [relevance 71%]
* [PATCH 01/15] t/lei-{auto-watch,export-kw}: extra diagnostics on failure
@ 2021-10-21 21:10 57% ` Eric Wong
2021-10-21 21:10 71% ` [PATCH 02/15] t/lei-import-maildir: rename fix (SR -> RS) Eric Wong
` (9 subsequent siblings)
10 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-10-21 21:10 UTC (permalink / raw)
To: meta
Maybe these will help track down some failures and make
diagnosing bugs easier. "lei export-kw" should also become
optional, even, so allow disabling it easily in the test.
---
t/lei-auto-watch.t | 3 ++-
t/lei-export-kw.t | 39 ++++++++++++++++++++++++++++-----------
2 files changed, 30 insertions(+), 12 deletions(-)
diff --git a/t/lei-auto-watch.t b/t/lei-auto-watch.t
index e5e132eb3bfd..d5661ae5d618 100644
--- a/t/lei-auto-watch.t
+++ b/t/lei-auto-watch.t
@@ -41,7 +41,8 @@ test_lei(sub {
$ins = json_utf8->decode($lei_out);
$exp = { "maildir:$x" => [ map { basename($_) } glob("$x/*/*") ],
"maildir:$y" => [ map { basename($_) } glob("$y/*/*") ] };
- is_deeply($ins->{'mail-sync'}, $exp, 'mail_sync matches FS');
+ is_deeply($ins->{'mail-sync'}, $exp, 'mail_sync matches FS') or
+ diag explain($ins);
});
done_testing;
diff --git a/t/lei-export-kw.t b/t/lei-export-kw.t
index 1fe940bb6d89..55730e87c050 100644
--- a/t/lei-export-kw.t
+++ b/t/lei-export-kw.t
@@ -7,28 +7,45 @@ use File::Path qw(make_path);
require_mods(qw(lei -imapd Mail::IMAPClient));
my ($tmpdir, $for_destroy) = tmpdir;
my $expect = eml_load('t/data/0001.patch');
+my $do_export_kw = 1;
+my $wait_for = sub {
+ my ($f) = @_;
+ lei_ok qw(export-kw --all=local) if $do_export_kw;
+ my $x = $f;
+ $x =~ s!\Q$tmpdir\E/!\$TMPDIR/!;
+ for (0..10) {
+ last if -f $f;
+ diag "tick #$_ $x";
+ tick(0.1);
+ }
+ ok(-f $f, "$x exists") or xbail;
+};
+
test_lei({ tmpdir => $tmpdir }, sub {
my $home = $ENV{HOME};
my $md = "$home/md";
+ my $f;
make_path("$md/new", "$md/cur", "$md/tmp");
cp('t/data/0001.patch', "$md/new/y") or xbail "cp $md $!";
cp('t/data/message_embed.eml', "$md/cur/x:2,S") or xbail "cp $md $!";
- lei_ok qw(index -q), $md;
+ lei_ok qw(index), $md;
lei_ok qw(tag t/data/0001.patch +kw:seen);
- lei_ok qw(export-kw --all=local);
- ok(!-e "$md/new/y", 'original gone');
- is_deeply(eml_load("$md/cur/y:2,S"), $expect,
- "`seen' kw exported");
+ $wait_for->($f = "$md/cur/y:2,S");
+ ok(!-e "$md/new/y", 'original gone') or
+ diag explain([glob("$md/*/*")]);
+ is_deeply(eml_load($f), $expect, "`seen' kw exported");
lei_ok qw(tag t/data/0001.patch +kw:answered);
- lei_ok qw(export-kw --all=local);
- ok(!-e "$md/cur/y:2,S", 'seen-only file gone');
- is_deeply(eml_load("$md/cur/y:2,RS"), $expect, "`R' added");
+ $wait_for->($f = "$md/cur/y:2,RS");
+ ok(!-e "$md/cur/y:2,S", 'seen-only file gone') or
+ diag explain([glob("$md/*/*")]);
+ is_deeply(eml_load($f), $expect, "`R' added");
lei_ok qw(tag t/data/0001.patch -kw:answered -kw:seen);
- lei_ok qw(export-kw --mode=set --all=local);
- ok(!-e "$md/cur/y:2,RS", 'seen+answered file gone');
- is_deeply(eml_load("$md/cur/y:2,"), $expect, 'no keywords left');
+ $wait_for->($f = "$md/cur/y:2,");
+ ok(!-e "$md/cur/y:2,RS", 'seen+answered file gone') or
+ diag explain([glob("$md/*/*")]);
+ is_deeply(eml_load($f), $expect, 'no keywords left');
});
done_testing;
^ permalink raw reply related [relevance 57%]
* [PATCH 08/11] doc: lei: describe lei-daemon-kill and upgrades
2021-10-19 9:33 70% [PATCH 00/11] refining lei up+inspect Eric Wong
` (5 preceding siblings ...)
2021-10-19 9:33 71% ` [PATCH 07/11] lei: remove unused ->busy time arg Eric Wong
@ 2021-10-19 9:33 80% ` Eric Wong
2021-10-19 9:33 71% ` [PATCH 09/11] lei inspect: add atfork hook Eric Wong
2021-10-19 9:33 64% ` [PATCH 10/11] lei inspect: show ISO8601 {rt} and {dt}, too Eric Wong
8 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-10-19 9:33 UTC (permalink / raw)
To: meta
While we're at it, start dropping copyright years
since it seems acceptable to not have them:
https://www.linuxfoundation.org/blog/copyright-notices-in-open-source-software-projects/
Copyright years are also a noisy to update every year (maybe,
just maybe, we'll make it to 2022...)
---
Documentation/lei-daemon-kill.pod | 29 ++++++++++++++++++++++++++---
Documentation/lei-overview.pod | 8 +++++++-
2 files changed, 33 insertions(+), 4 deletions(-)
diff --git a/Documentation/lei-daemon-kill.pod b/Documentation/lei-daemon-kill.pod
index 7fb0fb25ad0e..48c237b8d3a6 100644
--- a/Documentation/lei-daemon-kill.pod
+++ b/Documentation/lei-daemon-kill.pod
@@ -8,7 +8,30 @@ lei daemon-kill [-SIGNAL | -s SIGNAL | --signal SIGNAL]
=head1 DESCRIPTION
-Send a signal to the lei-daemon. C<SIGNAL> defaults to C<TERM>.
+Send a signal to the L<lei-daemon(8)>. C<SIGNAL> defaults to C<TERM>.
+
+This command should be run after updating the code of lei.
+
+=head1 SIGNALS
+
+=over 8
+
+=item SIGTERM
+
+Send a graceful termination signal. L<lei-daemon(8)> will exit
+when all currently running lei commands are done. The listen
+socket will be released as soon as the signal is processed
+so another L<lei-daemon(8)> process can take its place.
+
+=item SIGKILL
+
+Kills L<lei-daemon(8)> immediately. Some worker processes may
+remain running after a short while after this takes effect.
+
+=back
+
+=for comment
+SIGQUIT and SIGINT currently do what SIGTERM do, may change...
=head1 CONTACT
@@ -19,10 +42,10 @@ L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/>
=head1 COPYRIGHT
-Copyright 2021 all contributors L<mailto:meta@public-inbox.org>
+Copyright all contributors L<mailto:meta@public-inbox.org>
License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>
=head1 SEE ALSO
-L<lei-daemon-pid(1)>
+L<lei-daemon-pid(1)>, L<lei-daemon(8)>
diff --git a/Documentation/lei-overview.pod b/Documentation/lei-overview.pod
index 40a7b0aadd04..bb2fe50f7cd9 100644
--- a/Documentation/lei-overview.pod
+++ b/Documentation/lei-overview.pod
@@ -131,6 +131,12 @@ C<contrib/completion/>. Contributions adding support for other
shells, as well as improvements to the existing Bash completion, are
welcome.
+=head1 UPGRADING
+
+Since lei runs as a daemon, L<lei-daemon-kill(1)> is required to kill
+the daemon so it can load new code. It will be restarted with the
+next invocation of any lei command.
+
=head1 CONTACT
Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org>
@@ -140,6 +146,6 @@ L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/>
=head1 COPYRIGHT
-Copyright 2021 all contributors L<mailto:meta@public-inbox.org>
+Copyright all contributors L<mailto:meta@public-inbox.org>
License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>
^ permalink raw reply related [relevance 80%]
* [PATCH 07/11] lei: remove unused ->busy time arg
2021-10-19 9:33 70% [PATCH 00/11] refining lei up+inspect Eric Wong
` (4 preceding siblings ...)
2021-10-19 9:33 47% ` [PATCH 06/11] lei up: support --exclude=, --no-(external|remote|local) Eric Wong
@ 2021-10-19 9:33 71% ` Eric Wong
2021-10-19 9:33 80% ` [PATCH 08/11] doc: lei: describe lei-daemon-kill and upgrades Eric Wong
` (2 subsequent siblings)
8 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-10-19 9:33 UTC (permalink / raw)
To: meta
Our graceful shutdown doesn't time out clients.
---
lib/PublicInbox/LEI.pm | 5 ++---
1 file changed, 2 insertions(+), 3 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 5b726f71382e..c1f28f7b31bf 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -19,7 +19,7 @@ use IO::Handle ();
use Fcntl qw(SEEK_SET);
use PublicInbox::Config;
use PublicInbox::Syscall qw(EPOLLIN);
-use PublicInbox::DS qw(now dwaitpid);
+use PublicInbox::DS qw(dwaitpid);
use PublicInbox::Spawn qw(spawn popen_rd);
use PublicInbox::Lock;
use PublicInbox::Eml;
@@ -1315,11 +1315,10 @@ sub lazy_start {
$quit->();
}
return 1 if defined($path);
- my $now = now();
my $n = 0;
for my $s (values %$dmap) {
$s->can('busy') or next;
- if ($s->busy($now)) {
+ if ($s->busy) {
++$n;
} else {
$s->close;
^ permalink raw reply related [relevance 71%]
* [PATCH 09/11] lei inspect: add atfork hook
2021-10-19 9:33 70% [PATCH 00/11] refining lei up+inspect Eric Wong
` (6 preceding siblings ...)
2021-10-19 9:33 80% ` [PATCH 08/11] doc: lei: describe lei-daemon-kill and upgrades Eric Wong
@ 2021-10-19 9:33 71% ` Eric Wong
2021-10-19 9:33 64% ` [PATCH 10/11] lei inspect: show ISO8601 {rt} and {dt}, too Eric Wong
8 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-10-19 9:33 UTC (permalink / raw)
To: meta
This is necessary for in case an inspect command is run
in a parallel with other commands.
---
lib/PublicInbox/LeiInspect.pm | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/lib/PublicInbox/LeiInspect.pm b/lib/PublicInbox/LeiInspect.pm
index 05b6e21d298d..38ef3ad96df2 100644
--- a/lib/PublicInbox/LeiInspect.pm
+++ b/lib/PublicInbox/LeiInspect.pm
@@ -289,4 +289,10 @@ sub _complete_inspect {
# TODO: message-ids?, blobs? could get expensive...
}
+sub input_only_atfork_child {
+ my ($self) = @_;
+ $self->{lei}->_lei_atfork_child;
+ $self->SUPER::ipc_atfork_child;
+}
+
1;
^ permalink raw reply related [relevance 71%]
* [PATCH 10/11] lei inspect: show ISO8601 {rt} and {dt}, too
2021-10-19 9:33 70% [PATCH 00/11] refining lei up+inspect Eric Wong
` (7 preceding siblings ...)
2021-10-19 9:33 71% ` [PATCH 09/11] lei inspect: add atfork hook Eric Wong
@ 2021-10-19 9:33 64% ` Eric Wong
8 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-10-19 9:33 UTC (permalink / raw)
To: meta
While inspect is intended for debugging, the Unix epoch in
seconds requires extra steps for human consumption; just
steal what we used for "lei q -f json" output.
---
lib/PublicInbox/LeiInspect.pm | 5 +++++
lib/PublicInbox/LeiOverview.pm | 6 +++---
2 files changed, 8 insertions(+), 3 deletions(-)
diff --git a/lib/PublicInbox/LeiInspect.pm b/lib/PublicInbox/LeiInspect.pm
index 38ef3ad96df2..5ea32ccb7e66 100644
--- a/lib/PublicInbox/LeiInspect.pm
+++ b/lib/PublicInbox/LeiInspect.pm
@@ -12,10 +12,15 @@ use parent qw(PublicInbox::IPC);
use PublicInbox::Config;
use PublicInbox::MID qw(mids);
use PublicInbox::NetReader qw(imap_uri nntp_uri);
+use POSIX qw(strftime);
+use PublicInbox::LeiOverview;
+*iso8601 = \&PublicInbox::LeiOverview::iso8601;
sub _json_prep ($) {
my ($smsg) = @_;
$smsg->{$_} += 0 for qw(bytes lines); # integerize
+ $smsg->{dt} = iso8601($smsg->{ds}) if defined($smsg->{ds});
+ $smsg->{rt} = iso8601($smsg->{ts}) if defined($smsg->{ts});
+{ %$smsg } # unbless and scalarize
}
diff --git a/lib/PublicInbox/LeiOverview.pm b/lib/PublicInbox/LeiOverview.pm
index 1b9dc9701c95..2d3db9f4ab92 100644
--- a/lib/PublicInbox/LeiOverview.pm
+++ b/lib/PublicInbox/LeiOverview.pm
@@ -21,7 +21,7 @@ use PublicInbox::LeiToMail;
# cf. https://en.wikipedia.org/wiki/JSON_streaming
my $JSONL = 'ldjson|ndjson|jsonl'; # 3 names for the same thing
-sub _iso8601 ($) { strftime('%Y-%m-%dT%H:%M:%SZ', gmtime($_[0])) }
+sub iso8601 ($) { strftime('%Y-%m-%dT%H:%M:%SZ', gmtime($_[0])) }
# we open this in the parent process before ->wq_io_do handoff
sub ovv_out_lk_init ($) {
@@ -139,8 +139,8 @@ sub _unbless_smsg {
# num/tid are nonsensical with multi-inbox search,
# lines/bytes are not generally useful
delete @$smsg{qw(num tid lines bytes)};
- $smsg->{rt} = _iso8601(delete $smsg->{ts}); # JMAP receivedAt
- $smsg->{dt} = _iso8601(delete $smsg->{ds}); # JMAP UTCDate
+ $smsg->{rt} = iso8601(delete $smsg->{ts}); # JMAP receivedAt
+ $smsg->{dt} = iso8601(delete $smsg->{ds}); # JMAP UTCDate
$smsg->{pct} = get_pct($mitem) if $mitem;
if (my $r = delete $smsg->{references}) {
$smsg->{refs} = [ map { $_ } ($r =~ m/$MID_EXTRACT/go) ];
^ permalink raw reply related [relevance 64%]
* [PATCH 04/11] lei up: propagate redispatch_all failure via exit code
2021-10-19 9:33 70% [PATCH 00/11] refining lei up+inspect Eric Wong
2021-10-19 9:33 65% ` [PATCH 02/11] lei up: prefix `remote' and `local' with `o_' Eric Wong
2021-10-19 9:33 50% ` [PATCH 03/11] lei: use die for external and query handling Eric Wong
@ 2021-10-19 9:33 71% ` Eric Wong
2021-10-19 9:33 70% ` [PATCH 05/11] lei: conditionally add "\n" to error messages Eric Wong
` (5 subsequent siblings)
8 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-10-19 9:33 UTC (permalink / raw)
To: meta
We can still continue with some local externals, maybe;
but the error needs to be propagated to the calling process
for scripting purposes.
---
lib/PublicInbox/LeiUp.pm | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/lib/PublicInbox/LeiUp.pm b/lib/PublicInbox/LeiUp.pm
index 4fd0290c7612..fcdd535dc118 100644
--- a/lib/PublicInbox/LeiUp.pm
+++ b/lib/PublicInbox/LeiUp.pm
@@ -124,7 +124,7 @@ sub net_merge_all_done {
$lei->{net} = delete($self->{-net_new}) if $self->{-net_new};
$self->wq_close;
eval { redispatch_all($self, $lei) };
- warn "E: $@" if $@;
+ $lei->child_error(0, "E: $@") if $@;
}
sub _complete_up { # lei__complete hook
^ permalink raw reply related [relevance 71%]
* [PATCH 00/11] refining lei up+inspect
@ 2021-10-19 9:33 70% Eric Wong
2021-10-19 9:33 65% ` [PATCH 02/11] lei up: prefix `remote' and `local' with `o_' Eric Wong
` (8 more replies)
0 siblings, 9 replies; 200+ results
From: Eric Wong @ 2021-10-19 9:33 UTC (permalink / raw)
To: meta
"lei up" gains some flexibility in dealing with offline
situations. "inspect" is slightly nicer-to-use
11/11 is preparation for "lei fsck"...
Eric Wong (11):
test_common: lazy-require AutoReap
lei up: prefix `remote' and `local' with `o_'
lei: use die for external and query handling
lei up: propagate redispatch_all failure via exit code
lei: conditionally add "\n" to error messages
lei up: support --exclude=, --no-(external|remote|local)
lei: remove unused ->busy time arg
doc: lei: describe lei-daemon-kill and upgrades
lei inspect: add atfork hook
lei inspect: show ISO8601 {rt} and {dt}, too
lei_mail_sync: show non-matching SHA
Documentation/lei-daemon-kill.pod | 29 +++++++++++++++++--
Documentation/lei-overview.pod | 8 +++++-
Documentation/lei-up.pod | 28 ++++++++++++++----
lib/PublicInbox/LEI.pm | 15 +++++-----
lib/PublicInbox/LeiExternal.pm | 15 ++++++++--
lib/PublicInbox/LeiInspect.pm | 11 +++++++
lib/PublicInbox/LeiMailSync.pm | 9 ++++--
lib/PublicInbox/LeiOverview.pm | 6 ++--
lib/PublicInbox/LeiQuery.pm | 27 ++++++-----------
lib/PublicInbox/LeiSavedSearch.pm | 8 +++---
lib/PublicInbox/LeiUp.pm | 48 ++++++++++++++++++++++---------
lib/PublicInbox/LeiXSearch.pm | 7 +++--
lib/PublicInbox/TestCommon.pm | 3 +-
13 files changed, 149 insertions(+), 65 deletions(-)
^ permalink raw reply [relevance 70%]
* [PATCH 05/11] lei: conditionally add "\n" to error messages
2021-10-19 9:33 70% [PATCH 00/11] refining lei up+inspect Eric Wong
` (2 preceding siblings ...)
2021-10-19 9:33 71% ` [PATCH 04/11] lei up: propagate redispatch_all failure via exit code Eric Wong
@ 2021-10-19 9:33 70% ` Eric Wong
2021-10-19 9:33 47% ` [PATCH 06/11] lei up: support --exclude=, --no-(external|remote|local) Eric Wong
` (4 subsequent siblings)
8 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-10-19 9:33 UTC (permalink / raw)
To: meta
Some error messages already include "\n" (w/ file+line info),
so don't add another one. (`warn' will automatically add its
caller location unless there's a final "\n").
---
lib/PublicInbox/LEI.pm | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 6b989b33647e..553379e404fc 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -510,10 +510,10 @@ sub sigpipe_handler { # handles SIGPIPE from @WQ_KEYS workers
}
sub fail ($$;$) {
- my ($self, $buf, $exit_code) = @_;
+ my ($self, $msg, $exit_code) = @_;
local $current_lei = $self;
$self->{failed}++;
- warn($buf, "\n") if defined $buf;
+ warn(substr($msg, -1, 1) eq "\n" ? $msg : "$msg\n") if defined $msg;
$self->{pkt_op_p}->pkt_do('fail_handler') if $self->{pkt_op_p};
x_it($self, ($exit_code // 1) << 8);
undef;
@@ -534,7 +534,7 @@ sub child_error { # passes non-fatal curl exit codes to user
my ($self, $child_error, $msg) = @_; # child_error is $?
local $current_lei = $self;
$child_error ||= 1 << 8;
- warn($msg, "\n") if defined $msg;
+ warn(substr($msg, -1, 1) eq "\n" ? $msg : "$msg\n") if defined $msg;
if ($self->{pkt_op_p}) { # to top lei-daemon
$self->{pkt_op_p}->pkt_do('child_error', $child_error);
} elsif ($self->{sock}) { # to lei(1) client
^ permalink raw reply related [relevance 70%]
* [PATCH 02/11] lei up: prefix `remote' and `local' with `o_'
2021-10-19 9:33 70% [PATCH 00/11] refining lei up+inspect Eric Wong
@ 2021-10-19 9:33 65% ` Eric Wong
2021-10-19 9:33 50% ` [PATCH 03/11] lei: use die for external and query handling Eric Wong
` (7 subsequent siblings)
8 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-10-19 9:33 UTC (permalink / raw)
To: meta
This will help distinguish between mail outputs and external
public-inboxes.
---
lib/PublicInbox/LeiUp.pm | 20 ++++++++++----------
1 file changed, 10 insertions(+), 10 deletions(-)
diff --git a/lib/PublicInbox/LeiUp.pm b/lib/PublicInbox/LeiUp.pm
index 396041771ff9..c35b2e42d49f 100644
--- a/lib/PublicInbox/LeiUp.pm
+++ b/lib/PublicInbox/LeiUp.pm
@@ -46,7 +46,7 @@ sub up1 ($$) {
sub redispatch_all ($$) {
my ($self, $lei) = @_;
- my $upq = [ (@{$self->{local} // []}, @{$self->{remote} // []}) ];
+ my $upq = [ (@{$self->{o_local} // []}, @{$self->{o_remote} // []}) ];
return up1($lei, $upq->[0]) if @$upq == 1; # just one, may start MUA
# FIXME: this is also used per-query, see lei->_start_query
@@ -81,12 +81,12 @@ sub lei_up {
$lei->fail('--all and --mua= are incompatible');
@outs = PublicInbox::LeiSavedSearch::list($lei);
if ($all eq 'local') {
- $self->{local} = [ grep(!/$REMOTE_RE/, @outs) ];
+ $self->{o_local} = [ grep(!/$REMOTE_RE/, @outs) ];
} elsif ($all eq 'remote') {
- $self->{remote} = [ grep(/$REMOTE_RE/, @outs) ];
+ $self->{o_remote} = [ grep(/$REMOTE_RE/, @outs) ];
} elsif ($all eq '') {
- $self->{remote} = [ grep(/$REMOTE_RE/, @outs) ];
- $self->{local} = [ grep(!/$REMOTE_RE/, @outs) ];
+ $self->{o_remote} = [ grep(/$REMOTE_RE/, @outs) ];
+ $self->{o_local} = [ grep(!/$REMOTE_RE/, @outs) ];
} else {
$lei->fail("only --all=$all not understood");
}
@@ -94,16 +94,16 @@ sub lei_up {
scalar(@outs) == 1 or die "BUG: lse set w/ >1 out[@outs]";
return up1($lei, $outs[0]);
} else {
- $self->{remote} = [ grep(/$REMOTE_RE/, @outs) ];
- $self->{local} = [ grep(!/$REMOTE_RE/, @outs) ];
+ $self->{o_remote} = [ grep(/$REMOTE_RE/, @outs) ];
+ $self->{o_local} = [ grep(!/$REMOTE_RE/, @outs) ];
}
$lei->{lse} = $lei->_lei_store(1)->write_prepare($lei)->search;
- ((@{$self->{local} // []} + @{$self->{remote} // []}) > 1 &&
+ ((@{$self->{o_local} // []} + @{$self->{o_remote} // []}) > 1 &&
defined($opt->{mua})) and return $lei->fail(<<EOM);
multiple outputs and --mua= are incompatible
EOM
- if ($self->{remote}) { # setup lei->{auth}
- $self->prepare_inputs($lei, $self->{remote}) or return;
+ if ($self->{o_remote}) { # setup lei->{auth}
+ $self->prepare_inputs($lei, $self->{o_remote}) or return;
}
if ($lei->{auth}) { # start auth worker
require PublicInbox::NetWriter;
^ permalink raw reply related [relevance 65%]
* [PATCH 03/11] lei: use die for external and query handling
2021-10-19 9:33 70% [PATCH 00/11] refining lei up+inspect Eric Wong
2021-10-19 9:33 65% ` [PATCH 02/11] lei up: prefix `remote' and `local' with `o_' Eric Wong
@ 2021-10-19 9:33 50% ` Eric Wong
2021-10-19 9:33 71% ` [PATCH 04/11] lei up: propagate redispatch_all failure via exit code Eric Wong
` (6 subsequent siblings)
8 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-10-19 9:33 UTC (permalink / raw)
To: meta
This allows "lei up" to continue processing unrelated externals
if on output fails.
---
lib/PublicInbox/LeiExternal.pm | 4 ++--
lib/PublicInbox/LeiQuery.pm | 12 +++++-------
lib/PublicInbox/LeiSavedSearch.pm | 8 ++++----
lib/PublicInbox/LeiUp.pm | 4 ++--
lib/PublicInbox/LeiXSearch.pm | 7 ++++---
5 files changed, 17 insertions(+), 18 deletions(-)
diff --git a/lib/PublicInbox/LeiExternal.pm b/lib/PublicInbox/LeiExternal.pm
index 701d1ad53adf..851715d7099c 100644
--- a/lib/PublicInbox/LeiExternal.pm
+++ b/lib/PublicInbox/LeiExternal.pm
@@ -101,9 +101,9 @@ sub get_externals {
return (ext_canonicalize($loc));
}
if (scalar(@m) == 0) {
- $self->fail("`$loc' is unknown");
+ die "`$loc' is unknown\n";
} else {
- $self->fail("`$loc' is ambiguous:\n", map { "\t$_\n" } @m);
+ die("`$loc' is ambiguous:\n", map { "\t$_\n" } @m, "\n");
}
();
}
diff --git a/lib/PublicInbox/LeiQuery.pm b/lib/PublicInbox/LeiQuery.pm
index ec2ece051492..56b82acc8b5b 100644
--- a/lib/PublicInbox/LeiQuery.pm
+++ b/lib/PublicInbox/LeiQuery.pm
@@ -18,17 +18,15 @@ sub _start_query { # used by "lei q" and "lei up"
PublicInbox::LeiOverview->new($self) or return;
my $opt = $self->{opt};
my ($xj, $mj) = split(/,/, $opt->{jobs} // '');
- if (defined($xj) && $xj ne '' && $xj !~ /\A[1-9][0-9]*\z/) {
- return $self->fail("`$xj' search jobs must be >= 1");
- }
+ (defined($xj) && $xj ne '' && $xj !~ /\A[1-9][0-9]*\z/) and
+ die "`$xj' search jobs must be >= 1\n";
my $lxs = $self->{lxs};
$xj ||= $lxs->concurrency($opt); # allow: "--jobs ,$WRITER_ONLY"
my $nproc = $lxs->detect_nproc || 1; # don't memoize, schedtool(1) exists
$xj = $nproc if $xj > $nproc;
$lxs->{-wq_nr_workers} = $xj;
- if (defined($mj) && $mj !~ /\A[1-9][0-9]*\z/) {
- return $self->fail("`$mj' writer jobs must be >= 1");
- }
+ (defined($mj) && $mj !~ /\A[1-9][0-9]*\z/) and
+ die "`$mj' writer jobs must be >= 1\n";
my $l2m = $self->{l2m};
# we use \1 (a ref) to distinguish between default vs. user-supplied
if ($l2m && grep { $opt->{$_} //= \1 } (qw(mail-sync import-remote
@@ -112,7 +110,7 @@ sub lxs_prepare {
}
}
($lxs->locals || $lxs->remotes) ? ($self->{lxs} = $lxs) :
- $self->fail('no local or remote inboxes to search');
+ die("no local or remote inboxes to search\n");
}
# the main "lei q SEARCH_TERMS" method
diff --git a/lib/PublicInbox/LeiSavedSearch.pm b/lib/PublicInbox/LeiSavedSearch.pm
index 3e10f780ad02..b2f1ad1027d3 100644
--- a/lib/PublicInbox/LeiSavedSearch.pm
+++ b/lib/PublicInbox/LeiSavedSearch.pm
@@ -96,7 +96,7 @@ sub translate_dedupe ($$) {
my $dd = $lei->{opt}->{dedupe} // 'content';
return 1 if $dd eq 'content'; # the default
return $self->{"-dedupe_$dd"} = 1 if ($dd eq 'oid' || $dd eq 'mid');
- $lei->fail("--dedupe=$dd requires --no-save");
+ die("--dedupe=$dd requires --no-save\n");
}
sub up { # updating existing saved search via "lei up"
@@ -105,9 +105,9 @@ sub up { # updating existing saved search via "lei up"
my $self = bless { ale => $lei->ale }, $cls;
my $dir = $dst;
output2lssdir($self, $lei, \$dir, \$f) or
- return $lei->fail("--no-save was used with $dst cwd=".
- $lei->rel2abs('.'));
- $self->{-cfg} = $lei->cfg_dump($f) // return $lei->fail;
+ return die("--no-save was used with $dst cwd=".
+ $lei->rel2abs('.')."\n");
+ $self->{-cfg} = $lei->cfg_dump($f) // return $lei->child_error;
$self->{-ovf} = "$dir/over.sqlite3";
$self->{'-f'} = $f;
$self->{lock_path} = "$self->{-f}.flock";
diff --git a/lib/PublicInbox/LeiUp.pm b/lib/PublicInbox/LeiUp.pm
index c35b2e42d49f..4fd0290c7612 100644
--- a/lib/PublicInbox/LeiUp.pm
+++ b/lib/PublicInbox/LeiUp.pm
@@ -19,7 +19,7 @@ sub up1 ($$) {
my $f = $lss->{'-f'};
my $mset_opt = $lei->{mset_opt} = { relevance => -2 };
my $q = $mset_opt->{q_raw} = $lss->{-cfg}->{'lei.q'} //
- return $lei->fail("lei.q unset in $f");
+ die("lei.q unset in $f (out=$out)\n");
my $lse = $lei->{lse} // die 'BUG: {lse} missing';
if (ref($q)) {
$mset_opt->{qstr} = $lse->query_argv_to_string($lse->git, $q);
@@ -36,7 +36,7 @@ sub up1 ($$) {
$lei->{opt}->{$k} //= $v;
}
my $o = $lei->{opt}->{output} // '';
- return $lei->fail("lei.q.output unset in $f (out=$out)") if $o eq '';
+ return die("lei.q.output unset in $f (out=$out)\n") if $o eq '';
$lss->translate_dedupe($lei) or return;
$lei->{lss} = $lss; # for LeiOverview->new and query_remote_mboxrd
my $lxs = $lei->lxs_prepare or return;
diff --git a/lib/PublicInbox/LeiXSearch.pm b/lib/PublicInbox/LeiXSearch.pm
index 8ab84b15c00b..119070a289de 100644
--- a/lib/PublicInbox/LeiXSearch.pm
+++ b/lib/PublicInbox/LeiXSearch.pm
@@ -142,11 +142,11 @@ sub wait_startq ($) {
delete $lei->{opt}->{verbose};
delete $lei->{-progress};
} else {
- $lei->fail("$$ WTF `$do_augment_done'");
+ die "BUG: do_augment_done=`$do_augment_done'";
}
return;
}
- return $lei->fail("$$ wait_startq: $!") unless $!{EINTR};
+ die "wait_startq: $!" unless $!{EINTR};
}
}
@@ -473,7 +473,8 @@ sub do_post_augment {
$lei->fail("$err");
}
if (!$err && delete $lei->{early_mua}) { # non-augment case
- $lei->start_mua;
+ eval { $lei->start_mua };
+ $lei->fail($@) if $@;
}
close(delete $lei->{au_done}); # triggers wait_startq in lei_xsearch
}
^ permalink raw reply related [relevance 50%]
* [PATCH 06/11] lei up: support --exclude=, --no-(external|remote|local)
2021-10-19 9:33 70% [PATCH 00/11] refining lei up+inspect Eric Wong
` (3 preceding siblings ...)
2021-10-19 9:33 70% ` [PATCH 05/11] lei: conditionally add "\n" to error messages Eric Wong
@ 2021-10-19 9:33 47% ` Eric Wong
2021-10-19 9:33 71% ` [PATCH 07/11] lei: remove unused ->busy time arg Eric Wong
` (3 subsequent siblings)
8 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-10-19 9:33 UTC (permalink / raw)
To: meta
These can be used to temporarily disable using certain
externals in case of temporary network failure or mount point
unavailability.
---
Documentation/lei-up.pod | 28 +++++++++++++++++++++++-----
lib/PublicInbox/LEI.pm | 4 ++--
lib/PublicInbox/LeiExternal.pm | 11 ++++++++++-
lib/PublicInbox/LeiQuery.pm | 15 ++++-----------
lib/PublicInbox/LeiUp.pm | 22 ++++++++++++++++++++++
5 files changed, 61 insertions(+), 19 deletions(-)
diff --git a/Documentation/lei-up.pod b/Documentation/lei-up.pod
index f06ee5eb62b7..8fba0953b1ef 100644
--- a/Documentation/lei-up.pod
+++ b/Documentation/lei-up.pod
@@ -37,11 +37,26 @@ e.g C<1.hour> or C<3.days>
Default: 2.days
-=back
+=item --no-external
-The following options, described in L<lei-q(1)>, are supported.
+=item --no-local
-=over
+=item --no-remote
+
+These disable the use of all externals, local externals, or
+remote externals respectively. They are useful during
+temporary network or mount-point outages.
+
+Unlike C<lei q>, these switches override the original C<lei q --only>
+options saved as C<lei.q.only>.
+
+The combination C<--all=remote --no-remote> is supported for
+offline use in case a user is updating an IMAP folder on localhost.
+
+=item --exclude=LOCATION
+
+As with L<lei-q(1)>, but may also exclude externals originally
+specified via C<lei q --only>.
=item --lock=METHOD
@@ -49,7 +64,10 @@ The following options, described in L<lei-q(1)>, are supported.
=item --mua=CMD
-This option is incompatible with C<--all>.
+C<--lock>, C<--alert>, and C<--mua> are all supported and
+documented in L<lei-q(1)>.
+
+C<--mua> is incompatible with C<--all>.
=back
@@ -62,7 +80,7 @@ and L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta
=head1 COPYRIGHT
-Copyright 2021 all contributors L<mailto:meta@public-inbox.org>
+Copyright all contributors L<mailto:meta@public-inbox.org>
License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 553379e404fc..5b726f71382e 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -178,8 +178,8 @@ our %CMD = ( # sorted in order of importance/use:
shared color! mail-sync!), @c_opt, opt_dash('limit|n=i', '[0-9]+') ],
'up' => [ 'OUTPUT...|--all', 'update saved search',
- qw(jobs|j=s lock=s@ alert=s@ mua=s verbose|v+
- remote-fudge-time=s all:s), @c_opt ],
+ qw(jobs|j=s lock=s@ alert=s@ mua=s verbose|v+ exclude=s@
+ remote-fudge-time=s all:s remote! local! external!), @c_opt ],
'lcat' => [ '--stdin|MSGID_OR_URL...', 'display local copy of message(s)',
'stdin|', # /|\z/ must be first for lone dash
diff --git a/lib/PublicInbox/LeiExternal.pm b/lib/PublicInbox/LeiExternal.pm
index 851715d7099c..30bb1a4579c7 100644
--- a/lib/PublicInbox/LeiExternal.pm
+++ b/lib/PublicInbox/LeiExternal.pm
@@ -105,7 +105,16 @@ sub get_externals {
} else {
die("`$loc' is ambiguous:\n", map { "\t$_\n" } @m, "\n");
}
- ();
+}
+
+sub canonicalize_excludes {
+ my ($lei, $excludes) = @_;
+ my %x;
+ for my $loc (@$excludes) {
+ my @l = get_externals($lei, $loc, 1);
+ $x{$_} = 1 for @l;
+ }
+ \%x;
}
# returns an anonymous sub which returns an array of potential results
diff --git a/lib/PublicInbox/LeiQuery.pm b/lib/PublicInbox/LeiQuery.pm
index 56b82acc8b5b..effc572f6aaf 100644
--- a/lib/PublicInbox/LeiQuery.pm
+++ b/lib/PublicInbox/LeiQuery.pm
@@ -95,18 +95,11 @@ sub lxs_prepare {
}
# --external is enabled by default, but allow --no-external
if ($opt->{external} //= 1) {
- my %x;
- for my $loc (@{$opt->{exclude} // []}) {
- my @l = $self->get_externals($loc, 1) or return;
- $x{$_} = 1 for @l;
- }
- my $ne = $self->externals_each(\&prep_ext, $lxs, \%x);
+ my $ex = $self->canonicalize_excludes($opt->{exclude});
+ $self->externals_each(\&prep_ext, $lxs, $ex);
$opt->{remote} //= !($lxs->locals - $opt->{'local'});
- if ($opt->{'local'}) {
- $lxs->{remotes} = \@iremotes if !$opt->{remote};
- } else {
- $lxs->{locals} = \@ilocals;
- }
+ $lxs->{locals} = \@ilocals if !$opt->{'local'};
+ $lxs->{remotes} = \@iremotes if !$opt->{remote};
}
}
($lxs->locals || $lxs->remotes) ? ($self->{lxs} = $lxs) :
diff --git a/lib/PublicInbox/LeiUp.pm b/lib/PublicInbox/LeiUp.pm
index fcdd535dc118..dac0fc287885 100644
--- a/lib/PublicInbox/LeiUp.pm
+++ b/lib/PublicInbox/LeiUp.pm
@@ -15,6 +15,14 @@ my $REMOTE_RE = qr!\A(?:imap|http)s?://!i; # http(s) will be for JMAP
sub up1 ($$) {
my ($lei, $out) = @_;
+ # precedence note for CLI switches between lei q and up:
+ # `lei q --only' > `lei q --no-(remote|local|external)'
+ # `lei up --no-(remote|local|external)' > `lei.q.only' in saved search
+ my %no = map {
+ my $v = $lei->{opt}->{$_}; # set by CLI
+ (defined($v) && !$v) ? ($_ => 1) : ();
+ } qw(remote local external);
+ my $cli_exclude = delete $lei->{opt}->{exclude};
my $lss = PublicInbox::LeiSavedSearch->up($lei, $out) or return;
my $f = $lss->{'-f'};
my $mset_opt = $lei->{mset_opt} = { relevance => -2 };
@@ -31,6 +39,20 @@ sub up1 ($$) {
my $v = $lss->{-cfg}->get_all("lei.q.$k") // next;
$lei->{opt}->{$k} //= $v;
}
+
+ # --no-(local|remote) CLI flags overrided saved `lei.q.only'
+ my $only = $lei->{opt}->{only};
+ @$only = map { $lei->get_externals($_) } @$only if $only;
+ if (scalar keys %no && $only) {
+ @$only = grep(!m!\Ahttps?://!i, @$only) if $no{remote};
+ @$only = grep(m!\Ahttps?://!i, @$only) if $no{'local'};
+ }
+ if ($cli_exclude) {
+ my $ex = $lei->canonicalize_excludes($cli_exclude);
+ @$only = grep { !$ex->{$_} } @$only if $only;
+ push @{$lei->{opt}->{exclude}}, @$cli_exclude;
+ }
+ delete $lei->{opt}->{only} if $no{external} || ($only && !@$only);
for my $k ($lss->BOOL_FIELDS, $lss->SINGLE_FIELDS) {
my $v = $lss->{-cfg}->get_1("lei.q.$k") // next;
$lei->{opt}->{$k} //= $v;
^ permalink raw reply related [relevance 47%]
* Re: [PATCH 2/2] doc: lei: add manpages for remaining commands
2021-10-16 17:03 71% ` Eric Wong
@ 2021-10-16 17:21 71% ` Kyle Meyer
0 siblings, 0 replies; 200+ results
From: Kyle Meyer @ 2021-10-16 17:21 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
Eric Wong writes:
> The failure is probably caused by
> 00d5dff2cce9d2c9 (eml: avoid Encode 2.87..3.12 leak, 2021-10-13)
> but I can't reproduce it across CentOS 7.x, FreeBSD 11.x, nor
> Debian 10 & 11.
Hmm, yeah, as mentioned in my other reply, I'm now not having any luck
triggering this either.
> Which versions of Encode and Perl are you using?
Perl v5.32.1 from Debian 11 and...
> At least in Debian, libencode-perl is available as a separate package
> but it's also part of libperl5.xx (possibly w/ a different version);
> only the latter is required for us, but two packages offering
> the same thing gets confusing :/
>
> I use: perl -MEncode -E 'say $Encode::VERSION'
> to determine which gets loaded.
... it looks like I'm using the on that's ships with libperl5:
$ perl -MEncode -E 'say $Encode::VERSION'
3.06
$ apt-cache policy libencode-perl
libencode-perl:
Installed: (none)
Candidate: 3.08-1+deb11u1
Version table:
3.08-1+deb11u1 500
500 http://ftp.us.debian.org/debian bullseye/main amd64 Packages
500 http://security.debian.org/debian-security bullseye-security/main amd64 Packages
^ permalink raw reply [relevance 71%]
* Re: [PATCH 2/2] doc: lei: add manpages for remaining commands
2021-10-16 15:13 67% ` Kyle Meyer
2021-10-16 16:58 71% ` Kyle Meyer
@ 2021-10-16 17:03 71% ` Eric Wong
2021-10-16 17:21 71% ` Kyle Meyer
1 sibling, 1 reply; 200+ results
From: Eric Wong @ 2021-10-16 17:03 UTC (permalink / raw)
To: Kyle Meyer; +Cc: meta
Kyle Meyer <kyle@kyleam.com> wrote:
> Thanks, sorry about the mix up.
No worries, will push a regen.
> [*] I did run `make check' before sending, and it looks like that also
> checks MANIFEST _after_ running the test suite. That didn't help me
> catch the MANIFEST sorting issue in this case because the test suite
> is failing on my end.
>
> I believe these failures are recent and was planning on looking into
> them today, either sending a patch or just reporting, depending on
> whether I could figure out a fix. Anyway, here they are:
>
> t/psgi_multipart_not.t .......
> ok 1 - use HTTP::Request::Common;
> ok 2 - use Plack::Test;
> ok 3 - use PublicInbox::WWW;
> not ok 4 - /v2test/?q=%22ain't what it used to be%22&x=t
> not ok 5 - /v2test/?q=%22ain't what it used to be%22&x=t warns
> Failed 2/5 subtests
The failure is probably caused by
00d5dff2cce9d2c9 (eml: avoid Encode 2.87..3.12 leak, 2021-10-13)
but I can't reproduce it across CentOS 7.x, FreeBSD 11.x, nor
Debian 10 & 11.
Which versions of Encode and Perl are you using?
At least in Debian, libencode-perl is available as a separate package
but it's also part of libperl5.xx (possibly w/ a different version);
only the latter is required for us, but two packages offering
the same thing gets confusing :/
I use: perl -MEncode -E 'say $Encode::VERSION'
to determine which gets loaded.
^ permalink raw reply [relevance 71%]
* Re: [PATCH 2/2] doc: lei: add manpages for remaining commands
2021-10-16 15:13 67% ` Kyle Meyer
@ 2021-10-16 16:58 71% ` Kyle Meyer
2021-10-16 17:03 71% ` Eric Wong
1 sibling, 0 replies; 200+ results
From: Kyle Meyer @ 2021-10-16 16:58 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
Kyle Meyer writes:
> t/psgi_multipart_not.t .......
> ok 1 - use HTTP::Request::Common;
> ok 2 - use Plack::Test;
> ok 3 - use PublicInbox::WWW;
> not ok 4 - /v2test/?q=%22ain't what it used to be%22&x=t
> not ok 5 - /v2test/?q=%22ain't what it used to be%22&x=t warns
> Failed 2/5 subtests
Hmm, trying a few times on 9ee1798e (2021-10-16), I'm not having any
luck triggering it again. All tests are now passing for me.
^ permalink raw reply [relevance 71%]
* Re: [PATCH 2/2] doc: lei: add manpages for remaining commands
2021-10-16 7:07 71% ` Eric Wong
@ 2021-10-16 15:13 67% ` Kyle Meyer
2021-10-16 16:58 71% ` Kyle Meyer
2021-10-16 17:03 71% ` Eric Wong
0 siblings, 2 replies; 200+ results
From: Kyle Meyer @ 2021-10-16 15:13 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
Eric Wong writes:
>> @@ -47,6 +53,7 @@ Documentation/lei-q.pod
>> Documentation/lei-rediff.pod
>> Documentation/lei-refresh-mail-sync.pod
>> Documentation/lei-rm.pod
>> +Documentation/lei-rm-watch.pod
>> Documentation/lei-security.pod
>> Documentation/lei-store-format.pod
>> Documentation/lei-tag.pod
>
> Curious, was that from `git ls-files >MANIFEST' ?
No, it wasn't. I hadn't realized that the content of MANIFEST
corresponded directly to git-ls-files output, but poking around I see
there is even a check-manifest make target [*]. Doh.
So, I manually added this one, like the others I added. I think my past
changes to this file have ended up in the expected order because I tend
to run Emacs's sort-lines to check my manual placement, but I must not
have done it here because sort-lines agrees with ls-files.
[...]
> No big deal, I can just flip and push it; I just don't want a
> reproducibility issue popping up.
Thanks, sorry about the mix up.
[*] I did run `make check' before sending, and it looks like that also
checks MANIFEST _after_ running the test suite. That didn't help me
catch the MANIFEST sorting issue in this case because the test suite
is failing on my end.
I believe these failures are recent and was planning on looking into
them today, either sending a patch or just reporting, depending on
whether I could figure out a fix. Anyway, here they are:
t/psgi_multipart_not.t .......
ok 1 - use HTTP::Request::Common;
ok 2 - use Plack::Test;
ok 3 - use PublicInbox::WWW;
not ok 4 - /v2test/?q=%22ain't what it used to be%22&x=t
not ok 5 - /v2test/?q=%22ain't what it used to be%22&x=t warns
Failed 2/5 subtests
^ permalink raw reply [relevance 67%]
* [PATCH 0/4] lei: prioritize signals
@ 2021-10-16 9:29 71% Eric Wong
2021-10-16 9:29 57% ` [PATCH 4/4] lei sockets: favor level-triggered epoll for fairness Eric Wong
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2021-10-16 9:29 UTC (permalink / raw)
To: meta
We drop Edge-Triggered kevent/epoll in nearly all places
to give signalfd / EVFILT_SIGNAL priority.
Eric Wong (4):
wqworker: favor level-triggered epoll for fairness
pkt_op: favor level-triggered epoll for fairness
input_pipe: do not loop in ->event_step for fairness
lei sockets: favor level-triggered epoll for fairness
lib/PublicInbox/InputPipe.pm | 17 +++++-----
lib/PublicInbox/LEI.pm | 20 +++++-------
lib/PublicInbox/LeiSelfSocket.pm | 27 +++++++---------
lib/PublicInbox/PktOp.pm | 53 +++++++++++++++-----------------
lib/PublicInbox/WQWorker.pm | 18 +++++------
5 files changed, 60 insertions(+), 75 deletions(-)
^ permalink raw reply [relevance 71%]
* [PATCH 4/4] lei sockets: favor level-triggered epoll for fairness
2021-10-16 9:29 71% [PATCH 0/4] lei: prioritize signals Eric Wong
@ 2021-10-16 9:29 57% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-10-16 9:29 UTC (permalink / raw)
To: meta
Sigfd->event_step needs priority over script/lei clients,
LeiSelfSocket, and everything else.
---
lib/PublicInbox/LEI.pm | 20 ++++++++------------
lib/PublicInbox/LeiSelfSocket.pm | 27 +++++++++++----------------
2 files changed, 19 insertions(+), 28 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 876598f9530e..6b989b33647e 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -12,13 +12,13 @@ use parent qw(PublicInbox::DS PublicInbox::LeiExternal
PublicInbox::LeiQuery);
use Getopt::Long ();
use Socket qw(AF_UNIX SOCK_SEQPACKET MSG_EOR pack_sockaddr_un);
-use Errno qw(EPIPE EAGAIN EINTR ECONNREFUSED ENOENT ECONNRESET);
+use Errno qw(EPIPE EAGAIN ECONNREFUSED ENOENT ECONNRESET);
use Cwd qw(getcwd);
use POSIX qw(strftime);
use IO::Handle ();
use Fcntl qw(SEEK_SET);
use PublicInbox::Config;
-use PublicInbox::Syscall qw(EPOLLIN EPOLLET);
+use PublicInbox::Syscall qw(EPOLLIN);
use PublicInbox::DS qw(now dwaitpid);
use PublicInbox::Spawn qw(spawn popen_rd);
use PublicInbox::Lock;
@@ -1125,16 +1125,12 @@ sub event_step {
local %ENV = %{$self->{env}};
local $current_lei = $self;
eval {
- my $buf;
- while (my @fds = $recv_cmd->($self->{sock}, $buf, 4096)) {
- if (scalar(@fds) == 1 && !defined($fds[0])) {
- return if $! == EAGAIN;
- next if $! == EINTR;
- last if $! == ECONNRESET;
- die "recvmsg: $!";
- }
- for (@fds) { open my $rfh, '+<&=', $_ }
+ my @fds = $recv_cmd->($self->{sock}, my $buf, 4096);
+ if (scalar(@fds) == 1 && !defined($fds[0])) {
+ return if $! == EAGAIN;
+ die "recvmsg: $!" if $! != ECONNRESET;
}
+ for (@fds) { open my $rfh, '+<&=', $_ }
if ($buf eq '') {
_drop_wq($self); # EOF, client disconnected
dclose($self);
@@ -1162,7 +1158,7 @@ sub event_step_init {
my $sock = $self->{sock} or return;
$self->{-event_init_done} //= do { # persist til $ops done
$sock->blocking(0);
- $self->SUPER::new($sock, EPOLLIN|EPOLLET);
+ $self->SUPER::new($sock, EPOLLIN);
$sock;
};
}
diff --git a/lib/PublicInbox/LeiSelfSocket.pm b/lib/PublicInbox/LeiSelfSocket.pm
index 3d847649e3b1..dd64b6cfd491 100644
--- a/lib/PublicInbox/LeiSelfSocket.pm
+++ b/lib/PublicInbox/LeiSelfSocket.pm
@@ -10,7 +10,7 @@ use v5.10.1;
use parent qw(PublicInbox::DS);
use Data::Dumper;
$Data::Dumper::Useqq = 1; # should've been the Perl default :P
-use PublicInbox::Syscall qw(EPOLLIN EPOLLET);
+use PublicInbox::Syscall qw(EPOLLIN);
use PublicInbox::Spawn;
my $recv_cmd;
@@ -20,26 +20,21 @@ sub new {
$r->blocking(0);
no warnings 'once';
$recv_cmd = $PublicInbox::LEI::recv_cmd;
- $self->SUPER::new($r, EPOLLIN|EPOLLET);
+ $self->SUPER::new($r, EPOLLIN);
}
sub event_step {
my ($self) = @_;
- while (1) {
- my (@fds) = $recv_cmd->($self->{sock}, my $buf, 4096 * 33);
- if (scalar(@fds) == 1 && !defined($fds[0])) {
- return if $!{EAGAIN};
- next if $!{EINTR};
- die "recvmsg: $!";
- }
- # open so perl can auto-close them:
- for my $fd (@fds) {
- open(my $newfh, '+<&=', $fd) or die "open +<&=$fd: $!";
- }
- return $self->close if $buf eq '';
- warn Dumper({ 'unexpected self msg' => $buf, fds => \@fds });
- # TODO: figure out what to do with these messages...
+ my (@fds) = $recv_cmd->($self->{sock}, my $buf, 4096 * 33);
+ if (scalar(@fds) == 1 && !defined($fds[0])) {
+ return if $!{EAGAIN};
+ die "recvmsg: $!" unless $!{ECONNRESET};
+ } else { # just in case open so perl can auto-close them:
+ for (@fds) { open my $fh, '+<&=', $_ };
}
+ return $self->close if $buf eq '';
+ warn Dumper({ 'unexpected self msg' => $buf, fds => \@fds });
+ # TODO: figure out what to do with these messages...
}
1;
^ permalink raw reply related [relevance 57%]
* [PATCH] t/lei*: set EDITOR for dumb terminals
@ 2021-10-16 7:54 70% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-10-16 7:54 UTC (permalink / raw)
To: meta
Running tests over a non-interactive ssh session fails,
otherwise.
---
t/lei-q-remote-import.t | 2 +-
t/lei.t | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/t/lei-q-remote-import.t b/t/lei-q-remote-import.t
index fdf6a11e..92d8c9b6 100644
--- a/t/lei-q-remote-import.t
+++ b/t/lei-q-remote-import.t
@@ -99,7 +99,7 @@ EOF
lei_ok('up', "$ENV{HOME}/md");
is_deeply(\@f, [ glob("$ENV{HOME}/md/*/*") ],
'lei up remote dedupe works on maildir');
- my $edit_env = { VISUAL => 'cat' };
+ my $edit_env = { VISUAL => 'cat', EDITOR => 'cat' };
lei_ok([qw(edit-search), "$ENV{HOME}/md"], $edit_env);
like($lei_out, qr/^\Q[external "$url"]\E\n\s*lastresult = \d+/sm,
'lastresult set');
diff --git a/t/lei.t b/t/lei.t
index 53fc43fb..f7de1b71 100644
--- a/t/lei.t
+++ b/t/lei.t
@@ -101,7 +101,7 @@ my $test_config = sub {
lei_ok(qw(-c imap.debug=a -c imap.debug=b config --get-all imap.debug));
is($lei_out, "a\nb\n", '-c and --get-all work together');
- lei_ok([qw(config -e)], { VISUAL => 'cat' });
+ lei_ok([qw(config -e)], { VISUAL => 'cat', EDITOR => 'cat' });
is($lei_out, "[a]\n\tb = c\n", '--edit works');
};
^ permalink raw reply related [relevance 70%]
* Re: [PATCH 2/2] doc: lei: add manpages for remaining commands
2021-10-16 5:39 29% ` [PATCH 2/2] doc: lei: add manpages for remaining commands Kyle Meyer
@ 2021-10-16 7:07 71% ` Eric Wong
2021-10-16 15:13 67% ` Kyle Meyer
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2021-10-16 7:07 UTC (permalink / raw)
To: Kyle Meyer; +Cc: meta
Kyle Meyer <kyle@kyleam.com> wrote:
> At this point all of the current lei commands, aside from -help and
> -sucks, should be covered.
Thanks, pushed as commit 9d72cc3f876e3d2bd1ecb2fc0f33c43a9a72b933
> diff --git a/MANIFEST b/MANIFEST
> index b89513d5..c4cc6e33 100644
> --- a/MANIFEST
> +++ b/MANIFEST
> @@ -47,6 +53,7 @@ Documentation/lei-q.pod
> Documentation/lei-rediff.pod
> Documentation/lei-refresh-mail-sync.pod
> Documentation/lei-rm.pod
> +Documentation/lei-rm-watch.pod
> Documentation/lei-security.pod
> Documentation/lei-store-format.pod
> Documentation/lei-tag.pod
Curious, was that from `git ls-files >MANIFEST' ?
Because that puts lei-rm.pod after lei-rm-watch.pod on different
systems I tested. I don't see git using LC_COLLATE anywhere,
but locale(1) gives:
FreeBSD:
LANG=
LC_CTYPE="C"
LC_COLLATE="C"
LC_TIME="C"
LC_NUMERIC="C"
LC_MONETARY="C"
LC_MESSAGES="C"
LC_ALL=
Debian:
LANG=en_US.UTF-8
LANGUAGE=
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=
No big deal, I can just flip and push it; I just don't want a
reproducibility issue popping up.
^ permalink raw reply [relevance 71%]
* [PATCH 0/2] doc: lei manpages for remaining commands
@ 2021-10-16 5:39 70% Kyle Meyer
2021-10-16 5:39 65% ` [PATCH 1/2] doc: lei: restore alphabetical order to some listings Kyle Meyer
2021-10-16 5:39 29% ` [PATCH 2/2] doc: lei: add manpages for remaining commands Kyle Meyer
0 siblings, 2 replies; 200+ results
From: Kyle Meyer @ 2021-10-16 5:39 UTC (permalink / raw)
To: meta
This lei manpage update should take care of the remaining lei commands
(aside from -help and -sucks) that don't have manpages.
[1/2] doc: lei: restore alphabetical order to some listings
[2/2] doc: lei: add manpages for remaining commands
Documentation/lei-add-watch.pod | 33 ++++++++++++++
Documentation/lei-forget-mail-sync.pod | 31 +++++++++++++
Documentation/lei-inspect.pod | 57 ++++++++++++++++++++++++
Documentation/lei-ls-mail-source.pod | 58 +++++++++++++++++++++++++
Documentation/lei-ls-mail-sync.pod | 2 +-
Documentation/lei-ls-watch.pod | 29 +++++++++++++
Documentation/lei-mail-diff.pod | 33 ++++++++++++++
Documentation/lei-refresh-mail-sync.pod | 2 +-
Documentation/lei-rm-watch.pod | 30 +++++++++++++
Documentation/lei.pod | 14 ++++++
Documentation/txt2pre | 9 +++-
MANIFEST | 7 +++
Makefile.PL | 14 +++---
13 files changed, 309 insertions(+), 10 deletions(-)
create mode 100644 Documentation/lei-add-watch.pod
create mode 100644 Documentation/lei-forget-mail-sync.pod
create mode 100644 Documentation/lei-inspect.pod
create mode 100644 Documentation/lei-ls-mail-source.pod
create mode 100644 Documentation/lei-ls-watch.pod
create mode 100644 Documentation/lei-mail-diff.pod
create mode 100644 Documentation/lei-rm-watch.pod
base-commit: 299b40d252cf4d4db6fa29ad18cb78777f1f55fc
--
2.33.0
^ permalink raw reply [relevance 70%]
* [PATCH 2/2] doc: lei: add manpages for remaining commands
2021-10-16 5:39 70% [PATCH 0/2] doc: lei manpages for remaining commands Kyle Meyer
2021-10-16 5:39 65% ` [PATCH 1/2] doc: lei: restore alphabetical order to some listings Kyle Meyer
@ 2021-10-16 5:39 29% ` Kyle Meyer
2021-10-16 7:07 71% ` Eric Wong
1 sibling, 1 reply; 200+ results
From: Kyle Meyer @ 2021-10-16 5:39 UTC (permalink / raw)
To: meta
At this point all of the current lei commands, aside from -help and
-sucks, should be covered.
---
Documentation/lei-add-watch.pod | 33 ++++++++++++++
Documentation/lei-forget-mail-sync.pod | 31 +++++++++++++
Documentation/lei-inspect.pod | 57 ++++++++++++++++++++++++
Documentation/lei-ls-mail-source.pod | 58 +++++++++++++++++++++++++
Documentation/lei-ls-mail-sync.pod | 2 +-
Documentation/lei-ls-watch.pod | 29 +++++++++++++
Documentation/lei-mail-diff.pod | 33 ++++++++++++++
Documentation/lei-refresh-mail-sync.pod | 2 +-
Documentation/lei-rm-watch.pod | 30 +++++++++++++
Documentation/lei.pod | 14 ++++++
Documentation/txt2pre | 7 +++
MANIFEST | 7 +++
Makefile.PL | 12 ++---
13 files changed, 307 insertions(+), 8 deletions(-)
create mode 100644 Documentation/lei-add-watch.pod
create mode 100644 Documentation/lei-forget-mail-sync.pod
create mode 100644 Documentation/lei-inspect.pod
create mode 100644 Documentation/lei-ls-mail-source.pod
create mode 100644 Documentation/lei-ls-watch.pod
create mode 100644 Documentation/lei-mail-diff.pod
create mode 100644 Documentation/lei-rm-watch.pod
diff --git a/Documentation/lei-add-watch.pod b/Documentation/lei-add-watch.pod
new file mode 100644
index 00000000..60984618
--- /dev/null
+++ b/Documentation/lei-add-watch.pod
@@ -0,0 +1,33 @@
+=head1 NAME
+
+lei-add-watch - watch for new messages and flag changes
+
+=head1 SYNOPSIS
+
+lei add-watch [OPTIONS] LOCATION [LOCATION...]
+
+=head1 DESCRIPTION
+
+Tell lei to watch C<LOCATION> for new messages and flag changes.
+Currently only Maildir locations are supported.
+
+=for comment
+TODO: Document --state? Believe valid values are pause, import-ro,
+
+=head1 CONTACT
+
+Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org>
+
+The mail archives are hosted at L<https://public-inbox.org/meta/> and
+L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/>
+
+=head1 COPYRIGHT
+
+Copyright 2021 all contributors L<mailto:meta@public-inbox.org>
+
+License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>
+
+
+=head1 SEE ALSO
+
+L<lei-ls-watch(1)>, L<lei-rm-watch(1)>
diff --git a/Documentation/lei-forget-mail-sync.pod b/Documentation/lei-forget-mail-sync.pod
new file mode 100644
index 00000000..e70b4d33
--- /dev/null
+++ b/Documentation/lei-forget-mail-sync.pod
@@ -0,0 +1,31 @@
+=head1 NAME
+
+lei-forget-mail-sync - forget sync information for a mail folder
+
+=head1 SYNOPSIS
+
+lei forget-mail-sync [OPTIONS] LOCATION [LOCATION...]
+
+=head1 DESCRIPTION
+
+Forget synchronization information for C<LOCATION>, an IMAP or Maildir
+folder. Note that this won't delete any messages stored in Git,
+leaving C<lei-index(1)> users with dangling references.
+
+=head1 CONTACT
+
+Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org>
+
+The mail archives are hosted at L<https://public-inbox.org/meta/> and
+L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/>
+
+=head1 COPYRIGHT
+
+Copyright 2021 all contributors L<mailto:meta@public-inbox.org>
+
+License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>
+
+
+=head1 SEE ALSO
+
+L<lei-ls-mail-sync(1)>, L<lei-index(1)>
diff --git a/Documentation/lei-inspect.pod b/Documentation/lei-inspect.pod
new file mode 100644
index 00000000..19dd8ab5
--- /dev/null
+++ b/Documentation/lei-inspect.pod
@@ -0,0 +1,57 @@
+=head1 NAME
+
+lei-inspect - general purpose inspector
+
+=head1 SYNOPSIS
+
+lei inspect [OPTIONS] ITEM [ITEM...]
+
+lei inspect [OPTIONS] (--stdin|-)
+
+=head1 DESCRIPTION
+
+This is a diagnostic command that provides a general purpose inspector
+of various things, including blobs, message IDs, Xapian document IDs,
+and mail sync sources.
+
+=head1 OPTIONS
+
+=over
+
+=item -d DIR
+
+=item --dir=DIR
+
+An inboxdir, extindex topdir, or Xapian shard
+
+=item --pretty
+
+Pretty print output. If stdout is opened to a tty, C<--pretty> is
+enabled by default.
+
+=item -
+
+=item --stdin
+
+Read message from stdin. This is implicit if no arguments are given
+and stdin is a pipe or regular file.
+
+=back
+
+=head1 CONTACT
+
+Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org>
+
+The mail archives are hosted at L<https://public-inbox.org/meta/> and
+L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/>
+
+=head1 COPYRIGHT
+
+Copyright 2021 all contributors L<mailto:meta@public-inbox.org>
+
+License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>
+
+
+=head1 SEE ALSO
+
+L<lei-mail-diff(1)>
diff --git a/Documentation/lei-ls-mail-source.pod b/Documentation/lei-ls-mail-source.pod
new file mode 100644
index 00000000..926bbe2c
--- /dev/null
+++ b/Documentation/lei-ls-mail-source.pod
@@ -0,0 +1,58 @@
+=head1 NAME
+
+lei-ls-mail-source - list IMAP or NNTP mail source folders
+
+=head1 SYNOPSIS
+
+lei ls-mail-source [OPTIONS] URL
+
+=head1 DESCRIPTION
+
+List information about the IMAP or NNTP mail source at C<URL>.
+
+=head1 OPTIONS
+
+=over
+
+=item -z
+
+=item -0
+
+Use C<\0> (NUL) instead of newline (CR) to delimit lines.
+
+=item -l
+
+Format output as JSON and include more information.
+
+=item --pretty
+
+Pretty print JSON output. If stdout is opened to a tty, C<--pretty>
+is enabled by default.
+
+=item --ascii
+
+Escape non-ASCII characters.
+
+=item --url
+
+Show full URL of newsgroup or IMAP folder.
+
+=back
+
+=head1 CONTACT
+
+Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org>
+
+The mail archives are hosted at L<https://public-inbox.org/meta/> and
+L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/>
+
+=head1 COPYRIGHT
+
+Copyright 2021 all contributors L<mailto:meta@public-inbox.org>
+
+License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>
+
+
+=head1 SEE ALSO
+
+L<lei-import(1)>
diff --git a/Documentation/lei-ls-mail-sync.pod b/Documentation/lei-ls-mail-sync.pod
index 86aede40..883eeead 100644
--- a/Documentation/lei-ls-mail-sync.pod
+++ b/Documentation/lei-ls-mail-sync.pod
@@ -52,4 +52,4 @@ License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>
=head1 SEE ALSO
-L<lei-q(1)>, L<lei-up(1)>
+L<lei-refresh-mail-sync(1)>, L<lei-export-kw(1)>
diff --git a/Documentation/lei-ls-watch.pod b/Documentation/lei-ls-watch.pod
new file mode 100644
index 00000000..b1681ee4
--- /dev/null
+++ b/Documentation/lei-ls-watch.pod
@@ -0,0 +1,29 @@
+=head1 NAME
+
+lei-ls-watch - list active watches
+
+=head1 SYNOPSIS
+
+lei ls-watch
+
+=head1 DESCRIPTION
+
+List locations that lei is configured to watch.
+
+=head1 CONTACT
+
+Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org>
+
+The mail archives are hosted at L<https://public-inbox.org/meta/> and
+L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/>
+
+=head1 COPYRIGHT
+
+Copyright 2021 all contributors L<mailto:meta@public-inbox.org>
+
+License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>
+
+
+=head1 SEE ALSO
+
+L<lei-add-watch(1)>, L<lei-rm-watch(1)>
diff --git a/Documentation/lei-mail-diff.pod b/Documentation/lei-mail-diff.pod
new file mode 100644
index 00000000..96e49a8b
--- /dev/null
+++ b/Documentation/lei-mail-diff.pod
@@ -0,0 +1,33 @@
+=head1 NAME
+
+lei-mail-diff - diff the contents of emails
+
+=head1 SYNOPSIS
+
+
+lei mail-diff [OPTIONS] LOCATION
+
+lei mail-diff [OPTIONS] (--stdin|-)
+
+=head1 DESCRIPTION
+
+This is a diagnostic command that's useful for finding deduplication
+bugs.
+
+=head1 CONTACT
+
+Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org>
+
+The mail archives are hosted at L<https://public-inbox.org/meta/> and
+L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/>
+
+=head1 COPYRIGHT
+
+Copyright 2021 all contributors L<mailto:meta@public-inbox.org>
+
+License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>
+
+
+=head1 SEE ALSO
+
+L<lei-inspect(1)>
diff --git a/Documentation/lei-refresh-mail-sync.pod b/Documentation/lei-refresh-mail-sync.pod
index 92ca9044..65150ae3 100644
--- a/Documentation/lei-refresh-mail-sync.pod
+++ b/Documentation/lei-refresh-mail-sync.pod
@@ -54,4 +54,4 @@ License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>
=head1 SEE ALSO
-L<lei-index(1)>, L<lei-export-kw(1)>
+L<lei-index(1)>, L<lei-export-kw(1)>, L<lei-ls-mail-sync(1)>
diff --git a/Documentation/lei-rm-watch.pod b/Documentation/lei-rm-watch.pod
new file mode 100644
index 00000000..711d7dc4
--- /dev/null
+++ b/Documentation/lei-rm-watch.pod
@@ -0,0 +1,30 @@
+=head1 NAME
+
+lei-rm-watch - stop watching locations
+
+=head1 SYNOPSIS
+
+lei rm-watch [OPTIONS] LOCATION [LOCATION...]
+
+=head1 DESCRIPTION
+
+Tell lei to stop watching C<LOCATION> for new messages and flag
+changes. Currently only Maildir locations are supported.
+
+=head1 CONTACT
+
+Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org>
+
+The mail archives are hosted at L<https://public-inbox.org/meta/> and
+L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/>
+
+=head1 COPYRIGHT
+
+Copyright 2021 all contributors L<mailto:meta@public-inbox.org>
+
+License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>
+
+
+=head1 SEE ALSO
+
+L<lei-add-watch(1)>, L<lei-ls-watch(1)>
diff --git a/Documentation/lei.pod b/Documentation/lei.pod
index 63d5ee69..24a585da 100644
--- a/Documentation/lei.pod
+++ b/Documentation/lei.pod
@@ -96,6 +96,8 @@ Other subcommands include
=over
+=item * L<lei-add-watch(1)>
+
=item * L<lei-config(1)>
=item * L<lei-convert(1)>
@@ -104,10 +106,22 @@ Other subcommands include
=item * L<lei-daemon-pid(1)>
+=item * lei-forget-mail-sync(1)
+
+=item * L<lei-mail-diff(1)>
+
+=item * L<lei-inspect(1)>
+
=item * L<lei-ls-label(1)>
+=item * L<lei-ls-mail-source(1)>
+
=item * L<lei-ls-mail-sync(1)>
+=item * L<lei-ls-watch(1)>
+
+=item * L<lei-rm-watch(1)>
+
=back
=head1 FILES
diff --git a/Documentation/txt2pre b/Documentation/txt2pre
index bc94d404..fb07579a 100755
--- a/Documentation/txt2pre
+++ b/Documentation/txt2pre
@@ -12,6 +12,7 @@ use PublicInbox::Hval qw(ascii_html);
my %xurls;
for (qw[lei(1)
lei-add-external(1)
+ lei-add-watch(1)
lei-blob(1)
lei-config(1)
lei-convert(1)
@@ -19,20 +20,26 @@ for (qw[lei(1)
lei-daemon-pid(1)
lei-edit-search(1)
lei-forget-external(1)
+ lei-forget-mail-sync(1)
lei-forget-search(1)
lei-import(1)
lei-index(1)
lei-init(1)
+ lei-inspect(1)
lei-lcat(1)
lei-ls-external(1)
lei-ls-label(1)
+ lei-ls-mail-source(1)
lei-ls-mail-sync(1)
lei-ls-search(1)
+ lei-ls-watch(1)
+ lei-mail-diff(1)
lei-overview(7)
lei-p2q(1)
lei-q(1)
lei-rediff(1)
lei-rm(1)
+ lei-rm-watch(1)
lei-security(7)
lei-store-format(5)
lei-tag(1)
diff --git a/MANIFEST b/MANIFEST
index b89513d5..c4cc6e33 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -22,6 +22,7 @@ Documentation/flow.txt
Documentation/hosted.txt
Documentation/include.mk
Documentation/lei-add-external.pod
+Documentation/lei-add-watch.pod
Documentation/lei-blob.pod
Documentation/lei-config.pod
Documentation/lei-convert.pod
@@ -31,15 +32,20 @@ Documentation/lei-daemon.pod
Documentation/lei-edit-search.pod
Documentation/lei-export-kw.pod
Documentation/lei-forget-external.pod
+Documentation/lei-forget-mail-sync.pod
Documentation/lei-forget-search.pod
Documentation/lei-import.pod
Documentation/lei-index.pod
Documentation/lei-init.pod
+Documentation/lei-inspect.pod
Documentation/lei-lcat.pod
Documentation/lei-ls-external.pod
Documentation/lei-ls-label.pod
+Documentation/lei-ls-mail-source.pod
Documentation/lei-ls-mail-sync.pod
Documentation/lei-ls-search.pod
+Documentation/lei-ls-watch.pod
+Documentation/lei-mail-diff.pod
Documentation/lei-mail-formats.pod
Documentation/lei-overview.pod
Documentation/lei-p2q.pod
@@ -47,6 +53,7 @@ Documentation/lei-q.pod
Documentation/lei-rediff.pod
Documentation/lei-refresh-mail-sync.pod
Documentation/lei-rm.pod
+Documentation/lei-rm-watch.pod
Documentation/lei-security.pod
Documentation/lei-store-format.pod
Documentation/lei-tag.pod
diff --git a/Makefile.PL b/Makefile.PL
index 22a58bee..348a343d 100644
--- a/Makefile.PL
+++ b/Makefile.PL
@@ -45,13 +45,13 @@ $v->{-m1} = [ map {
}
} @EXE_FILES,
qw(
- lei-add-external lei-blob lei-config lei-convert
+ lei-add-external lei-add-watch lei-blob lei-config lei-convert
lei-daemon-kill lei-daemon-pid lei-edit-search lei-export-kw
- lei-forget-external lei-forget-search
- lei-import lei-index lei-init lei-lcat
- lei-ls-external lei-ls-label lei-ls-mail-sync
- lei-ls-search lei-p2q lei-q
- lei-rediff lei-refresh-mail-sync lei-rm lei-tag
+ lei-forget-external lei-forget-mail-sync lei-forget-search
+ lei-import lei-index lei-init lei-inspect lei-lcat
+ lei-ls-external lei-ls-label lei-ls-mail-source lei-ls-mail-sync
+ lei-ls-search lei-ls-watch lei-mail-diff lei-p2q lei-q
+ lei-rediff lei-refresh-mail-sync lei-rm lei-rm-watch lei-tag
lei-up)];
$v->{-m5} = [ qw(public-inbox-config public-inbox-v1-format
public-inbox-v2-format public-inbox-extindex-format
--
2.33.0
^ permalink raw reply related [relevance 29%]
* [PATCH 1/2] doc: lei: restore alphabetical order to some listings
2021-10-16 5:39 70% [PATCH 0/2] doc: lei manpages for remaining commands Kyle Meyer
@ 2021-10-16 5:39 65% ` Kyle Meyer
2021-10-16 5:39 29% ` [PATCH 2/2] doc: lei: add manpages for remaining commands Kyle Meyer
1 sibling, 0 replies; 200+ results
From: Kyle Meyer @ 2021-10-16 5:39 UTC (permalink / raw)
To: meta
Most the lei-related entries in txt2pre and Makefile.PL are in
alphabetical order. Reorder the few that aren't.
While at it, reflow the Makefile.PL entries in preparation for the
entries that will be added in the next commit.
---
Documentation/txt2pre | 2 +-
Makefile.PL | 14 +++++++-------
2 files changed, 8 insertions(+), 8 deletions(-)
diff --git a/Documentation/txt2pre b/Documentation/txt2pre
index 7bc31d23..bc94d404 100755
--- a/Documentation/txt2pre
+++ b/Documentation/txt2pre
@@ -33,8 +33,8 @@ for (qw[lei(1)
lei-q(1)
lei-rediff(1)
lei-rm(1)
- lei-store-format(5)
lei-security(7)
+ lei-store-format(5)
lei-tag(1)
lei-up(1)
public-inbox.cgi(1)
diff --git a/Makefile.PL b/Makefile.PL
index c41c408a..22a58bee 100644
--- a/Makefile.PL
+++ b/Makefile.PL
@@ -45,13 +45,13 @@ $v->{-m1} = [ map {
}
} @EXE_FILES,
qw(
- lei-add-external lei-blob lei-config lei-convert lei-edit-search
- lei-export-kw
- lei-daemon-kill lei-daemon-pid lei-forget-external lei-forget-search
- lei-import lei-index lei-init lei-lcat lei-ls-external lei-ls-label
- lei-ls-mail-sync lei-ls-search lei-p2q lei-q lei-rediff
- lei-refresh-mail-sync
- lei-rm lei-tag
+ lei-add-external lei-blob lei-config lei-convert
+ lei-daemon-kill lei-daemon-pid lei-edit-search lei-export-kw
+ lei-forget-external lei-forget-search
+ lei-import lei-index lei-init lei-lcat
+ lei-ls-external lei-ls-label lei-ls-mail-sync
+ lei-ls-search lei-p2q lei-q
+ lei-rediff lei-refresh-mail-sync lei-rm lei-tag
lei-up)];
$v->{-m5} = [ qw(public-inbox-config public-inbox-v1-format
public-inbox-v2-format public-inbox-extindex-format
--
2.33.0
^ permalink raw reply related [relevance 65%]
* [PATCH 09/12] lei_overview: die rather than lei->fail
` (2 preceding siblings ...)
2021-10-16 1:00 55% ` [PATCH 07/12] lei: more eval guards for die on failure Eric Wong
@ 2021-10-16 1:01 59% ` Eric Wong
3 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-10-16 1:01 UTC (permalink / raw)
To: meta
This will make our code more flexible in case it gets used in
non-lei things.
---
lib/PublicInbox/LEI.pm | 2 +-
lib/PublicInbox/LeiOverview.pm | 24 +++++++++++-------------
2 files changed, 12 insertions(+), 14 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 511b2c1d03a7..876598f9530e 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -1108,7 +1108,7 @@ sub accept_dispatch { # Listener {post_accept} callback
my %env = map { split(/=/, $_, 2) } splice(@argv, $argc);
$self->{env} = \%env;
eval { dispatch($self, @argv) };
- send($sock, $@, MSG_EOR) if $@;
+ $self->fail($@) if $@;
}
sub dclose {
diff --git a/lib/PublicInbox/LeiOverview.pm b/lib/PublicInbox/LeiOverview.pm
index 223db22244ec..1b9dc9701c95 100644
--- a/lib/PublicInbox/LeiOverview.pm
+++ b/lib/PublicInbox/LeiOverview.pm
@@ -37,16 +37,16 @@ sub ovv_out_lk_cancel ($) {
unlink($lock_path);
}
-sub detect_fmt ($$) {
- my ($lei, $dst) = @_;
+sub detect_fmt ($) {
+ my ($dst) = @_;
if ($dst =~ m!\A([:/]+://)!) {
- $lei->fail("$1 support not implemented, yet\n");
+ die "$1 support not implemented, yet\n";
} elsif (!-e $dst || -d _) {
'maildir'; # the default TODO: MH?
} elsif (-f _ || -p _) {
- $lei->fail("unable to determine mbox family of $dst\n");
+ die "unable to determine mbox family of $dst\n";
} else {
- $lei->fail("unable to determine format of $dst\n");
+ die "unable to determine format of $dst\n";
}
}
@@ -60,20 +60,19 @@ sub new {
my $fmt = $opt->{$ofmt_key};
$fmt = lc($fmt) if defined $fmt;
if ($dst =~ m!\A([a-z0-9\+]+)://!is) {
- defined($fmt) and return $lei->fail(<<"");
+ defined($fmt) and die <<"";
--$ofmt_key=$fmt invalid with URL $dst
$fmt = lc $1;
} elsif ($dst =~ s/\A([a-z0-9]+)://is) { # e.g. Maildir:/home/user/Mail/
my $ofmt = lc $1;
$fmt //= $ofmt;
- return $lei->fail(<<"") if $fmt ne $ofmt;
+ die <<"" if $fmt ne $ofmt;
--$ofmt_key=$fmt and --output=$ofmt conflict
}
-
my $devfd = $lei->path_to_fd($dst) // return;
- $fmt //= $devfd >= 0 ? 'json' : (detect_fmt($lei, $dst) or return);
+ $fmt //= $devfd >= 0 ? 'json' : detect_fmt($dst);
if (index($dst, '://') < 0) { # not a URL, so assume path
$dst = $lei->canonpath_harder($dst);
@@ -90,7 +89,7 @@ sub new {
$opt->{pretty} //= $isatty;
if (!$isatty && -f _) {
my $fl = fcntl($lei->{$devfd}, F_GETFL, 0) //
- return $lei->fail("fcntl(stdout): $!");
+ die("fcntl(/dev/fd/$devfd): $!\n");
ovv_out_lk_init($self) unless ($fl & O_APPEND);
} else {
ovv_out_lk_init($self);
@@ -101,14 +100,13 @@ sub new {
if ($json) {
$lei->{dedupe} //= PublicInbox::LeiDedupe->new($lei);
} else {
- $lei->{l2m} = eval { PublicInbox::LeiToMail->new($lei) };
- return $lei->fail($@) if $@;
+ $lei->{l2m} = PublicInbox::LeiToMail->new($lei);
if ($opt->{mua} && $lei->{l2m}->lock_free) {
$lei->{early_mua} = 1;
$opt->{alert} //= [ ':WINCH,:bell' ] if -t $lei->{1};
}
}
- return $lei->fail('--shared is only for v2 inbox output') if
+ die("--shared is only for v2 inbox output\n") if
$self->{fmt} ne 'v2' && $lei->{opt}->{shared};
$self;
}
^ permalink raw reply related [relevance 59%]
* [PATCH 07/12] lei: more eval guards for die on failure
2021-10-16 1:00 71% ` [PATCH 05/12] lei: golf PATH2CFG cleanup Eric Wong
2021-10-16 1:00 63% ` [PATCH 06/12] lei: always keep cwd fd {3} for ->fchdir Eric Wong
@ 2021-10-16 1:00 55% ` Eric Wong
2021-10-16 1:01 59% ` [PATCH 09/12] lei_overview: die rather than lei->fail Eric Wong
3 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-10-16 1:00 UTC (permalink / raw)
To: meta
Relying on $lei->fail is unsustainable since there'll always
be parts of our code and dependencies which can trigger die()
and break the event loop.
---
lib/PublicInbox/LEI.pm | 6 +++---
lib/PublicInbox/LeiLcat.pm | 21 +++++++++------------
lib/PublicInbox/LeiQuery.pm | 24 +++++++++++-------------
lib/PublicInbox/LeiXSearch.pm | 9 +++++----
4 files changed, 28 insertions(+), 32 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 0cdcf4492885..511b2c1d03a7 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -556,7 +556,7 @@ sub _lei_atfork_child {
# we need to explicitly close things which are on stack
if ($persist) {
open $self->{3}, '<', '/' or die "open(/) $!";
- fchdir($self) or die;
+ fchdir($self);
close($_) for (grep(defined, delete @$self{qw(0 1 2 sock)}));
if (my $cfg = $self->{cfg}) {
delete @$cfg{qw(-lei_store -watches -lei_note_event)};
@@ -779,7 +779,7 @@ sub lazy_cb ($$$) {
sub dispatch {
my ($self, $cmd, @argv) = @_;
- fchdir($self) or return;
+ fchdir($self);
local %ENV = %{$self->{env}};
local $current_lei = $self; # for __WARN__
$self->{2}->autoflush(1); # keep stdout buffered until x_it|DESTROY
@@ -1381,7 +1381,7 @@ sub wq_done_wait { # dwaitpid callback
sub fchdir {
my ($lei) = @_;
my $dh = $lei->{3} // die 'BUG: lei->{3} (CWD) gone';
- chdir($dh) || $lei->fail("fchdir: $!");
+ chdir($dh) || die "fchdir: $!";
}
sub wq_eof { # EOF callback for main daemon
diff --git a/lib/PublicInbox/LeiLcat.pm b/lib/PublicInbox/LeiLcat.pm
index d553b18733da..191f6f244857 100644
--- a/lib/PublicInbox/LeiLcat.pm
+++ b/lib/PublicInbox/LeiLcat.pm
@@ -124,18 +124,15 @@ could not extract Message-ID from $x
sub _stdin { # PublicInbox::InputPipe::consume callback for --stdin
my ($lei) = @_; # $_[1] = $rbuf
- if (defined($_[1])) {
- $_[1] eq '' and return eval {
- $lei->fchdir or return;
- my @argv = split(/\s+/, $lei->{mset_opt}->{qstr});
- $lei->{mset_opt}->{qstr} = extract_all($lei, @argv)
- or return;
- $lei->_start_query;
- };
- $lei->{mset_opt}->{qstr} .= $_[1];
- } else {
- $lei->fail("error reading stdin: $!");
- }
+ $_[1] // return $lei->fail("error reading stdin: $!");
+ return $lei->{mset_opt}->{qstr} .= $_[1] if $_[1] ne '';
+ eval {
+ $lei->fchdir;
+ my @argv = split(/\s+/, $lei->{mset_opt}->{qstr});
+ $lei->{mset_opt}->{qstr} = extract_all($lei, @argv) or return;
+ $lei->_start_query;
+ };
+ $lei->fail($@) if $@;
}
sub lei_lcat {
diff --git a/lib/PublicInbox/LeiQuery.pm b/lib/PublicInbox/LeiQuery.pm
index c65b00ca0986..ec2ece051492 100644
--- a/lib/PublicInbox/LeiQuery.pm
+++ b/lib/PublicInbox/LeiQuery.pm
@@ -55,19 +55,17 @@ sub _start_query { # used by "lei q" and "lei up"
}
sub qstr_add { # PublicInbox::InputPipe::consume callback for --stdin
- my ($self) = @_; # $_[1] = $rbuf
- if (defined($_[1])) {
- $_[1] eq '' and return eval {
- $self->fchdir or return;
- $self->{mset_opt}->{q_raw} = $self->{mset_opt}->{qstr};
- $self->{lse}->query_approxidate($self->{lse}->git,
- $self->{mset_opt}->{qstr});
- _start_query($self);
- };
- $self->{mset_opt}->{qstr} .= $_[1];
- } else {
- $self->fail("error reading stdin: $!");
- }
+ my ($lei) = @_; # $_[1] = $rbuf
+ $_[1] // $lei->fail("error reading stdin: $!");
+ return $lei->{mset_opt}->{qstr} .= $_[1] if $_[1] ne '';
+ eval {
+ $lei->fchdir;
+ $lei->{mset_opt}->{q_raw} = $lei->{mset_opt}->{qstr};
+ $lei->{lse}->query_approxidate($lei->{lse}->git,
+ $lei->{mset_opt}->{qstr});
+ _start_query($lei);
+ };
+ $lei->fail($@) if $@;
}
sub lxs_prepare {
diff --git a/lib/PublicInbox/LeiXSearch.pm b/lib/PublicInbox/LeiXSearch.pm
index 4aa2a81c0025..8ab84b15c00b 100644
--- a/lib/PublicInbox/LeiXSearch.pm
+++ b/lib/PublicInbox/LeiXSearch.pm
@@ -460,10 +460,11 @@ sub do_post_augment {
my ($lei) = @_;
local $PublicInbox::LEI::current_lei = $lei;
my $l2m = $lei->{l2m} or return; # client disconnected
- $lei->fchdir or return;
- my $err;
- eval { $l2m->post_augment($lei) };
- $err = $@;
+ eval {
+ $lei->fchdir;
+ $l2m->post_augment($lei);
+ };
+ my $err = $@;
if ($err) {
if (my $lxs = delete $lei->{lxs}) {
$lxs->wq_kill('-TERM');
^ permalink raw reply related [relevance 55%]
* [PATCH 05/12] lei: golf PATH2CFG cleanup
@ 2021-10-16 1:00 71% ` Eric Wong
2021-10-16 1:00 63% ` [PATCH 06/12] lei: always keep cwd fd {3} for ->fchdir Eric Wong
` (2 subsequent siblings)
3 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-10-16 1:00 UTC (permalink / raw)
To: meta
More code means more bugs.
---
lib/PublicInbox/LEI.pm | 4 +---
1 file changed, 1 insertion(+), 3 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index a526a91f8035..e7f37efaf0a3 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -850,9 +850,7 @@ sub _lei_cfg ($;$) {
}
if (scalar(keys %PATH2CFG) > 5) {
# FIXME: use inotify/EVFILT_VNODE to detect unlinked configs
- for my $k (keys %PATH2CFG) {
- delete($PATH2CFG{$k}) unless -f $k
- }
+ delete(@PATH2CFG{grep(!-f, keys %PATH2CFG)});
}
$self->{cfg} = $PATH2CFG{$f} = $cfg;
refresh_watches($self);
^ permalink raw reply related [relevance 71%]
* [PATCH 06/12] lei: always keep cwd fd {3} for ->fchdir
2021-10-16 1:00 71% ` [PATCH 05/12] lei: golf PATH2CFG cleanup Eric Wong
@ 2021-10-16 1:00 63% ` Eric Wong
2021-10-16 1:00 55% ` [PATCH 07/12] lei: more eval guards for die on failure Eric Wong
2021-10-16 1:01 59% ` [PATCH 09/12] lei_overview: die rather than lei->fail Eric Wong
3 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-10-16 1:00 UTC (permalink / raw)
To: meta
The extra FD shouldn't cause noticeable overhead in short-lived
workers, and it lets us simplify lei->rel2abs. Get rid of a
2-argument form of open() while we're at it, since it's been
considered for warning+deprecation by Perl for safety reasons.
---
lib/PublicInbox/LEI.pm | 14 ++++++--------
1 file changed, 6 insertions(+), 8 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index e7f37efaf0a3..0cdcf4492885 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -77,19 +77,16 @@ sub rel2abs {
return $p;
}
my $pwd = $self->{env}->{PWD};
- my $cwd;
if (defined $pwd) {
- my $xcwd = $self->{3} //
- ($cwd = getcwd() // die "getcwd(PWD=$pwd): $!");
if (my @st_pwd = stat($pwd)) {
- my @st_cwd = stat($xcwd) or die "stat($xcwd): $!";
+ my @st_cwd = stat($self->{3}) or die "stat({3}): $!";
"@st_pwd[1,0]" eq "@st_cwd[1,0]" or
$self->{env}->{PWD} = $pwd = undef;
} else { # PWD was invalid
$self->{env}->{PWD} = $pwd = undef;
}
}
- $pwd //= $self->{env}->{PWD} = $cwd // getcwd() // die "getcwd: $!";
+ $pwd //= $self->{env}->{PWD} = getcwd() // die "getcwd: $!";
File::Spec->rel2abs($p, $pwd);
}
@@ -558,7 +555,8 @@ sub _lei_atfork_child {
my ($self, $persist) = @_;
# we need to explicitly close things which are on stack
if ($persist) {
- chdir '/' or die "chdir(/): $!";
+ open $self->{3}, '<', '/' or die "open(/) $!";
+ fchdir($self) or die;
close($_) for (grep(defined, delete @$self{qw(0 1 2 sock)}));
if (my $cfg = $self->{cfg}) {
delete @$cfg{qw(-lei_store -watches -lei_note_event)};
@@ -568,7 +566,7 @@ sub _lei_atfork_child {
STDERR->autoflush(1);
POSIX::setpgid(0, $$) // die "setpgid(0, $$): $!";
}
- close($_) for (grep(defined, delete @$self{qw(3 old_1 au_done)}));
+ close($_) for (grep(defined, delete @$self{qw(old_1 au_done)}));
delete $self->{-socks};
if (my $op_c = delete $self->{pkt_op_c}) {
close(delete $op_c->{sock});
@@ -1190,7 +1188,7 @@ sub cfg2lei ($) {
open($lei->{0}, '<&', \*STDIN) or die "dup 0: $!";
open($lei->{1}, '>>&', \*STDOUT) or die "dup 1: $!";
open($lei->{2}, '>>&', \*STDERR) or die "dup 2: $!";
- open($lei->{3}, '/') or die "open /: $!";
+ open($lei->{3}, '<', '/') or die "open /: $!";
my ($x, $y);
socketpair($x, $y, AF_UNIX, SOCK_SEQPACKET, 0) or die "socketpair: $!";
$lei->{sock} = $x;
^ permalink raw reply related [relevance 63%]
* [PATCH] lei q: guard query_done against die()
@ 2021-10-15 15:52 61% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-10-15 15:52 UTC (permalink / raw)
To: meta
v2w->wq_do('done') may die on I/O errors, and likely other
places. Just guard the entire block with an eval and ->fail
as appropriate.
---
lib/PublicInbox/LeiXSearch.pm | 75 ++++++++++++++++++-----------------
1 file changed, 38 insertions(+), 37 deletions(-)
diff --git a/lib/PublicInbox/LeiXSearch.pm b/lib/PublicInbox/LeiXSearch.pm
index fd2c8a37..4aa2a81c 100644
--- a/lib/PublicInbox/LeiXSearch.pm
+++ b/lib/PublicInbox/LeiXSearch.pm
@@ -412,47 +412,48 @@ sub xsearch_done_wait { # dwaitpid callback
sub query_done { # EOF callback for main daemon
my ($lei) = @_;
local $PublicInbox::LEI::current_lei = $lei;
- my $l2m = delete $lei->{l2m};
- delete $lei->{lxs};
- ($lei->{opt}->{'mail-sync'} && !$lei->{sto}) and
- warn "BUG: {sto} missing with --mail-sync";
- $lei->sto_done_request if $lei->{sto};
- if (my $v2w = delete $lei->{v2w}) {
- $v2w->wq_do('done');
- $v2w->wq_close;
- }
- $lei->{ovv}->ovv_end($lei);
- my $start_mua;
- if ($l2m) { # close() calls LeiToMail reap_compress
- if (my $out = delete $lei->{old_1}) {
- if (my $mbout = $lei->{1}) {
- close($mbout) or return $lei->fail(<<"");
-Error closing $lei->{ovv}->{dst}: $!
+ eval {
+ my $l2m = delete $lei->{l2m};
+ delete $lei->{lxs};
+ ($lei->{opt}->{'mail-sync'} && !$lei->{sto}) and
+ warn "BUG: {sto} missing with --mail-sync";
+ $lei->sto_done_request if $lei->{sto};
+ if (my $v2w = delete $lei->{v2w}) {
+ my $wait = $v2w->wq_do('done'); # may die
+ $v2w->wq_close;
+ }
+ $lei->{ovv}->ovv_end($lei);
+ if ($l2m) { # close() calls LeiToMail reap_compress
+ if (my $out = delete $lei->{old_1}) {
+ if (my $mbout = $lei->{1}) {
+ close($mbout) or die <<"";
+Error closing $lei->{ovv}->{dst}: \$!=$! \$?=$?
+ }
+ $lei->{1} = $out;
+ }
+ if ($l2m->lock_free) {
+ $l2m->poke_dst;
+ $lei->poke_mua;
+ } else { # mbox users
+ delete $l2m->{mbl}; # drop dotlock
}
- $lei->{1} = $out;
- }
- if ($l2m->lock_free) {
- $l2m->poke_dst;
- $lei->poke_mua;
- } else { # mbox users
- delete $l2m->{mbl}; # drop dotlock
- $start_mua = 1;
}
- }
- if ($lei->{-progress}) {
- my $tot = $lei->{-mset_total} // 0;
- my $nr = $lei->{-nr_write} // 0;
- if ($l2m) {
- my $m = "# $nr written to " .
- "$lei->{ovv}->{dst} ($tot matches)";
- $nr ? $lei->qfin($m) : $lei->qerr($m);
- } else {
- $lei->qerr("# $tot matches");
+ if ($lei->{-progress}) {
+ my $tot = $lei->{-mset_total} // 0;
+ my $nr = $lei->{-nr_write} // 0;
+ if ($l2m) {
+ my $m = "# $nr written to " .
+ "$lei->{ovv}->{dst} ($tot matches)";
+ $nr ? $lei->qfin($m) : $lei->qerr($m);
+ } else {
+ $lei->qerr("# $tot matches");
+ }
}
- }
- $lei->start_mua if $start_mua;
- $lei->dclose;
+ $lei->start_mua if $l2m && !$l2m->lock_free;
+ $lei->dclose;
+ };
+ $lei->fail($@) if $@;
}
sub do_post_augment {
^ permalink raw reply related [relevance 61%]
* [PATCH] lei forget-search: support multiple args
@ 2021-10-15 14:02 64% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-10-15 14:02 UTC (permalink / raw)
To: meta
I've been testing a lot of searches which I don't want to keep
around, so make it easy to remove a bunch at once. We'll behave
like rm(1) and keep going in the face of failure.
---
lib/PublicInbox/LEI.pm | 2 +-
lib/PublicInbox/LeiForgetSearch.pm | 30 ++++++++++++++++++------------
2 files changed, 19 insertions(+), 13 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 83534878..7dfd3398 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -225,7 +225,7 @@ our %CMD = ( # sorted in order of importance/use:
'ls-search' => [ '[PREFIX]', 'list saved search queries',
qw(format|f=s pretty l ascii z|0), @c_opt ],
-'forget-search' => [ 'OUTPUT', 'forget a saved search',
+'forget-search' => [ 'OUTPUT...', 'forget a saved search',
qw(verbose|v+), @c_opt ],
'edit-search' => [ 'OUTPUT', "edit saved search via `git config --edit'",
@c_opt ],
diff --git a/lib/PublicInbox/LeiForgetSearch.pm b/lib/PublicInbox/LeiForgetSearch.pm
index 717fa5e9..0db9c75b 100644
--- a/lib/PublicInbox/LeiForgetSearch.pm
+++ b/lib/PublicInbox/LeiForgetSearch.pm
@@ -11,20 +11,26 @@ use File::Path ();
use SelectSaver;
sub lei_forget_search {
- my ($lei, $out) = @_;
- my $d = PublicInbox::LeiSavedSearch::lss_dir_for($lei, \$out, 1);
- if (-e $d) {
- my $save;
- my $opt = { safe => 1 };
- if ($lei->{opt}->{verbose}) {
- $opt->{verbose} = 1;
- $save = SelectSaver->new($lei->{2});
+ my ($lei, @outs) = @_;
+ my @dirs; # paths in ~/.local/share/lei/saved-search/
+ my $cwd;
+ for my $o (@outs) {
+ my $d = PublicInbox::LeiSavedSearch::lss_dir_for($lei, \$o, 1);
+ if (-e $d) {
+ push @dirs, $d
+ } else { # keep going, like rm(1):
+ $cwd //= $lei->rel2abs('.');
+ warn "--save was not used with $o cwd=$cwd\n";
}
- File::Path::remove_tree($d, $opt);
- } else {
- $lei->fail("--save was not used with $out cwd=".
- $lei->rel2abs('.'));
}
+ my $save;
+ my $opt = { safe => 1 };
+ if ($lei->{opt}->{verbose}) {
+ $opt->{verbose} = 1;
+ $save = SelectSaver->new($lei->{2});
+ }
+ File::Path::remove_tree(@dirs, $opt);
+ $lei->fail if defined $cwd;
}
*_complete_forget_search = \&PublicInbox::LeiUp::_complete_up;
^ permalink raw reply related [relevance 64%]
* [SQUASH PATCH 4/3] lei q: ensure all workers die on Ctrl-C
2021-10-15 13:30 33% ` [PATCH 2/3] lei + ipc: simplify process reaping Eric Wong
@ 2021-10-15 13:45 71% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-10-15 13:45 UTC (permalink / raw)
To: meta
We never handled v2:// writers properly before. And we still
need to kill the entire process group of each worker :x
Oops
---
lib/PublicInbox/LEI.pm | 7 +++++--
1 file changed, 5 insertions(+), 2 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 4a1f1652..83534878 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -455,11 +455,14 @@ my %CONFIG_KEYS = (
'leistore.dir' => 'top-level storage location',
);
-my @WQ_KEYS = qw(lxs l2m ikw pmd wq1 lne); # internal workers
+my @WQ_KEYS = qw(lxs l2m ikw pmd wq1 lne v2w); # internal workers
sub _drop_wq {
my ($self) = @_;
- for my $wq (grep(defined, delete(@$self{@WQ_KEYS}))) { $wq->DESTROY }
+ for my $wq (grep(defined, delete(@$self{@WQ_KEYS}))) {
+ $wq->wq_kill('-TERM');
+ $wq->DESTROY;
+ }
}
# pronounced "exit": x_it(1 << 8) => exit(1); x_it(13) => SIGPIPE
^ permalink raw reply related [relevance 71%]
* [PATCH 3/3] lei note-event: fix explicit flush reliability
2021-10-15 13:30 71% [PATCH 0/3] lei bugfixes and simplifications Eric Wong
2021-10-15 13:30 71% ` [PATCH 1/3] lei forget-search: fix for symlink-ed paths Eric Wong
2021-10-15 13:30 33% ` [PATCH 2/3] lei + ipc: simplify process reaping Eric Wong
@ 2021-10-15 13:30 67% ` Eric Wong
2 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-10-15 13:30 UTC (permalink / raw)
To: meta
We need to send the socket over to lei/store and wait for the
kernel to drop the socket refcount down to zero before
script/lei can exit.
This is not a new bug and only caused very sporadic test
failures. I only noticed it while simplifying IPC stuff.
---
lib/PublicInbox/LeiNoteEvent.pm | 15 ++++++++-------
1 file changed, 8 insertions(+), 7 deletions(-)
diff --git a/lib/PublicInbox/LeiNoteEvent.pm b/lib/PublicInbox/LeiNoteEvent.pm
index ba4dfd49..1749c98f 100644
--- a/lib/PublicInbox/LeiNoteEvent.pm
+++ b/lib/PublicInbox/LeiNoteEvent.pm
@@ -11,10 +11,11 @@ use PublicInbox::DS;
our $to_flush; # { cfgpath => $lei }
-sub flush_lei ($) {
- my ($lei) = @_;
- my $lne = delete $lei->{cfg}->{-lei_note_event};
- $lne->wq_close if $lne; # runs _lei_wq_eof;
+sub flush_lei ($;$) {
+ my ($lei, $manual) = @_;
+ my $lne = delete $lei->{cfg}->{-lei_note_event} // return;
+ $lne->{lei_sock} = $lei->{sock} if $manual;
+ $lne->wq_close; # runs _lei_wq_eof;
}
# we batch up writes and flush every 5s (matching Linux default
@@ -67,7 +68,7 @@ sub lei_note_event {
die "BUG: unexpected: @rest" if @rest;
my $cfg = $lei->_lei_cfg or return; # gone (race)
my $sto = $lei->_lei_store or return; # gone
- return flush_lei($lei) if $folder eq 'done'; # special case
+ return flush_lei($lei, 1) if $folder eq 'done'; # special case
my $lms = $lei->lms or return;
$lms->lms_write_prepare if $new_cur eq ''; # for ->clear_src below
$lei->{opt}->{quiet} = 1;
@@ -111,8 +112,8 @@ sub ipc_atfork_child {
sub _lei_wq_eof { # EOF callback for main lei daemon
my ($lei) = @_;
- delete $lei->{lne} or return $lei->fail;
- $lei->sto_done_request;
+ my $lne = delete $lei->{lne} or return $lei->fail;
+ $lei->sto_done_request($lne->{lei_sock});
}
1;
^ permalink raw reply related [relevance 67%]
* [PATCH 2/3] lei + ipc: simplify process reaping
2021-10-15 13:30 71% [PATCH 0/3] lei bugfixes and simplifications Eric Wong
2021-10-15 13:30 71% ` [PATCH 1/3] lei forget-search: fix for symlink-ed paths Eric Wong
@ 2021-10-15 13:30 33% ` Eric Wong
2021-10-15 13:45 71% ` [SQUASH PATCH 4/3] lei q: ensure all workers die on Ctrl-C Eric Wong
2021-10-15 13:30 67% ` [PATCH 3/3] lei note-event: fix explicit flush reliability Eric Wong
2 siblings, 1 reply; 200+ results
From: Eric Wong @ 2021-10-15 13:30 UTC (permalink / raw)
To: meta
Simplify our APIs and force dwaitpid() to work in async mode for
all lei workers. This avoids having lingering zombies for
parallel searches if one worker finishes soon before another.
The old distinction between "old" and "new" workers was
needlessly complex, error-prone, and embarrasingly bad.
---
lib/PublicInbox/IPC.pm | 46 ++++++++++++---------------------
lib/PublicInbox/LEI.pm | 17 ++++--------
lib/PublicInbox/LeiBlob.pm | 2 +-
lib/PublicInbox/LeiConvert.pm | 2 +-
lib/PublicInbox/LeiImportKw.pm | 1 -
lib/PublicInbox/LeiInput.pm | 2 +-
lib/PublicInbox/LeiInspect.pm | 2 +-
lib/PublicInbox/LeiLsSearch.pm | 2 +-
lib/PublicInbox/LeiMirror.pm | 2 +-
lib/PublicInbox/LeiNoteEvent.pm | 5 ++--
lib/PublicInbox/LeiP2q.pm | 2 +-
lib/PublicInbox/LeiPmdir.pm | 1 -
lib/PublicInbox/LeiStore.pm | 1 +
lib/PublicInbox/LeiToMail.pm | 10 +++++--
lib/PublicInbox/LeiUp.pm | 2 +-
lib/PublicInbox/LeiXSearch.pm | 16 +++++++-----
16 files changed, 50 insertions(+), 63 deletions(-)
diff --git a/lib/PublicInbox/IPC.pm b/lib/PublicInbox/IPC.pm
index 6c189b64..3e299448 100644
--- a/lib/PublicInbox/IPC.pm
+++ b/lib/PublicInbox/IPC.pm
@@ -134,16 +134,22 @@ sub ipc_worker_spawn {
sub ipc_worker_reap { # dwaitpid callback
my ($args, $pid) = @_;
+ my ($self, @uargs) = @$args;
+ delete $self->{-wq_workers}->{$pid};
+ return $self->{-reap_do}->($args, $pid) if $self->{-reap_do};
return if !$?;
- # TERM(15) is our default exit signal, PIPE(13) is likely w/ pager
my $s = $? & 127;
- warn "PID:$pid died with \$?=$?\n" if $s != 15 && $s != 13;
+ # TERM(15) is our default exit signal, PIPE(13) is likely w/ pager
+ warn "$self->{-wq_ident} PID:$pid died \$?=$?\n" if $s != 15 && $s != 13
}
-sub wq_wait_old {
- my ($self, $cb, @args) = @_;
- my $pids = delete $self->{"-wq_old_pids.$$"} or return;
- dwaitpid($_, $cb // \&ipc_worker_reap, [$self, @args]) for @$pids;
+sub wq_wait_async {
+ my ($self, $cb, @uargs) = @_;
+ local $PublicInbox::DS::in_loop = 1;
+ $self->{-reap_async} = 1;
+ $self->{-reap_do} = $cb;
+ my @pids = keys %{$self->{-wq_workers}};
+ dwaitpid($_, \&ipc_worker_reap, [ $self, @uargs ]) for @pids;
}
# for base class, override in sub classes
@@ -394,42 +400,24 @@ sub wq_workers_start {
}
sub wq_close {
- my ($self, $nohang, $cb, @args) = @_;
+ my ($self) = @_;
delete @$self{qw(-wq_s1 -wq_s2)} or return;
- my $ppid = delete $self->{-wq_ppid} or return;
- my $workers = delete $self->{-wq_workers} // die 'BUG: no wq_workers';
- return if $ppid != $$; # can't reap siblings or parents
- my @pids = map { $_ + 0 } keys %$workers;
- if ($nohang) {
- push @{$self->{"-wq_old_pids.$$"}}, @pids;
- } else {
- $cb //= \&ipc_worker_reap;
- unshift @args, $self;
- dwaitpid($_, $cb, \@args) for @pids;
- }
-}
-
-sub wq_kill_old {
- my ($self, $sig) = @_;
- my $pids = $self->{"-wq_old_pids.$$"} or return;
- kill($sig // 'TERM', @$pids);
+ return if $self->{-reap_async};
+ my @pids = keys %{$self->{-wq_workers}};
+ dwaitpid($_, \&ipc_worker_reap, [ $self ]) for @pids;
}
sub wq_kill {
my ($self, $sig) = @_;
- my $workers = $self->{-wq_workers} or return;
- kill($sig // 'TERM', keys %$workers);
+ kill($sig // 'TERM', keys %{$self->{-wq_workers}});
}
sub DESTROY {
my ($self) = @_;
my $ppid = $self->{-wq_ppid};
wq_kill($self) if $ppid && $ppid == $$;
- my $err = $?;
wq_close($self);
- wq_wait_old($self);
ipc_worker_stop($self);
- $? = $err if $err;
}
sub detect_nproc () {
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index b6338377..4a1f1652 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -459,14 +459,7 @@ my @WQ_KEYS = qw(lxs l2m ikw pmd wq1 lne); # internal workers
sub _drop_wq {
my ($self) = @_;
- for my $wq (grep(defined, delete(@$self{@WQ_KEYS}))) {
- if ($wq->wq_kill('-TERM')) {
- $wq->wq_close(0, undef, $self);
- } elsif ($wq->wq_kill_old('-TERM')) {
- $wq->wq_wait_old(undef, $self);
- }
- $wq->DESTROY;
- }
+ for my $wq (grep(defined, delete(@$self{@WQ_KEYS}))) { $wq->DESTROY }
}
# pronounced "exit": x_it(1 << 8) => exit(1); x_it(13) => SIGPIPE
@@ -644,6 +637,7 @@ sub workers_start {
my $op_c = delete $lei->{pkt_op_c};
@$end = ();
$lei->event_step_init;
+ $wq->wq_wait_async($wq->can('_wq_done_wait') // \&wq_done_wait, $lei);
($op_c, $ops);
}
@@ -651,7 +645,7 @@ sub workers_start {
sub wait_wq_events {
my ($lei, $op_c, $ops) = @_;
for my $wq (grep(defined, @$lei{qw(ikw pmd)})) { # auxiliary WQs
- $wq->wq_close(1);
+ $wq->wq_close;
}
$op_c->{ops} = $ops;
}
@@ -1150,7 +1144,7 @@ sub event_step {
if ($buf =~ /\A(?:STOP|CONT|TERM)\z/) {
my $sig = "-$buf";
for my $wq (grep(defined, @$self{@WQ_KEYS})) {
- $wq->wq_kill($sig) or $wq->wq_kill_old($sig);
+ $wq->wq_kill($sig);
}
} else {
die "unrecognized client signal: $buf";
@@ -1393,8 +1387,7 @@ sub fchdir {
sub wq_eof { # EOF callback for main daemon
my ($lei) = @_;
local $current_lei = $lei;
- my $wq1 = delete $lei->{wq1} // return $lei->fail; # already failed
- $wq1->wq_wait_old($wq1->can('_wq_done_wait') // \&wq_done_wait, $lei);
+ delete $lei->{wq1} // return $lei->fail; # already failed
}
sub watch_state_ok ($) {
diff --git a/lib/PublicInbox/LeiBlob.pm b/lib/PublicInbox/LeiBlob.pm
index b6a62d24..004b156c 100644
--- a/lib/PublicInbox/LeiBlob.pm
+++ b/lib/PublicInbox/LeiBlob.pm
@@ -166,7 +166,7 @@ sub lei_blob {
my ($op_c, $ops) = $lei->workers_start($self, 1);
$lei->{wq1} = $self;
$self->wq_io_do('do_solve_blob', []);
- $self->wq_close(1);
+ $self->wq_close;
$lei->wait_wq_events($op_c, $ops);
}
diff --git a/lib/PublicInbox/LeiConvert.pm b/lib/PublicInbox/LeiConvert.pm
index 9e98edc3..68fc7c0b 100644
--- a/lib/PublicInbox/LeiConvert.pm
+++ b/lib/PublicInbox/LeiConvert.pm
@@ -58,7 +58,7 @@ sub lei_convert { # the main "lei convert" method
my ($op_c, $ops) = $lei->workers_start($self, 1);
$lei->{wq1} = $self;
$self->wq_io_do('process_inputs', []);
- $self->wq_close(1);
+ $self->wq_close;
$lei->wait_wq_events($op_c, $ops);
}
diff --git a/lib/PublicInbox/LeiImportKw.pm b/lib/PublicInbox/LeiImportKw.pm
index 8359f338..54454511 100644
--- a/lib/PublicInbox/LeiImportKw.pm
+++ b/lib/PublicInbox/LeiImportKw.pm
@@ -50,7 +50,6 @@ sub _lei_wq_eof { # EOF callback for main lei daemon
my ($lei) = @_;
my $ikw = delete $lei->{ikw} or return $lei->fail;
$lei->sto_done_request($ikw->{lei_sock});
- $ikw->wq_wait_old($lei->can('wq_done_wait'), $lei);
}
1;
diff --git a/lib/PublicInbox/LeiInput.pm b/lib/PublicInbox/LeiInput.pm
index dd40d838..2621fc1f 100644
--- a/lib/PublicInbox/LeiInput.pm
+++ b/lib/PublicInbox/LeiInput.pm
@@ -402,7 +402,7 @@ sub input_only_atfork_child {
sub input_only_net_merge_all_done {
my ($self) = @_;
$self->wq_io_do('process_inputs');
- $self->wq_close(1);
+ $self->wq_close;
}
# like Getopt::Long, but for +kw:FOO and -kw:FOO to prepare
diff --git a/lib/PublicInbox/LeiInspect.pm b/lib/PublicInbox/LeiInspect.pm
index 5ba96056..05b6e21d 100644
--- a/lib/PublicInbox/LeiInspect.pm
+++ b/lib/PublicInbox/LeiInspect.pm
@@ -242,7 +242,7 @@ sub inspect_start ($$) {
$lei->{wq1} = $self;
$lei->wait_wq_events($op_c, $ops);
$self->wq_do('inspect_argv');
- $self->wq_close(1);
+ $self->wq_close;
}
sub ins_add { # InputPipe->consume callback
diff --git a/lib/PublicInbox/LeiLsSearch.pm b/lib/PublicInbox/LeiLsSearch.pm
index aebf0184..0193e590 100644
--- a/lib/PublicInbox/LeiLsSearch.pm
+++ b/lib/PublicInbox/LeiLsSearch.pm
@@ -75,7 +75,7 @@ sub bg_worker ($$$) {
my ($op_c, $ops) = $lei->workers_start($self, 1);
$lei->{wq1} = $self;
$self->wq_io_do('do_ls_search_long', [], $pfx);
- $self->wq_close(1);
+ $self->wq_close;
$lei->wait_wq_events($op_c, $ops);
}
diff --git a/lib/PublicInbox/LeiMirror.pm b/lib/PublicInbox/LeiMirror.pm
index a75c99c4..e20d30b4 100644
--- a/lib/PublicInbox/LeiMirror.pm
+++ b/lib/PublicInbox/LeiMirror.pm
@@ -451,7 +451,7 @@ sub start {
my ($op_c, $ops) = $lei->workers_start($self, 1);
$lei->{wq1} = $self;
$self->wq_io_do('do_mirror', []);
- $self->wq_close(1);
+ $self->wq_close;
$lei->wait_wq_events($op_c, $ops);
}
diff --git a/lib/PublicInbox/LeiNoteEvent.pm b/lib/PublicInbox/LeiNoteEvent.pm
index 1b714dae..ba4dfd49 100644
--- a/lib/PublicInbox/LeiNoteEvent.pm
+++ b/lib/PublicInbox/LeiNoteEvent.pm
@@ -14,7 +14,7 @@ our $to_flush; # { cfgpath => $lei }
sub flush_lei ($) {
my ($lei) = @_;
my $lne = delete $lei->{cfg}->{-lei_note_event};
- $lne->wq_close(1, undef, $lei) if $lne; # runs _lei_wq_eof;
+ $lne->wq_close if $lne; # runs _lei_wq_eof;
}
# we batch up writes and flush every 5s (matching Linux default
@@ -111,9 +111,8 @@ sub ipc_atfork_child {
sub _lei_wq_eof { # EOF callback for main lei daemon
my ($lei) = @_;
- my $lne = delete $lei->{lne} or return $lei->fail;
+ delete $lei->{lne} or return $lei->fail;
$lei->sto_done_request;
- $lne->wq_wait_old($lei->can('wq_done_wait'), $lei);
}
1;
diff --git a/lib/PublicInbox/LeiP2q.pm b/lib/PublicInbox/LeiP2q.pm
index 5c2ce0a1..08ec81c5 100644
--- a/lib/PublicInbox/LeiP2q.pm
+++ b/lib/PublicInbox/LeiP2q.pm
@@ -191,7 +191,7 @@ sub lei_p2q { # the "lei patch-to-query" entry point
my ($op_c, $ops) = $lei->workers_start($self, 1);
$lei->{wq1} = $self;
$self->wq_io_do('do_p2q', []);
- $self->wq_close(1);
+ $self->wq_close;
$lei->wait_wq_events($op_c, $ops);
}
diff --git a/lib/PublicInbox/LeiPmdir.pm b/lib/PublicInbox/LeiPmdir.pm
index 2d3b9755..f9b68fc2 100644
--- a/lib/PublicInbox/LeiPmdir.pm
+++ b/lib/PublicInbox/LeiPmdir.pm
@@ -51,7 +51,6 @@ sub _lei_wq_eof { # EOF callback for main lei daemon
my ($lei) = @_;
my $pmd = delete $lei->{pmd} or return $lei->fail;
$lei->sto_done_request($pmd->{lei_sock});
- $pmd->wq_wait_old($lei->can('wq_done_wait'), $lei);
}
1;
diff --git a/lib/PublicInbox/LeiStore.pm b/lib/PublicInbox/LeiStore.pm
index c45380d1..82104570 100644
--- a/lib/PublicInbox/LeiStore.pm
+++ b/lib/PublicInbox/LeiStore.pm
@@ -587,6 +587,7 @@ sub write_prepare {
-err_wr => $w,
to_close => [ $r ],
});
+ $self->wq_wait_async; # outlives $lei
require PublicInbox::LeiStoreErr;
PublicInbox::LeiStoreErr->new($r, $lei);
}
diff --git a/lib/PublicInbox/LeiToMail.pm b/lib/PublicInbox/LeiToMail.pm
index 9c748dea..76e103c7 100644
--- a/lib/PublicInbox/LeiToMail.pm
+++ b/lib/PublicInbox/LeiToMail.pm
@@ -637,6 +637,12 @@ sub _do_augment_mbox {
$dedupe->pause_dedupe if $dedupe;
}
+sub v2w_done_wait { # dwaitpid callback
+ my ($arg, $pid) = @_;
+ my ($v2w, $lei) = @$arg;
+ $lei->child_error($?, "error for $v2w->{ibx}->{inboxdir}") if $?;
+}
+
sub _pre_augment_v2 {
my ($self, $lei) = @_;
my $dir = $self->{dst};
@@ -659,8 +665,8 @@ sub _pre_augment_v2 {
PublicInbox::InboxWritable->new($ibx, @creat);
$ibx->init_inbox if @creat;
my $v2w = $ibx->importer;
- $v2w->{-wq_no_bcast} = 1;
$v2w->wq_workers_start("lei/v2w $dir", 1, $lei->oldset, {lei => $lei});
+ $v2w->wq_wait_async(\&v2w_done_wait, $lei);
$lei->{v2w} = $v2w;
return if !$lei->{opt}->{shared};
my $d = "$lei->{ale}->{git}->{git_dir}/objects";
@@ -811,7 +817,7 @@ sub net_merge_all_done {
$self->{dst}, \$self->{-au_noted});
}
$self->wq_broadcast('do_post_auth');
- $self->wq_close(1);
+ $self->wq_close;
}
1;
diff --git a/lib/PublicInbox/LeiUp.pm b/lib/PublicInbox/LeiUp.pm
index df65cb9b..39604177 100644
--- a/lib/PublicInbox/LeiUp.pm
+++ b/lib/PublicInbox/LeiUp.pm
@@ -122,7 +122,7 @@ EOM
sub net_merge_all_done {
my ($self, $lei) = @_;
$lei->{net} = delete($self->{-net_new}) if $self->{-net_new};
- $self->wq_close(1);
+ $self->wq_close;
eval { redispatch_all($self, $lei) };
warn "E: $@" if $@;
}
diff --git a/lib/PublicInbox/LeiXSearch.pm b/lib/PublicInbox/LeiXSearch.pm
index 3ec75528..fd2c8a37 100644
--- a/lib/PublicInbox/LeiXSearch.pm
+++ b/lib/PublicInbox/LeiXSearch.pm
@@ -413,14 +413,14 @@ sub query_done { # EOF callback for main daemon
my ($lei) = @_;
local $PublicInbox::LEI::current_lei = $lei;
my $l2m = delete $lei->{l2m};
- $l2m->wq_wait_old(\&xsearch_done_wait, $lei) if $l2m;
- if (my $lxs = delete $lei->{lxs}) {
- $lxs->wq_wait_old(\&xsearch_done_wait, $lei);
- }
+ delete $lei->{lxs};
($lei->{opt}->{'mail-sync'} && !$lei->{sto}) and
warn "BUG: {sto} missing with --mail-sync";
$lei->sto_done_request if $lei->{sto};
- my $wait = $lei->{v2w} ? $lei->{v2w}->wq_do('done') : undef;
+ if (my $v2w = delete $lei->{v2w}) {
+ $v2w->wq_do('done');
+ $v2w->wq_close;
+ }
$lei->{ovv}->ovv_end($lei);
my $start_mua;
if ($l2m) { # close() calls LeiToMail reap_compress
@@ -466,7 +466,7 @@ sub do_post_augment {
if ($err) {
if (my $lxs = delete $lei->{lxs}) {
$lxs->wq_kill('-TERM');
- $lxs->wq_close(0, undef, $lei);
+ $lxs->wq_close;
}
$lei->fail("$err");
}
@@ -514,7 +514,7 @@ sub start_query ($$) { # always runs in main (lei-daemon) process
if ($self->{-do_lcat}) {
$self->wq_io_do('lcat_dump', []);
}
- $self->wq_close(1); # lei_xsearch workers stop when done
+ $self->wq_close; # lei_xsearch workers stop when done
}
sub incr_start_query { # called whenever an l2m shard starts do_post_auth
@@ -569,12 +569,14 @@ sub do_query {
}
$l2m->wq_workers_start('lei2mail', undef,
$lei->oldset, { lei => $lei });
+ $l2m->wq_wait_async(\&xsearch_done_wait, $lei);
pipe($lei->{startq}, $lei->{au_done}) or die "pipe: $!";
fcntl($lei->{startq}, $F_SETPIPE_SZ, 4096) if $F_SETPIPE_SZ;
delete $l2m->{au_peers};
}
$self->wq_workers_start('lei_xsearch', undef,
$lei->oldset, { lei => $lei });
+ $self->wq_wait_async(\&xsearch_done_wait, $lei);
my $op_c = delete $lei->{pkt_op_c};
delete $lei->{pkt_op_p};
@$end = ();
^ permalink raw reply related [relevance 33%]
* [PATCH 1/3] lei forget-search: fix for symlink-ed paths
2021-10-15 13:30 71% [PATCH 0/3] lei bugfixes and simplifications Eric Wong
@ 2021-10-15 13:30 71% ` Eric Wong
2021-10-15 13:30 33% ` [PATCH 2/3] lei + ipc: simplify process reaping Eric Wong
2021-10-15 13:30 67% ` [PATCH 3/3] lei note-event: fix explicit flush reliability Eric Wong
2 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-10-15 13:30 UTC (permalink / raw)
To: meta
If lei up and edit-search work on something, so should forget-search.
---
lib/PublicInbox/LeiForgetSearch.pm | 2 +-
t/lei-q-save.t | 1 +
2 files changed, 2 insertions(+), 1 deletion(-)
diff --git a/lib/PublicInbox/LeiForgetSearch.pm b/lib/PublicInbox/LeiForgetSearch.pm
index b5fe5fb1..717fa5e9 100644
--- a/lib/PublicInbox/LeiForgetSearch.pm
+++ b/lib/PublicInbox/LeiForgetSearch.pm
@@ -12,7 +12,7 @@ use SelectSaver;
sub lei_forget_search {
my ($lei, $out) = @_;
- my $d = PublicInbox::LeiSavedSearch::lss_dir_for($lei, \$out);
+ my $d = PublicInbox::LeiSavedSearch::lss_dir_for($lei, \$out, 1);
if (-e $d) {
my $save;
my $opt = { safe => 1 };
diff --git a/t/lei-q-save.t b/t/lei-q-save.t
index 5940018c..05d5d9f4 100644
--- a/t/lei-q-save.t
+++ b/t/lei-q-save.t
@@ -173,6 +173,7 @@ test_lei(sub {
symlink($o, "$home/ln -s") or
skip "symlinks not supported in $home?: $!", 1;
lei_ok('up', "$home/ln -s");
+ lei_ok('forget-search', "$home/ln -s");
};
my $v2 = "$home/v2"; # v2: as an output destination
^ permalink raw reply related [relevance 71%]
* [PATCH 0/3] lei bugfixes and simplifications
@ 2021-10-15 13:30 71% Eric Wong
2021-10-15 13:30 71% ` [PATCH 1/3] lei forget-search: fix for symlink-ed paths Eric Wong
` (2 more replies)
0 siblings, 3 replies; 200+ results
From: Eric Wong @ 2021-10-15 13:30 UTC (permalink / raw)
To: meta
I think I finally tracked down the cause of very sporadic
test failures in 3/3.
Eric Wong (3):
lei forget-search: fix for symlink-ed paths
lei + ipc: simplify process reaping
lei note-event: fix explicit flush reliability
lib/PublicInbox/IPC.pm | 46 +++++++++++-------------------
lib/PublicInbox/LEI.pm | 17 ++++-------
lib/PublicInbox/LeiBlob.pm | 2 +-
lib/PublicInbox/LeiConvert.pm | 2 +-
lib/PublicInbox/LeiForgetSearch.pm | 2 +-
lib/PublicInbox/LeiImportKw.pm | 1 -
lib/PublicInbox/LeiInput.pm | 2 +-
lib/PublicInbox/LeiInspect.pm | 2 +-
lib/PublicInbox/LeiLsSearch.pm | 2 +-
lib/PublicInbox/LeiMirror.pm | 2 +-
lib/PublicInbox/LeiNoteEvent.pm | 14 ++++-----
lib/PublicInbox/LeiP2q.pm | 2 +-
lib/PublicInbox/LeiPmdir.pm | 1 -
lib/PublicInbox/LeiStore.pm | 1 +
lib/PublicInbox/LeiToMail.pm | 10 +++++--
lib/PublicInbox/LeiUp.pm | 2 +-
lib/PublicInbox/LeiXSearch.pm | 16 ++++++-----
t/lei-q-save.t | 1 +
18 files changed, 57 insertions(+), 68 deletions(-)
^ permalink raw reply [relevance 71%]
* [PATCH] lei q: avoid kw lookup failure on remote mboxrd
2021-10-14 5:31 71% ` Eric Wong
@ 2021-10-15 9:52 54% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-10-15 9:52 UTC (permalink / raw)
To: meta
> > > 2/3 is probably a fix for a long-standing problem, 3/3 was
> > > noticed while working on it. If 2/3 doesn't fix it, then maybe
> > > 1/3 will help us narrow it down.
> >
> > s/is probably/was almost certainly/ \o/
>
> No. :<
>
> It still happens when there's overlap between various search
> sources, but I think I know how to fix it...
Ok, higher certainty for now :P
-----8<-----
Subject: [PATCH] lei q: avoid kw lookup failure on remote mboxrd
When importing several sources in parallel via http(s) mboxrd,
we need to be able to get keywords of uncommitted documents
directly from shard workers. Otherwise, Xapian DocNotFound
errors happen because the read-only LeiSearch won't see
documents from uncomitted transactions. Keep in mind that it's
possible the keywords can be changed on-the-fly even for
uncommitted documents because of inotify watches from LeiNoteEvent.
---
lib/PublicInbox/LeiStore.pm | 28 +++++++++++++++++++++++-----
lib/PublicInbox/LeiXSearch.pm | 8 +++-----
lib/PublicInbox/SearchIdx.pm | 6 ++++++
t/lei_store.t | 3 ++-
4 files changed, 34 insertions(+), 11 deletions(-)
diff --git a/lib/PublicInbox/LeiStore.pm b/lib/PublicInbox/LeiStore.pm
index bf41dcf5..c45380d1 100644
--- a/lib/PublicInbox/LeiStore.pm
+++ b/lib/PublicInbox/LeiStore.pm
@@ -328,6 +328,20 @@ sub _add_vmd ($$$$) {
sto_export_kw($self, $docid, $vmd);
}
+sub _docids_and_maybe_kw ($$) {
+ my ($self, $docids) = @_;
+ return $docids unless wantarray;
+ my $kw = {};
+ for my $num (@$docids) { # likely only 1, unless ContentHash changes
+ # can't use ->search->msg_keywords on uncommitted docs
+ my $idx = $self->{priv_eidx}->idx_shard($num);
+ my $tmp = eval { $idx->ipc_do('get_terms', 'K', $num) };
+ if ($@) { warn "#$num get_terms: $@" }
+ else { @$kw{keys %$tmp} = values(%$tmp) };
+ }
+ ($docids, [ sort keys %$kw ]);
+}
+
sub add_eml {
my ($self, $eml, $vmd, $xoids) = @_;
my $im = $self->{-fake_im} // $self->importer; # may create new epoch
@@ -339,7 +353,11 @@ sub add_eml {
if ($vmd && $vmd->{sync_info}) {
set_sync_info($self, $smsg->{blob}, @{$vmd->{sync_info}});
}
- $im_mark or return; # duplicate blob returns undef
+ unless ($im_mark) { # duplicate blob returns undef
+ return unless wantarray;
+ my @docids = $oidx->blob_exists($smsg->{blob});
+ return _docids_and_maybe_kw $self, \@docids;
+ }
local $self->{current_info} = $smsg->{blob};
my $vivify_xvmd = delete($smsg->{-vivify_xvmd}) // []; # exact matches
@@ -373,7 +391,7 @@ sub add_eml {
}
_add_vmd($self, $idx, $docid, $vmd) if $vmd;
}
- $vivify_xvmd;
+ _docids_and_maybe_kw $self, $vivify_xvmd;
} elsif (my @docids = _docids_for($self, $eml)) {
# fuzzy match from within lei/store
for my $docid (@docids) {
@@ -383,8 +401,8 @@ sub add_eml {
$idx->ipc_do('add_eidx_info', $docid, '.', $eml);
_add_vmd($self, $idx, $docid, $vmd) if $vmd;
}
- \@docids;
- } else { # totally new message
+ _docids_and_maybe_kw $self, \@docids;
+ } else { # totally new message, no keywords
delete $smsg->{-oidx}; # for IPC-friendliness
$smsg->{num} = $oidx->adj_counter('eidx_docid', '+');
$oidx->add_overview($eml, $smsg);
@@ -392,7 +410,7 @@ sub add_eml {
my $idx = $eidx->idx_shard($smsg->{num});
$idx->index_eml($eml, $smsg);
_add_vmd($self, $idx, $smsg->{num}, $vmd) if $vmd;
- $smsg;
+ wantarray ? ($smsg, []) : $smsg;
}
}
diff --git a/lib/PublicInbox/LeiXSearch.pm b/lib/PublicInbox/LeiXSearch.pm
index fba16861..3ec75528 100644
--- a/lib/PublicInbox/LeiXSearch.pm
+++ b/lib/PublicInbox/LeiXSearch.pm
@@ -282,11 +282,9 @@ sub each_remote_eml { # callback for MboxReader->mboxrd
my $xoids = $lei->{ale}->xoids_for($eml, 1);
my $smsg = bless {}, 'PublicInbox::Smsg';
if ($self->{import_sto} && !$xoids) {
- my $res = $self->{import_sto}->wq_do('add_eml', $eml);
- if (ref($res) eq ref($smsg)) { # totally new message
- $smsg = $res;
- $smsg->{kw} = []; # short-circuit xsmsg_vmd
- }
+ my ($res, $kw) = $self->{import_sto}->wq_do('add_eml', $eml);
+ $smsg = $res if ref($res) eq ref($smsg); # totally new message
+ $smsg->{kw} = $kw; # short-circuit xsmsg_vmd
}
$smsg->{blob} //= $xoids ? (keys(%$xoids))[0]
: $lei->git_oid($eml)->hexdigest;
diff --git a/lib/PublicInbox/SearchIdx.pm b/lib/PublicInbox/SearchIdx.pm
index 928152ec..585f28f5 100644
--- a/lib/PublicInbox/SearchIdx.pm
+++ b/lib/PublicInbox/SearchIdx.pm
@@ -517,6 +517,12 @@ sub add_eidx_info {
$self->{xdb}->replace_document($docid, $doc);
}
+sub get_terms {
+ my ($self, $pfx, $docid) = @_;
+ begin_txn_lazy($self);
+ xap_terms($pfx, $self->{xdb}, $docid);
+}
+
sub remove_eidx_info {
my ($self, $docid, $eidx_key, $eml) = @_;
begin_txn_lazy($self);
diff --git a/t/lei_store.t b/t/lei_store.t
index c31e27a2..40ad7800 100644
--- a/t/lei_store.t
+++ b/t/lei_store.t
@@ -138,7 +138,8 @@ Subject: timezone-dependent test
WHAT IS TIME ANYMORE?
EOM
- ok($sto->add_eml($eml), 'recently received message');
+ my $smsg = $sto->add_eml($eml);
+ ok($smsg && $smsg->{blob}, 'recently received message');
$sto->done;
local $ENV{TZ} = 'GMT+5';
my $lse = $sto->search;
^ permalink raw reply related [relevance 54%]
* [PATCH 6/7] lei up: actually rely on DESTROY for --alllll
2021-10-14 13:16 70% ` [PATCH 0/7] lei: more process handling fixes Eric Wong
2021-10-14 13:16 71% ` [PATCH 1/7] lei: use send() perlop for signals Eric Wong
2021-10-14 13:16 34% ` [PATCH 5/7] lei: TSTP affects all curl and related subprocesses Eric Wong
@ 2021-10-14 13:16 67% ` Eric Wong
2021-10-14 13:16 62% ` [PATCH 7/7] lei up --all: send signals to workers, receive errors Eric Wong
3 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-10-14 13:16 UTC (permalink / raw)
To: meta
We need to use DESTROY here to ensure we wait for workers, too;
not just the initial dispatch.
Fixes: cafbd77b3c82167d ("lei up: avoid excessively parallel --all")
---
lib/PublicInbox/LeiUp.pm | 11 ++++++++---
1 file changed, 8 insertions(+), 3 deletions(-)
diff --git a/lib/PublicInbox/LeiUp.pm b/lib/PublicInbox/LeiUp.pm
index 3011300dd836..719736e8597e 100644
--- a/lib/PublicInbox/LeiUp.pm
+++ b/lib/PublicInbox/LeiUp.pm
@@ -63,6 +63,7 @@ sub redispatch_all ($$) {
$op_c->{ops} = { '' => [ $lei->can('dclose'), $lei ] };
my @first_batch = splice(@$upq, 0, $j); # initial parallelism
$lei->{-upq} = $upq;
+ $lei->{daemon_pid} = $$;
$lei->event_step_init; # wait for client disconnects
for my $out (@first_batch) {
PublicInbox::DS::requeue(
@@ -158,18 +159,22 @@ sub event_step { # runs via PublicInbox::DS::requeue
$l->{opt} = { %{$l->{opt}} }; # deep copy
delete $l->{opt}->{all};
$l->qerr("# updating $self->{out}");
- $l->{up_op_p} = $self->{op_p}; # ($l => $lei => script/lei)
+ my $o = " (output: $self->{out})"; # add to all warnings
my $cb = $SIG{__WARN__} // \&CORE::warn;
- my $o = " (output: $self->{out})";
local $SIG{__WARN__} = sub {
my @m = @_;
push(@m, $o) if !@m || $m[-1] !~ s/\n\z/$o\n/;
$cb->(@m);
};
+ $l->{-up1} = $self;
eval { $l->dispatch('up', $self->{out}) };
$lei->child_error(0, $@) if $@ || $l->{failed}; # lei->fail()
+}
- # onto the next:
+sub DESTROY {
+ my ($self) = @_;
+ my $lei = $self->{lei}; # the original, from lei_up
+ return if $lei->{daemon_pid} != $$;
my $out = shift(@{$lei->{-upq}}) or return;
PublicInbox::DS::requeue(nxt($lei, $out, $self->{op_p}));
}
^ permalink raw reply related [relevance 67%]
* [PATCH 7/7] lei up --all: send signals to workers, receive errors
2021-10-14 13:16 70% ` [PATCH 0/7] lei: more process handling fixes Eric Wong
` (2 preceding siblings ...)
2021-10-14 13:16 67% ` [PATCH 6/7] lei up: actually rely on DESTROY for --alllll Eric Wong
@ 2021-10-14 13:16 62% ` Eric Wong
3 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-10-14 13:16 UTC (permalink / raw)
To: meta
The redispatch mechanism wasn't routing signals and messages
between redispatched workers and script/lei properly. We now
rely on PktOp to do bidirectional message forwarding and
carefully avoiding circular references by using PktOp.
---
lib/PublicInbox/LEI.pm | 7 ++++++-
lib/PublicInbox/LeiUp.pm | 13 ++++++++++++-
2 files changed, 18 insertions(+), 2 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index d0905562f616..b6338377328f 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -573,6 +573,7 @@ sub _lei_atfork_child {
POSIX::setpgid(0, $$) // die "setpgid(0, $$): $!";
}
close($_) for (grep(defined, delete @$self{qw(3 old_1 au_done)}));
+ delete $self->{-socks};
if (my $op_c = delete $self->{pkt_op_c}) {
close(delete $op_c->{sock});
}
@@ -1144,7 +1145,9 @@ sub event_step {
if ($buf eq '') {
_drop_wq($self); # EOF, client disconnected
dclose($self);
- } elsif ($buf =~ /\A(?:STOP|CONT)\z/) {
+ $buf = 'TERM';
+ }
+ if ($buf =~ /\A(?:STOP|CONT|TERM)\z/) {
my $sig = "-$buf";
for my $wq (grep(defined, @$self{@WQ_KEYS})) {
$wq->wq_kill($sig) or $wq->wq_kill_old($sig);
@@ -1152,6 +1155,8 @@ sub event_step {
} else {
die "unrecognized client signal: $buf";
}
+ my $s = $self->{-socks} // []; # lei up --all
+ @$s = grep { send($_, $buf, MSG_EOR) } @$s;
};
if (my $err = $@) {
eval { $self->fail($err) };
diff --git a/lib/PublicInbox/LeiUp.pm b/lib/PublicInbox/LeiUp.pm
index 719736e8597e..df65cb9b8474 100644
--- a/lib/PublicInbox/LeiUp.pm
+++ b/lib/PublicInbox/LeiUp.pm
@@ -166,7 +166,15 @@ sub event_step { # runs via PublicInbox::DS::requeue
push(@m, $o) if !@m || $m[-1] !~ s/\n\z/$o\n/;
$cb->(@m);
};
- $l->{-up1} = $self;
+ $l->{-up1} = $self; # for LeiUp1->DESTROY
+ delete @$l{qw(-socks -event_init_done)};
+ my ($op_c, $op_p) = PublicInbox::PktOp->pair;
+ $self->{unref_on_destroy} = $op_c->{sock}; # to cleanup $lei->{-socks}
+ $lei->pkt_ops($op_c->{ops} //= {}); # errors from $l -> script/lei
+ push @{$lei->{-socks}}, $op_c->{sock}; # script/lei signals to $l
+ $l->{sock} = $op_p->{op_p}; # receive signals from op_c->{sock}
+ $op_c = $op_p = undef;
+
eval { $l->dispatch('up', $self->{out}) };
$lei->child_error(0, $@) if $@ || $l->{failed}; # lei->fail()
}
@@ -175,6 +183,9 @@ sub DESTROY {
my ($self) = @_;
my $lei = $self->{lei}; # the original, from lei_up
return if $lei->{daemon_pid} != $$;
+ my $sock = delete $self->{unref_on_destroy};
+ my $s = $lei->{-socks} // [];
+ @$s = grep { $_ != $sock } @$s;
my $out = shift(@{$lei->{-upq}}) or return;
PublicInbox::DS::requeue(nxt($lei, $out, $self->{op_p}));
}
^ permalink raw reply related [relevance 62%]
* [PATCH 5/7] lei: TSTP affects all curl and related subprocesses
2021-10-14 13:16 70% ` [PATCH 0/7] lei: more process handling fixes Eric Wong
2021-10-14 13:16 71% ` [PATCH 1/7] lei: use send() perlop for signals Eric Wong
@ 2021-10-14 13:16 34% ` Eric Wong
2021-10-14 13:16 67% ` [PATCH 6/7] lei up: actually rely on DESTROY for --alllll Eric Wong
2021-10-14 13:16 62% ` [PATCH 7/7] lei up --all: send signals to workers, receive errors Eric Wong
3 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-10-14 13:16 UTC (permalink / raw)
To: meta
By relying more on pgroups for remaining remaining processes,
this lets us pause all curl+tail subprocesses with a single
kill(2) to avoid cluttering stderr.
We won't bother pausing the pigz/gzip/bzip2/xz compressor
process not cat-file processes, though, since those don't write
to the terminal and they idle soon after the workers react to
SIGSTOP.
AutoReap is hoisted out from TestCommon.pm. CLONE_SKIP
is gone since we won't be using Perl threads any time
soon (they're discouraged by the maintainers of Perl).
---
MANIFEST | 1 +
lib/PublicInbox/AutoReap.pm | 34 +++++++++++++++++++++++++++++++++
lib/PublicInbox/LEI.pm | 7 +------
lib/PublicInbox/LeiInput.pm | 7 ++++---
lib/PublicInbox/LeiMirror.pm | 8 +++-----
lib/PublicInbox/LeiRemote.pm | 13 +++++--------
lib/PublicInbox/LeiToMail.pm | 2 +-
lib/PublicInbox/LeiXSearch.pm | 20 +++++++++----------
lib/PublicInbox/TestCommon.pm | 36 +++--------------------------------
9 files changed, 61 insertions(+), 67 deletions(-)
create mode 100644 lib/PublicInbox/AutoReap.pm
diff --git a/MANIFEST b/MANIFEST
index 122ceda0a761..b89513d5afb5 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -147,6 +147,7 @@ lib/PublicInbox/AddressPP.pm
lib/PublicInbox/Admin.pm
lib/PublicInbox/AdminEdit.pm
lib/PublicInbox/AltId.pm
+lib/PublicInbox/AutoReap.pm
lib/PublicInbox/Cgit.pm
lib/PublicInbox/CmdIPC4.pm
lib/PublicInbox/CompressNoop.pm
diff --git a/lib/PublicInbox/AutoReap.pm b/lib/PublicInbox/AutoReap.pm
new file mode 100644
index 000000000000..23ecce772186
--- /dev/null
+++ b/lib/PublicInbox/AutoReap.pm
@@ -0,0 +1,34 @@
+# Copyright (C) all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+
+# automatically kill + reap children when this goes out-of-scope
+package PublicInbox::AutoReap;
+use v5.10.1;
+use strict;
+
+sub new {
+ my (undef, $pid, $cb) = @_;
+ bless { pid => $pid, cb => $cb, owner => $$ }, __PACKAGE__
+}
+
+sub kill {
+ my ($self, $sig) = @_;
+ CORE::kill($sig // 'TERM', $self->{pid});
+}
+
+sub join {
+ my ($self, $sig) = @_;
+ my $pid = delete $self->{pid} or return;
+ $self->{cb}->() if defined $self->{cb};
+ CORE::kill($sig, $pid) if defined $sig;
+ my $ret = waitpid($pid, 0) // die "waitpid($pid): $!";
+ $ret == $pid or die "BUG: waitpid($pid) != $ret";
+}
+
+sub DESTROY {
+ my ($self) = @_;
+ return if $self->{owner} != $$;
+ $self->join('TERM');
+}
+
+1;
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 9620e2642213..d0905562f616 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -516,12 +516,6 @@ sub sigpipe_handler { # handles SIGPIPE from @WQ_KEYS workers
fail_handler($_[0], 13, delete $_[0]->{1});
}
-# PublicInbox::OnDestroy callback for SIGINT to take out the entire pgid
-sub sigint_reap {
- my ($pgid) = @_;
- dwaitpid($pgid) if kill('-INT', $pgid);
-}
-
sub fail ($$;$) {
my ($self, $buf, $exit_code) = @_;
local $current_lei = $self;
@@ -600,6 +594,7 @@ sub _lei_atfork_child {
$cb->(@_) unless PublicInbox::Eml::warn_ignore(@_)
};
}
+ $SIG{TERM} = sub { exit(128 + 15) };
$current_lei = $persist ? undef : $self; # for SIG{__WARN__}
}
diff --git a/lib/PublicInbox/LeiInput.pm b/lib/PublicInbox/LeiInput.pm
index 6a90e7e1e756..dd40d83840c5 100644
--- a/lib/PublicInbox/LeiInput.pm
+++ b/lib/PublicInbox/LeiInput.pm
@@ -8,6 +8,7 @@ use v5.10.1;
use PublicInbox::DS;
use PublicInbox::Spawn qw(which popen_rd);
use PublicInbox::InboxWritable qw(eml_from_path);
+use PublicInbox::AutoReap;
# JMAP RFC 8621 4.1.1
# https://www.iana.org/assignments/imap-jmap-keywords/imap-jmap-keywords.xhtml
@@ -102,13 +103,13 @@ sub handle_http_input ($$@) {
push @$curl, '-s', @$curl_opt;
my $cmd = $curl->for_uri($lei, $uri);
$lei->qerr("# $cmd");
- my $rdr = { 2 => $lei->{2}, pgid => 0 };
- my ($fh, $pid) = popen_rd($cmd, undef, $rdr);
+ my ($fh, $pid) = popen_rd($cmd, undef, { 2 => $lei->{2} });
+ my $ar = PublicInbox::AutoReap->new($pid);
grep(/\A--compressed\z/, @$curl) or
$fh = IO::Uncompress::Gunzip->new($fh, MultiStream => 1);
eval { $self->input_fh('mboxrd', $fh, $url, @args) };
my $err = $@;
- waitpid($pid, 0);
+ $ar->join;
$? || $err and
$lei->child_error($?, "@$cmd failed".$err ? " $err" : '');
}
diff --git a/lib/PublicInbox/LeiMirror.pm b/lib/PublicInbox/LeiMirror.pm
index f1bc82e27205..a75c99c4987f 100644
--- a/lib/PublicInbox/LeiMirror.pm
+++ b/lib/PublicInbox/LeiMirror.pm
@@ -7,6 +7,7 @@ use strict;
use v5.10.1;
use parent qw(PublicInbox::IPC);
use PublicInbox::Config;
+use PublicInbox::AutoReap;
use IO::Uncompress::Gunzip qw(gunzip $GunzipError);
use IO::Compress::Gzip qw(gzip $GzipError);
use PublicInbox::Spawn qw(popen_rd spawn);
@@ -192,10 +193,8 @@ sub index_cloned_inbox {
sub run_reap {
my ($lei, $cmd, $opt) = @_;
$lei->qerr("# @$cmd");
- my $pid = spawn($cmd, undef, $opt);
- my $reap = PublicInbox::OnDestroy->new($lei->can('sigint_reap'), $pid);
- waitpid($pid, 0) == $pid or die "waitpid @$cmd: $!";
- @$reap = (); # cancel reap
+ my $ar = PublicInbox::AutoReap->new(spawn($cmd, undef, $opt));
+ $ar->join;
my $ret = $?;
$? = 0; # don't let it influence normal exit
$ret;
@@ -459,7 +458,6 @@ sub start {
sub ipc_atfork_child {
my ($self) = @_;
$self->{lei}->_lei_atfork_child;
- $SIG{TERM} = sub { exit(128 + 15) }; # trigger OnDestroy $reap
$self->SUPER::ipc_atfork_child;
}
diff --git a/lib/PublicInbox/LeiRemote.pm b/lib/PublicInbox/LeiRemote.pm
index 7782aa9dbfa1..54750062fd5f 100644
--- a/lib/PublicInbox/LeiRemote.pm
+++ b/lib/PublicInbox/LeiRemote.pm
@@ -9,10 +9,10 @@ package PublicInbox::LeiRemote;
use v5.10.1;
use strict;
use IO::Uncompress::Gunzip;
-use PublicInbox::OnDestroy;
use PublicInbox::MboxReader;
use PublicInbox::Spawn qw(popen_rd);
use PublicInbox::LeiCurl;
+use PublicInbox::AutoReap;
use PublicInbox::ContentHash qw(git_sha);
sub new {
@@ -47,17 +47,14 @@ sub mset {
$uri->query_form(q => $qstr, x => 'm', r => 1); # r=1: relevance
my $cmd = $curl->for_uri($self->{lei}, $uri);
$self->{lei}->qerr("# $cmd");
- my $rdr = { 2 => $lei->{2}, pgid => 0 };
- my ($fh, $pid) = popen_rd($cmd, undef, $rdr);
- my $reap = PublicInbox::OnDestroy->new($lei->can('sigint_reap'), $pid);
+ my ($fh, $pid) = popen_rd($cmd, undef, { 2 => $lei->{2} });
+ my $ar = PublicInbox::AutoReap->new($pid);
$self->{smsg} = [];
$fh = IO::Uncompress::Gunzip->new($fh, MultiStream => 1);
PublicInbox::MboxReader->mboxrd($fh, \&_each_mboxrd_eml, $self);
- my $err = waitpid($pid, 0) == $pid ? undef
- : "BUG: waitpid($cmd): $!";
- @$reap = (); # cancel OnDestroy
my $wait = $self->{lei}->{sto}->wq_do('done');
- die $err if $err;
+ $ar->join;
+ $lei->child_error($?) if $?;
$self; # we are the mset (and $ibx, and $self)
}
diff --git a/lib/PublicInbox/LeiToMail.pm b/lib/PublicInbox/LeiToMail.pm
index 5a220ba39735..9c748deaed16 100644
--- a/lib/PublicInbox/LeiToMail.pm
+++ b/lib/PublicInbox/LeiToMail.pm
@@ -157,7 +157,7 @@ sub _post_augment_mbox { # open a compressor process from top-level process
my $zsfx = $self->{zsfx} or return;
my $cmd = PublicInbox::MboxReader::zsfx2cmd($zsfx, undef, $lei);
my ($r, $w) = @{delete $lei->{zpipe}};
- my $rdr = { 0 => $r, 1 => $lei->{1}, 2 => $lei->{2} };
+ my $rdr = { 0 => $r, 1 => $lei->{1}, 2 => $lei->{2}, pgid => 0 };
my $pid = spawn($cmd, undef, $rdr);
my $pp = gensym;
my $dup = bless { "pid.$pid" => $cmd }, ref($lei);
diff --git a/lib/PublicInbox/LeiXSearch.pm b/lib/PublicInbox/LeiXSearch.pm
index 668d0b6e5df3..fba168613d96 100644
--- a/lib/PublicInbox/LeiXSearch.pm
+++ b/lib/PublicInbox/LeiXSearch.pm
@@ -15,6 +15,7 @@ use PublicInbox::Search qw(xap_terms);
use PublicInbox::Spawn qw(popen_rd spawn which);
use PublicInbox::MID qw(mids);
use PublicInbox::Smsg;
+use PublicInbox::AutoReap;
use PublicInbox::Eml;
use PublicInbox::LEI;
use Fcntl qw(SEEK_SET F_SETFL O_APPEND O_RDWR);
@@ -346,18 +347,17 @@ sub query_remote_mboxrd {
my @qform = (x => 'm');
push(@qform, t => 1) if $opt->{threads};
my $verbose = $opt->{verbose};
- my ($reap_tail, $reap_curl);
+ my $reap_tail;
my $cerr = File::Temp->new(TEMPLATE => 'curl.err-XXXX', TMPDIR => 1);
fcntl($cerr, F_SETFL, O_APPEND|O_RDWR) or warn "set O_APPEND: $!";
- my $rdr = { 2 => $cerr, pgid => 0 };
- my $sigint_reap = $lei->can('sigint_reap');
+ my $rdr = { 2 => $cerr };
if ($verbose) {
# spawn a process to force line-buffering, otherwise curl
# will write 1 character at-a-time and parallel outputs
# mmmaaayyy llloookkk llliiikkkeee ttthhhiiisss
- my $o = { 1 => $lei->{2}, 2 => $lei->{2}, pgid => 0 };
+ my $o = { 1 => $lei->{2}, 2 => $lei->{2} };
my $pid = spawn(['tail', '-f', $cerr->filename], undef, $o);
- $reap_tail = PublicInbox::OnDestroy->new($sigint_reap, $pid);
+ $reap_tail = PublicInbox::AutoReap->new($pid);
}
my $curl = PublicInbox::LeiCurl->new($lei, $self->{curl}) or return;
push @$curl, '-s', '-d', '';
@@ -372,16 +372,13 @@ sub query_remote_mboxrd {
my $cmd = $curl->for_uri($lei, $uri);
$lei->qerr("# $cmd");
my ($fh, $pid) = popen_rd($cmd, undef, $rdr);
- $reap_curl = PublicInbox::OnDestroy->new($sigint_reap, $pid);
+ my $reap_curl = PublicInbox::AutoReap->new($pid);
$fh = IO::Uncompress::Gunzip->new($fh, MultiStream => 1);
PublicInbox::MboxReader->mboxrd($fh, \&each_remote_eml, $self,
$lei, $each_smsg);
- my $err = waitpid($pid, 0) == $pid ? undef
- : "BUG: waitpid($cmd): $!";
- @$reap_curl = (); # cancel OnDestroy
- die $err if $err;
my $nr = $lei->{-nr_remote_eml};
my $wait = $lei->{sto}->wq_do('done') if $nr && $lei->{sto};
+ $reap_curl->join;
if ($? == 0) {
# don't update if no results, maybe MTA is down
$key && $nr and
@@ -389,7 +386,7 @@ sub query_remote_mboxrd {
mset_progress($lei, $lei->{-current_url}, $nr, $nr);
next;
}
- $err = '';
+ my $err;
if (-s $cerr) {
seek($cerr, 0, SEEK_SET) //
warn "seek($cmd stderr): $!";
@@ -397,6 +394,7 @@ sub query_remote_mboxrd {
warn "read($cmd stderr): $!";
truncate($cerr, 0) // warn "truncate($cmd stderr): $!";
}
+ $err //= '';
next if (($? >> 8) == 22 && $err =~ /\b404\b/);
$uri->query_form(q => $qstr);
$lei->child_error($?, "E: <$uri> $err");
diff --git a/lib/PublicInbox/TestCommon.pm b/lib/PublicInbox/TestCommon.pm
index 57f1db952e49..835779996d56 100644
--- a/lib/PublicInbox/TestCommon.pm
+++ b/lib/PublicInbox/TestCommon.pm
@@ -6,6 +6,7 @@ package PublicInbox::TestCommon;
use strict;
use parent qw(Exporter);
use v5.10.1;
+use PublicInbox::AutoReap;
use Fcntl qw(FD_CLOEXEC F_SETFD F_GETFD :seek);
use POSIX qw(dup2);
use IO::Socket::INET;
@@ -429,7 +430,7 @@ sub tail_f (@) {
require PublicInbox::Spawn;
my $pid = PublicInbox::Spawn::spawn($cmd, undef, { 1 => 2 });
wait_for_tail($pid, scalar @_);
- PublicInboxTestProcess->new($pid, \&wait_for_tail);
+ PublicInbox::AutoReap->new($pid, \&wait_for_tail);
}
sub start_script {
@@ -492,7 +493,7 @@ sub start_script {
die "FAIL: ",join(' ', $key, @argv), ": $!\n";
}
}
- my $td = PublicInboxTestProcess->new($pid);
+ my $td = PublicInbox::AutoReap->new($pid);
$td->{-extra} = $tail;
$td;
}
@@ -742,37 +743,6 @@ sub test_httpd ($$;$) {
};
-package PublicInboxTestProcess;
-use strict;
-
-# prevent new threads from inheriting these objects
-sub CLONE_SKIP { 1 }
-
-sub new {
- my ($cls, $pid, $cb) = @_;
- bless { pid => $pid, cb => $cb, owner => $$ }, $cls;
-}
-
-sub kill {
- my ($self, $sig) = @_;
- CORE::kill($sig // 'TERM', $self->{pid});
-}
-
-sub join {
- my ($self, $sig) = @_;
- my $pid = delete $self->{pid} or return;
- $self->{cb}->() if defined $self->{cb};
- CORE::kill($sig, $pid) if defined $sig;
- my $ret = waitpid($pid, 0) // die "waitpid($pid): $!";
- $ret == $pid or die "waitpid($pid) != $ret";
-}
-
-sub DESTROY {
- my ($self) = @_;
- return if $self->{owner} != $$;
- $self->join('TERM');
-}
-
package PublicInbox::TestCommon::InboxWakeup;
use strict;
sub on_inbox_unlock { ${$_[0]}->($_[1]) }
^ permalink raw reply related [relevance 34%]
* [PATCH 1/7] lei: use send() perlop for signals
2021-10-14 13:16 70% ` [PATCH 0/7] lei: more process handling fixes Eric Wong
@ 2021-10-14 13:16 71% ` Eric Wong
2021-10-14 13:16 34% ` [PATCH 5/7] lei: TSTP affects all curl and related subprocesses Eric Wong
` (2 subsequent siblings)
3 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-10-14 13:16 UTC (permalink / raw)
To: meta
This may save us a small bit of startup time since there's
fewer args and opcodes should be smaller.
---
script/lei | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/script/lei b/script/lei
index 8f6e8aacb86b..5cad19d77603 100755
--- a/script/lei
+++ b/script/lei
@@ -107,8 +107,8 @@ my $buf = join("\0", scalar(@ARGV), @ARGV);
while (my ($k, $v) = each %ENV) { $buf .= "\0$k=$v" }
$buf .= "\0\0";
$send_cmd->($sock, [0, 1, 2, fileno($dh)], $buf, MSG_EOR) or die "sendmsg: $!";
-$SIG{TSTP} = sub { $send_cmd->($sock, [], 'STOP', MSG_EOR); kill 'STOP', $$ };
-$SIG{CONT} = sub { $send_cmd->($sock, [], 'CONT', MSG_EOR) };
+$SIG{TSTP} = sub { send($sock, 'STOP', MSG_EOR); kill 'STOP', $$ };
+$SIG{CONT} = sub { send($sock, 'CONT', MSG_EOR) };
my $x_it_code = 0;
while (1) {
^ permalink raw reply related [relevance 71%]
* [PATCH 0/7] lei: more process handling fixes
2021-10-14 4:32 61% ` [PATCH 3/3] lei: give workers their own process group Eric Wong
@ 2021-10-14 13:16 70% ` Eric Wong
2021-10-14 13:16 71% ` [PATCH 1/7] lei: use send() perlop for signals Eric Wong
` (3 more replies)
0 siblings, 4 replies; 200+ results
From: Eric Wong @ 2021-10-14 13:16 UTC (permalink / raw)
To: meta
"lei up --all" SIGTSTP/CONT/INT handling was totally
broken and now fixed. And we put cat-file processes
into their own pgrp, so it avoids scary errors when
hitting Ctrl-C on -extindex, too.
Automated testing of interactive stuff is tricky, so
it's not being done, currently :<
Eric Wong (7):
lei: use send() perlop for signals
git: async_err shows retried requests properly
git: ->fail invokes current callback
git: cat-file --batch are their own pgrp
lei: TSTP affects all curl and related subprocesses
lei up: actually rely on DESTROY for --all
lei up --all: send signals to workers, receive errors
MANIFEST | 1 +
lib/PublicInbox/AutoReap.pm | 34 +++++++++++++++++++++++++++++++++
lib/PublicInbox/Git.pm | 36 +++++++++++++++++++----------------
lib/PublicInbox/LEI.pm | 14 +++++++-------
lib/PublicInbox/LeiInput.pm | 7 ++++---
lib/PublicInbox/LeiMirror.pm | 8 +++-----
lib/PublicInbox/LeiRemote.pm | 13 +++++--------
lib/PublicInbox/LeiToMail.pm | 2 +-
lib/PublicInbox/LeiUp.pm | 22 ++++++++++++++++++---
lib/PublicInbox/LeiXSearch.pm | 20 +++++++++----------
lib/PublicInbox/TestCommon.pm | 36 +++--------------------------------
script/lei | 4 ++--
12 files changed, 108 insertions(+), 89 deletions(-)
create mode 100644 lib/PublicInbox/AutoReap.pm
^ permalink raw reply [relevance 70%]
* [PATCH] lei: -d (--dir) and -O (only) shortcuts
@ 2021-10-14 9:54 60% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-10-14 9:54 UTC (permalink / raw)
To: meta
`-d' seems like a non-brainer for --dir with inspect.
I find myself using `--only' a bit, too, and `-O' seems like
a reasonable shortcut for it.
---
Documentation/lei-q.pod | 2 ++
lib/PublicInbox/LEI.pm | 9 +++++----
2 files changed, 7 insertions(+), 4 deletions(-)
diff --git a/Documentation/lei-q.pod b/Documentation/lei-q.pod
index e1e3666d797a..574c12eb9d3a 100644
--- a/Documentation/lei-q.pod
+++ b/Documentation/lei-q.pod
@@ -164,6 +164,8 @@ multiple times.
=item --only=LOCATION
+=item -O LOCATION
+
Use only the specified external for search. This option may be given
multiple times, in which case the search uses only the specified set.
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 145af7e2cb59..9620e2642213 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -148,7 +148,7 @@ sub index_opt {
my @c_opt = qw(c=s@ C=s@ quiet|q);
my @net_opt = (qw(no-torsocks torsocks=s), PublicInbox::LeiQuery::curl_opt());
-my @lxs_opt = qw(remote! local! external! include|I=s@ exclude=s@ only=s@
+my @lxs_opt = qw(remote! local! external! include|I=s@ exclude=s@ only|O=s@
import-remote!);
# we don't support -C as an alias for --find-copies since it's already
@@ -284,7 +284,7 @@ our %CMD = ( # sorted in order of importance/use:
}, qw(config-file|system|global|file|f=s), # for conflict detection
qw(edit|e c=s@ C=s@), pass_through('git config') ],
'inspect' => [ 'ITEMS...|--stdin', 'inspect lei/store and/or local external',
- qw(stdin| pretty ascii dir=s), @c_opt ],
+ qw(stdin| pretty ascii dir|d=s), @c_opt ],
'init' => [ '[DIRNAME]', sub {
"initialize storage, default: ".store_path($_[0]);
@@ -337,7 +337,8 @@ my %OPTDESC = (
'path-a|a=s' => 'pre-image pathname associated with OID',
'path-b|b=s' => 'post-image pathname associated with OID',
'git-dir=s@' => 'additional git repository to scan',
-'dir=s inspect' => 'specify a inboxdir, extindex topdir or Xapian shard',
+'dir|d=s inspect' =>
+ 'specify a inboxdir, extindex topdir or Xapian shard',
'proxy=s' => [ 'PROTO://HOST[:PORT]', # shared with curl(1)
"proxy for (e.g. `socks5h://0:9050')" ],
'torsocks=s' => ['VAL|auto|no|yes',
@@ -395,7 +396,7 @@ my %OPTDESC = (
'exclude specified external(s) from search' ],
'include|I=s@ q' => [ 'LOCATION',
'include specified external(s) in search' ],
-'only=s@ q' => [ 'LOCATION',
+'only|O=s@ q' => [ 'LOCATION',
'only use specified external(s) for search' ],
'jobs=s q' => [ '[SEARCH_JOBS][,WRITER_JOBS]',
'control number of search and writer jobs' ],
^ permalink raw reply related [relevance 60%]
* Re: [PATCH 0/3] lei: hopefully^W kill /Document \d+ not found/ errors
@ 2021-10-14 5:31 71% ` Eric Wong
2021-10-15 9:52 54% ` [PATCH] lei q: avoid kw lookup failure on remote mboxrd Eric Wong
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2021-10-14 5:31 UTC (permalink / raw)
To: meta
Eric Wong <e@80x24.org> wrote:
> Eric Wong <e@80x24.org> wrote:
> > 2/3 is probably a fix for a long-standing problem, 3/3 was
> > noticed while working on it. If 2/3 doesn't fix it, then maybe
> > 1/3 will help us narrow it down.
>
> s/is probably/was almost certainly/ \o/
No. :<
It still happens when there's overlap between various search
sources, but I think I know how to fix it...
^ permalink raw reply [relevance 71%]
* [PATCH 2/3] lei add-external --mirror: respect client umask
@ 2021-10-14 4:32 65% ` Eric Wong
2021-10-14 4:32 61% ` [PATCH 3/3] lei: give workers their own process group Eric Wong
1 sibling, 0 replies; 200+ results
From: Eric Wong @ 2021-10-14 4:32 UTC (permalink / raw)
To: meta
While lei is intended for non-public mail and runs umask(077)
by default, externals are one area which can safely defer to
the user's umask.
Instead of sending it unconditionally with every command, only
have lei-daemon request it when necessary.
---
lib/PublicInbox/LEI.pm | 11 +++++++++++
lib/PublicInbox/LeiMirror.pm | 2 ++
script/lei | 2 ++
3 files changed, 15 insertions(+)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index bd8a6bef632b..635cd0c5508a 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -1518,4 +1518,15 @@ sub cfg_dump ($$) {
undef;
}
+sub request_umask {
+ my ($lei) = @_;
+ my $s = $lei->{sock} // return;
+ send($s, 'umask', MSG_EOR) // die "send: $!";
+ vec(my $rvec = '', fileno($s), 1) = 1;
+ select($rvec, undef, undef, 2) or die 'timeout waiting for umask';
+ recv($s, my $v, 5, 0) // die "recv: $!";
+ (my $u, $lei->{client_umask}) = unpack('AV', $v);
+ $u eq 'u' or warn "E: recv $v has no umask";
+}
+
1;
diff --git a/lib/PublicInbox/LeiMirror.pm b/lib/PublicInbox/LeiMirror.pm
index 1369c00c57fd..fb73d8631670 100644
--- a/lib/PublicInbox/LeiMirror.pm
+++ b/lib/PublicInbox/LeiMirror.pm
@@ -424,6 +424,7 @@ sub start_clone_url {
sub do_mirror { # via wq_io_do
my ($self) = @_;
my $lei = $self->{lei};
+ umask($lei->{client_umask}) if defined $lei->{client_umask};
eval {
my $iv = $lei->{opt}->{'inbox-version'};
if (defined $iv) {
@@ -448,6 +449,7 @@ sub start {
require PublicInbox::Inbox;
require PublicInbox::Admin;
require PublicInbox::InboxWritable;
+ $lei->request_umask;
my ($op_c, $ops) = $lei->workers_start($self, 1);
$lei->{wq1} = $self;
$self->wq_io_do('do_mirror', []);
diff --git a/script/lei b/script/lei
index bc43779821e7..8f6e8aacb86b 100755
--- a/script/lei
+++ b/script/lei
@@ -122,6 +122,8 @@ while (1) {
$exec_cmd->(\@fds, split(/\0/, $1));
} elsif ($buf eq '-WINCH') {
kill($buf, @parent); # for MUA
+ } elsif ($buf eq 'umask') {
+ send($sock, 'u'.pack('V', umask), MSG_EOR) or die "send: $!"
} elsif ($buf =~ /\Ax_it ([0-9]+)\z/) {
$x_it_code ||= $1 + 0;
last;
^ permalink raw reply related [relevance 65%]
* [PATCH 3/3] lei: give workers their own process group
2021-10-14 4:32 65% ` [PATCH 2/3] lei add-external --mirror: respect client umask Eric Wong
@ 2021-10-14 4:32 61% ` Eric Wong
2021-10-14 13:16 70% ` [PATCH 0/7] lei: more process handling fixes Eric Wong
1 sibling, 1 reply; 200+ results
From: Eric Wong @ 2021-10-14 4:32 UTC (permalink / raw)
To: meta
This lets users Ctrl-Z from their terminal to pause an entire
git-clone process hierarchy.
---
lib/PublicInbox/LEI.pm | 10 ++++++----
lib/PublicInbox/LeiMirror.pm | 3 +--
lib/PublicInbox/LeiXSearch.pm | 2 +-
3 files changed, 8 insertions(+), 7 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 635cd0c5508a..145af7e2cb59 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -459,9 +459,9 @@ my @WQ_KEYS = qw(lxs l2m ikw pmd wq1 lne); # internal workers
sub _drop_wq {
my ($self) = @_;
for my $wq (grep(defined, delete(@$self{@WQ_KEYS}))) {
- if ($wq->wq_kill) {
+ if ($wq->wq_kill('-TERM')) {
$wq->wq_close(0, undef, $self);
- } elsif ($wq->wq_kill_old) {
+ } elsif ($wq->wq_kill_old('-TERM')) {
$wq->wq_wait_old(undef, $self);
}
$wq->DESTROY;
@@ -575,6 +575,7 @@ sub _lei_atfork_child {
} else { # worker, Net::NNTP (Net::Cmd) uses STDERR directly
open STDERR, '+>&='.fileno($self->{2}) or warn "open $!";
STDERR->autoflush(1);
+ POSIX::setpgid(0, $$) // die "setpgid(0, $$): $!";
}
close($_) for (grep(defined, delete @$self{qw(3 old_1 au_done)}));
if (my $op_c = delete $self->{pkt_op_c}) {
@@ -1147,9 +1148,10 @@ sub event_step {
if ($buf eq '') {
_drop_wq($self); # EOF, client disconnected
dclose($self);
- } elsif ($buf =~ /\A(STOP|CONT)\z/) {
+ } elsif ($buf =~ /\A(?:STOP|CONT)\z/) {
+ my $sig = "-$buf";
for my $wq (grep(defined, @$self{@WQ_KEYS})) {
- $wq->wq_kill($buf) or $wq->wq_kill_old($buf);
+ $wq->wq_kill($sig) or $wq->wq_kill_old($sig);
}
} else {
die "unrecognized client signal: $buf";
diff --git a/lib/PublicInbox/LeiMirror.pm b/lib/PublicInbox/LeiMirror.pm
index fb73d8631670..f1bc82e27205 100644
--- a/lib/PublicInbox/LeiMirror.pm
+++ b/lib/PublicInbox/LeiMirror.pm
@@ -9,7 +9,7 @@ use parent qw(PublicInbox::IPC);
use PublicInbox::Config;
use IO::Uncompress::Gunzip qw(gunzip $GunzipError);
use IO::Compress::Gzip qw(gzip $GzipError);
-use PublicInbox::Spawn qw(popen_rd spawn run_die);
+use PublicInbox::Spawn qw(popen_rd spawn);
use File::Temp ();
use Fcntl qw(SEEK_SET O_CREAT O_EXCL O_WRONLY);
use Carp qw(croak);
@@ -192,7 +192,6 @@ sub index_cloned_inbox {
sub run_reap {
my ($lei, $cmd, $opt) = @_;
$lei->qerr("# @$cmd");
- $opt->{pgid} = 0 if $lei->{sock};
my $pid = spawn($cmd, undef, $opt);
my $reap = PublicInbox::OnDestroy->new($lei->can('sigint_reap'), $pid);
waitpid($pid, 0) == $pid or die "waitpid @$cmd: $!";
diff --git a/lib/PublicInbox/LeiXSearch.pm b/lib/PublicInbox/LeiXSearch.pm
index ee9216feeb23..668d0b6e5df3 100644
--- a/lib/PublicInbox/LeiXSearch.pm
+++ b/lib/PublicInbox/LeiXSearch.pm
@@ -469,7 +469,7 @@ sub do_post_augment {
$err = $@;
if ($err) {
if (my $lxs = delete $lei->{lxs}) {
- $lxs->wq_kill;
+ $lxs->wq_kill('-TERM');
$lxs->wq_close(0, undef, $lei);
}
$lei->fail("$err");
^ permalink raw reply related [relevance 61%]
* [PATCH] lei inspect: account for non-extindex inboxes
@ 2021-10-14 3:12 57% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-10-14 3:12 UTC (permalink / raw)
To: meta
Inbox->xdb does not exist, but this code path was apparently
never tested :x I noticed this on basic v2 inbox, but it could
happen with any v1/v2 inbox. Move ->num2docid into Search
so it's less awkward to use.
---
lib/PublicInbox/LeiInspect.pm | 6 ++++--
lib/PublicInbox/LeiSearch.pm | 14 +++-----------
lib/PublicInbox/Search.pm | 8 ++++++++
t/lei-mirror.t | 6 ++++++
4 files changed, 21 insertions(+), 13 deletions(-)
diff --git a/lib/PublicInbox/LeiInspect.pm b/lib/PublicInbox/LeiInspect.pm
index 590dfdab..5ba96056 100644
--- a/lib/PublicInbox/LeiInspect.pm
+++ b/lib/PublicInbox/LeiInspect.pm
@@ -143,8 +143,10 @@ sub inspect_num ($$) {
my $ent = { num => $num };
if (defined(my $dir = $lei->{opt}->{dir})) {
$ibx = dir2ibx($lei, $dir) or return;
- $ent->{xdb} = $ibx->xdb and # for inspect_docid
- $docid = PublicInbox::LeiSearch::num2docid($ibx, $num);
+ if (my $srch = $ibx->search) {
+ $ent->{xdb} = $srch->xdb and
+ $docid = $srch->num2docid($num);
+ }
} elsif ($lei->{lse}) {
$ibx = $lei->{lse};
$lei->{lse}->xdb; # set {nshard} for num2docid
diff --git a/lib/PublicInbox/LeiSearch.pm b/lib/PublicInbox/LeiSearch.pm
index 4e048e9a..3e046b21 100644
--- a/lib/PublicInbox/LeiSearch.pm
+++ b/lib/PublicInbox/LeiSearch.pm
@@ -11,18 +11,10 @@ use PublicInbox::ContentHash qw(content_digest content_hash);
use PublicInbox::MID qw(mids mids_for_index);
use Carp qw(croak);
-# get combined docid from over.num:
-# (not generic Xapian, only works with our sharding scheme)
-sub num2docid ($$) {
- my ($self, $num) = @_;
- my $nshard = $self->{nshard};
- ($num - 1) * $nshard + $num % $nshard + 1;
-}
-
sub _msg_kw { # retry_reopen callback
my ($self, $num) = @_;
my $xdb = $self->xdb; # set {nshard} for num2docid;
- xap_terms('K', $xdb, num2docid($self, $num));
+ xap_terms('K', $xdb, $self->num2docid($num));
}
sub msg_keywords { # array or hashref
@@ -35,7 +27,7 @@ sub _oid_kw { # retry_reopen callback
my $xdb = $self->xdb; # set {nshard};
my %kw;
for my $num (@$nums) { # there should only be one...
- my $doc = $xdb->get_document(num2docid($self, $num));
+ my $doc = $xdb->get_document($self->num2docid($num));
my $x = xap_terms('K', $doc);
%kw = (%kw, %$x);
}
@@ -56,7 +48,7 @@ sub _xsmsg_vmd { # retry_reopen
$kw{flagged} = 1 if delete($smsg->{lei_q_tt_flagged});
my @num = $self->over->blob_exists($smsg->{blob});
for my $num (@num) { # there should only be one...
- $doc = $xdb->get_document(num2docid($self, $num));
+ $doc = $xdb->get_document($self->num2docid($num));
$x = xap_terms('K', $doc);
%kw = (%kw, %$x);
if ($want_label) { # JSON/JMAP only
diff --git a/lib/PublicInbox/Search.pm b/lib/PublicInbox/Search.pm
index f0e7ed0c..d89bf545 100644
--- a/lib/PublicInbox/Search.pm
+++ b/lib/PublicInbox/Search.pm
@@ -570,4 +570,12 @@ sub xap_terms ($$;@) {
wantarray ? sort(keys(%ret)) : \%ret;
}
+# get combined docid from over.num:
+# (not generic Xapian, only works with our sharding scheme)
+sub num2docid ($$) {
+ my ($self, $num) = @_;
+ my $nshard = $self->{nshard};
+ ($num - 1) * $nshard + $num % $nshard + 1;
+}
+
1;
diff --git a/t/lei-mirror.t b/t/lei-mirror.t
index 294eb654..646ff2b1 100644
--- a/t/lei-mirror.t
+++ b/t/lei-mirror.t
@@ -176,6 +176,12 @@ SKIP: {
$f = "$d/t2/msgmap.sqlite3";
$ca = PublicInbox::Msgmap->new_file($f)->created_at;
is($ca, $created{v2}, 'clone + index v1 synced ->created_at');
+ test_lei(sub {
+ lei_ok qw(inspect num:1 --dir), "$d/t1";
+ ok(ref(json_utf8->decode($lei_out)), 'inspect num: on v1');
+ lei_ok qw(inspect num:1 --dir), "$d/t2";
+ ok(ref(json_utf8->decode($lei_out)), 'inspect num: on v2');
+ });
}
ok($td->kill, 'killed -httpd');
^ permalink raw reply related [relevance 57%]
* [PATCH 5/7] t/lei-mirror: avoid reading ~/.public-inbox/config in test
@ 2021-10-13 10:16 71% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-10-13 10:16 UTC (permalink / raw)
To: meta
Oops, we shouldn't attempt to read a users' actual HOME when
running -index, since mine has a bunch of invalid entries in
there.
---
t/lei-mirror.t | 1 +
1 file changed, 1 insertion(+)
diff --git a/t/lei-mirror.t b/t/lei-mirror.t
index b449e0b4..294eb654 100644
--- a/t/lei-mirror.t
+++ b/t/lei-mirror.t
@@ -167,6 +167,7 @@ SKIP: {
my $after = [ glob("$d/t1/*") ];
is_deeply($before, $after, 'no new files created');
+ local $ENV{HOME} = $tmpdir;
ok(run_script([qw(-index -Lbasic), "$d/t1"]), 'index v1');
ok(run_script([qw(-index -Lbasic), "$d/t2"]), 'index v2');
my $f = "$d/t1/public-inbox/msgmap.sqlite3";
^ permalink raw reply related [relevance 71%]
* [PATCH 2/5] lei: use standard warn() in more places
@ 2021-10-13 7:00 28% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-10-13 7:00 UTC (permalink / raw)
To: meta
warn() is easier to augment with context information, and
frankly unavoidable in the presence of 3rd-party libraries
we don't control.
---
lib/PublicInbox/Fetch.pm | 2 +-
lib/PublicInbox/LEI.pm | 18 +++++++++++++-----
lib/PublicInbox/LeiBlob.pm | 2 +-
lib/PublicInbox/LeiConvert.pm | 2 +-
lib/PublicInbox/LeiForgetExternal.pm | 5 ++---
lib/PublicInbox/LeiImport.pm | 7 +++----
lib/PublicInbox/LeiImportKw.pm | 2 +-
lib/PublicInbox/LeiInput.pm | 4 ++--
lib/PublicInbox/LeiMailSync.pm | 4 ++--
lib/PublicInbox/LeiMirror.pm | 8 ++++----
lib/PublicInbox/LeiP2q.pm | 2 +-
lib/PublicInbox/LeiRediff.pm | 2 +-
lib/PublicInbox/LeiRemote.pm | 2 +-
lib/PublicInbox/LeiViewText.pm | 2 +-
lib/PublicInbox/LeiXSearch.pm | 25 +++++++++++++++----------
15 files changed, 49 insertions(+), 38 deletions(-)
diff --git a/lib/PublicInbox/Fetch.pm b/lib/PublicInbox/Fetch.pm
index 5ada1f49e4dc..e5756fb664e4 100644
--- a/lib/PublicInbox/Fetch.pm
+++ b/lib/PublicInbox/Fetch.pm
@@ -52,7 +52,7 @@ sub do_manifest ($$$) {
$m0 = eval {
PublicInbox::LeiMirror::decode_manifest($fh, $mf, $mf)
};
- $lei->err($@) if $@;
+ warn($@) if $@;
}
my ($bn) = ($fn =~ m!/([^/]+)\z!);
my $curl_cmd = $lei->{curl}->for_uri($lei, $muri, qw(-R -o), $bn);
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 183cb545fe55..bd8a6bef632b 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -471,6 +471,7 @@ sub _drop_wq {
# pronounced "exit": x_it(1 << 8) => exit(1); x_it(13) => SIGPIPE
sub x_it ($$) {
my ($self, $code) = @_;
+ local $current_lei = $self;
# make sure client sees stdout before exit
$self->{1}->autoflush(1) if $self->{1};
stop_pager($self);
@@ -504,6 +505,7 @@ sub qfin { # show message on finalization (LeiFinmsg)
sub fail_handler ($;$$) {
my ($lei, $code, $io) = @_;
+ local $current_lei = $lei;
close($io) if $io; # needed to avoid warnings on SIGPIPE
_drop_wq($lei);
x_it($lei, $code // (1 << 8));
@@ -521,6 +523,7 @@ sub sigint_reap {
sub fail ($$;$) {
my ($self, $buf, $exit_code) = @_;
+ local $current_lei = $self;
$self->{failed}++;
warn($buf, "\n") if defined $buf;
$self->{pkt_op_p}->pkt_do('fail_handler') if $self->{pkt_op_p};
@@ -541,6 +544,7 @@ sub puts ($;@) { out(shift, map { "$_\n" } @_) }
sub child_error { # passes non-fatal curl exit codes to user
my ($self, $child_error, $msg) = @_; # child_error is $?
+ local $current_lei = $self;
$child_error ||= 1 << 8;
warn($msg, "\n") if defined $msg;
if ($self->{pkt_op_p}) { # to top lei-daemon
@@ -1019,7 +1023,7 @@ sub poke_mua { # forces terminal MUAs to wake up and hopefully notice new mail
$cmd = [ Text::ParseWords::shellwords($cmd) ];
send($sock, exec_buf($cmd, {}), MSG_EOR) if $sock;
} else {
- err($self, "W: unsupported --alert=$op"); # non-fatal
+ warn("W: unsupported --alert=$op\n"); # non-fatal
}
}
}
@@ -1068,7 +1072,7 @@ sub start_pager {
# display a message for user before spawning full-screen $VISUAL
sub pgr_err {
my ($self, @msg) = @_;
- return $self->err(@msg) unless $self->{sock} && -t $self->{2};
+ return warn(@msg) unless $self->{sock} && -t $self->{2};
start_pager($self, { LESS => 'RX' }); # no 'F' so we prompt
print { $self->{2} } @msg;
$self->{2}->autoflush(1);
@@ -1118,6 +1122,7 @@ sub accept_dispatch { # Listener {post_accept} callback
sub dclose {
my ($self) = @_;
+ local $current_lei = $self;
delete $self->{-progress};
_drop_wq($self) if $self->{failed};
$self->close if $self->{-event_init_done}; # PublicInbox::DS::close
@@ -1369,6 +1374,7 @@ sub DESTROY {
sub wq_done_wait { # dwaitpid callback
my ($arg, $pid) = @_;
my ($wq, $lei) = @$arg;
+ local $current_lei = $lei;
my $err_type = $lei->{-err_type};
$? and $lei->child_error($?,
$err_type ? "$err_type errors during $lei->{cmd}" : ());
@@ -1383,6 +1389,7 @@ sub fchdir {
sub wq_eof { # EOF callback for main daemon
my ($lei) = @_;
+ local $current_lei = $lei;
my $wq1 = delete $lei->{wq1} // return $lei->fail; # already failed
$wq1->wq_wait_old($wq1->can('_wq_done_wait') // \&wq_done_wait, $lei);
}
@@ -1423,7 +1430,7 @@ sub refresh_watches {
$seen{$url} = undef;
my $state = $cfg->get_1("watch.$url.state");
if (!watch_state_ok($state)) {
- $lei->err("watch.$url.state=$state not supported");
+ warn("watch.$url.state=$state not supported\n");
next;
}
if ($url =~ /\Amaildir:(.+)/i) {
@@ -1492,6 +1499,7 @@ sub lms {
sub sto_done_request {
my ($lei, $sock) = @_;
+ local $current_lei = $lei;
eval {
if ($sock //= $lei->{sock}) { # issue, async wait
$lei->{sto}->wq_io_do('done', [ $sock ]);
@@ -1499,14 +1507,14 @@ sub sto_done_request {
my $wait = $lei->{sto}->wq_do('done');
}
};
- $lei->err($@) if $@;
+ warn($@) if $@;
}
sub cfg_dump ($$) {
my ($lei, $f) = @_;
my $ret = eval { PublicInbox::Config->git_config_dump($f, $lei->{2}) };
return $ret if !$@;
- $lei->err($@);
+ warn($@);
undef;
}
diff --git a/lib/PublicInbox/LeiBlob.pm b/lib/PublicInbox/LeiBlob.pm
index b94f67a029ff..a3ddbbcec211 100644
--- a/lib/PublicInbox/LeiBlob.pm
+++ b/lib/PublicInbox/LeiBlob.pm
@@ -40,7 +40,7 @@ sub solver_user_cb { # called by solver when done
# don't try to support all the git-show(1) options for non-blob,
# this is just a convenience:
$type ne 'blob' and
- $lei->err("# $oid is a $type of $size bytes in:\n#\t$gd");
+ warn "# $oid is a $type of $size bytes in:\n#\t$gd\n";
my $cmd = [ 'git', "--git-dir=$gd", 'show', $oid ];
my $rdr = { 1 => $lei->{1}, 2 => $lei->{2} };
diff --git a/lib/PublicInbox/LeiConvert.pm b/lib/PublicInbox/LeiConvert.pm
index 6550c242b1a2..9e98edc391a0 100644
--- a/lib/PublicInbox/LeiConvert.pm
+++ b/lib/PublicInbox/LeiConvert.pm
@@ -41,7 +41,7 @@ sub process_inputs { # via wq_do
delete $lei->{1};
delete $self->{wcb}; # commit
my $nr = delete($lei->{-nr_write}) // 0;
- $lei->err("# converted $nr messages") if $lei->{opt}->{verbose};
+ $lei->qerr("# converted $nr messages");
}
sub lei_convert { # the main "lei convert" method
diff --git a/lib/PublicInbox/LeiForgetExternal.pm b/lib/PublicInbox/LeiForgetExternal.pm
index 7a4bbcf80a3a..07f0ac803c88 100644
--- a/lib/PublicInbox/LeiForgetExternal.pm
+++ b/lib/PublicInbox/LeiForgetExternal.pm
@@ -20,10 +20,9 @@ sub lei_forget_external {
if ($? == 0) {
$lei->qerr("# $l forgotten ");
} elsif (($? >> 8) == 5) {
- $lei->err("# $l not found");
+ warn("# $l not found\n");
} else {
- $lei->err("# --unset $key error");
- return $lei->x_it($?);
+ $lei->child_error($?, "# --unset $key error");
}
}
}
diff --git a/lib/PublicInbox/LeiImport.pm b/lib/PublicInbox/LeiImport.pm
index 69d63ab6b397..2f8fd6c64f69 100644
--- a/lib/PublicInbox/LeiImport.pm
+++ b/lib/PublicInbox/LeiImport.pm
@@ -38,7 +38,7 @@ sub pmdir_cb { # called via wq_io_do from LeiPmdir->each_mdir_fn
my $lse = $self->{lse} //= $self->{lei}->{sto}->search;
my $lms = $self->{-lms_ro} //= $self->{lei}->lms; # may be 0 or undef
my @oidbin = $lms ? $lms->name_oidbin($folder, $bn) : ();
- @oidbin > 1 and $self->{lei}->err("W: $folder/*/$$bn not unique:\n",
+ @oidbin > 1 and warn("W: $folder/*/$$bn not unique:\n",
map { "\t".unpack('H*', $_)."\n" } @oidbin);
my %seen;
my @docids = sort { $a <=> $b } grep { !$seen{$_}++ }
@@ -100,9 +100,8 @@ sub do_import_index ($$@) {
my $nproc = $self->detect_nproc;
$j = $nproc if $j > $nproc;
}
- if ($lei->{opt}->{'new-only'} && (!$net || !$net->{imap_order})) {
- $lei->err('# --new-only is only for IMAP');
- }
+ ($lei->{opt}->{'new-only'} && (!$net || !$net->{imap_order})) and
+ warn "# --new-only is only for IMAP\n";
my $ops = {};
$lei->{auth}->op_merge($ops, $self) if $lei->{auth};
$lei->{-eml_noisy} = 1;
diff --git a/lib/PublicInbox/LeiImportKw.pm b/lib/PublicInbox/LeiImportKw.pm
index c35c5c266c4b..8359f3386668 100644
--- a/lib/PublicInbox/LeiImportKw.pm
+++ b/lib/PublicInbox/LeiImportKw.pm
@@ -36,7 +36,7 @@ sub ck_update_kw { # via wq_io_do
my ($self, $url, $uid, $kw) = @_;
my @oidbin = $self->{-lms_ro}->num_oidbin($url, $uid);
my $uid_url = "$url/;UID=$uid";
- @oidbin > 1 and $self->{lei}->err("W: $uid_url not unique:\n",
+ @oidbin > 1 and warn("W: $uid_url not unique:\n",
map { "\t".unpack('H*', $_)."\n" } @oidbin);
my %seen;
my @docids = sort { $a <=> $b } grep { !$seen{$_}++ }
diff --git a/lib/PublicInbox/LeiInput.pm b/lib/PublicInbox/LeiInput.pm
index 83479221000b..6a90e7e1e756 100644
--- a/lib/PublicInbox/LeiInput.pm
+++ b/lib/PublicInbox/LeiInput.pm
@@ -343,8 +343,8 @@ $input is `eml', not --in-format=$in_fmt
--mail-sync specified but no inputs support it
# non-fatal if some inputs support support sync
- $lei->err("# --mail-sync will only be used for @{$sync->{ok}}");
- $lei->err("# --mail-sync is not supported for: @{$sync->{no}}");
+ warn("# --mail-sync will only be used for @{$sync->{ok}}\n");
+ warn("# --mail-sync is not supported for: @{$sync->{no}}\n");
}
if ($net) {
$net->{-can_die} = 1;
diff --git a/lib/PublicInbox/LeiMailSync.pm b/lib/PublicInbox/LeiMailSync.pm
index f7e37ad9ca80..f2f1e3ed2658 100644
--- a/lib/PublicInbox/LeiMailSync.pm
+++ b/lib/PublicInbox/LeiMailSync.pm
@@ -466,7 +466,7 @@ sub arg2folder {
# using `$res' instead of `$orig'
EOM
} else {
- $lei->err($res) if defined $res;
+ warn($res, "\n") if defined $res;
push @no, $orig;
}
} elsif (m!\A(?:nntps?|s?news)://!i) {
@@ -478,7 +478,7 @@ EOM
# using `$res' instead of `$orig'
EOM
} else {
- $lei->err($res) if defined $res;
+ warn($res, "\n") if defined $res;
push @no, $orig;
}
} else {
diff --git a/lib/PublicInbox/LeiMirror.pm b/lib/PublicInbox/LeiMirror.pm
index 4be8f70ae401..ec41bec6f16b 100644
--- a/lib/PublicInbox/LeiMirror.pm
+++ b/lib/PublicInbox/LeiMirror.pm
@@ -20,7 +20,7 @@ sub _wq_done_wait { # dwaitpid callback (via wq_eof)
if ($?) {
$lei->child_error($?);
} elsif (!unlink($f)) {
- $lei->err("unlink($f): $!") unless $!{ENOENT};
+ warn("unlink($f): $!\n") unless $!{ENOENT};
} else {
if ($lei->{cmd} ne 'public-inbox-clone') {
$lei->lazy_cb('add-external', '_finish_'
@@ -120,7 +120,7 @@ sub _try_config {
-d $dst or die "mkpath($dst): $!\n";
}
my $err = _get_txt($self, qw(_/text/config/raw inbox.config.example));
- return $self->{lei}->err($err) if $err;
+ return warn($err, "\n") if $err;
my $f = "$self->{dst}/inbox.config.example";
chmod((stat($f))[2] & 0444, $f) or die "chmod(a-w, $f): $!";
my $cfg = PublicInbox::Config->git_config_dump($f, $self->{lei}->{2});
@@ -151,7 +151,7 @@ sub index_cloned_inbox {
my ($self, $iv) = @_;
my $lei = $self->{lei};
my $err = _get_txt($self, qw(description description));
- $lei->err($err) if $err; # non fatal
+ warn($err, "\n") if $err; # non fatal
eval { set_description($self) };
warn $@ if $@;
@@ -380,7 +380,7 @@ sub try_manifest {
my ($path_pfx, $v1_path, @v2_epochs) = deduce_epochs($m, $path);
if (@v2_epochs) {
# It may be possible to have v1 + v2 in parallel someday:
- $lei->err(<<EOM) if defined $v1_path;
+ warn(<<EOM) if defined $v1_path;
# `$v1_path' appears to be a v1 inbox while v2 epochs exist:
# @v2_epochs
# ignoring $v1_path (use --inbox-version=1 to force v1 instead)
diff --git a/lib/PublicInbox/LeiP2q.pm b/lib/PublicInbox/LeiP2q.pm
index c0c4563d6fbb..5c2ce0a13dfc 100644
--- a/lib/PublicInbox/LeiP2q.pm
+++ b/lib/PublicInbox/LeiP2q.pm
@@ -135,7 +135,7 @@ sub do_p2q { # via wq_do
if ($lei->{opt}->{debug}) {
my $json = ref(PublicInbox::Config->json)->new;
$json->utf8->canonical->pretty;
- $lei->err($json->encode($lei->{qterms}));
+ print { $lei->{2} } $json->encode($lei->{qterms});
}
my (@q, %seen);
for my $pfx (@want) {
diff --git a/lib/PublicInbox/LeiRediff.pm b/lib/PublicInbox/LeiRediff.pm
index decb721b7a4c..740cbcee334f 100644
--- a/lib/PublicInbox/LeiRediff.pm
+++ b/lib/PublicInbox/LeiRediff.pm
@@ -30,7 +30,7 @@ sub rediff_user_cb { # called by solver when done
# don't try to support all the git-show(1) options for non-blob,
# this is just a convenience:
- $type ne 'blob' and return $lei->err(<<EOF);
+ $type ne 'blob' and return warn(<<EOF);
# $oid is a $type of $size bytes in:
# $git->{git_dir} (wanted: $oid_want)
EOF
diff --git a/lib/PublicInbox/LeiRemote.pm b/lib/PublicInbox/LeiRemote.pm
index 346aa6a40132..7782aa9dbfa1 100644
--- a/lib/PublicInbox/LeiRemote.pm
+++ b/lib/PublicInbox/LeiRemote.pm
@@ -75,7 +75,7 @@ sub smsg_eml {
if (my $bref = $self->{lei}->ale->git->cat_file($smsg->{blob})) {
return PublicInbox::Eml->new($bref);
}
- $self->{lei}->err("E: $self->{uri} $smsg->{blob} gone <$smsg->{mid}>");
+ warn("E: $self->{uri} $smsg->{blob} gone <$smsg->{mid}>\n");
undef;
}
diff --git a/lib/PublicInbox/LeiViewText.pm b/lib/PublicInbox/LeiViewText.pm
index c469d1ea5b1c..2dad3b780fa1 100644
--- a/lib/PublicInbox/LeiViewText.pm
+++ b/lib/PublicInbox/LeiViewText.pm
@@ -77,7 +77,7 @@ sub new {
my $cfg = PublicInbox::Config::config_fh_parse($r, "\0", "\n");
waitpid($pid, 0);
if ($?) {
- $lei->err("# git-config failed, no color (non-fatal)");
+ warn "# git-config failed, no color (non-fatal)\n";
return $self;
}
$self->{-colored} = \&my_colored;
diff --git a/lib/PublicInbox/LeiXSearch.pm b/lib/PublicInbox/LeiXSearch.pm
index ae9f5881676d..fee1b859f9c6 100644
--- a/lib/PublicInbox/LeiXSearch.pm
+++ b/lib/PublicInbox/LeiXSearch.pm
@@ -16,6 +16,7 @@ use PublicInbox::Spawn qw(popen_rd spawn which);
use PublicInbox::MID qw(mids);
use PublicInbox::Smsg;
use PublicInbox::Eml;
+use PublicInbox::LEI;
use Fcntl qw(SEEK_SET F_SETFL O_APPEND O_RDWR);
use PublicInbox::ContentHash qw(git_sha);
use POSIX qw(strftime);
@@ -392,11 +393,11 @@ sub query_remote_mboxrd {
$err = '';
if (-s $cerr) {
seek($cerr, 0, SEEK_SET) or
- $lei->err("seek($cmd stderr): $!");
+ warn "seek($cmd stderr): $!";
$err = do { local $/; <$cerr> } //
- "read($cmd stderr): $!";
+ warn "read($cmd stderr): $!";
truncate($cerr, 0) or
- $lei->err("truncate($cmd stderr): $!");
+ warn "truncate($cmd stderr): $!";
}
next if (($? >> 8) == 22 && $err =~ /\b404\b/);
$uri->query_form(q => $qstr);
@@ -416,6 +417,7 @@ sub xsearch_done_wait { # dwaitpid callback
sub query_done { # EOF callback for main daemon
my ($lei) = @_;
+ local $PublicInbox::LEI::current_lei = $lei;
my $l2m = delete $lei->{l2m};
$l2m->wq_wait_old(\&xsearch_done_wait, $lei) if $l2m;
if (my $lxs = delete $lei->{lxs}) {
@@ -462,6 +464,7 @@ Error closing $lei->{ovv}->{dst}: $!
sub do_post_augment {
my ($lei) = @_;
+ local $PublicInbox::LEI::current_lei = $lei;
my $l2m = $lei->{l2m} or return; # client disconnected
$lei->fchdir or return;
my $err;
@@ -497,9 +500,10 @@ sub concurrency {
$nl + $nr;
}
-sub start_query ($;$) { # always runs in main (lei-daemon) process
- my ($self, $l2m) = @_;
- if ($self->{opt_threads} || ($l2m && !$self->{opt_sort})) {
+sub start_query ($$) { # always runs in main (lei-daemon) process
+ my ($self, $lei) = @_;
+ local $PublicInbox::LEI::current_lei = $lei;
+ if ($self->{opt_threads} || ($lei->{l2m} && !$self->{opt_sort})) {
for my $ibxish (locals($self)) {
$self->wq_io_do('query_one_mset', [], $ibxish);
}
@@ -521,9 +525,10 @@ sub start_query ($;$) { # always runs in main (lei-daemon) process
}
sub incr_start_query { # called whenever an l2m shard starts do_post_auth
- my ($self, $l2m) = @_;
+ my ($self, $lei) = @_;
+ my $l2m = $lei->{l2m};
return if ++$self->{nr_start_query} != $l2m->{-wq_nr_workers};
- start_query($self, $l2m);
+ start_query($self, $lei);
}
sub ipc_atfork_child {
@@ -545,7 +550,7 @@ sub do_query {
'l2m_progress' => [ \&l2m_progress, $lei ],
'x_it' => [ $lei ],
'child_error' => [ $lei ],
- 'incr_start_query' => [ $self, $l2m ],
+ 'incr_start_query' => [ $self, $lei ],
};
$lei->{auth}->op_merge($ops, $l2m) if $l2m && $lei->{auth};
my $end = $lei->pkt_op_pair;
@@ -586,7 +591,7 @@ sub do_query {
if ($l2m) {
$l2m->net_merge_all_done($lei) unless $lei->{auth};
} else {
- start_query($self);
+ start_query($self, $lei);
}
$lei->event_step_init; # wait for shutdowns
$lei->wait_wq_events($op_c, $ops);
^ permalink raw reply related [relevance 28%]
* [PATCH 5/5] lei up --all: show output for warnings
2021-10-12 22:44 71% ` [PATCH 2/5] lei/store: use remove_doc to save some LoC Eric Wong
@ 2021-10-12 22:45 63% ` Eric Wong
1 sibling, 0 replies; 200+ results
From: Eric Wong @ 2021-10-12 22:45 UTC (permalink / raw)
To: meta
This helps users make sense of which saved searches some
warnings were coming from.
Since I often create and discard externals, some warnings
from saved searches were confusing to me without output context:
"`$FOO' is unknown"
"$FOO not indexed by Xapian"
---
lib/PublicInbox/LEI.pm | 12 ++++++++----
lib/PublicInbox/LeiUp.pm | 7 +++++++
2 files changed, 15 insertions(+), 4 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 51b0e95e1728..183cb545fe55 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -522,7 +522,7 @@ sub sigint_reap {
sub fail ($$;$) {
my ($self, $buf, $exit_code) = @_;
$self->{failed}++;
- err($self, $buf) if defined $buf;
+ warn($buf, "\n") if defined $buf;
$self->{pkt_op_p}->pkt_do('fail_handler') if $self->{pkt_op_p};
x_it($self, ($exit_code // 1) << 8);
undef;
@@ -542,7 +542,7 @@ sub puts ($;@) { out(shift, map { "$_\n" } @_) }
sub child_error { # passes non-fatal curl exit codes to user
my ($self, $child_error, $msg) = @_; # child_error is $?
$child_error ||= 1 << 8;
- $self->err($msg) if $msg;
+ warn($msg, "\n") if defined $msg;
if ($self->{pkt_op_p}) { # to top lei-daemon
$self->{pkt_op_p}->pkt_do('child_error', $child_error);
} elsif ($self->{sock}) { # to lei(1) client
@@ -588,8 +588,12 @@ sub _lei_atfork_child {
eval 'no warnings; undef $PublicInbox::LeiNoteEvent::to_flush';
undef $errors_log;
$quit = \&CORE::exit;
- $self->{-eml_noisy} or # only "lei import" sets this atm
- $SIG{__WARN__} = PublicInbox::Eml::warn_ignore_cb();
+ if (!$self->{-eml_noisy}) { # only "lei import" sets this atm
+ my $cb = $SIG{__WARN__} // \&CORE::warn;
+ $SIG{__WARN__} = sub {
+ $cb->(@_) unless PublicInbox::Eml::warn_ignore(@_)
+ };
+ }
$current_lei = $persist ? undef : $self; # for SIG{__WARN__}
}
diff --git a/lib/PublicInbox/LeiUp.pm b/lib/PublicInbox/LeiUp.pm
index 3e1ca21e29e7..3011300dd836 100644
--- a/lib/PublicInbox/LeiUp.pm
+++ b/lib/PublicInbox/LeiUp.pm
@@ -159,6 +159,13 @@ sub event_step { # runs via PublicInbox::DS::requeue
delete $l->{opt}->{all};
$l->qerr("# updating $self->{out}");
$l->{up_op_p} = $self->{op_p}; # ($l => $lei => script/lei)
+ my $cb = $SIG{__WARN__} // \&CORE::warn;
+ my $o = " (output: $self->{out})";
+ local $SIG{__WARN__} = sub {
+ my @m = @_;
+ push(@m, $o) if !@m || $m[-1] !~ s/\n\z/$o\n/;
+ $cb->(@m);
+ };
eval { $l->dispatch('up', $self->{out}) };
$lei->child_error(0, $@) if $@ || $l->{failed}; # lei->fail()
^ permalink raw reply related [relevance 63%]
* [PATCH 2/5] lei/store: use remove_doc to save some LoC
@ 2021-10-12 22:44 71% ` Eric Wong
2021-10-12 22:45 63% ` [PATCH 5/5] lei up --all: show output for warnings Eric Wong
1 sibling, 0 replies; 200+ results
From: Eric Wong @ 2021-10-12 22:44 UTC (permalink / raw)
To: meta
---
lib/PublicInbox/LeiStore.pm | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/lib/PublicInbox/LeiStore.pm b/lib/PublicInbox/LeiStore.pm
index 613d1d31f581..bf41dcf53094 100644
--- a/lib/PublicInbox/LeiStore.pm
+++ b/lib/PublicInbox/LeiStore.pm
@@ -281,8 +281,7 @@ sub remove_docids ($;@) {
my ($self, @docids) = @_;
my $eidx = eidx_init($self);
for my $docid (@docids) {
- $eidx->idx_shard($docid)->ipc_do('xdb_remove', $docid);
- $eidx->{oidx}->delete_by_num($docid);
+ $eidx->remove_doc($docid);
$eidx->{oidx}->{dbh}->do(<<EOF, undef, $docid);
DELETE FROM xref3 WHERE docid = ?
EOF
^ permalink raw reply related [relevance 71%]
* Re: [PATCH] doc: lei-refresh-mail-sync: drop repeated word
2021-10-11 4:53 71% [PATCH] doc: lei-refresh-mail-sync: drop repeated word Kyle Meyer
@ 2021-10-11 5:09 71% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-10-11 5:09 UTC (permalink / raw)
To: Kyle Meyer; +Cc: meta
Thanks, pushed as c9567f1e142931cf4c5f092ad1ec5904f7c5bdc1
^ permalink raw reply [relevance 71%]
* [PATCH] doc: lei-refresh-mail-sync: drop repeated word
@ 2021-10-11 4:53 71% Kyle Meyer
2021-10-11 5:09 71% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Kyle Meyer @ 2021-10-11 4:53 UTC (permalink / raw)
To: meta
---
Spotted while (slowly :/) studying these commands and working on
adding missing lei manpages (add-watch, ls-watch, rm-watch,
ls-mail-source, forget-mail-sync, inspect, and mail-diff).
Documentation/lei-refresh-mail-sync.pod | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/Documentation/lei-refresh-mail-sync.pod b/Documentation/lei-refresh-mail-sync.pod
index aefd516f..92ca9044 100644
--- a/Documentation/lei-refresh-mail-sync.pod
+++ b/Documentation/lei-refresh-mail-sync.pod
@@ -10,7 +10,7 @@ lei refresh-mail-sync MFOLDER [MFOLDER...]
=head1 DESCRIPTION
-C<lei refresh-mail-sync> is intended to to keep old messages
+C<lei refresh-mail-sync> is intended to keep old messages
indexed with L<lei-index(1)> retrievable if Maildir flags change
a filename. It will prune invalid entries for messages which no
longer exist in a Maildir.
base-commit: dea4495677bbbdbce959551de34ef8f70fe2bc7c
--
2.33.0
^ permalink raw reply related [relevance 71%]
* [PATCH 7/8] lei/store: keep ".err-XXXX" in stderr tmpfile
@ 2021-10-10 14:25 71% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-10-10 14:25 UTC (permalink / raw)
To: meta
This is slighly more meaningful since the file is already
in ~/.local/share/lei/store, so "lei_store" was redundant
(and the "XXXX" are random characters replaced by File::Temp)
---
lib/PublicInbox/LeiStore.pm | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/lib/PublicInbox/LeiStore.pm b/lib/PublicInbox/LeiStore.pm
index 52a1456f..613d1d31 100644
--- a/lib/PublicInbox/LeiStore.pm
+++ b/lib/PublicInbox/LeiStore.pm
@@ -512,7 +512,7 @@ sub xchg_stderr {
return unless -e $dir;
my $old = delete $self->{-tmp_err};
my $pfx = POSIX::strftime('%Y%m%d%H%M%S', gmtime(time));
- my $err = File::Temp->new(TEMPLATE => "$pfx.$$.lei_storeXXXX",
+ my $err = File::Temp->new(TEMPLATE => "$pfx.$$.err-XXXX",
SUFFIX => '.err', DIR => $dir);
open STDERR, '>>', $err->filename or die "dup2: $!";
STDERR->autoflush(1); # shared with shard subprocesses
^ permalink raw reply related [relevance 71%]
* Re: [PATCH 9/9] doc: lei-daemon: new manpage
@ 2021-10-04 3:16 71% ` Kyle Meyer
0 siblings, 0 replies; 200+ results
From: Kyle Meyer @ 2021-10-04 3:16 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
Eric Wong writes:
> Btw, did you have time to work on some of the other manpages?
> No worries if not, I might take a stab at them sometime in the
> next few days. [...]
Sorry, I haven't worked on manpages for the new commands. (Actually I
still need to catch up on what the new commands are). I suppose at this
point you might prefer to handle it yourself for a quicker turnaround,
but I think I could find time this week.
^ permalink raw reply [relevance 71%]
* [PATCH 2/4] lei mail-diff: diagnostic command to diff mail contents
2021-10-02 11:18 71% [PATCH 0/4] lei mail-diff and other small things Eric Wong
@ 2021-10-02 11:18 38% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-10-02 11:18 UTC (permalink / raw)
To: meta
This is useful in finding the cause of deduplication bugs,
and possibly the cause of missing threads reported by
Konstantin in <20211001130527.z7eivotlgqbgetzz@meerkat.local>
usage:
u=https://yhbt.net/lore/all/87czop5j33.fsf@tynnyri.adurom.net/raw
lei mail-diff $u
---
MANIFEST | 1 +
lib/PublicInbox/ContentHash.pm | 6 +-
lib/PublicInbox/LEI.pm | 5 ++
lib/PublicInbox/LeiInput.pm | 6 ++
lib/PublicInbox/LeiMailDiff.pm | 111 +++++++++++++++++++++++++++++++++
lib/PublicInbox/LeiRediff.pm | 63 ++++++++++---------
6 files changed, 160 insertions(+), 32 deletions(-)
create mode 100644 lib/PublicInbox/LeiMailDiff.pm
diff --git a/MANIFEST b/MANIFEST
index 74b28d2d54ab..22b7df9bb89d 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -237,6 +237,7 @@ lib/PublicInbox/LeiLsMailSource.pm
lib/PublicInbox/LeiLsMailSync.pm
lib/PublicInbox/LeiLsSearch.pm
lib/PublicInbox/LeiLsWatch.pm
+lib/PublicInbox/LeiMailDiff.pm
lib/PublicInbox/LeiMailSync.pm
lib/PublicInbox/LeiMirror.pm
lib/PublicInbox/LeiNoteEvent.pm
diff --git a/lib/PublicInbox/ContentHash.pm b/lib/PublicInbox/ContentHash.pm
index cc4a54c922d0..f6ae9011c1bf 100644
--- a/lib/PublicInbox/ContentHash.pm
+++ b/lib/PublicInbox/ContentHash.pm
@@ -52,9 +52,9 @@ sub content_dig_i {
$dig->add($s);
}
-sub content_digest ($) {
- my ($eml) = @_;
- my $dig = Digest::SHA->new(256);
+sub content_digest ($;$) {
+ my ($eml, $dig) = @_;
+ $dig //= Digest::SHA->new(256);
# References: and In-Reply-To: get used interchangeably
# in some "duplicates" in LKML. We treat them the same
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index fd59235846ae..51b0e95e1728 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -203,6 +203,11 @@ our %CMD = ( # sorted in order of importance/use:
qw(git-dir=s@ cwd! verbose|v+ color:s no-color drq:1 dequote-only:1),
@diff_opt, @lxs_opt, @net_opt, @c_opt ],
+'mail-diff' => [ '--stdin|LOCATION...', 'diff the contents of emails',
+ 'stdin|', # /|\z/ must be first for lone dash
+ qw(verbose|v+ color:s no-color raw-header),
+ @diff_opt, @net_opt, @c_opt ],
+
'add-external' => [ 'LOCATION',
'add/set priority of a publicinbox|extindex for extra matches',
qw(boost=i mirror=s inbox-version=i epoch=s verbose|v+),
diff --git a/lib/PublicInbox/LeiInput.pm b/lib/PublicInbox/LeiInput.pm
index 22bedba6e6c3..83479221000b 100644
--- a/lib/PublicInbox/LeiInput.pm
+++ b/lib/PublicInbox/LeiInput.pm
@@ -57,6 +57,12 @@ sub check_input_format ($;$) {
1;
}
+sub input_mbox_cb { # base MboxReader callback
+ my ($eml, $self) = @_;
+ $eml->header_set($_) for (qw(Status X-Status));
+ $self->input_eml_cb($eml);
+}
+
# import a single file handle of $name
# Subclass must define ->input_eml_cb and ->input_mbox_cb
sub input_fh {
diff --git a/lib/PublicInbox/LeiMailDiff.pm b/lib/PublicInbox/LeiMailDiff.pm
new file mode 100644
index 000000000000..a29ae2259f71
--- /dev/null
+++ b/lib/PublicInbox/LeiMailDiff.pm
@@ -0,0 +1,111 @@
+# Copyright (C) all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+
+# The "lei mail-diff" sub-command, diffs input contents against
+# the first message of input
+package PublicInbox::LeiMailDiff;
+use strict;
+use v5.10.1;
+use parent qw(PublicInbox::IPC PublicInbox::LeiInput);
+use File::Temp 0.19 (); # 0.19 for ->newdir
+use PublicInbox::Spawn qw(spawn which);
+use PublicInbox::MsgIter qw(msg_part_text);
+use File::Path qw(remove_tree);
+use PublicInbox::ContentHash qw(content_digest);
+require PublicInbox::LeiRediff;
+use Data::Dumper ();
+
+sub write_part { # Eml->each_part callback
+ my ($ary, $self) = @_;
+ my ($part, $depth, $idx) = @$ary;
+ if ($idx ne '1' || $self->{lei}->{opt}->{'raw-header'}) {
+ open my $fh, '>', "$self->{curdir}/$idx.hdr" or die "open: $!";
+ print $fh ${$part->{hdr}} or die "print $!";
+ close $fh or die "close $!";
+ }
+ my $ct = $part->content_type || 'text/plain';
+ my ($s, $err) = msg_part_text($part, $ct);
+ my $sfx = defined($s) ? 'txt' : 'bin';
+ open my $fh, '>', "$self->{curdir}/$idx.$sfx" or die "open: $!";
+ print $fh ($s // $part->body) or die "print $!";
+ close $fh or die "close $!";
+}
+
+sub dump_eml ($$$) {
+ my ($self, $dir, $eml) = @_;
+ local $self->{curdir} = $dir;
+ mkdir $dir or die "mkdir($dir): $!";
+ $eml->each_part(\&write_part, $self);
+
+ open my $fh, '>', "$dir/content_digest" or die "open: $!";
+ my $dig = PublicInbox::ContentDigestDbg->new($fh);
+ local $Data::Dumper::Useqq = 1;
+ local $Data::Dumper::Terse = 1;
+ content_digest($eml, $dig);
+ print $fh "\n", $dig->hexdigest, "\n" or die "print $!";
+ close $fh or die "close: $!";
+}
+
+sub prep_a ($$) {
+ my ($self, $eml) = @_;
+ $self->{tmp} = File::Temp->newdir('lei-mail-diff-XXXX', TMPDIR => 1);
+ dump_eml($self, "$self->{tmp}/a", $eml);
+}
+
+sub diff_a ($$) {
+ my ($self, $eml) = @_;
+ ++$self->{nr};
+ my $dir = "$self->{tmp}/N$self->{nr}";
+ dump_eml($self, $dir, $eml);
+ my $cmd = [ qw(git diff --no-index) ];
+ my $lei = $self->{lei};
+ PublicInbox::LeiRediff::_lei_diff_prepare($lei, $cmd);
+ push @$cmd, qw(-- a), "N$self->{nr}";
+ my $rdr = { -C => "$self->{tmp}" };
+ @$rdr{1, 2} = @$lei{1, 2};
+ my $pid = spawn($cmd, $lei->{env}, $rdr);
+ waitpid($pid, 0);
+ $lei->child_error($?) if $?; # for git diff --exit-code
+ File::Path::remove_tree($self->{curdir});
+}
+
+sub input_eml_cb { # used by PublicInbox::LeiInput::input_fh
+ my ($self, $eml) = @_;
+ $self->{tmp} ? diff_a($self, $eml) : prep_a($self, $eml);
+}
+
+sub lei_mail_diff {
+ my ($lei, @argv) = @_;
+ $lei->{opt}->{'in-format'} //= 'eml';
+ my $self = bless {}, __PACKAGE__;
+ $self->prepare_inputs($lei, \@argv) or return;
+ my $isatty = -t $lei->{1};
+ $lei->{opt}->{color} //= $isatty;
+ $lei->start_pager if $isatty;
+ my $ops = {};
+ $lei->{auth}->op_merge($ops, $self) if $lei->{auth};
+ (my $op_c, $ops) = $lei->workers_start($self, 1, $ops);
+ $lei->{wq1} = $self;
+ $lei->{-err_type} = 'non-fatal';
+ net_merge_all_done($self) unless $lei->{auth};
+ $lei->wait_wq_events($op_c, $ops);
+}
+
+no warnings 'once';
+*net_merge_all_done = \&PublicInbox::LeiInput::input_only_net_merge_all_done;
+
+package PublicInbox::ContentDigestDbg;
+use strict;
+use v5.10.1;
+use Data::Dumper;
+
+sub new { bless { dig => Digest::SHA->new(256), fh => $_[1] }, __PACKAGE__ }
+
+sub add {
+ $_[0]->{dig}->add($_[1]);
+ print { $_[0]->{fh} } Dumper($_[1]) or die "print $!";
+}
+
+sub hexdigest { $_[0]->{dig}->hexdigest; }
+
+1;
diff --git a/lib/PublicInbox/LeiRediff.pm b/lib/PublicInbox/LeiRediff.pm
index 1e95e55ac1cc..decb721b7a4c 100644
--- a/lib/PublicInbox/LeiRediff.pm
+++ b/lib/PublicInbox/LeiRediff.pm
@@ -56,6 +56,34 @@ sub solve_1 ($$$) {
$self->{blob}->{$oid_want}; # full OID
}
+sub _lei_diff_prepare ($$) {
+ my ($lei, $cmd) = @_;
+ my $opt = $lei->{opt};
+ push @$cmd, '--'.($opt->{color} && !$opt->{'no-color'} ? '' : 'no-').
+ 'color';
+ for my $o (@PublicInbox::LEI::diff_opt) {
+ my $c = '';
+ # remove single char short option
+ $o =~ s/\|([a-z0-9])\b//i and $c = $1;
+ if ($o =~ s/=[is]@\z//) {
+ my $v = $opt->{$o} or next;
+ push @$cmd, map { $c ? "-$c$_" : "--$o=$_" } @$v;
+ } elsif ($o =~ s/=[is]\z//) {
+ my $v = $opt->{$o} // next;
+ push @$cmd, $c ? "-$c$v" : "--$o=$v";
+ } elsif ($o =~ s/:[is]\z//) {
+ my $v = $opt->{$o} // next;
+ push @$cmd, $c ? "-$c$v" :
+ ($v eq '' ? "--$o" : "--$o=$v");
+ } elsif ($o =~ s/!\z//) {
+ my $v = $opt->{$o} // next;
+ push @$cmd, $v ? "--$o" : "--no-$o";
+ } elsif ($opt->{$o}) {
+ push @$cmd, $c ? "-$c" : "--$o";
+ }
+ }
+}
+
sub diff_ctxq ($$) {
my ($self, $ctxq) = @_;
return unless $ctxq;
@@ -103,35 +131,12 @@ EOM
waitpid($pid, 0);
die "fast-import failed: \$?=$?" if $?;
- my @cmd = qw(diff);
- my $opt = $lei->{opt};
- push @cmd, '--'.($opt->{color} && !$opt->{'no-color'} ? '' : 'no-').
- 'color';
- for my $o (@PublicInbox::LEI::diff_opt) {
- my $c = '';
- # remove single char short option
- $o =~ s/\|([a-z0-9])\b//i and $c = $1;
- if ($o =~ s/=[is]@\z//) {
- my $v = $opt->{$o} or next;
- push @cmd, map { $c ? "-$c$_" : "--$o=$_" } @$v;
- } elsif ($o =~ s/=[is]\z//) {
- my $v = $opt->{$o} // next;
- push @cmd, $c ? "-$c$v" : "--$o=$v";
- } elsif ($o =~ s/:[is]\z//) {
- my $v = $opt->{$o} // next;
- push @cmd, $c ? "-$c$v" :
- ($v eq '' ? "--$o" : "--$o=$v");
- } elsif ($o =~ s/!\z//) {
- my $v = $opt->{$o} // next;
- push @cmd, $v ? "--$o" : "--no-$o";
- } elsif ($opt->{$o}) {
- push @cmd, $c ? "-$c" : "--$o";
- }
- }
- $lei->qerr("# git @cmd");
- push @cmd, qw(A B);
- unshift @cmd, 'git', "--git-dir=$rw->{git_dir}";
- $pid = spawn(\@cmd, $lei->{env}, { 2 => $lei->{2}, 1 => $lei->{1} });
+ my $cmd = [ 'diff' ];
+ _lei_diff_prepare($lei, $cmd);
+ $lei->qerr("# git @$cmd");
+ push @$cmd, qw(A B);
+ unshift @$cmd, 'git', "--git-dir=$rw->{git_dir}";
+ $pid = spawn($cmd, $lei->{env}, { 2 => $lei->{2}, 1 => $lei->{1} });
waitpid($pid, 0);
$lei->child_error($?) if $?; # for git diff --exit-code
undef;
^ permalink raw reply related [relevance 38%]
* [PATCH 0/4] lei mail-diff and other small things
@ 2021-10-02 11:18 71% Eric Wong
2021-10-02 11:18 38% ` [PATCH 2/4] lei mail-diff: diagnostic command to diff mail contents Eric Wong
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2021-10-02 11:18 UTC (permalink / raw)
To: meta
1/4 doesn't matter, atm;
2/4 is something I finally got around to doing :x
3/4 is long overdue, I think (and a result of 2/4)
Not sure why 4/4 wasn't done earlier, either, I guess
I never notice missing blobs.
Eric Wong (4):
extsearchidx: attach_config: set {ibx_map} value to $ibx
lei mail-diff: diagnostic command to diff mail contents
content_hash: normalize whitespace before hashing addresses
extsearchidx: emit diagnostics for missing blobs
MANIFEST | 1 +
lib/PublicInbox/ContentHash.pm | 7 +-
lib/PublicInbox/ExtSearchIdx.pm | 8 ++-
lib/PublicInbox/LEI.pm | 5 ++
lib/PublicInbox/LeiInput.pm | 6 ++
lib/PublicInbox/LeiMailDiff.pm | 111 ++++++++++++++++++++++++++++++++
lib/PublicInbox/LeiRediff.pm | 63 +++++++++---------
7 files changed, 167 insertions(+), 34 deletions(-)
create mode 100644 lib/PublicInbox/LeiMailDiff.pm
^ permalink raw reply [relevance 71%]
* [PATCH 1/2] lei inspect: integerize "bytes" and "lines" fields
2021-10-02 8:16 71% [PATCH 0/2] lei inspect stuffs Eric Wong
@ 2021-10-02 8:16 71% ` Eric Wong
2021-10-02 8:16 53% ` [PATCH 2/2] lei inspect: fix "mid:" prefix, expand to Xapian Eric Wong
1 sibling, 0 replies; 200+ results
From: Eric Wong @ 2021-10-02 8:16 UTC (permalink / raw)
To: meta
These are always numeric, but none of the Perl code cares;
but we want to prevent JSON from quoting them.
---
lib/PublicInbox/LeiInspect.pm | 10 ++++++++--
1 file changed, 8 insertions(+), 2 deletions(-)
diff --git a/lib/PublicInbox/LeiInspect.pm b/lib/PublicInbox/LeiInspect.pm
index 2158b996a3a4..f18e31c5c8f4 100644
--- a/lib/PublicInbox/LeiInspect.pm
+++ b/lib/PublicInbox/LeiInspect.pm
@@ -13,6 +13,12 @@ use PublicInbox::Config;
use PublicInbox::MID qw(mids);
use PublicInbox::NetReader qw(imap_uri nntp_uri);
+sub _json_prep ($) {
+ my ($smsg) = @_;
+ $smsg->{$_} += 0 for qw(bytes lines); # integerize
+ +{ %$smsg } # unbless and scalarize
+}
+
sub inspect_blob ($$) {
my ($lei, $oidhex) = @_;
my $ent = {};
@@ -143,7 +149,7 @@ sub inspect_num ($$) {
}
if ($ibx && $ibx->over) {
my $smsg = $ibx->over->get_art($num);
- $ent->{smsg} = { %$smsg } if $smsg;
+ $ent->{smsg} = _json_prep($smsg) if $smsg;
}
defined($docid) ? inspect_docid($lei, $docid, $ent) : $ent;
}
@@ -164,7 +170,7 @@ sub inspect_mid ($$) {
if ($ibx && $ibx->over) {
my ($id, $prev);
while (my $smsg = $ibx->over->next_by_mid($mid, \$id, \$prev)) {
- push @{$ent->{smsg}}, { %$smsg }
+ push @{$ent->{smsg}}, _json_prep($smsg);
}
}
$ent;
^ permalink raw reply related [relevance 71%]
* [PATCH 0/2] lei inspect stuffs
@ 2021-10-02 8:16 71% Eric Wong
2021-10-02 8:16 71% ` [PATCH 1/2] lei inspect: integerize "bytes" and "lines" fields Eric Wong
2021-10-02 8:16 53% ` [PATCH 2/2] lei inspect: fix "mid:" prefix, expand to Xapian Eric Wong
0 siblings, 2 replies; 200+ results
From: Eric Wong @ 2021-10-02 8:16 UTC (permalink / raw)
To: meta
Some obvious fixes while working extindex stuff..
Eric Wong (2):
lei inspect: integerize "bytes" and "lines" fields
lei inspect: fix "mid:" prefix, expand to Xapian
MANIFEST | 1 +
lib/PublicInbox/LeiInspect.pm | 69 +++++++++++++++++++++--------------
t/lei-inspect.t | 14 +++++++
3 files changed, 56 insertions(+), 28 deletions(-)
create mode 100644 t/lei-inspect.t
^ permalink raw reply [relevance 71%]
* [PATCH 2/2] lei inspect: fix "mid:" prefix, expand to Xapian
2021-10-02 8:16 71% [PATCH 0/2] lei inspect stuffs Eric Wong
2021-10-02 8:16 71% ` [PATCH 1/2] lei inspect: integerize "bytes" and "lines" fields Eric Wong
@ 2021-10-02 8:16 53% ` Eric Wong
1 sibling, 0 replies; 200+ results
From: Eric Wong @ 2021-10-02 8:16 UTC (permalink / raw)
To: meta; +Cc: Konstantin Ryabitsev
This fixes inspect for uninitialized instances, and adds Xapian
("xdoc") output if available.
Reported-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
Message-ID: <20211001204943.l4yl6xvc45c5eapz@meerkat.local>
---
MANIFEST | 1 +
lib/PublicInbox/LeiInspect.pm | 59 ++++++++++++++++++++---------------
t/lei-inspect.t | 14 +++++++++
3 files changed, 48 insertions(+), 26 deletions(-)
create mode 100644 t/lei-inspect.t
diff --git a/MANIFEST b/MANIFEST
index 929f5f869c5b..74b28d2d54ab 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -450,6 +450,7 @@ t/lei-import-maildir.t
t/lei-import-nntp.t
t/lei-import.t
t/lei-index.t
+t/lei-inspect.t
t/lei-lcat.t
t/lei-mirror.psgi
t/lei-mirror.t
diff --git a/lib/PublicInbox/LeiInspect.pm b/lib/PublicInbox/LeiInspect.pm
index f18e31c5c8f4..590dfdabca56 100644
--- a/lib/PublicInbox/LeiInspect.pm
+++ b/lib/PublicInbox/LeiInspect.pm
@@ -78,22 +78,9 @@ sub inspect_sync_folder ($$) {
$ent
}
-sub inspect_docid ($$;$) {
- my ($lei, $docid, $ent) = @_;
- require PublicInbox::Search;
- $ent //= {};
- my $xdb;
- if ($xdb = delete $ent->{xdb}) { # from inspect_num
- } elsif (defined(my $dir = $lei->{opt}->{dir})) {
- no warnings 'once';
- $xdb = $PublicInbox::Search::X{Database}->new($dir);
- } else {
- $xdb = $lei->{lse}->xdb;
- }
- $xdb or return $lei->fail('no Xapian DB');
- my $doc = $xdb->get_document($docid); # raises
+sub _inspect_doc ($$) {
+ my ($ent, $doc) = @_;
my $data = $doc->get_data;
- $ent->{docid} = $docid;
$ent->{data_length} = length($data);
$ent->{description} = $doc->get_description;
$ent->{$_} = $doc->$_ for (qw(termlist_count values_count));
@@ -119,6 +106,24 @@ sub inspect_docid ($$;$) {
$ent;
}
+sub inspect_docid ($$;$) {
+ my ($lei, $docid, $ent) = @_;
+ require PublicInbox::Search;
+ $ent //= {};
+ my $xdb;
+ if ($xdb = delete $ent->{xdb}) { # from inspect_num
+ } elsif (defined(my $dir = $lei->{opt}->{dir})) {
+ no warnings 'once';
+ $xdb = $PublicInbox::Search::X{Database}->new($dir);
+ } elsif ($lei->{lse}) {
+ $xdb = $lei->{lse}->xdb;
+ }
+ $xdb or return $lei->fail('no Xapian DB');
+ my $doc = $xdb->get_document($docid); # raises
+ $ent->{docid} = $docid;
+ _inspect_doc($ent, $doc);
+}
+
sub dir2ibx ($$) {
my ($lei, $dir) = @_;
if (-f "$dir/ei.lock") {
@@ -138,11 +143,9 @@ sub inspect_num ($$) {
my $ent = { num => $num };
if (defined(my $dir = $lei->{opt}->{dir})) {
$ibx = dir2ibx($lei, $dir) or return;
- if ($ent->{xdb} = $ibx->xdb) {
- my $num2docid = $lei->{lse}->can('num2docid');
- $docid = $num2docid->($ibx, $num);
- }
- } else {
+ $ent->{xdb} = $ibx->xdb and # for inspect_docid
+ $docid = PublicInbox::LeiSearch::num2docid($ibx, $num);
+ } elsif ($lei->{lse}) {
$ibx = $lei->{lse};
$lei->{lse}->xdb; # set {nshard} for num2docid
$docid = $lei->{lse}->num2docid($num);
@@ -156,16 +159,12 @@ sub inspect_num ($$) {
sub inspect_mid ($$) {
my ($lei, $mid) = @_;
- my ($ibx, $over);
+ my $ibx;
my $ent = { mid => $mid };
if (defined(my $dir = $lei->{opt}->{dir})) {
- my $num2docid = $lei->{lse}->can('num mid => [ $mid ] 2docid');
- $ibx = dir2ibx($lei, $dir) or return;
- # $ent->{xdb} = $ibx->xdb //
- # return $lei->fail("no Xapian DB for $dir");
+ $ibx = dir2ibx($lei, $dir)
} else {
$ibx = $lei->{lse};
- $lei->{lse}->xdb; # set {nshard} for num2docid
}
if ($ibx && $ibx->over) {
my ($id, $prev);
@@ -173,6 +172,14 @@ sub inspect_mid ($$) {
push @{$ent->{smsg}}, _json_prep($smsg);
}
}
+ if ($ibx && $ibx->search) {
+ my $mset = $ibx->search->mset(qq{mid:"$mid"});
+ for (sort { $a->get_docid <=> $b->get_docid } $mset->items) {
+ my $tmp = { docid => $_->get_docid };
+ _inspect_doc($tmp, $_->get_document);
+ push @{$ent->{xdoc}}, $tmp;
+ }
+ }
$ent;
}
diff --git a/t/lei-inspect.t b/t/lei-inspect.t
new file mode 100644
index 000000000000..077d0d13e84d
--- /dev/null
+++ b/t/lei-inspect.t
@@ -0,0 +1,14 @@
+#!perl -w
+# Copyright all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+use strict; use v5.10.1; use PublicInbox::TestCommon;
+
+test_lei(sub {
+ my ($ro_home, $cfg_path) = setup_public_inboxes;
+ lei_ok qw(inspect --dir), "$ro_home/t1", 'mid:testmessage@example.com';
+ my $ent = json_utf8->decode($lei_out);
+ is(ref($ent->{smsg}), 'ARRAY', 'smsg array');
+ is(ref($ent->{xdoc}), 'ARRAY', 'xdoc array');
+});
+
+done_testing;
^ permalink raw reply related [relevance 53%]
Results 201-400 of ~1311 next (older) | prev (newer) | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2021-08-14 0:29 [PATCH 0/3] lei: hopefully kill /Document \d+ not found/ errors Eric Wong
2021-08-24 20:14 ` [PATCH 0/3] lei: hopefully^W " Eric Wong
2021-10-14 5:31 71% ` Eric Wong
2021-10-15 9:52 54% ` [PATCH] lei q: avoid kw lookup failure on remote mboxrd Eric Wong
2021-10-01 9:54 [PATCH 0/9] daemon-related things Eric Wong
2021-10-01 9:54 ` [PATCH 9/9] doc: lei-daemon: new manpage Eric Wong
2021-10-02 0:19 ` Kyle Meyer
2021-10-02 8:08 ` Eric Wong
2021-10-04 3:16 71% ` Kyle Meyer
2021-10-02 8:16 71% [PATCH 0/2] lei inspect stuffs Eric Wong
2021-10-02 8:16 71% ` [PATCH 1/2] lei inspect: integerize "bytes" and "lines" fields Eric Wong
2021-10-02 8:16 53% ` [PATCH 2/2] lei inspect: fix "mid:" prefix, expand to Xapian Eric Wong
2021-10-02 11:18 71% [PATCH 0/4] lei mail-diff and other small things Eric Wong
2021-10-02 11:18 38% ` [PATCH 2/4] lei mail-diff: diagnostic command to diff mail contents Eric Wong
2021-10-10 14:25 [PATCH 0/8] extindex and then some Eric Wong
2021-10-10 14:25 71% ` [PATCH 7/8] lei/store: keep ".err-XXXX" in stderr tmpfile Eric Wong
2021-10-11 4:53 71% [PATCH] doc: lei-refresh-mail-sync: drop repeated word Kyle Meyer
2021-10-11 5:09 71% ` Eric Wong
2021-10-12 22:44 [PATCH 0/5] fix extindex reindex harder Eric Wong
2021-10-12 22:44 71% ` [PATCH 2/5] lei/store: use remove_doc to save some LoC Eric Wong
2021-10-12 22:45 63% ` [PATCH 5/5] lei up --all: show output for warnings Eric Wong
2021-10-13 7:00 [PATCH 0/5] warnings and such Eric Wong
2021-10-13 7:00 28% ` [PATCH 2/5] lei: use standard warn() in more places Eric Wong
2021-10-13 10:16 [PATCH 0/7] workaround Encode leak, several test fixes Eric Wong
2021-10-13 10:16 71% ` [PATCH 5/7] t/lei-mirror: avoid reading ~/.public-inbox/config in test Eric Wong
2021-10-14 3:12 57% [PATCH] lei inspect: account for non-extindex inboxes Eric Wong
2021-10-14 4:32 [PATCH 0/3] clone+fetch stuff Eric Wong
2021-10-14 4:32 65% ` [PATCH 2/3] lei add-external --mirror: respect client umask Eric Wong
2021-10-14 4:32 61% ` [PATCH 3/3] lei: give workers their own process group Eric Wong
2021-10-14 13:16 70% ` [PATCH 0/7] lei: more process handling fixes Eric Wong
2021-10-14 13:16 71% ` [PATCH 1/7] lei: use send() perlop for signals Eric Wong
2021-10-14 13:16 34% ` [PATCH 5/7] lei: TSTP affects all curl and related subprocesses Eric Wong
2021-10-14 13:16 67% ` [PATCH 6/7] lei up: actually rely on DESTROY for --alllll Eric Wong
2021-10-14 13:16 62% ` [PATCH 7/7] lei up --all: send signals to workers, receive errors Eric Wong
2021-10-14 9:54 60% [PATCH] lei: -d (--dir) and -O (only) shortcuts Eric Wong
2021-10-15 13:30 71% [PATCH 0/3] lei bugfixes and simplifications Eric Wong
2021-10-15 13:30 71% ` [PATCH 1/3] lei forget-search: fix for symlink-ed paths Eric Wong
2021-10-15 13:30 33% ` [PATCH 2/3] lei + ipc: simplify process reaping Eric Wong
2021-10-15 13:45 71% ` [SQUASH PATCH 4/3] lei q: ensure all workers die on Ctrl-C Eric Wong
2021-10-15 13:30 67% ` [PATCH 3/3] lei note-event: fix explicit flush reliability Eric Wong
2021-10-15 14:02 64% [PATCH] lei forget-search: support multiple args Eric Wong
2021-10-15 15:52 61% [PATCH] lei q: guard query_done against die() Eric Wong
2021-10-16 1:00 [PATCH 00/16] some yak-shaving and annoyance fixes Eric Wong
2021-10-16 1:00 71% ` [PATCH 05/12] lei: golf PATH2CFG cleanup Eric Wong
2021-10-16 1:00 63% ` [PATCH 06/12] lei: always keep cwd fd {3} for ->fchdir Eric Wong
2021-10-16 1:00 55% ` [PATCH 07/12] lei: more eval guards for die on failure Eric Wong
2021-10-16 1:01 59% ` [PATCH 09/12] lei_overview: die rather than lei->fail Eric Wong
2021-10-16 5:39 70% [PATCH 0/2] doc: lei manpages for remaining commands Kyle Meyer
2021-10-16 5:39 65% ` [PATCH 1/2] doc: lei: restore alphabetical order to some listings Kyle Meyer
2021-10-16 5:39 29% ` [PATCH 2/2] doc: lei: add manpages for remaining commands Kyle Meyer
2021-10-16 7:07 71% ` Eric Wong
2021-10-16 15:13 67% ` Kyle Meyer
2021-10-16 16:58 71% ` Kyle Meyer
2021-10-16 17:03 71% ` Eric Wong
2021-10-16 17:21 71% ` Kyle Meyer
2021-10-16 7:54 70% [PATCH] t/lei*: set EDITOR for dumb terminals Eric Wong
2021-10-16 9:29 71% [PATCH 0/4] lei: prioritize signals Eric Wong
2021-10-16 9:29 57% ` [PATCH 4/4] lei sockets: favor level-triggered epoll for fairness Eric Wong
2021-10-19 9:33 70% [PATCH 00/11] refining lei up+inspect Eric Wong
2021-10-19 9:33 65% ` [PATCH 02/11] lei up: prefix `remote' and `local' with `o_' Eric Wong
2021-10-19 9:33 50% ` [PATCH 03/11] lei: use die for external and query handling Eric Wong
2021-10-19 9:33 71% ` [PATCH 04/11] lei up: propagate redispatch_all failure via exit code Eric Wong
2021-10-19 9:33 70% ` [PATCH 05/11] lei: conditionally add "\n" to error messages Eric Wong
2021-10-19 9:33 47% ` [PATCH 06/11] lei up: support --exclude=, --no-(external|remote|local) Eric Wong
2021-10-19 9:33 71% ` [PATCH 07/11] lei: remove unused ->busy time arg Eric Wong
2021-10-19 9:33 80% ` [PATCH 08/11] doc: lei: describe lei-daemon-kill and upgrades Eric Wong
2021-10-19 9:33 71% ` [PATCH 09/11] lei inspect: add atfork hook Eric Wong
2021-10-19 9:33 64% ` [PATCH 10/11] lei inspect: show ISO8601 {rt} and {dt}, too Eric Wong
2021-10-21 21:10 [PATCH 00/15] use RENAME_NOREPLACE on Linux 3.15+ Eric Wong
2021-10-21 21:10 57% ` [PATCH 01/15] t/lei-{auto-watch,export-kw}: extra diagnostics on failure Eric Wong
2021-10-21 21:10 71% ` [PATCH 02/15] t/lei-import-maildir: rename fix (SR -> RS) Eric Wong
2021-10-21 21:10 71% ` [PATCH 03/15] t/lei-p2q: extra diagnostics Eric Wong
2021-10-21 21:10 71% ` [PATCH 04/15] lei/store: check for any unexpected process death Eric Wong
2021-10-21 21:10 71% ` [PATCH 05/15] lei note-event: drop unnecessary eval guard Eric Wong
2021-10-21 21:10 71% ` [PATCH 06/15] lei note-event: wq_io_do => wq_do Eric Wong
2021-10-21 21:10 69% ` [PATCH 07/15] lei_search: try harder to associate "lei index"-ed messages Eric Wong
2021-10-21 21:10 71% ` [PATCH 10/15] doc: lei-overview: add CAVEATS section Eric Wong
2021-10-21 21:10 71% ` [PATCH 11/15] lei note-event: clear_src on ENOENT Eric Wong
2021-10-21 21:10 71% ` [PATCH 13/15] lei: no Perl FileHandle for `undef' w/ ECONNRESET Eric Wong
2021-10-21 21:10 37% ` [PATCH 15/15] lei: use RENAME_NOREPLACE on Linux 3.15+ Eric Wong
2021-10-22 8:22 71% [PATCH 0/3] lei: some cleanup stuff Eric Wong
2021-10-22 8:22 60% ` [PATCH 1/3] lei export-kw: don't recreate deleted IMAP folders Eric Wong
2021-10-22 8:22 71% ` [PATCH 2/3] lei export-kw: completion returns all Maildir+IMAP Eric Wong
2021-10-22 8:22 47% ` [PATCH 3/3] lei forget-search: support --prune=<local|remote> Eric Wong
2021-10-23 0:22 71% [PATCH] doc: lei-forget-search: fix option name in --prune description Kyle Meyer
2021-10-23 2:54 71% ` Eric Wong
2021-10-24 0:20 [PATCH 0/7] misc tweaks and fixes Eric Wong
2021-10-24 0:20 75% ` [PATCH 1/7] lei: always pass $lei to LeiAuth->op_merge Eric Wong
2021-10-24 0:20 61% ` [PATCH 2/7] lei export-kw: skip read-only IMAP folders Eric Wong
2021-10-25 19:31 64% [PATCH] lei p2q: document --uri, add examples Eric Wong
2021-10-25 23:39 71% ` Kyle Meyer
2021-10-25 22:24 [PATCH] t/index-git-times: support non-master default branch Thomas Weißschuh
2021-10-25 22:58 ` Eric Wong
2021-10-26 5:05 ` Thomas Weißschuh
2021-10-26 5:28 ` Eric Wong
2021-10-26 18:43 ` Thomas Weißschuh
2021-10-27 21:15 71% ` lei-q-remote-import failures [was: [PATCH] t/index-git-times: support non-master default branch] Eric Wong
2021-10-27 21:24 39% ` Thomas Weißschuh
2021-10-27 23:48 63% ` Eric Wong
2021-10-28 14:03 71% ` Thomas Weißschuh
2021-10-28 19:16 68% ` [PATCH] test_common: clear XDG_CACHE_HOME before lei tests Eric Wong
2021-10-28 19:22 71% ` Thomas Weißschuh
2021-10-26 10:35 69% [PATCH 0/9] lei p2q: more capable than originally thought Eric Wong
2021-10-26 10:35 67% ` [PATCH 2/9] doc: lei-store-format: mail sync section, update IPC Eric Wong
2021-10-26 10:35 71% ` [PATCH 4/9] lei q: enable expensive Xapian flags Eric Wong
2021-10-26 10:35 71% ` [PATCH 5/9] lei inspect: fix atfork hook Eric Wong
2021-10-26 10:35 66% ` [PATCH 6/9] lei: add net getopt spec to various commands Eric Wong
2021-10-26 10:35 35% ` [PATCH 7/9] lei p2q: use LeiInput for multi-patch series Eric Wong
2021-10-26 10:35 68% ` [PATCH 8/9] lei rm|tag: drop redundant mbox+net callbacks Eric Wong
2021-10-26 10:47 71% [PATCH] t/lei-watch: add diagnostics for failure Eric Wong
2021-10-26 21:18 64% [PATCH] lei mail-diff: support more inputs, split newlines Eric Wong
2021-10-27 21:09 64% [PATCH] lei q: fix remote import accounting Eric Wong
2021-10-27 21:28 71% ` Thomas Weißschuh
2021-10-28 6:17 71% [PATCH] lei add-watch: ensure folders are known to mail_sync.sqlite3 Eric Wong
2021-10-28 11:14 71% [PATCH 0/8] lei: docs and cleanups Eric Wong
2021-10-28 11:14 68% ` [PATCH 1/8] xt/net_writer_imap: test "lei convert" w/ IMAP source Eric Wong
2021-10-28 11:14 71% ` [PATCH 2/8] lei convert: use "--output" in failure message Eric Wong
2021-10-28 11:14 62% ` [PATCH 3/8] doc: lei-convert: various updates and cleanups Eric Wong
2021-10-28 11:14 70% ` [PATCH 4/8] doc: lei blob: wording fixups, describe --remote Eric Wong
2021-10-28 11:14 71% ` [PATCH 5/8] lei convert: remove redundant input_net_cb Eric Wong
2021-10-28 11:14 70% ` [PATCH 6/8] doc: lei-add-watch: add warning about unreliability Eric Wong
2021-10-28 11:15 71% ` [PATCH 7/8] lei sucks: show nproc in CPU info Eric Wong
2021-10-28 11:15 63% ` [PATCH 8/8] lei rm: move generic input_maildir_cb to LeiInput parent class Eric Wong
2021-10-30 8:11 71% [PATCH 0/5] lei: fix various SIGPIPE problems Eric Wong
2021-10-30 8:11 71% ` [PATCH 1/5] lei: do not access {sock} after SIGPIPE Eric Wong
2021-10-30 8:11 71% ` [PATCH 5/5] doc: lei-security: add a note about core dumps Eric Wong
2021-10-31 9:10 71% [PATCH 0/2] lei: mail-sync docs + compat fix Eric Wong
2021-10-31 9:10 29% ` [PATCH 1/2] doc: add lei-mail-sync-overview manpage Eric Wong
2021-10-31 16:32 71% ` Kyle Meyer
2021-10-31 17:30 64% ` Eric Wong
2021-11-01 19:00 71% [PATCH] doc: lei-config: fix missing =back Eric Wong
2021-11-02 9:24 60% [PATCH] t/lei-refresh-mail-sync: speed up test on FreeBSD 12 Eric Wong
2021-11-02 18:14 71% [PATCH 0/3] lei: more coherent input handling Eric Wong
2021-11-02 18:14 71% ` [PATCH 1/3] lei mail-diff: do not default to 'eml' Eric Wong
2021-11-02 18:14 41% ` [PATCH 2/3] lei: simplify common LeiInput users with ->wq1_start Eric Wong
2021-11-02 18:47 71% ` Eric Wong
2021-11-02 18:14 65% ` [PATCH 3/3] lei <rediff|rm|tag>: stdin implies `-F eml' Eric Wong
2021-11-02 23:55 43% [PATCH] doc: lei-q: document SEARCH TERMS prefixes Eric Wong
2021-11-03 0:28 71% ` Kyle Meyer
2021-11-03 1:28 71% Initial Fedora packaging for lei Konstantin Ryabitsev
2021-11-03 8:34 [PATCH 0/3] doc: extindex-related updates Eric Wong
2021-11-03 8:34 68% ` [PATCH 2/3] doc: -clone|lei add-external: add bit about the Makefile Eric Wong
[not found] <lorelei.part1.202111051304.mdtebsxahljcrxak@meerkat.local>
[not found] ` <CAL_JsqJBh1O3H2-P07AHzVq0x89BoP_N6P=rT5up6=3QyF_B0Q@mail.gmail.com>
2021-11-08 20:22 86% ` lei: incorrect quoting on saved searches (was Re: lore+lei: getting started) Konstantin Ryabitsev
2021-11-08 20:49 90% ` Eric Wong
2021-11-08 21:36 90% ` Konstantin Ryabitsev
2021-11-08 21:48 90% ` Eric Wong
2021-11-08 22:36 88% ` Konstantin Ryabitsev
2021-11-08 22:57 90% ` Eric Wong
2021-11-08 20:53 90% ` Rob Herring
2021-11-08 23:39 53% [PATCH] lei q|up: fix saved searches for single-phrase search Eric Wong
2021-11-10 10:25 71% [PATCH 0/4] "lei q" rawstr fallout + fixes Eric Wong
2021-11-10 10:28 70% ` [PATCH 3/4] lei q: disallow "\n" in argv[] elements Eric Wong
2021-11-10 10:28 68% ` [PATCH 2/4] lei up: infer rawstr from old searches via trailing "\n" Eric Wong
2021-11-10 10:28 71% ` [PATCH 4/4] lei q: make HTTP(S) query strings even less ugly Eric Wong
2021-11-10 10:33 71% [PATCH] t/lei-watch: test with with higher sleep Eric Wong
2021-11-12 11:08 71% [PATCH] lei forget-search: add help for --prune Eric Wong
2021-11-14 20:41 71% lei spawns mua before results are written Leah Neukirchen
2021-11-15 0:31 71% ` Eric Wong
2021-11-19 12:42 'make test' failures while packaging for openSUSE Jörg Rödel
2021-11-19 18:54 ` Eric Wong
2021-11-19 21:46 ` Jörg Rödel
2021-11-22 6:55 ` [PATCH] searchidx: add some extra diagnostics for odd message Eric Wong
2021-11-22 7:42 96% ` [PATCH] t/lei-mirror: skip lei comparisons if lei missing Eric Wong
2021-11-22 13:47 99% ` Jörg Rödel
2021-11-22 17:24 99% ` Eric Wong
2021-11-19 20:49 71% RFC: should lei inject its own "Received:" header? Konstantin Ryabitsev
2021-11-21 10:13 71% ` Eric Wong
2021-11-20 15:30 How to delete caches Johannes Altmanninger
2021-11-20 16:36 71% ` lei q error after deleting cache (can't find mail_sync.sqlite3) Johannes Altmanninger
2021-11-22 18:38 71% [PATCH] lei: always use 3-arg open perlop Eric Wong
2021-12-08 1:07 Test failures with 1.7.0 Julien Moutinho
2021-12-08 4:08 ` Eric Wong
2021-12-08 10:56 ` Dominique Martinet
2021-12-09 1:37 ` Julien Moutinho
2022-02-17 21:02 63% ` [PATCH] t/lei-sigpipe: attempt to improve diagnostics for stuck test Eric Wong
2022-02-20 1:38 51% ` Julien Moutinho
2022-02-22 6:44 71% ` Eric Wong
2022-02-27 4:15 71% ` Julien Moutinho
2022-02-27 6:41 71% ` Julien Moutinho
2022-02-27 7:23 71% ` Dominique Martinet
2022-02-27 8:04 71% ` Julien Moutinho
2022-02-27 11:17 62% ` [PATCH] t/lei-sigpipe: ensure SIGPIPE is unblocked for this test Eric Wong
2022-03-11 10:42 69% ` [PATCH] t/lei-sigpipe.t: ensure SIGPIPE is not ignored instead of not blocked Julien Moutinho
2022-03-14 22:14 71% ` Eric Wong
2022-03-15 2:56 71% ` Julien Moutinho
2021-12-30 23:04 60% lei q: importing messages when specifying '-f *json*'? Kyle Meyer
[not found] <CABPp-BF_xsOpQ6GSaWs9u9JcnPQT_OXP-gCsAuxPtMj-X1tgOg@mail.gmail.com>
[not found] ` <211203.86sfv9qwdm.gmgdl@evledraar.gmail.com>
[not found] ` <20211203202427.o575sgrx4auqkmjp@meerkat.local>
2021-12-06 16:12 ` Large delays in mailing list delivery? Ævar Arnfjörð Bjarmason
2021-12-06 16:36 ` Eric Wong
2022-02-02 9:34 48% ` Using public-inbox+lei+Emacs+mu+mu4e (was: Large delays in mailing list delivery?) Ævar Arnfjörð Bjarmason
2022-02-07 21:27 69% ` Eric Wong
2022-03-12 21:04 87% Failed 'lei q' blocks 'lei init' from working Nícolas F. R. A. Prado
2022-03-13 0:06 90% ` Kyle Meyer
2022-04-05 8:18 37% [PATCH] lei: always open mail_sync.sqlite3 R/W Eric Wong
2022-04-18 9:50 71% [PATCH 0/4] lei: finish wiring up pure-Perl stuff for Linux Eric Wong
2022-04-18 9:50 71% ` [PATCH 1/4] lei: clobber recvmsg buffer on errors Eric Wong
2022-04-18 9:50 54% ` [PATCH 3/4] lei: wire up pure Perl sendmsg/recvmsg for Linux users Eric Wong
2022-04-21 11:59 58% [PATCH] lei: commit store on interrupted partial imports Eric Wong
2022-04-23 22:03 [PATCH 0/2] version bumps, Perl 5.12 in _some_ places Eric Wong
2022-04-23 22:03 61% ` [PATCH 2/2] lei: move to v5.12 to avoid "use strict" Eric Wong
2022-04-30 21:04 71% [PATCH] lei: improve diagnosis of errors from children Eric Wong
2022-04-30 21:29 71% [PATCH] lei refresh-mail-sync: filter NNTP(S) from --all Eric Wong
2022-05-02 18:10 71% [PATCH] lei import: add label completions (+L:$LABEL) Eric Wong
2022-05-03 11:15 70% Trouble running lei Filipe Manana
2022-05-03 11:37 71% ` Eric Wong
2022-05-03 12:50 71% ` Filipe Manana
2022-05-03 15:24 71% ` Konstantin Ryabitsev
2022-05-03 20:32 71% ` Eric Wong
2022-06-20 19:27 [PATCH 0/3] search indexing improvements Eric Wong
2022-06-20 19:27 ` [PATCH 2/3] search: support "patchid:" prefix (git patch-id --stable) Eric Wong
2022-06-20 20:01 ` Kyle Meyer
2022-06-21 10:37 ` [PATCH 4/3] search: add help for patchid: prefix Eric Wong
2022-06-22 7:50 58% ` [PATCH 5/3] doc: lei-q: regenerate for patchid: help Eric Wong
2022-06-29 16:15 71% lei missing mails Rob Herring
2022-06-29 16:30 71% ` Eric Wong
2022-06-29 16:53 69% ` Rob Herring
2022-06-29 17:27 71% ` Eric Wong
2022-06-29 22:01 59% ` Rob Herring
2022-06-30 8:55 71% ` Eric Wong
2022-07-07 9:48 71% ` Eric Wong
2022-07-11 21:17 71% ` Rob Herring
2022-07-11 21:59 68% ` Rob Herring
2022-07-18 23:41 71% ` Eric Wong
2022-07-07 9:40 71% [PATCH 0/2] lei: minor diagnostic improvement Eric Wong
2022-07-07 9:40 71% ` [PATCH 1/2] lei_xsearch: simplify lei/store import check Eric Wong
2022-07-07 9:40 53% ` [PATCH 2/2] lei: track seen messages to note duplicates Eric Wong
2022-07-19 22:42 71% [PATCH 0/2] lei inotify/EVFILT_VNODE deadlock fix Eric Wong
2022-07-19 22:42 51% ` [PATCH 1/2] lei: avoid deadlock on inotify/EVFILT_VNODE wakeups Eric Wong
2022-07-19 22:42 71% ` [PATCH 2/2] lei note-event: inline note_event_arm_done Eric Wong
2022-08-16 3:44 71% [PATCH] lei: do not wait for sto->done on disconnected EOF Eric Wong
2022-08-17 9:33 71% [PATCH 0/4] lei reindex, minor tweaks Eric Wong
2022-08-17 9:33 71% ` [PATCH 2/4] lei inspect: less scary exception for invalid "docid:" inspect Eric Wong
2022-08-17 9:33 71% ` [PATCH 3/4] lei/store: reduce work when accessing mail_sync.sqlite3 Eric Wong
2022-08-17 9:33 69% ` [PATCH 4/4] lei reindex: new command to reindex lei/store Eric Wong
2022-08-18 7:22 90% ` Eric Wong
2022-08-19 9:07 71% [PATCH 0/4] lei reindex-related stuff Eric Wong
2022-08-19 9:07 62% ` [PATCH 2/4] tests: add some basic "lei reindex" tests Eric Wong
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).