* Re: [PATCH] lei q: fix remote import accounting
@ 2021-10-27 21:28 71% ` Thomas Weißschuh
0 siblings, 0 replies; 200+ results
From: Thomas Weißschuh @ 2021-10-27 21:28 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
On 2021-10-27 21:09+0000, Eric Wong wrote:
> We need to update the {-nr_remote_eml} counter regardless
> of progress display being enabled since it's needed for
> saved searches. We'll also split out the {-imported} flag
> separately and only call LeiStore->done if a new message
> was imported.
>
> Note: this change is NOT expected to fix errors reported by
> Thomas in <ebf92218-1470-4602-b534-6dae59639dc6@t-8ch.de>
For the record: This patch did in fact *not* fix the issue.
^ permalink raw reply [relevance 71%]
* Re: lei-q-remote-import failures [was: [PATCH] t/index-git-times: support non-master default branch]
@ 2021-10-27 23:48 63% ` Eric Wong
2021-10-28 14:03 71% ` Thomas Weißschuh
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2021-10-27 23:48 UTC (permalink / raw)
To: Thomas Weißschuh; +Cc: meta
Thomas Weißschuh <thomas@t-8ch.de> wrote:
> It didn't help, neither did larger sleeps.
Thanks for testing.
> I'll see if I can debug it in the coming days.
OK, maybe the patch below can get you started...
I've also been wondering if something like GIT_TRACE is
necessary, but it's also more work to support + document.
And Perl code is easily modified compared to AOT languages.
diff --git a/lib/PublicInbox/LeiXSearch.pm b/lib/PublicInbox/LeiXSearch.pm
index 2a037f2bd79b..a0982138966e 100644
--- a/lib/PublicInbox/LeiXSearch.pm
+++ b/lib/PublicInbox/LeiXSearch.pm
@@ -281,13 +281,19 @@ sub each_remote_eml { # callback for MboxReader->mboxrd
my ($eml, $self, $lei, $each_smsg) = @_;
my $xoids = $lei->{ale}->xoids_for($eml, 1);
my $smsg = bless {}, 'PublicInbox::Smsg';
+ use Data::Dumper;
if ($self->{import_sto} && !$xoids) {
my ($res, $kw) = $self->{import_sto}->wq_do('add_eml', $eml);
if (ref($res) eq ref($smsg)) { # totally new message
$smsg = $res;
+ $lei->qerr("# imported <$smsg->{mid}>") if $ENV{DBG};
$self->{-imported} = 1;
+ } elsif ($ENV{DBG}) {
+ $lei->qerr("# res, kw = ".Dumper([$res, $kw]));
}
$smsg->{kw} = $kw; # short-circuit xsmsg_vmd
+ } elsif ($ENV{DBG}) {
+ $lei->qerr("# no import $self->{import_sto}".Dumper($xoids));
}
$smsg->{blob} //= $xoids ? (keys(%$xoids))[0]
: $lei->git_oid($eml)->hexdigest;
@@ -378,6 +384,9 @@ sub query_remote_mboxrd {
$lei, $each_smsg);
if ($self->{import_sto} && delete($self->{-imported})) {
my $wait = $self->{import_sto}->wq_do('done');
+ $lei->qerr("# flushed imported messages") if $ENV{DBG};
+ } elsif ($ENV{DBG}) {
+ $lei->qerr("# nothing flushed $self->{import_sto}");
}
$reap_curl->join;
if ($? == 0) {
diff --git a/t/lei-q-remote-import.t b/t/lei-q-remote-import.t
index 92d8c9b6058c..e10af2eb7849 100644
--- a/t/lei-q-remote-import.t
+++ b/t/lei-q-remote-import.t
@@ -32,7 +32,11 @@ test_lei({ tmpdir => $tmpdir }, sub {
lei_ok(@cmd);
ok(-f $o && !-s _, 'output exists but is empty');
unlink $o or BAIL_OUT $!;
- lei_ok(@cmd, '-I', $url);
+ lei_ok([@cmd, '-I', $url], { DBG => 1 });
+
+# should say 'imported <qp@example.com>' + 'flushed imported messages'
+ diag $lei_err;
+return; # drop this if the above diag looks right
is_deeply($slurp_emls->($o), $exp1, 'got results after remote search');
unlink $o or BAIL_OUT $!;
lei_ok(@cmd);
^ permalink raw reply related [relevance 63%]
* [PATCH] lei add-watch: ensure folders are known to mail_sync.sqlite3
@ 2021-10-28 6:17 71% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-10-28 6:17 UTC (permalink / raw)
To: meta
This prevents noisy errors in syslog when running t/lei-watch.t
---
lib/PublicInbox/LeiAddWatch.pm | 2 ++
1 file changed, 2 insertions(+)
diff --git a/lib/PublicInbox/LeiAddWatch.pm b/lib/PublicInbox/LeiAddWatch.pm
index 671d54f92467..97e7a3422897 100644
--- a/lib/PublicInbox/LeiAddWatch.pm
+++ b/lib/PublicInbox/LeiAddWatch.pm
@@ -34,6 +34,8 @@ sub lei_add_watch {
next if defined $cfg->{"watch.$w.state"};
$lei->_config("watch.$w.state", $state);
}
+ $lei->_lei_store(1); # create
+ $lei->lms(1)->lms_write_prepare->add_folders(@{$self->{inputs}});
delete $lei->{cfg}; # force reload
$lei->refresh_watches;
}
^ permalink raw reply related [relevance 71%]
* [PATCH 0/8] lei: docs and cleanups
@ 2021-10-28 11:14 71% Eric Wong
2021-10-28 11:14 68% ` [PATCH 1/8] xt/net_writer_imap: test "lei convert" w/ IMAP source Eric Wong
` (7 more replies)
0 siblings, 8 replies; 200+ results
From: Eric Wong @ 2021-10-28 11:14 UTC (permalink / raw)
To: meta
Expect more on the way. As always some bugs and fixes were
found while attempting to write docs :x
Eric Wong (8):
xt/net_writer_imap: test "lei convert" w/ IMAP source
lei convert: use "--output" in failure message
doc: lei-convert: various updates and cleanups
doc: lei blob: wording fixups, describe --remote
lei convert: remove redundant input_net_cb
doc: lei-add-watch: add warning about unreliability
lei sucks: show nproc in CPU info
lei rm: move generic input_maildir_cb to LeiInput parent class
Documentation/lei-add-watch.pod | 8 ++++++--
Documentation/lei-blob.pod | 12 ++++++++----
Documentation/lei-convert.pod | 21 +++++++++------------
lib/PublicInbox/LeiConvert.pm | 8 ++------
lib/PublicInbox/LeiInput.pm | 5 +++++
lib/PublicInbox/LeiRm.pm | 5 -----
lib/PublicInbox/LeiSucks.pm | 4 +++-
t/lei-convert.t | 10 ++++++++++
xt/net_writer-imap.t | 7 +++++++
9 files changed, 50 insertions(+), 30 deletions(-)
^ permalink raw reply [relevance 71%]
* [PATCH 1/8] xt/net_writer_imap: test "lei convert" w/ IMAP source
2021-10-28 11:14 71% [PATCH 0/8] lei: docs and cleanups Eric Wong
@ 2021-10-28 11:14 68% ` Eric Wong
2021-10-28 11:14 71% ` [PATCH 2/8] lei convert: use "--output" in failure message Eric Wong
` (6 subsequent siblings)
7 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-10-28 11:14 UTC (permalink / raw)
To: meta
I just did a double-take and nearly thought authentication
was broken while reading LeiConvert.pm. Add a comment in
LeiConvert.pm to clarify things, too.
---
lib/PublicInbox/LeiConvert.pm | 1 +
xt/net_writer-imap.t | 7 +++++++
2 files changed, 8 insertions(+)
diff --git a/lib/PublicInbox/LeiConvert.pm b/lib/PublicInbox/LeiConvert.pm
index 68fc7c0ba2b0..424eab8e22e2 100644
--- a/lib/PublicInbox/LeiConvert.pm
+++ b/lib/PublicInbox/LeiConvert.pm
@@ -55,6 +55,7 @@ sub lei_convert { # the main "lei convert" method
my $devfd = $lei->path_to_fd($ovv->{dst}) // return;
$lei->{opt}->{augment} = 1 if $devfd < 0;
$self->prepare_inputs($lei, \@inputs) or return;
+ # n.b. {net} {auth} is handled by l2m worker
my ($op_c, $ops) = $lei->workers_start($self, 1);
$lei->{wq1} = $self;
$self->wq_io_do('process_inputs', []);
diff --git a/xt/net_writer-imap.t b/xt/net_writer-imap.t
index afa4bcc3e881..333e0e3b7ee0 100644
--- a/xt/net_writer-imap.t
+++ b/xt/net_writer-imap.t
@@ -137,6 +137,13 @@ test_lei(sub {
is_deeply($res->[0]->[1], $plack_qp_eml,
'lei q wrote expected result');
+ my $mdir = "$ENV{HOME}/t.mdir";
+ lei_ok 'convert', $folder_url, '-o', $mdir;
+ my @mdfiles = glob("$mdir/*/*");
+ is(scalar(@mdfiles), 1, '1 message from IMAP => Maildir conversion');
+ is_deeply(eml_load($mdfiles[0]), $plack_qp_eml,
+ 'conversion from IMAP to Maildir');
+
lei_ok qw(q f:matz -a -o), $folder_url;
$nwr->imap_each($folder_uri, $imap_slurp_all, my $aug = []);
is(scalar(@$aug), 2, '2 results after augment') or diag explain($aug);
^ permalink raw reply related [relevance 68%]
* [PATCH 2/8] lei convert: use "--output" in failure message
2021-10-28 11:14 71% [PATCH 0/8] lei: docs and cleanups Eric Wong
2021-10-28 11:14 68% ` [PATCH 1/8] xt/net_writer_imap: test "lei convert" w/ IMAP source Eric Wong
@ 2021-10-28 11:14 71% ` Eric Wong
2021-10-28 11:14 62% ` [PATCH 3/8] doc: lei-convert: various updates and cleanups Eric Wong
` (5 subsequent siblings)
7 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-10-28 11:14 UTC (permalink / raw)
To: meta
The extra dashes should help users find the correct option
more easily.
---
lib/PublicInbox/LeiConvert.pm | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/lib/PublicInbox/LeiConvert.pm b/lib/PublicInbox/LeiConvert.pm
index 424eab8e22e2..623599aef883 100644
--- a/lib/PublicInbox/LeiConvert.pm
+++ b/lib/PublicInbox/LeiConvert.pm
@@ -51,7 +51,7 @@ sub lei_convert { # the main "lei convert" method
my $self = bless {}, __PACKAGE__;
my $ovv = PublicInbox::LeiOverview->new($lei, 'out-format');
$lei->{l2m} or return
- $lei->fail("output not specified or is not a mail destination");
+ $lei->fail('--output unspecified or is not a mail destination');
my $devfd = $lei->path_to_fd($ovv->{dst}) // return;
$lei->{opt}->{augment} = 1 if $devfd < 0;
$self->prepare_inputs($lei, \@inputs) or return;
^ permalink raw reply related [relevance 71%]
* [PATCH 5/8] lei convert: remove redundant input_net_cb
2021-10-28 11:14 71% [PATCH 0/8] lei: docs and cleanups Eric Wong
` (3 preceding siblings ...)
2021-10-28 11:14 70% ` [PATCH 4/8] doc: lei blob: wording fixups, describe --remote Eric Wong
@ 2021-10-28 11:14 71% ` Eric Wong
2021-10-28 11:14 70% ` [PATCH 6/8] doc: lei-add-watch: add warning about unreliability Eric Wong
` (2 subsequent siblings)
7 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-10-28 11:14 UTC (permalink / raw)
To: meta
Use the one provided by the LeiInput parent class.
---
lib/PublicInbox/LeiConvert.pm | 5 -----
1 file changed, 5 deletions(-)
diff --git a/lib/PublicInbox/LeiConvert.pm b/lib/PublicInbox/LeiConvert.pm
index 623599aef883..906f30268344 100644
--- a/lib/PublicInbox/LeiConvert.pm
+++ b/lib/PublicInbox/LeiConvert.pm
@@ -23,11 +23,6 @@ sub input_eml_cb { # used by PublicInbox::LeiInput::input_fh
$self->{wcb}->(undef, {}, $eml);
}
-sub input_net_cb { # callback for ->imap_each, ->nntp_each
- my (undef, undef, $kw, $eml, $self) = @_; # @_[0,1]: url + uid ignored
- $self->{wcb}->(undef, { kw => $kw }, $eml);
-}
-
sub input_maildir_cb {
my (undef, $kw, $eml, $self) = @_; # $_[0] $filename ignored
$self->{wcb}->(undef, { kw => $kw }, $eml);
^ permalink raw reply related [relevance 71%]
* [PATCH 7/8] lei sucks: show nproc in CPU info
2021-10-28 11:14 71% [PATCH 0/8] lei: docs and cleanups Eric Wong
` (5 preceding siblings ...)
2021-10-28 11:14 70% ` [PATCH 6/8] doc: lei-add-watch: add warning about unreliability Eric Wong
@ 2021-10-28 11:15 71% ` Eric Wong
2021-10-28 11:15 63% ` [PATCH 8/8] lei rm: move generic input_maildir_cb to LeiInput parent class Eric Wong
7 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-10-28 11:15 UTC (permalink / raw)
To: meta
Some bugs are triggered with more CPUs, some with 1 CPU.
---
lib/PublicInbox/LeiSucks.pm | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/lib/PublicInbox/LeiSucks.pm b/lib/PublicInbox/LeiSucks.pm
index e832f95e0736..8e866fc96655 100644
--- a/lib/PublicInbox/LeiSucks.pm
+++ b/lib/PublicInbox/LeiSucks.pm
@@ -11,6 +11,7 @@ use Digest::SHA ();
use Config;
use POSIX ();
use PublicInbox::Config;
+use PublicInbox::IPC;
sub lei_sucks {
my ($lei, @argv) = @_;
@@ -22,9 +23,10 @@ sub lei_sucks {
}
eval { require PublicInbox };
my $pi_ver = eval('$PublicInbox::VERSION') // '(???)';
+ my $nproc = PublicInbox::IPC::detect_nproc() // '?';
my @out = ("lei $pi_ver\n",
"perl $Config{version} / $os $rel / $mac ".
- "ptrsize=$Config{ptrsize}\n");
+ "ptrsize=$Config{ptrsize} nproc=$nproc\n");
chomp(my $gv = `git --version` || "git missing");
$gv =~ s/ version / /;
my $json = ref(PublicInbox::Config->json);
^ permalink raw reply related [relevance 71%]
* [PATCH 6/8] doc: lei-add-watch: add warning about unreliability
2021-10-28 11:14 71% [PATCH 0/8] lei: docs and cleanups Eric Wong
` (4 preceding siblings ...)
2021-10-28 11:14 71% ` [PATCH 5/8] lei convert: remove redundant input_net_cb Eric Wong
@ 2021-10-28 11:14 70% ` Eric Wong
2021-10-28 11:15 71% ` [PATCH 7/8] lei sucks: show nproc in CPU info Eric Wong
2021-10-28 11:15 63% ` [PATCH 8/8] lei rm: move generic input_maildir_cb to LeiInput parent class Eric Wong
7 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-10-28 11:14 UTC (permalink / raw)
To: meta
This needs work at some point in the future.
---
Documentation/lei-add-watch.pod | 8 ++++++--
1 file changed, 6 insertions(+), 2 deletions(-)
diff --git a/Documentation/lei-add-watch.pod b/Documentation/lei-add-watch.pod
index 609846181578..1b48b43b589c 100644
--- a/Documentation/lei-add-watch.pod
+++ b/Documentation/lei-add-watch.pod
@@ -11,6 +11,11 @@ lei add-watch [OPTIONS] LOCATION [LOCATION...]
Tell lei to watch C<LOCATION> for new messages and flag changes.
Currently only Maildir locations are supported.
+WARNING: watches are not always reliable, occasional use
+of L<lei-index(1)> and L<lei-refresh-mail-sync(1)> is recommended
+if L<lei-daemon(8)> crashes or needs to be restarted. This will
+be improved in the future.
+
=for comment
TODO: Document --state? Believe valid values are pause, import-ro,
@@ -23,11 +28,10 @@ L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/>
=head1 COPYRIGHT
-Copyright 2021 all contributors L<mailto:meta@public-inbox.org>
+Copyright all contributors L<mailto:meta@public-inbox.org>
License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>
-
=head1 SEE ALSO
L<lei-ls-watch(1)>, L<lei-rm-watch(1)>
^ permalink raw reply related [relevance 70%]
* [PATCH 4/8] doc: lei blob: wording fixups, describe --remote
2021-10-28 11:14 71% [PATCH 0/8] lei: docs and cleanups Eric Wong
` (2 preceding siblings ...)
2021-10-28 11:14 62% ` [PATCH 3/8] doc: lei-convert: various updates and cleanups Eric Wong
@ 2021-10-28 11:14 70% ` Eric Wong
2021-10-28 11:14 71% ` [PATCH 5/8] lei convert: remove redundant input_net_cb Eric Wong
` (3 subsequent siblings)
7 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-10-28 11:14 UTC (permalink / raw)
To: meta
There's no current way to retrieve blobs by OID directly
from remote externals. Maybe the $INBOX_NAME/$OID/s/raw.eml
endpoint could be overloaded for that.
---
Documentation/lei-blob.pod | 12 ++++++++----
1 file changed, 8 insertions(+), 4 deletions(-)
diff --git a/Documentation/lei-blob.pod b/Documentation/lei-blob.pod
index 429d206e358e..e401bb4711a6 100644
--- a/Documentation/lei-blob.pod
+++ b/Documentation/lei-blob.pod
@@ -9,9 +9,10 @@ lei blob [OPTIONS] OID
=head1 DESCRIPTION
Display a git blob. The blob may correspond to a message from the
-local store, an existing blob in the current repository, or a
-not-yet-created blob in the current git project repository (if any)
-that can be reconstructed from a message.
+local store, any local external, or blobs associated with a
+project git repository (if run from a git (working) directory).
+For blobs which do not exist, it will attempt to recreate the blob
+using patch emails.
=head1 OPTIONS
@@ -64,7 +65,10 @@ L<lei-q(1)>.
=over
-=item --[no-]remote
+=item --remote
+
+Remote externals only get queried when the blob needs to be
+reconstructed from patch emails.
=item --no-local
^ permalink raw reply related [relevance 70%]
* [PATCH 8/8] lei rm: move generic input_maildir_cb to LeiInput parent class
2021-10-28 11:14 71% [PATCH 0/8] lei: docs and cleanups Eric Wong
` (6 preceding siblings ...)
2021-10-28 11:15 71% ` [PATCH 7/8] lei sucks: show nproc in CPU info Eric Wong
@ 2021-10-28 11:15 63% ` Eric Wong
7 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-10-28 11:15 UTC (permalink / raw)
To: meta
It's not much of a savings, right now, but maybe it can be in the
future. I wanted to eliminate the "lei convert" one, too, but
convert needs to preserve keywords which isn't possible with the
generic fallback, so new tests were written for convert, instead.
---
lib/PublicInbox/LeiInput.pm | 5 +++++
lib/PublicInbox/LeiRm.pm | 5 -----
t/lei-convert.t | 10 ++++++++++
3 files changed, 15 insertions(+), 5 deletions(-)
diff --git a/lib/PublicInbox/LeiInput.pm b/lib/PublicInbox/LeiInput.pm
index 540681e3ff6b..84fc579dc480 100644
--- a/lib/PublicInbox/LeiInput.pm
+++ b/lib/PublicInbox/LeiInput.pm
@@ -64,6 +64,11 @@ sub input_mbox_cb { # base MboxReader callback
$self->input_eml_cb($eml);
}
+sub input_maildir_cb {
+ my ($fn, $kw, $eml, $self) = @_;
+ $self->input_eml_cb($eml);
+}
+
sub input_net_cb { # imap_each, nntp_each cb
my ($url, $uid, $kw, $eml, $self) = @_;
$self->input_eml_cb($eml);
diff --git a/lib/PublicInbox/LeiRm.pm b/lib/PublicInbox/LeiRm.pm
index 524c178e3b47..cc1abbff66cb 100644
--- a/lib/PublicInbox/LeiRm.pm
+++ b/lib/PublicInbox/LeiRm.pm
@@ -13,11 +13,6 @@ sub input_eml_cb { # used by PublicInbox::LeiInput::input_fh
$self->{lei}->{sto}->wq_do('remove_eml', $eml);
}
-sub input_maildir_cb {
- my (undef, $kw, $eml, $self) = @_; # $_[0] $filename ignored
- input_eml_cb($self, $eml);
-}
-
sub lei_rm {
my ($lei, @inputs) = @_;
$lei->_lei_store(1)->write_prepare($lei);
diff --git a/t/lei-convert.t b/t/lei-convert.t
index 0ea860c82189..e1849ff796cd 100644
--- a/t/lei-convert.t
+++ b/t/lei-convert.t
@@ -115,5 +115,15 @@ test_lei({ tmpdir => $tmpdir }, sub {
@bar = ();
PublicInbox::MboxReader->mboxrd($fh, sub { push @bar, shift });
is_deeply(\@bar, [ $qp_eml ], 'readed gzipped mboxrd');
+
+ # Status => Maildir flag => Status round trip
+ $lei_out =~ s/^Status: O/Status: RO/sm or xbail "`seen' Status";
+ $rdr = { 0 => \($in = $lei_out), %$lei_opt };
+ lei_ok([qw(convert -F mboxrd -o), "$d/md2"], undef, $rdr);
+ @md = glob("$d/md2/*/*");
+ is(scalar(@md), 1, 'one message');
+ like($md[0], qr/:2,S\z/, "`seen' flag set in Maildir");
+ lei_ok(qw(convert -o mboxrd:/dev/stdout), "$d/md2");
+ like($lei_out, qr/^Status: RO/sm, "`seen' flag preserved");
});
done_testing;
^ permalink raw reply related [relevance 63%]
* [PATCH 3/8] doc: lei-convert: various updates and cleanups
2021-10-28 11:14 71% [PATCH 0/8] lei: docs and cleanups Eric Wong
2021-10-28 11:14 68% ` [PATCH 1/8] xt/net_writer_imap: test "lei convert" w/ IMAP source Eric Wong
2021-10-28 11:14 71% ` [PATCH 2/8] lei convert: use "--output" in failure message Eric Wong
@ 2021-10-28 11:14 62% ` Eric Wong
2021-10-28 11:14 70% ` [PATCH 4/8] doc: lei blob: wording fixups, describe --remote Eric Wong
` (4 subsequent siblings)
7 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-10-28 11:14 UTC (permalink / raw)
To: meta
Note that "-o OUTPUT" is required in the synopsis.
Leave out "eml:" for now since it doesn't work as an output and
I doubt anybody would use it as a prefix, and it's not really
useful.
--no-import-remote is also not accepted by convert, since it
doesn't touch lei/store at all.
---
Documentation/lei-convert.pod | 21 +++++++++------------
1 file changed, 9 insertions(+), 12 deletions(-)
diff --git a/Documentation/lei-convert.pod b/Documentation/lei-convert.pod
index 750ba54ffa1a..c113db18b1e4 100644
--- a/Documentation/lei-convert.pod
+++ b/Documentation/lei-convert.pod
@@ -1,23 +1,23 @@
=head1 NAME
-lei-convert - one-time conversion from URL or filesystem to another format
+lei-convert - one-time conversion from one mail format to another
=head1 SYNOPSIS
-lei convert [OPTIONS] LOCATION
+lei convert -o OUTPUT [OPTIONS] LOCATION
-lei convert [OPTIONS] (--stdin|-)
+lei convert -o OUTPUT [OPTIONS] (--stdin|-)
=head1 DESCRIPTION
Convert messages to another format. C<LOCATION> is a source of
-messages: a directory (Maildir), a file, or a URL (C<imap://>,
-C<imaps://>, C<nntp://>, or C<nntps://>). URLs requiring
-authentication use L<git-credential(1)> to
+messages: a directory (Maildir), a file (various mbox), or a URL
+(C<imap://>, C<imaps://>, C<nntp://>, or C<nntps://>). URLs
+requiring authentication use L<git-credential(1)> to
fill in the username and password.
For a regular file, the location must have a C<E<lt>formatE<gt>:>
-prefix specifying one of the following formats: C<eml>, C<mboxrd>,
+prefix specifying one of the following formats: C<mboxrd>,
C<mboxcl2>, C<mboxcl>, or C<mboxo>.
=head1 OPTIONS
@@ -44,8 +44,6 @@ L<lei-q(1)>.
=item --no-kw
-=item --no-import-remote
-
=item --torsocks=auto|no|yes
=item --no-torsocks
@@ -63,11 +61,10 @@ and L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta
=head1 COPYRIGHT
-Copyright 2021 all contributors L<mailto:meta@public-inbox.org>
+Copyright all contributors L<mailto:meta@public-inbox.org>
License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>
-
=head1 SEE ALSO
-L<lei-add-external(1)>
+L<lei-mail-formats(5)>
^ permalink raw reply related [relevance 62%]
* Re: lei-q-remote-import failures [was: [PATCH] t/index-git-times: support non-master default branch]
2021-10-27 23:48 63% ` Eric Wong
@ 2021-10-28 14:03 71% ` Thomas Weißschuh
2021-10-28 19:16 68% ` [PATCH] test_common: clear XDG_CACHE_HOME before lei tests Eric Wong
0 siblings, 1 reply; 200+ results
From: Thomas Weißschuh @ 2021-10-28 14:03 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
On 2021-10-27 23:48+0000, Eric Wong wrote:
> Thomas Weißschuh <thomas@t-8ch.de> wrote:
> > It didn't help, neither did larger sleeps.
>
> Thanks for testing.
>
> > I'll see if I can debug it in the coming days.
>
> OK, maybe the patch below can get you started...
>
> I've also been wondering if something like GIT_TRACE is
> necessary, but it's also more work to support + document.
> And Perl code is easily modified compared to AOT languages.
It turned out that in my $XDG_CONFIG_HOME/lei/all_locals_ever.git/lei_ale.state
there were entries for the repositories in
$PUBLIC_INBOX_SRC/t/home2/{t1,t2}.
I'm not sure how those entries came to be but probably because of some
debugging things I did before.
IMO it would make sense to prevent p-i to read this global state during
unittests.
^ permalink raw reply [relevance 71%]
* [PATCH] test_common: clear XDG_CACHE_HOME before lei tests
2021-10-28 14:03 71% ` Thomas Weißschuh
@ 2021-10-28 19:16 68% ` Eric Wong
2021-10-28 19:22 71% ` Thomas Weißschuh
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2021-10-28 19:16 UTC (permalink / raw)
To: Thomas Weißschuh; +Cc: meta
Thomas Weißschuh <thomas@t-8ch.de> wrote:
> It turned out that in my $XDG_CONFIG_HOME/lei/all_locals_ever.git/lei_ale.state
> there were entries for the repositories in
> $PUBLIC_INBOX_SRC/t/home2/{t1,t2}.
>
> I'm not sure how those entries came to be but probably because of some
> debugging things I did before.
The presence of t1 & t2 entries is harmless for normal lei
operation; but they shouldn't be there...
I wonder, do you have XDG_CACHE_HOME explicitly set in your env?
If so, that would've caused problems (fixed below).
> IMO it would make sense to prevent p-i to read this global state during
> unittests.
Yes, I think this was from our failure to clobber XDG_CACHE_HOME:
------------8<------------
Subject: [PATCH] test_common: clear XDG_CACHE_HOME before lei tests
We don't want to read a users'
$XDG_CACHE_HOME/lei/all_locals_ever.git during tests.
Reported-by: Thomas Weißschuh <thomas@t-8ch.de>
Link: https://public-inbox.org/meta/f239abac-4aee-4573-a0d6-e533c7a32662@t-8ch.de/
---
lib/PublicInbox/TestCommon.pm | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/lib/PublicInbox/TestCommon.pm b/lib/PublicInbox/TestCommon.pm
index c3820d3a..052d6e45 100644
--- a/lib/PublicInbox/TestCommon.pm
+++ b/lib/PublicInbox/TestCommon.pm
@@ -556,16 +556,20 @@ SKIP: {
require_git(2.6, 1) or skip('git 2.6+ required for lei test', 2);
my $mods = $test_opt->{mods} // [ 'lei' ];
require_mods(@$mods, 2);
+
+ # set PERL_INLINE_DIRECTORY before clobbering XDG_CACHE_HOME
+ require PublicInbox::Spawn;
require PublicInbox::Config;
require File::Path;
+
local %ENV = %ENV;
delete $ENV{XDG_DATA_HOME};
delete $ENV{XDG_CONFIG_HOME};
+ delete $ENV{XDG_CACHE_HOME};
$ENV{GIT_COMMITTER_EMAIL} = 'lei@example.com';
$ENV{GIT_COMMITTER_NAME} = 'lei user';
my (undef, $fn, $lineno) = caller(0);
my $t = "$fn:$lineno";
- require PublicInbox::Spawn;
state $lei_daemon = PublicInbox::Spawn->can('send_cmd4') ||
eval { require Socket::MsgHdr; 1 };
unless ($lei_daemon) {
^ permalink raw reply related [relevance 68%]
* Re: [PATCH] test_common: clear XDG_CACHE_HOME before lei tests
2021-10-28 19:16 68% ` [PATCH] test_common: clear XDG_CACHE_HOME before lei tests Eric Wong
@ 2021-10-28 19:22 71% ` Thomas Weißschuh
0 siblings, 0 replies; 200+ results
From: Thomas Weißschuh @ 2021-10-28 19:22 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
On 2021-10-28 19:16+0000, Eric Wong wrote:
> Thomas Weißschuh <thomas@t-8ch.de> wrote:
> > It turned out that in my $XDG_CONFIG_HOME/lei/all_locals_ever.git/lei_ale.state
> > there were entries for the repositories in
> > $PUBLIC_INBOX_SRC/t/home2/{t1,t2}.
> >
> > I'm not sure how those entries came to be but probably because of some
> > debugging things I did before.
>
> The presence of t1 & t2 entries is harmless for normal lei
> operation; but they shouldn't be there...
>
> I wonder, do you have XDG_CACHE_HOME explicitly set in your env?
> If so, that would've caused problems (fixed below).
Yes I have it set explicitly
> > IMO it would make sense to prevent p-i to read this global state during
> > unittests.
>
> Yes, I think this was from our failure to clobber XDG_CACHE_HOME:
That works, thanks!
Tested-by: Thomas Weißschuh <thomas@t-8ch.de>
> ------------8<------------
> Subject: [PATCH] test_common: clear XDG_CACHE_HOME before lei tests
>
> We don't want to read a users'
> $XDG_CACHE_HOME/lei/all_locals_ever.git during tests.
>
> Reported-by: Thomas Weißschuh <thomas@t-8ch.de>
> Link: https://public-inbox.org/meta/f239abac-4aee-4573-a0d6-e533c7a32662@t-8ch.de/
> ---
> lib/PublicInbox/TestCommon.pm | 6 +++++-
> 1 file changed, 5 insertions(+), 1 deletion(-)
>
> diff --git a/lib/PublicInbox/TestCommon.pm b/lib/PublicInbox/TestCommon.pm
> index c3820d3a..052d6e45 100644
> --- a/lib/PublicInbox/TestCommon.pm
> +++ b/lib/PublicInbox/TestCommon.pm
> @@ -556,16 +556,20 @@ SKIP: {
> require_git(2.6, 1) or skip('git 2.6+ required for lei test', 2);
> my $mods = $test_opt->{mods} // [ 'lei' ];
> require_mods(@$mods, 2);
> +
> + # set PERL_INLINE_DIRECTORY before clobbering XDG_CACHE_HOME
> + require PublicInbox::Spawn;
> require PublicInbox::Config;
> require File::Path;
> +
> local %ENV = %ENV;
> delete $ENV{XDG_DATA_HOME};
> delete $ENV{XDG_CONFIG_HOME};
> + delete $ENV{XDG_CACHE_HOME};
> $ENV{GIT_COMMITTER_EMAIL} = 'lei@example.com';
> $ENV{GIT_COMMITTER_NAME} = 'lei user';
> my (undef, $fn, $lineno) = caller(0);
> my $t = "$fn:$lineno";
> - require PublicInbox::Spawn;
> state $lei_daemon = PublicInbox::Spawn->can('send_cmd4') ||
> eval { require Socket::MsgHdr; 1 };
> unless ($lei_daemon) {
^ permalink raw reply [relevance 71%]
* [PATCH 1/5] lei: do not access {sock} after SIGPIPE
2021-10-30 8:11 71% [PATCH 0/5] lei: fix various SIGPIPE problems Eric Wong
@ 2021-10-30 8:11 71% ` Eric Wong
2021-10-30 8:11 71% ` [PATCH 5/5] doc: lei-security: add a note about core dumps Eric Wong
1 sibling, 0 replies; 200+ results
From: Eric Wong @ 2021-10-30 8:11 UTC (permalink / raw)
To: meta
It's possible for this to break out of the event loop if
note_sigpipe fires via PktOp in the same iteration.
---
lib/PublicInbox/LEI.pm | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 96f7c5e315a9..78b49a3bc1af 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -1127,7 +1127,7 @@ sub event_step {
local %ENV = %{$self->{env}};
local $current_lei = $self;
eval {
- my @fds = $recv_cmd->($self->{sock}, my $buf, 4096);
+ my @fds = $recv_cmd->($self->{sock} // return, my $buf, 4096);
if (scalar(@fds) == 1 && !defined($fds[0])) {
return if $! == EAGAIN;
die "recvmsg: $!" if $! != ECONNRESET;
^ permalink raw reply related [relevance 71%]
* [PATCH 0/5] lei: fix various SIGPIPE problems
@ 2021-10-30 8:11 71% Eric Wong
2021-10-30 8:11 71% ` [PATCH 1/5] lei: do not access {sock} after SIGPIPE Eric Wong
2021-10-30 8:11 71% ` [PATCH 5/5] doc: lei-security: add a note about core dumps Eric Wong
0 siblings, 2 replies; 200+ results
From: Eric Wong @ 2021-10-30 8:11 UTC (permalink / raw)
To: meta
Most worrying was the the bug fixed in 4/5; but at least there
wasn't data loss involved...
While the bug fixed in 4/5 didn't cause data loss, it was
dumping core files and filling up my disk while polluting the
kernel log buffer.
Eric Wong (5):
lei: do not access {sock} after SIGPIPE
lei_to_mail: limit workers for text, reply and v2 outputs
lei_xsearch: quiet error message on SIG{PIPE,TERM}
lei_to_mail: avoid SEGV on worker exit via SIGTERM
doc: lei-security: add a note about core dumps
Documentation/lei-security.pod | 6 ++++++
lib/PublicInbox/LEI.pm | 2 +-
lib/PublicInbox/LeiQuery.pm | 2 +-
lib/PublicInbox/LeiToMail.pm | 10 +++++++++-
lib/PublicInbox/LeiXSearch.pm | 5 ++++-
t/lei-sigpipe.t | 7 +++++--
6 files changed, 26 insertions(+), 6 deletions(-)
^ permalink raw reply [relevance 71%]
* [PATCH 5/5] doc: lei-security: add a note about core dumps
2021-10-30 8:11 71% [PATCH 0/5] lei: fix various SIGPIPE problems Eric Wong
2021-10-30 8:11 71% ` [PATCH 1/5] lei: do not access {sock} after SIGPIPE Eric Wong
@ 2021-10-30 8:11 71% ` Eric Wong
1 sibling, 0 replies; 200+ results
From: Eric Wong @ 2021-10-30 8:11 UTC (permalink / raw)
To: meta
Maybe we can avoid them if we stop having buggy code :P
---
Documentation/lei-security.pod | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/Documentation/lei-security.pod b/Documentation/lei-security.pod
index 8cbd89934568..104bfb48a26c 100644
--- a/Documentation/lei-security.pod
+++ b/Documentation/lei-security.pod
@@ -64,6 +64,12 @@ public-facing L<public-inbox-daemon(8)> processes. They may
reside on shared storage and may be made world-readable to
other users on the local system.
+=head1 CORE DUMPS
+
+In case any process crashes, a core dumps may contain passwords or
+contents of sensitive messages. Please report these so they can be
+fixed (see L</CONTACT>).
+
=head1 NETWORK ACCESS
lei currently uses the L<curl(1)> and L<git(1)> executables in
^ permalink raw reply related [relevance 71%]
* [PATCH 0/2] lei: mail-sync docs + compat fix
@ 2021-10-31 9:10 71% Eric Wong
2021-10-31 9:10 29% ` [PATCH 1/2] doc: add lei-mail-sync-overview manpage Eric Wong
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2021-10-31 9:10 UTC (permalink / raw)
To: meta
Eric Wong (2):
doc: add lei-mail-sync-overview manpage
lei_input: disallow uppercase characters for labels
Documentation/lei-export-kw.pod | 3 ++
Documentation/lei-forget-mail-sync.pod | 5 ++-
Documentation/lei-import.pod | 34 +++++++++++++---
Documentation/lei-index.pod | 5 ++-
Documentation/lei-ls-label.pod | 1 +
Documentation/lei-ls-mail-source.pod | 5 ++-
Documentation/lei-ls-watch.pod | 3 +-
Documentation/lei-mail-formats.pod | 5 ++-
Documentation/lei-mail-sync-overview.pod | 52 ++++++++++++++++++++++++
Documentation/lei-overview.pod | 13 +++++-
Documentation/lei-tag.pod | 21 +++++++++-
Documentation/lei.pod | 4 +-
MANIFEST | 1 +
Makefile.PL | 4 +-
lib/PublicInbox/LeiInput.pm | 2 +-
15 files changed, 137 insertions(+), 21 deletions(-)
create mode 100644 Documentation/lei-mail-sync-overview.pod
^ permalink raw reply [relevance 71%]
* [PATCH 1/2] doc: add lei-mail-sync-overview manpage
2021-10-31 9:10 71% [PATCH 0/2] lei: mail-sync docs + compat fix Eric Wong
@ 2021-10-31 9:10 29% ` Eric Wong
2021-10-31 16:32 71% ` Kyle Meyer
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2021-10-31 9:10 UTC (permalink / raw)
To: meta
Mostly illustrating how clunky the process is :p
We'll also tweak some things in existing man pages around
mail synchronization.
---
Documentation/lei-export-kw.pod | 3 ++
Documentation/lei-forget-mail-sync.pod | 5 ++-
Documentation/lei-import.pod | 34 +++++++++++++---
Documentation/lei-index.pod | 5 ++-
Documentation/lei-ls-label.pod | 1 +
Documentation/lei-ls-mail-source.pod | 5 ++-
Documentation/lei-ls-watch.pod | 3 +-
Documentation/lei-mail-formats.pod | 5 ++-
Documentation/lei-mail-sync-overview.pod | 52 ++++++++++++++++++++++++
Documentation/lei-overview.pod | 13 +++++-
Documentation/lei-tag.pod | 21 +++++++++-
Documentation/lei.pod | 4 +-
MANIFEST | 1 +
Makefile.PL | 4 +-
14 files changed, 136 insertions(+), 20 deletions(-)
create mode 100644 Documentation/lei-mail-sync-overview.pod
diff --git a/Documentation/lei-export-kw.pod b/Documentation/lei-export-kw.pod
index cf482ca0d307..b6d175f14773 100644
--- a/Documentation/lei-export-kw.pod
+++ b/Documentation/lei-export-kw.pod
@@ -12,6 +12,9 @@ lei export-kw MFOLDER [MFOLDER...]
C<lei export-kw> propagates keywords (e.g. C<seen>, C<answered>,
C<flagged>, etc.) from lei/store to IMAP folders and/or Maildirs.
+It only works for messages lei knows about (e.g. was used as a
+C<lei q --output>, or imported via L<lei-import(1)>, or indexed
+via L<lei-index(1)>.
It does not delete, write, nor modify messages themselves;
it only sets metadata on Maildirs and IMAP folders.
diff --git a/Documentation/lei-forget-mail-sync.pod b/Documentation/lei-forget-mail-sync.pod
index e70b4d33a33a..29ff56711815 100644
--- a/Documentation/lei-forget-mail-sync.pod
+++ b/Documentation/lei-forget-mail-sync.pod
@@ -9,8 +9,9 @@ lei forget-mail-sync [OPTIONS] LOCATION [LOCATION...]
=head1 DESCRIPTION
Forget synchronization information for C<LOCATION>, an IMAP or Maildir
-folder. Note that this won't delete any messages stored in Git,
-leaving C<lei-index(1)> users with dangling references.
+folder. Note that this won't delete any messages on the filesystem.
+Users using L<lei-index(1)> without L<lei-import(1)> will be left
+with dangling references in search results.
=head1 CONTACT
diff --git a/Documentation/lei-import.pod b/Documentation/lei-import.pod
index 9322dfa8d34c..25f1daf5f32f 100644
--- a/Documentation/lei-import.pod
+++ b/Documentation/lei-import.pod
@@ -4,7 +4,7 @@ lei-import - one-time import of messages into local store
=head1 SYNOPSIS
-lei import [OPTIONS] LOCATION [LOCATION...]
+lei import [OPTIONS] LOCATION [LOCATION...] [+L:LABEL]
lei import [OPTIONS] (--stdin|-)
@@ -16,8 +16,8 @@ source of messages: a directory (Maildir), a file, or a URL
authentication use L<git-credential(1)> to
fill in the username and password.
-For a regular file, the location must have a C<E<lt>formatE<gt>:>
-prefix specifying one of the following formats: C<eml>, C<mboxrd>,
+For a regular file, the C<LOCATION> must have a C<E<lt>formatE<gt>:>
+prefix specifying one of the following formats: C<mboxrd>,
C<mboxcl2>, C<mboxcl>, or C<mboxo>.
=head1 OPTIONS
@@ -42,6 +42,29 @@ C<none>.
Default: fcntl,dotlock
+=item +L:LABEL
+
+Add the given C<LABEL> to all messages imported, where C<LABEL>
+is an arbitrary user-defined value consisting of lowercase and digits.
+See L<lei-tag(1)> for more info on labels.
+
+For example, specifying C<+L:inbox> applies the C<inbox> label
+to all messages being imported.
+
+May be specified multiple times to apply multiple labels.
+
+Default: none
+
+=item +kw:KEYWORD
+
+Apply C<KEYWORD> to all messages being imported in addition
+to any per-message keywords from the store (unless C<--no-kw>
+is specified). See L<lei-tag(1)> for more info on keywords.
+
+May be specified multiple times to apply multiple keywords.
+
+Default: none
+
=item --no-kw
Don't import message keywords (or "flags" in IMAP terminology).
@@ -73,11 +96,10 @@ L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/>
=head1 COPYRIGHT
-Copyright 2021 all contributors L<mailto:meta@public-inbox.org>
+Copyright all contributors L<mailto:meta@public-inbox.org>
License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>
-
=head1 SEE ALSO
-L<lei-add-external(1)>
+L<lei-index(1)>
diff --git a/Documentation/lei-index.pod b/Documentation/lei-index.pod
index 9e72026351a1..de088538e5ca 100644
--- a/Documentation/lei-index.pod
+++ b/Documentation/lei-index.pod
@@ -20,6 +20,9 @@ Combined with L<lei-q(1)>, C<lei index> allows Maildir users to
have similar functionality to L<mairix(1)> by not duplicating
messages into C<lei/store>.
+Occasional invocations of C<lei-refresh-mail-sync --all=local>
+is recommended to keep indexed messages retrievable.
+
=head1 OPTIONS
=over
@@ -58,4 +61,4 @@ License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>
=head1 SEE ALSO
-L<lei-store-format(5)>, L<lei-import(1)>
+L<lei-refresh-mail-sync(1)>, L<lei-store-format(5)>, L<lei-import(1)>
diff --git a/Documentation/lei-ls-label.pod b/Documentation/lei-ls-label.pod
index 41aa030d9932..8342705a6cf6 100644
--- a/Documentation/lei-ls-label.pod
+++ b/Documentation/lei-ls-label.pod
@@ -9,6 +9,7 @@ lei ls-label [OPTIONS]
=head1 DESCRIPTION
List all known message labels ("mailboxes" in JMAP terminology).
+This is handy for writing L<lei-import(1)> invocations.
=head1 OPTIONS
diff --git a/Documentation/lei-ls-mail-source.pod b/Documentation/lei-ls-mail-source.pod
index 926bbe2c02f6..59d14afe6880 100644
--- a/Documentation/lei-ls-mail-source.pod
+++ b/Documentation/lei-ls-mail-source.pod
@@ -9,6 +9,8 @@ lei ls-mail-source [OPTIONS] URL
=head1 DESCRIPTION
List information about the IMAP or NNTP mail source at C<URL>.
+This command populates the cache used for Bash shell completion
+and is handy for writing L<lei-import(1)> invocations.
=head1 OPTIONS
@@ -48,11 +50,10 @@ L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/>
=head1 COPYRIGHT
-Copyright 2021 all contributors L<mailto:meta@public-inbox.org>
+Copyright all contributors L<mailto:meta@public-inbox.org>
License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>
-
=head1 SEE ALSO
L<lei-import(1)>
diff --git a/Documentation/lei-ls-watch.pod b/Documentation/lei-ls-watch.pod
index b1681ee45782..8063d34d3273 100644
--- a/Documentation/lei-ls-watch.pod
+++ b/Documentation/lei-ls-watch.pod
@@ -8,7 +8,8 @@ lei ls-watch
=head1 DESCRIPTION
-List locations that lei is configured to watch.
+List locations that lei is configured to watch. This command is
+incomplete, mail-sync locations are implicitly watched.
=head1 CONTACT
diff --git a/Documentation/lei-mail-formats.pod b/Documentation/lei-mail-formats.pod
index 3c37c88046b6..930c5d7660ea 100644
--- a/Documentation/lei-mail-formats.pod
+++ b/Documentation/lei-mail-formats.pod
@@ -96,7 +96,10 @@ it worth supporting.
Depending on the IMAP server software and configuration, IMAP
servers may use any (or combination) of the aforementioned
-formats or a non-standard database backend.
+formats or a non-standard database backend. Currently, lei
+uses L<Mail::IMAPClient> which has acceptable performance
+over low-latency links. Performance over high-latency links
+is currently poor.
=head1 eml
diff --git a/Documentation/lei-mail-sync-overview.pod b/Documentation/lei-mail-sync-overview.pod
new file mode 100644
index 000000000000..dfe1034cea7e
--- /dev/null
+++ b/Documentation/lei-mail-sync-overview.pod
@@ -0,0 +1,52 @@
+=head1 NAME
+
+lei - an overview of lei mail synchronization
+
+=head1 DESCRIPTION
+
+L<lei(1)> provides several plumbing-level commands to synchronize
+mail and keywords (flags) between lei/store and existing IMAP
+and Maildir stores. Nothing documented in this manpage is required
+for day-to-day use against externals.
+
+Mail and keyword synchronization is currently a clunky process.
+Future work will be done to improve it and add IMAP IDLE support.
+
+=head1 TYPICAL WORKFLOW
+
+ # import mail from a user's IMAP inbox and give it the "inbox" label:
+ lei import +L:inbox imaps://user@example.com/INBOX
+
+ # dump "inbox" labeled files from the past week to a Maildir
+ lei q L:inbox rt:last.week.. -o /tmp/results
+
+ # open /tmp/results in your favorite mail agent. If inotify or kevent
+ # works, keyword changes (e.g. marking messages as `seen').
+
+ # If the inotify queue overflows, or if lei-daemon crashes,
+ # "lei index" will tell lei about keyword changes:
+ lei index /tmp/results
+
+ # Optional: cleanup stale entries from mail_sync.sqlite3
+ lei refresh-mail-sync /tmp/results
+
+ # to export keyword changes back to IMAP
+ lei export-kw imaps://user@example.com/INBOX
+
+=head1 CONTACT
+
+Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org>
+
+The mail archives are hosted at L<https://public-inbox.org/meta/> and
+L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/>
+
+=head1 COPYRIGHT
+
+Copyright all contributors L<mailto:meta@public-inbox.org>
+
+License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>
+
+=head1 SEE ALSO
+
+L<lei-import(1)>, L<lei-q(1)>, L<lei-index(1)>,
+L<lei-refresh-mail-sync(1)>, L<lei-export-kw(1)>
diff --git a/Documentation/lei-overview.pod b/Documentation/lei-overview.pod
index 99fd6ef72174..7095b504cdb8 100644
--- a/Documentation/lei-overview.pod
+++ b/Documentation/lei-overview.pod
@@ -4,8 +4,8 @@ lei - an overview of lei
=head1 DESCRIPTION
-L<lei(1)> is a local email interface for public-inbox. This document
-provides some basic examples.
+L<lei(1)> is a local email interface for public-inbox and personal mail.
+This document provides some basic examples.
=head1 LEI STORE
@@ -49,6 +49,11 @@ For existing local paths, the external needs to be indexed with
L<public-inbox-index(1)> (in the case of a regular inbox) or
L<public-inbox-extindex(1)> (in the case of an external index).
+=head1 SYNCHRONIZATION
+
+lei currently has primitive mail synchronization abilities;
+see L<lei-mail-sync-overview(7)> for more details.
+
=head2 EXAMPLES
=over
@@ -154,3 +159,7 @@ L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/>
Copyright all contributors L<mailto:meta@public-inbox.org>
License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>
+
+=head1 SEE ALSO
+
+L<lei-mail-sync-overview(7)>
diff --git a/Documentation/lei-tag.pod b/Documentation/lei-tag.pod
index b2509ba4ee26..8cb9e736b6e7 100644
--- a/Documentation/lei-tag.pod
+++ b/Documentation/lei-tag.pod
@@ -43,6 +43,26 @@ Suppress feedback messages.
=back
+=head1 LABELS
+
+Labels are user-defined values analogous to IMAP/JMAP mailbox
+names. They must only contain lowercase characters, digits, and
+a limited amount of punctuation (e.g. C<.>, C<->, C<@>).
+
+Messages may have multiple labels.
+
+=head1 KEYWORDS
+
+Keywords are "flags" in Maildir and IMAP terminology.
+Common keywords include: C<seen>, C<answered>, C<flagged>, and
+C<draft>, though C<forwarded>, C<phishing>, C<junk>, and C<notjunk>
+are also supported.
+
+When writing to various mboxes, the common keywords will be
+mapped to the C<Status> and C<X-Status> headers.
+
+Messages may have multiple keywords.
+
=head1 CONTACT
Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org>
@@ -56,7 +76,6 @@ Copyright 2021 all contributors L<mailto:meta@public-inbox.org>
License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>
-
=head1 SEE ALSO
L<lei-add-external(1)>
diff --git a/Documentation/lei.pod b/Documentation/lei.pod
index 24a585daf24c..f01f506af359 100644
--- a/Documentation/lei.pod
+++ b/Documentation/lei.pod
@@ -106,7 +106,7 @@ Other subcommands include
=item * L<lei-daemon-pid(1)>
-=item * lei-forget-mail-sync(1)
+=item * L<lei-forget-mail-sync(1)>
=item * L<lei-mail-diff(1)>
@@ -145,7 +145,7 @@ L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/>
=head1 COPYRIGHT
-Copyright 2021 all contributors L<mailto:meta@public-inbox.org>
+Copyright all contributors L<mailto:meta@public-inbox.org>
License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>
diff --git a/MANIFEST b/MANIFEST
index 9fd979ef02fb..1e8f60fb41da 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -47,6 +47,7 @@ Documentation/lei-ls-search.pod
Documentation/lei-ls-watch.pod
Documentation/lei-mail-diff.pod
Documentation/lei-mail-formats.pod
+Documentation/lei-mail-sync-overview.pod
Documentation/lei-overview.pod
Documentation/lei-p2q.pod
Documentation/lei-q.pod
diff --git a/Makefile.PL b/Makefile.PL
index b3ac59be32ee..8c8c0235f35b 100644
--- a/Makefile.PL
+++ b/Makefile.PL
@@ -1,5 +1,5 @@
#!/usr/bin/perl -w
-# Copyright (C) 2013-2021 all contributors <meta@public-inbox.org>
+# Copyright (C) all contributors <meta@public-inbox.org>
# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
use strict;
use ExtUtils::MakeMaker;
@@ -57,7 +57,7 @@ $v->{-m5} = [ qw(public-inbox-config public-inbox-v1-format
public-inbox-v2-format public-inbox-extindex-format
lei-mail-formats lei-store-format
) ];
-$v->{-m7} = [ qw(lei-overview lei-security
+$v->{-m7} = [ qw(lei-mail-sync-overview lei-overview lei-security
public-inbox-overview public-inbox-tuning
public-inbox-glossary) ];
$v->{-m8} = [ qw(public-inbox-daemon lei-daemon) ];
^ permalink raw reply related [relevance 29%]
* Re: [PATCH 1/2] doc: add lei-mail-sync-overview manpage
2021-10-31 9:10 29% ` [PATCH 1/2] doc: add lei-mail-sync-overview manpage Eric Wong
@ 2021-10-31 16:32 71% ` Kyle Meyer
2021-10-31 17:30 64% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Kyle Meyer @ 2021-10-31 16:32 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
Eric Wong writes:
> --- a/Documentation/lei-export-kw.pod
> +++ b/Documentation/lei-export-kw.pod
> @@ -12,6 +12,9 @@ lei export-kw MFOLDER [MFOLDER...]
>
> C<lei export-kw> propagates keywords (e.g. C<seen>, C<answered>,
> C<flagged>, etc.) from lei/store to IMAP folders and/or Maildirs.
> +It only works for messages lei knows about (e.g. was used as a
> +C<lei q --output>, or imported via L<lei-import(1)>, or indexed
> +via L<lei-index(1)>.
The closing paren is missing here.
> --- a/Documentation/lei-index.pod
> +++ b/Documentation/lei-index.pod
> @@ -20,6 +20,9 @@ Combined with L<lei-q(1)>, C<lei index> allows Maildir users to
> have similar functionality to L<mairix(1)> by not duplicating
> messages into C<lei/store>.
>
> +Occasional invocations of C<lei-refresh-mail-sync --all=local>
> +is recommended to keep indexed messages retrievable.
"Occasional invocations ... is" -> "Occasional invocations ... are"?
Or perhaps "Occasionally invoking ... is"
> +=head1 TYPICAL WORKFLOW
> +
> + # import mail from a user's IMAP inbox and give it the "inbox" label:
> + lei import +L:inbox imaps://user@example.com/INBOX
> +
> + # dump "inbox" labeled files from the past week to a Maildir
> + lei q L:inbox rt:last.week.. -o /tmp/results
> +
> + # open /tmp/results in your favorite mail agent. If inotify or kevent
> + # works, keyword changes (e.g. marking messages as `seen').
Is this part incomplete (maybe "are synchronized" is missing at the
end)?
^ permalink raw reply [relevance 71%]
* Re: [PATCH 1/2] doc: add lei-mail-sync-overview manpage
2021-10-31 16:32 71% ` Kyle Meyer
@ 2021-10-31 17:30 64% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-10-31 17:30 UTC (permalink / raw)
To: Kyle Meyer; +Cc: meta
Kyle Meyer <kyle@kyleam.com> wrote:
> Eric Wong writes:
> > +++ b/Documentation/lei-export-kw.pod
> > @@ -12,6 +12,9 @@ lei export-kw MFOLDER [MFOLDER...]
> >
> > C<lei export-kw> propagates keywords (e.g. C<seen>, C<answered>,
> > C<flagged>, etc.) from lei/store to IMAP folders and/or Maildirs.
> > +It only works for messages lei knows about (e.g. was used as a
> > +C<lei q --output>, or imported via L<lei-import(1)>, or indexed
> > +via L<lei-index(1)>.
>
> The closing paren is missing here.
Yup.
> > +++ b/Documentation/lei-index.pod
> > @@ -20,6 +20,9 @@ Combined with L<lei-q(1)>, C<lei index> allows Maildir users to
> > have similar functionality to L<mairix(1)> by not duplicating
> > messages into C<lei/store>.
> >
> > +Occasional invocations of C<lei-refresh-mail-sync --all=local>
> > +is recommended to keep indexed messages retrievable.
>
> "Occasional invocations ... is" -> "Occasional invocations ... are"?
>
> Or perhaps "Occasionally invoking ... is"
I think I'll go with "are"
> > +=head1 TYPICAL WORKFLOW
> > +
> > + # import mail from a user's IMAP inbox and give it the "inbox" label:
> > + lei import +L:inbox imaps://user@example.com/INBOX
> > +
> > + # dump "inbox" labeled files from the past week to a Maildir
> > + lei q L:inbox rt:last.week.. -o /tmp/results
> > +
> > + # open /tmp/results in your favorite mail agent. If inotify or kevent
> > + # works, keyword changes (e.g. marking messages as `seen').
>
> Is this part incomplete (maybe "are synchronized" is missing at the
> end)?
Yes, "are synchronized automatically". Will squash the
following in, thanks:
diff --git a/Documentation/lei-export-kw.pod b/Documentation/lei-export-kw.pod
index b6d175f1..4cb673ac 100644
--- a/Documentation/lei-export-kw.pod
+++ b/Documentation/lei-export-kw.pod
@@ -14,7 +14,7 @@ C<lei export-kw> propagates keywords (e.g. C<seen>, C<answered>,
C<flagged>, etc.) from lei/store to IMAP folders and/or Maildirs.
It only works for messages lei knows about (e.g. was used as a
C<lei q --output>, or imported via L<lei-import(1)>, or indexed
-via L<lei-index(1)>.
+via L<lei-index(1)>).
It does not delete, write, nor modify messages themselves;
it only sets metadata on Maildirs and IMAP folders.
diff --git a/Documentation/lei-index.pod b/Documentation/lei-index.pod
index de088538..f8ff6950 100644
--- a/Documentation/lei-index.pod
+++ b/Documentation/lei-index.pod
@@ -21,7 +21,7 @@ have similar functionality to L<mairix(1)> by not duplicating
messages into C<lei/store>.
Occasional invocations of C<lei-refresh-mail-sync --all=local>
-is recommended to keep indexed messages retrievable.
+are recommended to keep indexed messages retrievable.
=head1 OPTIONS
diff --git a/Documentation/lei-mail-sync-overview.pod b/Documentation/lei-mail-sync-overview.pod
index dfe1034c..e30674bb 100644
--- a/Documentation/lei-mail-sync-overview.pod
+++ b/Documentation/lei-mail-sync-overview.pod
@@ -21,7 +21,8 @@ Future work will be done to improve it and add IMAP IDLE support.
lei q L:inbox rt:last.week.. -o /tmp/results
# open /tmp/results in your favorite mail agent. If inotify or kevent
- # works, keyword changes (e.g. marking messages as `seen').
+ # works, keyword changes (e.g. marking messages as `seen') are
+ # synchronized automatically.
# If the inotify queue overflows, or if lei-daemon crashes,
# "lei index" will tell lei about keyword changes:
^ permalink raw reply related [relevance 64%]
* [PATCH] doc: lei-config: fix missing =back
@ 2021-11-01 19:00 71% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-11-01 19:00 UTC (permalink / raw)
To: meta
---
Documentation/lei-config.pod | 2 ++
1 file changed, 2 insertions(+)
diff --git a/Documentation/lei-config.pod b/Documentation/lei-config.pod
index 154295ce..663404fe 100644
--- a/Documentation/lei-config.pod
+++ b/Documentation/lei-config.pod
@@ -95,6 +95,8 @@ will also be parsed for diff coloring. git diff color slots
(C<color.diff.SLOT>) supported are C<new>, C<old>, C<meta>,
C<frag>, C<func>, and C<context>.
+=back
+
=head1 CONTACT
Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org>
^ permalink raw reply related [relevance 71%]
* [PATCH] t/lei-refresh-mail-sync: speed up test on FreeBSD 12
@ 2021-11-02 9:24 60% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-11-02 9:24 UTC (permalink / raw)
To: meta
And improve reliability while we're at it. It seems closing a
TCP listen socket on FreeBSD 12.2 doesn't cause connect()-ing
clients to fail. This happens regardless of whether a socket is
IPv4 or IPv6
This non-failure was causing tests to timeout slowly on the
client side instead of failing immediately. We now fork a new
process which does nothing but accept() + shutdown() to emulate
a dead server.
Reliability improves on all OSes since there's never a point in
time when another process can bind the socket.
---
t/lei-refresh-mail-sync.t | 35 +++++++++++++++++++----------------
1 file changed, 19 insertions(+), 16 deletions(-)
diff --git a/t/lei-refresh-mail-sync.t b/t/lei-refresh-mail-sync.t
index 43fbc50a..ea83a513 100644
--- a/t/lei-refresh-mail-sync.t
+++ b/t/lei-refresh-mail-sync.t
@@ -88,12 +88,9 @@ SKIP: {
$sock_cls //= ref($s);
my $cmd = [ "-$x", '-W0', "--stdout=$home/$x.out",
"--stderr=$home/$x.err" ];
- my $td = start_script($cmd, $env, { 3 => $s}) or xbail("-$x");
- $srv->{$x} = {
- addr => (my $scalar = tcp_host_port($s)),
- td => $td,
- cmd => $cmd,
- };
+ my $td = start_script($cmd, $env, { 3 => $s }) or xbail("-$x");
+ my $addr = tcp_host_port($s);
+ $srv->{$x} = { addr => $addr, td => $td, cmd => $cmd, s => $s };
}
my $url = "imap://$srv->{imapd}->{addr}/t.v1.0";
lei_ok 'import', $url, '+L:v1';
@@ -123,20 +120,26 @@ SKIP: {
$before = $lei_out;
delete $srv->{imapd}->{td}; # kill + join daemon
+ my $pid = fork // xbail "fork";
+ if ($pid == 0) { # dummy server to kill new connections
+ $SIG{TERM} = sub { POSIX::_exit(0) };
+ $srv->{imapd}->{s}->blocking(1);
+ while (1) {
+ my $caddr = accept(my $c, $srv->{imapd}->{s}) // next;
+ shutdown($c, 2);
+ }
+ POSIX::_exit(0);
+ }
+ my $ar = PublicInbox::AutoReap->new($pid);
ok(!(lei 'refresh-mail-sync', $url), 'URL fails on dead -imapd');
ok(!(lei 'refresh-mail-sync', '--all'), '--all fails on dead -imapd');
+ $ar->kill for qw(avoid sig wake miss-no signalfd or EVFILT_SIG);
+ $ar->join('TERM');
- # restart server (somewhat dangerous since we released the socket)
- my $listen = $sock_cls->new(
- ReuseAddr => 1,
- Proto => 'tcp',
- Type => Socket::SOCK_STREAM(),
- Listen => 1024,
- Blocking => 0,
- LocalAddr => $srv->{imapd}->{addr},
- ) or xbail "$sock_cls->new: $!";
my $cmd = $srv->{imapd}->{cmd};
- $srv->{imapd}->{td} = start_script($cmd, $env, { 3 => $listen }) or
+ my $s = $srv->{imapd}->{s};
+ $s->blocking(0);
+ $srv->{imapd}->{td} = start_script($cmd, $env, { 3 => $s }) or
xbail "@$cmd";
lei_ok 'refresh-mail-sync', '--all';
lei_ok 'inspect', "blob:$oid";
^ permalink raw reply related [relevance 60%]
* [PATCH 1/3] lei mail-diff: do not default to 'eml'
2021-11-02 18:14 71% [PATCH 0/3] lei: more coherent input handling Eric Wong
@ 2021-11-02 18:14 71% ` Eric Wong
2021-11-02 18:14 41% ` [PATCH 2/3] lei: simplify common LeiInput users with ->wq1_start Eric Wong
2021-11-02 18:14 65% ` [PATCH 3/3] lei <rediff|rm|tag>: stdin implies `-F eml' Eric Wong
2 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-11-02 18:14 UTC (permalink / raw)
To: meta
In retrospect, this doesn't make sense, since it needs at least
two messages to diff. So go about "normal" input rules and
require users to specify the format.
---
lib/PublicInbox/LeiMailDiff.pm | 1 -
1 file changed, 1 deletion(-)
diff --git a/lib/PublicInbox/LeiMailDiff.pm b/lib/PublicInbox/LeiMailDiff.pm
index b21a0c36..48ba74cf 100644
--- a/lib/PublicInbox/LeiMailDiff.pm
+++ b/lib/PublicInbox/LeiMailDiff.pm
@@ -76,7 +76,6 @@ sub input_eml_cb { # used by PublicInbox::LeiInput::input_fh
sub lei_mail_diff {
my ($lei, @argv) = @_;
- $lei->{opt}->{'in-format'} //= 'eml' if !grep(/\A[a-z0-9]+:/i, @argv);
my $self = bless {}, __PACKAGE__;
$self->prepare_inputs($lei, \@argv) or return;
my $isatty = -t $lei->{1};
^ permalink raw reply related [relevance 71%]
* [PATCH 0/3] lei: more coherent input handling
@ 2021-11-02 18:14 71% Eric Wong
2021-11-02 18:14 71% ` [PATCH 1/3] lei mail-diff: do not default to 'eml' Eric Wong
` (2 more replies)
0 siblings, 3 replies; 200+ results
From: Eric Wong @ 2021-11-02 18:14 UTC (permalink / raw)
To: meta
Hopefully everything makes more sense to new users.
Eric Wong (3):
lei mail-diff: do not default to 'eml'
lei: simplify common LeiInput users with ->wq1_start
lei <rediff|rm|tag>: stdin implies `-F eml'
lib/PublicInbox/LEI.pm | 15 ++++++++++++++-
lib/PublicInbox/LeiExportKw.pm | 7 +------
lib/PublicInbox/LeiForgetSearch.pm | 7 +------
lib/PublicInbox/LeiImport.pm | 7 +------
lib/PublicInbox/LeiLsMailSource.pm | 7 +------
lib/PublicInbox/LeiMailDiff.pm | 8 +-------
lib/PublicInbox/LeiP2q.pm | 7 +------
lib/PublicInbox/LeiRediff.pm | 7 ++-----
lib/PublicInbox/LeiRefreshMailSync.pm | 7 +------
lib/PublicInbox/LeiRm.pm | 7 ++-----
lib/PublicInbox/LeiTag.pm | 11 +++--------
lib/PublicInbox/LeiUp.pm | 5 +----
12 files changed, 29 insertions(+), 66 deletions(-)
^ permalink raw reply [relevance 71%]
* [PATCH 3/3] lei <rediff|rm|tag>: stdin implies `-F eml'
2021-11-02 18:14 71% [PATCH 0/3] lei: more coherent input handling Eric Wong
2021-11-02 18:14 71% ` [PATCH 1/3] lei mail-diff: do not default to 'eml' Eric Wong
2021-11-02 18:14 41% ` [PATCH 2/3] lei: simplify common LeiInput users with ->wq1_start Eric Wong
@ 2021-11-02 18:14 65% ` Eric Wong
2 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-11-02 18:14 UTC (permalink / raw)
To: meta
These commands are usually run on a single message, so saving
the user the trouble of typing `-F eml' on the command-line
seems reasonable. I don't think commands like "index" and
"import" will be too useful for single messages, though.
---
lib/PublicInbox/LeiRediff.pm | 2 +-
lib/PublicInbox/LeiRm.pm | 2 +-
lib/PublicInbox/LeiTag.pm | 4 ++--
3 files changed, 4 insertions(+), 4 deletions(-)
diff --git a/lib/PublicInbox/LeiRediff.pm b/lib/PublicInbox/LeiRediff.pm
index f0521bcc..c312d90f 100644
--- a/lib/PublicInbox/LeiRediff.pm
+++ b/lib/PublicInbox/LeiRediff.pm
@@ -256,7 +256,7 @@ sub lei_rediff {
($lei->{opt}->{drq} && !$lei->{opt}->{verbose}) and
$lei->{opt}->{quiet} //= 1;
$lei->_lei_store(1)->write_prepare($lei);
- $lei->{opt}->{'in-format'} //= 'eml';
+ $lei->{opt}->{'in-format'} //= 'eml' if $lei->{opt}->{stdin};
# maybe it's a non-email (code) blob from a coderepo
my $git_dirs = $lei->{opt}->{'git-dir'} //= [];
if ($lei->{opt}->{cwd} // 1) {
diff --git a/lib/PublicInbox/LeiRm.pm b/lib/PublicInbox/LeiRm.pm
index 62423ac9..00b12485 100644
--- a/lib/PublicInbox/LeiRm.pm
+++ b/lib/PublicInbox/LeiRm.pm
@@ -16,7 +16,7 @@ sub input_eml_cb { # used by PublicInbox::LeiInput::input_fh
sub lei_rm {
my ($lei, @inputs) = @_;
$lei->_lei_store(1)->write_prepare($lei);
- $lei->{opt}->{'in-format'} //= 'eml';
+ $lei->{opt}->{'in-format'} //= 'eml' if $lei->{opt}->{stdin};
my $self = bless {}, __PACKAGE__;
$self->prepare_inputs($lei, \@inputs) or return;
$lei->{-err_type} = 'non-fatal';
diff --git a/lib/PublicInbox/LeiTag.pm b/lib/PublicInbox/LeiTag.pm
index 2dc59f70..8ce96a10 100644
--- a/lib/PublicInbox/LeiTag.pm
+++ b/lib/PublicInbox/LeiTag.pm
@@ -27,8 +27,8 @@ sub pmdir_cb { # called via wq_io_do from LeiPmdir->each_mdir_fn
sub lei_tag { # the "lei tag" method
my ($lei, @argv) = @_;
- my $sto = $lei->_lei_store(1);
- $sto->write_prepare($lei);
+ $lei->{opt}->{'in-format'} //= 'eml' if $lei->{opt}->{stdin};
+ my $sto = $lei->_lei_store(1)->write_prepare($lei);
my $self = bless {}, __PACKAGE__;
$lei->ale; # refresh and prepare
my $vmd_mod = $self->vmd_mod_extract(\@argv);
^ permalink raw reply related [relevance 65%]
* [PATCH 2/3] lei: simplify common LeiInput users with ->wq1_start
2021-11-02 18:14 71% [PATCH 0/3] lei: more coherent input handling Eric Wong
2021-11-02 18:14 71% ` [PATCH 1/3] lei mail-diff: do not default to 'eml' Eric Wong
@ 2021-11-02 18:14 41% ` Eric Wong
2021-11-02 18:47 71% ` Eric Wong
2021-11-02 18:14 65% ` [PATCH 3/3] lei <rediff|rm|tag>: stdin implies `-F eml' Eric Wong
2 siblings, 1 reply; 200+ results
From: Eric Wong @ 2021-11-02 18:14 UTC (permalink / raw)
To: meta
This method replaces a common worker of starting workers,
preparing internal auth ops, and asynchronous waiting of
command completion.
It also adds missing LeiAuth support for rediff and rm
which rarely need auth.
---
lib/PublicInbox/LEI.pm | 15 ++++++++++++++-
lib/PublicInbox/LeiExportKw.pm | 7 +------
lib/PublicInbox/LeiForgetSearch.pm | 7 +------
lib/PublicInbox/LeiImport.pm | 7 +------
lib/PublicInbox/LeiLsMailSource.pm | 7 +------
lib/PublicInbox/LeiMailDiff.pm | 7 +------
lib/PublicInbox/LeiP2q.pm | 7 +------
lib/PublicInbox/LeiRediff.pm | 5 +----
lib/PublicInbox/LeiRefreshMailSync.pm | 7 +------
lib/PublicInbox/LeiRm.pm | 5 +----
lib/PublicInbox/LeiTag.pm | 7 +------
lib/PublicInbox/LeiUp.pm | 5 +----
12 files changed, 25 insertions(+), 61 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 3e1706a0..887025de 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -631,7 +631,10 @@ sub pkt_ops {
sub workers_start {
my ($lei, $wq, $jobs, $ops, $flds) = @_;
- $ops = pkt_ops($lei, { ($ops ? %$ops : ()) });
+ $ops //= {};
+ ($wq->can('net_merge_all_done') && $lei->{auth}) and
+ $lei->{auth}->op_merge($ops, $wq, $lei);
+ pkt_ops($lei, $ops);
$ops->{''} //= [ $wq->can('_lei_wq_eof') || \&wq_eof, $lei ];
my $end = $lei->pkt_op_pair;
my $ident = $wq->{-wq_ident} // "lei-$lei->{cmd} worker";
@@ -648,12 +651,22 @@ sub workers_start {
# call this when we're ready to wait on events and yield to other clients
sub wait_wq_events {
my ($lei, $op_c, $ops) = @_;
+ my $wq1 = $lei->{wq1};
+ ($wq1 && $wq1->can('net_merge_all_done') && !$lei->{auth}) and
+ $wq1->net_merge_all_done;
for my $wq (grep(defined, @$lei{qw(ikw pmd)})) { # auxiliary WQs
$wq->wq_close;
}
$op_c->{ops} = $ops;
}
+sub wq1_start {
+ my ($lei, $wq, $jobs) = @_;
+ my ($op_c, $ops) = workers_start($lei, $wq, $jobs // 1);
+ $lei->{wq1} = $wq;
+ wait_wq_events($lei, $op_c, $ops); # net_merge_all_done if !{auth}
+}
+
sub _help {
require PublicInbox::LeiHelp;
PublicInbox::LeiHelp::call($_[0], $_[1], \%CMD, \%OPTDESC);
diff --git a/lib/PublicInbox/LeiExportKw.pm b/lib/PublicInbox/LeiExportKw.pm
index 0ecfb782..d2396fa7 100644
--- a/lib/PublicInbox/LeiExportKw.pm
+++ b/lib/PublicInbox/LeiExportKw.pm
@@ -124,13 +124,8 @@ EOM
'imap_add_kw' : 'imap_set_kw');
$self->{nwr}->{-skip_creat} = 1;
}
- my $ops = {};
- $lei->{auth}->op_merge($ops, $self, $lei) if $lei->{auth};
- (my $op_c, $ops) = $lei->workers_start($self, 1, $ops);
- $lei->{wq1} = $self;
$lei->{-err_type} = 'non-fatal';
- net_merge_all_done($self) unless $lei->{auth};
- $lei->wait_wq_events($op_c, $ops); # net_merge_all_done if !{auth}
+ $lei->wq1_start($self);
}
sub _complete_export_kw {
diff --git a/lib/PublicInbox/LeiForgetSearch.pm b/lib/PublicInbox/LeiForgetSearch.pm
index dfeb0293..dd358ae1 100644
--- a/lib/PublicInbox/LeiForgetSearch.pm
+++ b/lib/PublicInbox/LeiForgetSearch.pm
@@ -45,12 +45,7 @@ sub lei_forget_search {
if ($self->{o_remote}) { # setup lei->{auth}
$self->prepare_inputs($lei, $self->{o_remote}) or return;
}
- my $ops = {};
- $lei->{auth}->op_merge($ops, $self, $lei) if $lei->{auth};
- (my $op_c, $ops) = $lei->workers_start($self, 1, $ops);
- $lei->{wq1} = $self;
- net_merge_all_done($self) unless $lei->{auth};
- $lei->wait_wq_events($op_c, $ops);
+ $lei->wq1_start($self);
}
sub do_prune {
diff --git a/lib/PublicInbox/LeiImport.pm b/lib/PublicInbox/LeiImport.pm
index d8f39fdf..bbc0634e 100644
--- a/lib/PublicInbox/LeiImport.pm
+++ b/lib/PublicInbox/LeiImport.pm
@@ -102,14 +102,9 @@ sub do_import_index ($$@) {
}
($lei->{opt}->{'new-only'} && (!$net || !$net->{imap_order})) and
warn "# --new-only is only for IMAP\n";
- my $ops = {};
- $lei->{auth}->op_merge($ops, $self, $lei) if $lei->{auth};
$lei->{-eml_noisy} = 1;
- (my $op_c, $ops) = $lei->workers_start($self, $j, $ops);
- $lei->{wq1} = $self;
$lei->{-err_type} = 'non-fatal';
- net_merge_all_done($self) unless $lei->{auth};
- $lei->wait_wq_events($op_c, $ops);
+ $lei->wq1_start($self, $j);
}
sub lei_import { # the main "lei import" method
diff --git a/lib/PublicInbox/LeiLsMailSource.pm b/lib/PublicInbox/LeiLsMailSource.pm
index 5eb7032d..50799270 100644
--- a/lib/PublicInbox/LeiLsMailSource.pm
+++ b/lib/PublicInbox/LeiLsMailSource.pm
@@ -95,13 +95,8 @@ sub lei_ls_mail_source {
$json->pretty(1)->indent(2) if $isatty || $lei->{opt}->{pretty};
}
$lei->start_pager if $isatty;
- my $ops = {};
- $lei->{auth}->op_merge($ops, $self, $lei);
- (my $op_c, $ops) = $lei->workers_start($self, 1, $ops);
- $lei->{wq1} = $self;
$lei->{-err_type} = 'non-fatal';
- net_merge_all_done($self) unless $lei->{auth};
- $lei->wait_wq_events($op_c, $ops); # net_merge_all_done if !{auth}
+ $lei->wq1_start($self);
}
sub _complete_ls_mail_source {
diff --git a/lib/PublicInbox/LeiMailDiff.pm b/lib/PublicInbox/LeiMailDiff.pm
index 48ba74cf..2b4cfd9e 100644
--- a/lib/PublicInbox/LeiMailDiff.pm
+++ b/lib/PublicInbox/LeiMailDiff.pm
@@ -81,13 +81,8 @@ sub lei_mail_diff {
my $isatty = -t $lei->{1};
$lei->{opt}->{color} //= $isatty;
$lei->start_pager if $isatty;
- my $ops = {};
- $lei->{auth}->op_merge($ops, $self, $lei) if $lei->{auth};
- (my $op_c, $ops) = $lei->workers_start($self, 1, $ops);
- $lei->{wq1} = $self;
$lei->{-err_type} = 'non-fatal';
- net_merge_all_done($self) unless $lei->{auth};
- $lei->wait_wq_events($op_c, $ops);
+ $lei->wq1_start($self);
}
no warnings 'once';
diff --git a/lib/PublicInbox/LeiP2q.pm b/lib/PublicInbox/LeiP2q.pm
index 09ec0a07..610adb78 100644
--- a/lib/PublicInbox/LeiP2q.pm
+++ b/lib/PublicInbox/LeiP2q.pm
@@ -183,12 +183,7 @@ sub lei_p2q { # the "lei patch-to-query" entry point
$lei->{opt}->{'in-format'} //= 'eml' if $lei->{opt}->{stdin};
my $self = bless { missing_ok => 1 }, __PACKAGE__;
$self->prepare_inputs($lei, \@inputs) or return;
- my $ops = {};
- $lei->{auth}->op_merge($ops, $self, $lei) if $lei->{auth};
- (my $op_c, $ops) = $lei->workers_start($self, 1, $ops);
- $lei->{wq1} = $self;
- net_merge_all_done($self) unless $lei->{auth};
- $lei->wait_wq_events($op_c, $ops);
+ $lei->wq1_start($self);
}
sub ipc_atfork_child {
diff --git a/lib/PublicInbox/LeiRediff.pm b/lib/PublicInbox/LeiRediff.pm
index 56c457fc..f0521bcc 100644
--- a/lib/PublicInbox/LeiRediff.pm
+++ b/lib/PublicInbox/LeiRediff.pm
@@ -279,10 +279,7 @@ sub lei_rediff {
my $isatty = -t $lei->{1};
$lei->{opt}->{color} //= $isatty;
$lei->start_pager if $isatty;
- my ($op_c, $ops) = $lei->workers_start($self, 1);
- $lei->{wq1} = $self;
- net_merge_all_done($self) unless $lei->{auth};
- $lei->wait_wq_events($op_c, $ops);
+ $lei->wq1_start($self);
}
sub ipc_atfork_child {
diff --git a/lib/PublicInbox/LeiRefreshMailSync.pm b/lib/PublicInbox/LeiRefreshMailSync.pm
index f516f572..7821008f 100644
--- a/lib/PublicInbox/LeiRefreshMailSync.pm
+++ b/lib/PublicInbox/LeiRefreshMailSync.pm
@@ -81,13 +81,8 @@ EOM
my $self = bless { missing_ok => 1, lms => $lms }, __PACKAGE__;
$lei->{opt}->{'mail-sync'} = 1; # for prepare_inputs
$self->prepare_inputs($lei, \@folders) or return;
- my $ops = {};
- $lei->{auth}->op_merge($ops, $self, $lei) if $lei->{auth};
- (my $op_c, $ops) = $lei->workers_start($self, 1, $ops);
- $lei->{wq1} = $self;
$lei->{-err_type} = 'non-fatal';
- net_merge_all_done($self) unless $lei->{auth};
- $lei->wait_wq_events($op_c, $ops); # net_merge_all_done if !{auth}
+ $lei->wq1_start($self);
}
sub ipc_atfork_child { # needed for PublicInbox::LeiPmdir
diff --git a/lib/PublicInbox/LeiRm.pm b/lib/PublicInbox/LeiRm.pm
index cc1abbff..62423ac9 100644
--- a/lib/PublicInbox/LeiRm.pm
+++ b/lib/PublicInbox/LeiRm.pm
@@ -19,11 +19,8 @@ sub lei_rm {
$lei->{opt}->{'in-format'} //= 'eml';
my $self = bless {}, __PACKAGE__;
$self->prepare_inputs($lei, \@inputs) or return;
- my ($op_c, $ops) = $lei->workers_start($self, 1);
- $lei->{wq1} = $self;
$lei->{-err_type} = 'non-fatal';
- net_merge_all_done($self) unless $lei->{auth};
- $lei->wait_wq_events($op_c, $ops);
+ $lei->wq1_start($self);
}
no warnings 'once';
diff --git a/lib/PublicInbox/LeiTag.pm b/lib/PublicInbox/LeiTag.pm
index d64a9f86..2dc59f70 100644
--- a/lib/PublicInbox/LeiTag.pm
+++ b/lib/PublicInbox/LeiTag.pm
@@ -37,13 +37,8 @@ sub lei_tag { # the "lei tag" method
$self->prepare_inputs($lei, \@argv) or return;
grep(defined, @$vmd_mod{qw(+kw +L -L -kw)}) or
return $lei->fail('no keywords or labels specified');
- my $ops = {};
- $lei->{auth}->op_merge($ops, $self, $lei) if $lei->{auth};
- (my $op_c, $ops) = $lei->workers_start($self, 1, $ops);
- $lei->{wq1} = $self;
$lei->{-err_type} = 'non-fatal';
- net_merge_all_done($self) unless $lei->{auth};
- $lei->wait_wq_events($op_c, $ops);
+ $lei->wq1_start($self);
}
sub note_unimported {
diff --git a/lib/PublicInbox/LeiUp.pm b/lib/PublicInbox/LeiUp.pm
index 79639d5e..66d950b2 100644
--- a/lib/PublicInbox/LeiUp.pm
+++ b/lib/PublicInbox/LeiUp.pm
@@ -136,10 +136,7 @@ EOM
if ($lei->{auth}) { # start auth worker
require PublicInbox::NetWriter;
bless $lei->{net}, 'PublicInbox::NetWriter';
- $lei->{auth}->op_merge(my $ops = {}, $self, $lei);
- (my $op_c, $ops) = $lei->workers_start($self, 1, $ops);
- $lei->{wq1} = $self;
- $lei->wait_wq_events($op_c, $ops);
+ $lei->wq1_start($self);
# net_merge_all_done will fire when auth is done
} else {
redispatch_all($self, $lei); # see below
^ permalink raw reply related [relevance 41%]
* Re: [PATCH 2/3] lei: simplify common LeiInput users with ->wq1_start
2021-11-02 18:14 41% ` [PATCH 2/3] lei: simplify common LeiInput users with ->wq1_start Eric Wong
@ 2021-11-02 18:47 71% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-11-02 18:47 UTC (permalink / raw)
To: meta
Eric Wong <e@80x24.org> wrote:
> This method replaces a common worker of starting workers,
s/common worker/common pattern/ :x
^ permalink raw reply [relevance 71%]
* [PATCH] doc: lei-q: document SEARCH TERMS prefixes
@ 2021-11-02 23:55 43% Eric Wong
2021-11-03 0:28 71% ` Kyle Meyer
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2021-11-02 23:55 UTC (permalink / raw)
To: meta
The new Documentation/common.perl file will be used for
all manpages in the future.
---
Documentation/common.perl | 65 +++++++++++++++++++++++++++++++++++++++
Documentation/include.mk | 3 ++
Documentation/lei-q.pod | 51 +++++++++++++++++++++++++++++-
MANIFEST | 1 +
lib/PublicInbox/Search.pm | 8 ++---
5 files changed, 123 insertions(+), 5 deletions(-)
create mode 100755 Documentation/common.perl
diff --git a/Documentation/common.perl b/Documentation/common.perl
new file mode 100755
index 000000000000..5fdbe8da055e
--- /dev/null
+++ b/Documentation/common.perl
@@ -0,0 +1,65 @@
+#!perl -w
+# Copyright (C) all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+use strict;
+use Fcntl qw(SEEK_SET);
+my $have_search = eval { require PublicInbox::Search; 1 };
+my $addr = 'meta@public-inbox.org';
+for my $pod (@ARGV) {
+ open my $fh, '+<', $pod or die "open($pod): $!";
+ my $s = do { local $/; <$fh> } // die "read $!";
+ my $orig = $s;
+ $s =~ s!^=head1 COPYRIGHT\n.+?^=head1([^\n]+)\n!=head1 COPYRIGHT
+
+Copyright all contributors L<mailto:$addr>
+
+License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>
+
+=head1$1
+ !ms;
+
+ $s =~ s!^=head1 CONTACT\n.+?^=head1([^\n]+)\n!=head1 CONTACT
+
+Feedback welcome via plain-text mail to L<mailto:$addr>
+
+The mail archives are hosted at L<https://public-inbox.org/meta/> and
+L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/>
+
+=head1$1
+ !ms;
+ $have_search and $s =~ s!^=for\scomment\n
+ ^AUTO-GENERATED-SEARCH-TERMS-BEGIN\n
+ .+?
+ ^=for\scomment\n
+ ^AUTO-GENERATED-SEARCH-TERMS-END\n
+ !search_terms()!emsx;
+ $s =~ s/[ \t]+$//sgm;
+ next if $s eq $orig;
+ seek($fh, 0, SEEK_SET) or die "seek: $!";
+ truncate($fh, 0) or die "truncate: $!";
+ print $fh $s or die "print: $!";
+ close $fh or die "close: $!";
+}
+
+sub search_terms {
+ my $help = eval('\@PublicInbox::Search::HELP');
+ my $s = '';
+ my $pad = 0;
+ my $i;
+ for ($i = 0; $i < @$help; $i += 2) {
+ my $pfx = $help->[$i];
+ my $n = length($pfx);
+ $pad = $n if $n > $pad;
+ $s .= $pfx . "\0";
+ $s .= $help->[$i + 1];
+ $s .= "\f\n";
+ }
+ $pad += 2;
+ my $padding = ' ' x ($pad + 4);
+ $s =~ s/^/$padding/gms;
+ $s =~ s/^$padding(\S+)\0/" $1".(' ' x ($pad - length($1)))/egms;
+ $s =~ s/\f\n/\n/gs;
+ $s =~ s/^ //gms;
+ substr($s, 0, 0, "=for comment\nAUTO-GENERATED-SEARCH-TERMS-BEGIN\n\n");
+ $s .= "\n=for comment\nAUTO-GENERATED-SEARCH-TERMS-END\n";
+}
diff --git a/Documentation/include.mk b/Documentation/include.mk
index 5f3ffcc56937..149f9e3ce42e 100644
--- a/Documentation/include.mk
+++ b/Documentation/include.mk
@@ -80,6 +80,9 @@ Documentation/flow.txt : Documentation/flow.ge
touch -r Documentation/flow.ge $@+
mv $@+ $@
+Documentation/lei-q.pod : lib/PublicInbox/Search.pm Documentation/common.perl
+ $(PERL) -I lib -w Documentation/common.perl $@
+
NEWS NEWS.atom NEWS.html : $(news_deps)
$(PERL) -I lib -w Documentation/mknews.perl $@ $(RELEASES)
diff --git a/Documentation/lei-q.pod b/Documentation/lei-q.pod
index 574c12eb9d3a..fe281d7b0121 100644
--- a/Documentation/lei-q.pod
+++ b/Documentation/lei-q.pod
@@ -245,6 +245,55 @@ Default: C<auto>
=back
+=head1 SEARCH TERMS
+
+C<lei q> supports the same search prefixes used by HTTP(S) public-inbox
+instances:
+
+=for comment
+AUTO-GENERATED-SEARCH-TERMS-BEGIN
+
+ s: match within Subject e.g. s:"a quick brown fox"
+ d: match date-time range, git "approxidate" formats supported
+ Open-ended ranges such as `d:last.week..' and
+ `d:..2.days.ago' are supported
+ b: match within message body, including text attachments
+ nq: match non-quoted text within message body
+ q: match quoted text within message body
+ n: match filename of attachment(s)
+ t: match within the To header
+ c: match within the Cc header
+ f: match within the From header
+ a: match within the To, Cc, and From headers
+ tc: match within the To and Cc headers
+ l: match contents of the List-Id header
+ bs: match within the Subject and body
+ dfn: match filename from diff
+ dfa: match diff removed (-) lines
+ dfb: match diff added (+) lines
+ dfhh: match diff hunk header context (usually a function name)
+ dfctx: match diff context lines
+ dfpre: match pre-image git blob ID
+ dfpost: match post-image git blob ID
+ dfblob: match either pre or post-image git blob ID
+ rt: match received time, like `d:' if sender's clock was correct
+
+=for comment
+AUTO-GENERATED-SEARCH-TERMS-END
+
+Additional search prefixes which only affect the local lei/store:
+
+ L: match the given label
+ kw: match the given keywords
+
+See L<lei-tag(1)> for more info on labels and keywords.
+
+Most prefixes are probabilistic, meaning they support stemming
+and wildcards (C<*>). Ranges (such as C<d:>) and boolean prefixes
+do not support stemming or wildcards.
+The upstream Xapian query parser documentation fully explains
+the query syntax: L<https://xapian.org/docs/queryparser.html>
+
=head1 TIPS
C<-f reply> is intended to aid in turning a cover letter
@@ -261,7 +310,7 @@ L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/>
=head1 COPYRIGHT
-Copyright 2021 all contributors L<mailto:meta@public-inbox.org>
+Copyright all contributors L<mailto:meta@public-inbox.org>
License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>
diff --git a/MANIFEST b/MANIFEST
index 1e8f60fb41da..47eadad2e403 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -13,6 +13,7 @@ Documentation/RelNotes/v1.6.0.eml
Documentation/RelNotes/v1.6.1.eml
Documentation/RelNotes/v1.7.0.wip
Documentation/clients.txt
+Documentation/common.perl
Documentation/dc-dlvr-spam-flow.txt
Documentation/design_notes.txt
Documentation/design_www.txt
diff --git a/lib/PublicInbox/Search.pm b/lib/PublicInbox/Search.pm
index 600e6400d4b6..523003b3c269 100644
--- a/lib/PublicInbox/Search.pm
+++ b/lib/PublicInbox/Search.pm
@@ -155,9 +155,9 @@ my %prob_prefix = (
our @HELP = (
's:' => 'match within Subject e.g. s:"a quick brown fox"',
'd:' => <<EOF,
-match date range, git "approxidate" formats supported
-Open-ended ranges such as `d:last.week..' and `d:..2.days.ago'
-are supported
+match date-time range, git "approxidate" formats supported
+Open-ended ranges such as `d:last.week..' and
+`d:..2.days.ago' are supported
EOF
'b:' => 'match within message body, including text attachments',
'nq:' => 'match non-quoted text within message body',
@@ -179,7 +179,7 @@ EOF
'dfpost:' => 'match post-image git blob ID',
'dfblob:' => 'match either pre or post-image git blob ID',
'rt:' => <<EOF,
-match received time, like `d:' unless sender's clock was broken
+match received time, like `d:' if sender's clock was correct
EOF
);
chomp @HELP;
^ permalink raw reply related [relevance 43%]
* Re: [PATCH] doc: lei-q: document SEARCH TERMS prefixes
2021-11-02 23:55 43% [PATCH] doc: lei-q: document SEARCH TERMS prefixes Eric Wong
@ 2021-11-03 0:28 71% ` Kyle Meyer
0 siblings, 0 replies; 200+ results
From: Kyle Meyer @ 2021-11-03 0:28 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
Eric Wong writes:
> +=for comment
> +AUTO-GENERATED-SEARCH-TERMS-BEGIN
Looks great, thanks. By the way, this takes care of a to-do in this
file:
=for comment
TODO: Give common prefixes, or at least a description/reference.
^ permalink raw reply [relevance 71%]
* Initial Fedora packaging for lei
@ 2021-11-03 1:28 71% Konstantin Ryabitsev
0 siblings, 0 replies; 200+ results
From: Konstantin Ryabitsev @ 2021-11-03 1:28 UTC (permalink / raw)
To: meta; +Cc: tools
[-- Attachment #1: Type: text/plain, Size: 622 bytes --]
Hi, all:
I did some initial work to package lei for Fedora 34 and 35 (out today). The
lei parts should be ready to use, though I'll continue to work on the server
parts (only needed if you're running httpd/nntpd/imapd daemons).
For now, you'll need to enable my copr repository to use it:
sudo dnf copr enable icon/b4
sudo dnf install lei
(you can also install python3-b4 from there if you're using b4)
I'm still working on some introductory docs, but you can use the following
resource to get yourself going with search-based lei goodness:
https://josefbacik.github.io/kernel/2021/10/18/lei-and-b4.html
-K
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]
^ permalink raw reply [relevance 71%]
* [PATCH 2/3] doc: -clone|lei add-external: add bit about the Makefile
@ 2021-11-03 8:34 68% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-11-03 8:34 UTC (permalink / raw)
To: meta
It's pretty useful, I think.
---
Documentation/lei-add-external.pod | 2 ++
Documentation/public-inbox-clone.pod | 5 +++++
2 files changed, 7 insertions(+)
diff --git a/Documentation/lei-add-external.pod b/Documentation/lei-add-external.pod
index 1ab65a1650a6..1761eed1ed78 100644
--- a/Documentation/lei-add-external.pod
+++ b/Documentation/lei-add-external.pod
@@ -29,6 +29,8 @@ Default: 0
=item --mirror=URL
Create C<LOCATION> by mirroring the public-inbox at C<URL>.
+C<LOCATION> will have a Makefile with a C<make update>
+target to update the external.
=item --epoch=RANGE
diff --git a/Documentation/public-inbox-clone.pod b/Documentation/public-inbox-clone.pod
index efee01eec592..103260913d92 100644
--- a/Documentation/public-inbox-clone.pod
+++ b/Documentation/public-inbox-clone.pod
@@ -18,6 +18,11 @@ L<public-inbox-index(1)>. Those commands must be run separately
if serving/searching the mirror is required. As-is,
public-inbox-clone is suitable for creating a git-only backup.
+public-inbox-clone creates a Makefile with handy targets to update the
+inbox once indexed. This Makefile may be edited by the user; it will
+not be rewritten by L<public-inbox-fetch(1)> unless it is removed
+completely.
+
public-inbox-clone does not use nor require any extra
configuration files (not even C<~/.public-inbox/config>).
^ permalink raw reply related [relevance 68%]
* lei: incorrect quoting on saved searches (was Re: lore+lei: getting started)
[not found] ` <CAL_JsqJBh1O3H2-P07AHzVq0x89BoP_N6P=rT5up6=3QyF_B0Q@mail.gmail.com>
@ 2021-11-08 20:22 86% ` Konstantin Ryabitsev
2021-11-08 20:49 90% ` Eric Wong
2021-11-08 20:53 90% ` Rob Herring
0 siblings, 2 replies; 200+ results
From: Konstantin Ryabitsev @ 2021-11-08 20:22 UTC (permalink / raw)
To: Rob Herring; +Cc: meta
On Mon, Nov 08, 2021 at 01:49:07PM -0600, Rob Herring wrote:
Moving this to meta.
> > lei q -I https://lore.kernel.org/all/ -o ~/Mail/floppy \
> > --threads --dedupe=mid \
> > '(dfn:drivers/block/floppy.c OR dfhh:floppy_* OR s:floppy \
> > OR ((nq:bug OR nq:regression) AND nq:floppy)) \
> > AND rt:1.month.ago..'
>
> I tried a similar one which I had working as a bookmark:
>
> $ lei q -I https://lore.kernel.org/all/ -o ~/Mail/my-patches
> --threads --dedupe=mid '(dfn:drivers OR dfn:arch OR dfn:Documentation
> OR dfn:include OR dfn:scripts) AND f:robh@kernel.org'
> # /home/rob/.local/share/lei/store 0/0
> # /usr/bin/curl -Sf -s -d ''
> https://lore.kernel.org/all/?x=m&t=1&q=(dfn%3A%22drivers+OR+dfn%3Aarch+OR+dfn%3ADocumentation+OR+dfn%3Ainclude+OR+dfn%3Ascripts)+AND+f%3Arobh%40kernel.org%22
> # 0 written to /home/rob/Mail/my-patches/ (0 matches)
It's true, I get the same thing if I omit "AND rt:" at the end.
$ lei q -I https://lore.kernel.org/all/ -o ~/work/temp/lei/robh-patches --threads --dedupe=mid '(dfn:drivers OR dfn:arch OR dfn:Documentation OR dfn:include OR dfn:scripts) AND f:robh@kernel.org'
# /home/user/.local/share/lei/store 0/0
# /usr/bin/curl -Sf -s -d '' https://lore.kernel.org/all/?x=m&t=1&q=(dfn%3A%22drivers+OR+dfn%3Aarch+OR+dfn%3ADocumentation+OR+dfn%3Ainclude+OR+dfn%3Ascripts)+AND+f%3Arobh%40kernel.org%22
# 0 written to /home/user/work/temp/lei/robh-patches/ (0 matches)
$ lei forget-search ~/work/temp/lei/robh-patches
$ lei q -I https://lore.kernel.org/all/ -o ~/work/temp/lei/robh-patches --threads --dedupe=mid '(dfn:drivers OR dfn:arch OR dfn:Documentation OR dfn:include OR dfn:scripts) AND f:robh@kernel.org AND rt:1.month.ago..'
# /usr/bin/curl -Sf -s -d '' https://lore.kernel.org/all/?x=m&t=1&q=(dfn%3Adrivers+OR+dfn%3Aarch+OR+dfn%3ADocumentation+OR+dfn%3Ainclude+OR+dfn%3Ascripts)+AND+f%3Arobh%40kernel.org+AND+rt%3A1633724105..
# /home/user/.local/share/lei/store 13/13
# https://lore.kernel.org/all/ 65/?
# https://lore.kernel.org/all/ 75/75
# 45 written to /home/user/work/temp/lei/robh-patches/ (88 matches)
> It seems there is some problem in quoting. Notice the '%22' that's
> inserted in the url.
Deferring to Eric here.
> Also, the above query is a bit of a work-around as what I really want
> is just all patches from me. I haven't been able to get something to
> work. I've tried things like 'dfn:*' or 'dfn:/' or 'dfn:b/'.
I think 's:patch AND nq:diff' is a good option here.
-K
^ permalink raw reply [relevance 86%]
* Re: lei: incorrect quoting on saved searches (was Re: lore+lei: getting started)
2021-11-08 20:22 86% ` lei: incorrect quoting on saved searches (was Re: lore+lei: getting started) Konstantin Ryabitsev
@ 2021-11-08 20:49 90% ` Eric Wong
2021-11-08 21:36 90% ` Konstantin Ryabitsev
2021-11-08 20:53 90% ` Rob Herring
1 sibling, 1 reply; 200+ results
From: Eric Wong @ 2021-11-08 20:49 UTC (permalink / raw)
To: Rob Herring; +Cc: Konstantin Ryabitsev, meta, workflows
Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> On Mon, Nov 08, 2021 at 01:49:07PM -0600, Rob Herring wrote:
>
> Moving this to meta.
I don't think workflows should've been dropped, though.
> > > lei q -I https://lore.kernel.org/all/ -o ~/Mail/floppy \
> > > --threads --dedupe=mid \
> > > '(dfn:drivers/block/floppy.c OR dfhh:floppy_* OR s:floppy \
> > > OR ((nq:bug OR nq:regression) AND nq:floppy)) \
> > > AND rt:1.month.ago..'
> >
> > I tried a similar one which I had working as a bookmark:
That's actually treating the entire single-quoted section as
a phrase search for Xapian.
The correct way to use '(', ')', and '*' on the command-line for
Xapian is to shell escape them:
lei q -I https://lore.kernel.org/all/ -o ~/Mail/floppy \
--threads --dedupe=mid \
\( dfn:drivers/block/floppy.c OR dfhh:floppy_\* OR s:floppy \
OR \(\(nq:bug OR nq:regression\) AND nq:floppy\)\) \
AND rt:1.month.ago...
Since shell escaping so many metacharacters is annoying,
stdin is supported (and implicit iff file|pipe):
echo '(dfn:drivers/block/floppy.c OR dfhh:floppy_* OR s:floppy \
OR ((nq:bug OR nq:regression) AND nq:floppy)) \
AND rt:1.month.ago..' | \
lei q -I https://lore.kernel.org/all/ -o ~/Mail/floppy \
--threads --dedupe=mid
^ permalink raw reply [relevance 90%]
* Re: lei: incorrect quoting on saved searches (was Re: lore+lei: getting started)
2021-11-08 20:22 86% ` lei: incorrect quoting on saved searches (was Re: lore+lei: getting started) Konstantin Ryabitsev
2021-11-08 20:49 90% ` Eric Wong
@ 2021-11-08 20:53 90% ` Rob Herring
1 sibling, 0 replies; 200+ results
From: Rob Herring @ 2021-11-08 20:53 UTC (permalink / raw)
To: Konstantin Ryabitsev; +Cc: meta
On Mon, Nov 8, 2021 at 2:22 PM Konstantin Ryabitsev
<konstantin@linuxfoundation.org> wrote:
>
> On Mon, Nov 08, 2021 at 01:49:07PM -0600, Rob Herring wrote:
>
> Moving this to meta.
>
> > > lei q -I https://lore.kernel.org/all/ -o ~/Mail/floppy \
> > > --threads --dedupe=mid \
> > > '(dfn:drivers/block/floppy.c OR dfhh:floppy_* OR s:floppy \
> > > OR ((nq:bug OR nq:regression) AND nq:floppy)) \
> > > AND rt:1.month.ago..'
> >
> > I tried a similar one which I had working as a bookmark:
> >
> > $ lei q -I https://lore.kernel.org/all/ -o ~/Mail/my-patches
> > --threads --dedupe=mid '(dfn:drivers OR dfn:arch OR dfn:Documentation
> > OR dfn:include OR dfn:scripts) AND f:robh@kernel.org'
> > # /home/rob/.local/share/lei/store 0/0
> > # /usr/bin/curl -Sf -s -d ''
> > https://lore.kernel.org/all/?x=m&t=1&q=(dfn%3A%22drivers+OR+dfn%3Aarch+OR+dfn%3ADocumentation+OR+dfn%3Ainclude+OR+dfn%3Ascripts)+AND+f%3Arobh%40kernel.org%22
> > # 0 written to /home/rob/Mail/my-patches/ (0 matches)
>
> It's true, I get the same thing if I omit "AND rt:" at the end.
>
> $ lei q -I https://lore.kernel.org/all/ -o ~/work/temp/lei/robh-patches --threads --dedupe=mid '(dfn:drivers OR dfn:arch OR dfn:Documentation OR dfn:include OR dfn:scripts) AND f:robh@kernel.org'
> # /home/user/.local/share/lei/store 0/0
> # /usr/bin/curl -Sf -s -d '' https://lore.kernel.org/all/?x=m&t=1&q=(dfn%3A%22drivers+OR+dfn%3Aarch+OR+dfn%3ADocumentation+OR+dfn%3Ainclude+OR+dfn%3Ascripts)+AND+f%3Arobh%40kernel.org%22
> # 0 written to /home/user/work/temp/lei/robh-patches/ (0 matches)
> $ lei forget-search ~/work/temp/lei/robh-patches
> $ lei q -I https://lore.kernel.org/all/ -o ~/work/temp/lei/robh-patches --threads --dedupe=mid '(dfn:drivers OR dfn:arch OR dfn:Documentation OR dfn:include OR dfn:scripts) AND f:robh@kernel.org AND rt:1.month.ago..'
> # /usr/bin/curl -Sf -s -d '' https://lore.kernel.org/all/?x=m&t=1&q=(dfn%3Adrivers+OR+dfn%3Aarch+OR+dfn%3ADocumentation+OR+dfn%3Ainclude+OR+dfn%3Ascripts)+AND+f%3Arobh%40kernel.org+AND+rt%3A1633724105..
> # /home/user/.local/share/lei/store 13/13
> # https://lore.kernel.org/all/ 65/?
> # https://lore.kernel.org/all/ 75/75
> # 45 written to /home/user/work/temp/lei/robh-patches/ (88 matches)
>
> > It seems there is some problem in quoting. Notice the '%22' that's
> > inserted in the url.
>
> Deferring to Eric here.
>
> > Also, the above query is a bit of a work-around as what I really want
> > is just all patches from me. I haven't been able to get something to
> > work. I've tried things like 'dfn:*' or 'dfn:/' or 'dfn:b/'.
>
> I think 's:patch AND nq:diff' is a good option here.
Not even close really. That mainly finds my replies with 'diff' in
them. I'm not sure why, but it misses most actual patches:
https://lore.kernel.org/all/?q=s%3Apatch+nq%3Adiff+f%3Arobh%40kernel.org
Rob
^ permalink raw reply [relevance 90%]
* Re: lei: incorrect quoting on saved searches (was Re: lore+lei: getting started)
2021-11-08 20:49 90% ` Eric Wong
@ 2021-11-08 21:36 90% ` Konstantin Ryabitsev
2021-11-08 21:48 90% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Konstantin Ryabitsev @ 2021-11-08 21:36 UTC (permalink / raw)
To: Eric Wong; +Cc: Rob Herring, meta, workflows
On Mon, Nov 08, 2021 at 08:49:23PM +0000, Eric Wong wrote:
> Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> > On Mon, Nov 08, 2021 at 01:49:07PM -0600, Rob Herring wrote:
> >
> > Moving this to meta.
>
> I don't think workflows should've been dropped, though.
>
> > > > lei q -I https://lore.kernel.org/all/ -o ~/Mail/floppy \
> > > > --threads --dedupe=mid \
> > > > '(dfn:drivers/block/floppy.c OR dfhh:floppy_* OR s:floppy \
> > > > OR ((nq:bug OR nq:regression) AND nq:floppy)) \
> > > > AND rt:1.month.ago..'
> > >
> > > I tried a similar one which I had working as a bookmark:
>
> That's actually treating the entire single-quoted section as
> a phrase search for Xapian.
Hmm... I noticed that when I `lei edit-search` the initial query that was
causing quoting issues, I get the following:
[lei]
q = (dfn:drivers OR dfn:arch OR dfn:Documentation OR dfn:include OR dfn:scripts) AND f:robh@kernel.org
So, the extra quotes didn't get added to the config file. Running `lei up` on
that saved search seems to do the right thing, so the erroneous quotes are
only added during the initial `lei q` call.
> The correct way to use '(', ')', and '*' on the command-line for
> Xapian is to shell escape them:
But putting them into single quotes should accomplish the same result, no? At
least, that's how I've always understood shell escaping.
-K
^ permalink raw reply [relevance 90%]
* Re: lei: incorrect quoting on saved searches (was Re: lore+lei: getting started)
2021-11-08 21:36 90% ` Konstantin Ryabitsev
@ 2021-11-08 21:48 90% ` Eric Wong
2021-11-08 22:36 88% ` Konstantin Ryabitsev
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2021-11-08 21:48 UTC (permalink / raw)
To: Konstantin Ryabitsev; +Cc: Rob Herring, meta, workflows
Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> On Mon, Nov 08, 2021 at 08:49:23PM +0000, Eric Wong wrote:
> > Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> > > On Mon, Nov 08, 2021 at 01:49:07PM -0600, Rob Herring wrote:
> >
> > > > > lei q -I https://lore.kernel.org/all/ -o ~/Mail/floppy \
> > > > > --threads --dedupe=mid \
> > > > > '(dfn:drivers/block/floppy.c OR dfhh:floppy_* OR s:floppy \
> > > > > OR ((nq:bug OR nq:regression) AND nq:floppy)) \
> > > > > AND rt:1.month.ago..'
> > > >
> > > > I tried a similar one which I had working as a bookmark:
> >
> > That's actually treating the entire single-quoted section as
> > a phrase search for Xapian.
>
> Hmm... I noticed that when I `lei edit-search` the initial query that was
> causing quoting issues, I get the following:
>
> [lei]
> q = (dfn:drivers OR dfn:arch OR dfn:Documentation OR dfn:include OR dfn:scripts) AND f:robh@kernel.org
>
> So, the extra quotes didn't get added to the config file. Running `lei up` on
> that saved search seems to do the right thing, so the erroneous quotes are
> only added during the initial `lei q` call.
Right, each entry in lei.q is actually an entry in argv[].
So the correct query should look something like:
[lei]
q = (
q = dfn:drivers/block/floppy.c
q = OR
...
> > The correct way to use '(', ')', and '*' on the command-line for
> > Xapian is to shell escape them:
>
> But putting them into single quotes should accomplish the same result, no? At
> least, that's how I've always understood shell escaping.
Yeah, that works, too. As long as spaces/tabs don't show up
within each argv[] element, it won't be interpreted as a phrase.
I really wanted: lei q s:"a quick brown fox"
to work from a shell like it would in the WWW UI;
and thus deprioritized '(' and ')' working properly :x
^ permalink raw reply [relevance 90%]
* Re: lei: incorrect quoting on saved searches (was Re: lore+lei: getting started)
2021-11-08 21:48 90% ` Eric Wong
@ 2021-11-08 22:36 88% ` Konstantin Ryabitsev
2021-11-08 22:57 90% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Konstantin Ryabitsev @ 2021-11-08 22:36 UTC (permalink / raw)
To: Eric Wong; +Cc: Rob Herring, meta, workflows
On Mon, Nov 08, 2021 at 09:48:36PM +0000, Eric Wong wrote:
> > Hmm... I noticed that when I `lei edit-search` the initial query that was
> > causing quoting issues, I get the following:
> >
> > [lei]
> > q = (dfn:drivers OR dfn:arch OR dfn:Documentation OR dfn:include OR dfn:scripts) AND f:robh@kernel.org
> >
> > So, the extra quotes didn't get added to the config file. Running `lei up` on
> > that saved search seems to do the right thing, so the erroneous quotes are
> > only added during the initial `lei q` call.
>
> Right, each entry in lei.q is actually an entry in argv[].
> So the correct query should look something like:
So, to be clear here... the following doesn't work because instead of multiple
query parameters to 'lei q' the single-quoted string becomes a single
parameter?
lei q -I https://lore.kernel.org/all/ -o ~/work/temp/lei/robh-patches \
--threads --dedupe=mid \
'(dfn:drivers OR dfn:arch OR dfn:Documentation OR dfn:include OR dfn:scripts) AND f:robh@kernel.org'
Any way to make this work? I find that it's more easily readable than the
"echo | lei q" version.
For bash users, the following should work as well:
lei q -I https://lore.kernel.org/all/ -o ~/work/temp/lei/robh-patches \
--threads --dedupe=mid <<< \
'(dfn:drivers OR dfn:arch OR dfn:Documentation OR dfn:include OR dfn:scripts) AND f:robh@kernel.org'
Suggestion, can -I accept the URL containing the query, so that the command
becomes:
lei q -o ~/mail/foo --threads --dedupe=mid -I \
https://lore.kernel.org/all/?q=f%3Atorvalds+AND+nq%3Agarbage
This way we pass both the location of the extindex to query AND the parameters
we should use, avoiding shell quoting problems?
-K
^ permalink raw reply [relevance 88%]
* Re: lei: incorrect quoting on saved searches (was Re: lore+lei: getting started)
2021-11-08 22:36 88% ` Konstantin Ryabitsev
@ 2021-11-08 22:57 90% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-11-08 22:57 UTC (permalink / raw)
To: Konstantin Ryabitsev; +Cc: Rob Herring, meta, workflows
Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> On Mon, Nov 08, 2021 at 09:48:36PM +0000, Eric Wong wrote:
> > > Hmm... I noticed that when I `lei edit-search` the initial query that was
> > > causing quoting issues, I get the following:
> > >
> > > [lei]
> > > q = (dfn:drivers OR dfn:arch OR dfn:Documentation OR dfn:include OR dfn:scripts) AND f:robh@kernel.org
> > >
> > > So, the extra quotes didn't get added to the config file. Running `lei up` on
> > > that saved search seems to do the right thing, so the erroneous quotes are
> > > only added during the initial `lei q` call.
> >
> > Right, each entry in lei.q is actually an entry in argv[].
> > So the correct query should look something like:
>
> So, to be clear here... the following doesn't work because instead of multiple
> query parameters to 'lei q' the single-quoted string becomes a single
> parameter?
>
> lei q -I https://lore.kernel.org/all/ -o ~/work/temp/lei/robh-patches \
> --threads --dedupe=mid \
> '(dfn:drivers OR dfn:arch OR dfn:Documentation OR dfn:include OR dfn:scripts) AND f:robh@kernel.org'
>
> Any way to make this work? I find that it's more easily readable than the
> "echo | lei q" version.
I can't think of a way to make it work w/o breaking phrase searches
(or asking users to use both single and double-quotes):
lei q 's:"a quick brown fox"' # yuck
> For bash users, the following should work as well:
>
> lei q -I https://lore.kernel.org/all/ -o ~/work/temp/lei/robh-patches \
> --threads --dedupe=mid <<< \
> '(dfn:drivers OR dfn:arch OR dfn:Documentation OR dfn:include OR dfn:scripts) AND f:robh@kernel.org'
Oh, not sure about bash and <<<; but this heredoc should work with any
POSIX sh:
lei q -I https://lore.kernel.org/all/ \
-o ~/work/temp/lei/robh-patches \
--threads --dedupe=mid <<'EOM'
(dfn:drivers OR dfn:arch OR dfn:Documentation OR dfn:include OR dfn:scripts)
AND f:robh@kernel.org
EOM
> Suggestion, can -I accept the URL containing the query, so that the command
> becomes:
>
> lei q -o ~/mail/foo --threads --dedupe=mid -I \
> https://lore.kernel.org/all/?q=f%3Atorvalds+AND+nq%3Agarbage
>
> This way we pass both the location of the extindex to query AND the parameters
> we should use, avoiding shell quoting problems?
Maybe, but URI escaping bothers the heck out of me, too.
I guess the heredoc example is actually good...
^ permalink raw reply [relevance 90%]
* [PATCH] lei q|up: fix saved searches for single-phrase search
@ 2021-11-08 23:39 53% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-11-08 23:39 UTC (permalink / raw)
To: meta
`"' (double-quote) needs to be quoted for stdin searches.
We also need to differentiate between "lei q --stdin" usage
when calling "lei up", do it by setting an internal "rawstr"
knob to ensure we can parse the config properly regardless
of whether the initial search used --stdin or not.
---
lib/PublicInbox/LeiSavedSearch.pm | 5 ++++
lib/PublicInbox/LeiUp.pm | 10 +++++---
t/lei-q-save.t | 41 +++++++++++++++++++++++++++++++
3 files changed, 52 insertions(+), 4 deletions(-)
diff --git a/lib/PublicInbox/LeiSavedSearch.pm b/lib/PublicInbox/LeiSavedSearch.pm
index b2f1ad10..1d13aef6 100644
--- a/lib/PublicInbox/LeiSavedSearch.pm
+++ b/lib/PublicInbox/LeiSavedSearch.pm
@@ -21,6 +21,7 @@ my %cquote = ("\n" => '\\n', "\t" => '\\t', "\b" => '\\b');
sub cquote_val ($) { # cf. git-config(1)
my ($val) = @_;
$val =~ s/([\n\t\b])/$cquote{$1}/g;
+ $val =~ s/\"/\\\"/g;
$val;
}
@@ -162,6 +163,10 @@ EOM
my $val = $lei->{opt}->{$k} // next;
print $fh "\t$k = $val\n";
}
+ $lei->{opt}->{stdin} and print $fh <<EOM;
+[lei "internal"]
+ rawstr = 1 # stdin was used initially
+EOM
close($fh) or return $lei->fail("close $f: $!");
$self->{lock_path} = "$self->{-f}.flock";
$self->{-ovf} = "$dir/over.sqlite3";
diff --git a/lib/PublicInbox/LeiUp.pm b/lib/PublicInbox/LeiUp.pm
index 66d950b2..d7873a3f 100644
--- a/lib/PublicInbox/LeiUp.pm
+++ b/lib/PublicInbox/LeiUp.pm
@@ -26,13 +26,15 @@ sub up1 ($$) {
my $lss = PublicInbox::LeiSavedSearch->up($lei, $out) or return;
my $f = $lss->{'-f'};
my $mset_opt = $lei->{mset_opt} = { relevance => -2 };
- my $q = $mset_opt->{q_raw} = $lss->{-cfg}->{'lei.q'} //
+ my $q = $lss->{-cfg}->get_all('lei.q') //
die("lei.q unset in $f (out=$out)\n");
my $lse = $lei->{lse} // die 'BUG: {lse} missing';
- if (ref($q)) {
- $mset_opt->{qstr} = $lse->query_argv_to_string($lse->git, $q);
+ if ($lss->{-cfg}->{'lei.internal.rawstr'}) {
+ scalar(@$q) > 1 and
+ die "$f: lei.q has multiple values (@$q) (out=$out)\n";
+ $lse->query_approxidate($lse->git, $mset_opt->{qstr} = $q->[0]);
} else {
- $lse->query_approxidate($lse->git, $mset_opt->{qstr} = $q);
+ $mset_opt->{qstr} = $lse->query_argv_to_string($lse->git, $q);
}
# n.b. only a few CLI args are accepted for "up", so //= usually sets
for my $k ($lss->ARRAY_FIELDS) {
diff --git a/t/lei-q-save.t b/t/lei-q-save.t
index cd35461c..3d09fe37 100644
--- a/t/lei-q-save.t
+++ b/t/lei-q-save.t
@@ -240,5 +240,46 @@ test_lei(sub {
lei_ok qw(forget-search --prune);
lei_ok qw(ls-search);
unlike($lei_out, qr!\Q$home/after\E!, "`after' pruned");
+
+ my $d = "$home/d";
+ lei_ok [qw(import -q -F eml)], undef,
+ {0 => \"Subject: do not call\n\n"};
+ lei_ok qw(q -o), $d, 's:do not call';
+
+ my @orig = glob("$d/*/*");
+ is(scalar(@orig), 1, 'got one message via argv');
+ lei_ok [qw(import -q -Feml)], undef,
+ {0 => \"Subject: do not ever call\n\n"};
+ lei_ok 'up', $d;
+ is_deeply([glob("$d/*/*")], \@orig, 'nothing written');
+ lei_ok [qw(import -q -Feml)], undef,
+ {0 => \"Subject: do not call, ever\n\n"};
+ lei_ok 'up', $d;
+ @after = glob("$d/*/*");
+ is(scalar(@after), 2, '2 total, messages, now');
+ is_deeply([glob("$d/cur/*")], \@orig, 'cur untouched');
+ my @new = glob("$d/new/*");
+ is(scalar(@new), 1, "new message written to `new'");
+ is(eml_load($new[0])->header('Subject'), 'do not call, ever',
+ 'up retrieved correct message');
+
+ $d = "$home/d-stdin";
+ lei_ok [ qw(q -q -o), $d ], undef, { 0 => \'s:"do not ever call"' };
+ @orig = glob("$d/*/*");
+ is(scalar(@orig), 1, 'got one message via stdin');
+
+ lei_ok [qw(import -q -Feml)], undef,
+ {0 => \"Subject: do not fall or ever call\n\n"};
+ lei_ok [qw(import -q -Feml)], undef,
+ {0 => \"Subject: do not ever call, again\n\n"};
+ lei_ok 'up', $d;
+ @new = glob("$d/new/*");
+ is(scalar(@new), 1, "new message written to `new'") or do {
+ for (@new) { diag "$_ ".eml_load($_)->header('Subject') }
+ };
+ is_deeply([glob("$d/cur/*")], \@orig, 'cur untouched');
+ is(eml_load($new[0])->header('Subject'), 'do not ever call, again',
+ 'up retrieved correct message');
+
});
done_testing;
^ permalink raw reply related [relevance 53%]
* [PATCH 0/4] "lei q" rawstr fallout + fixes
@ 2021-11-10 10:25 71% Eric Wong
2021-11-10 10:28 68% ` [PATCH 2/4] lei up: infer rawstr from old searches via trailing "\n" Eric Wong
` (2 more replies)
0 siblings, 3 replies; 200+ results
From: Eric Wong @ 2021-11-10 10:25 UTC (permalink / raw)
To: meta
The recent rawstr fix caused some errors for existing
--stdin searches, these patches fix them and make some
minor improvements along the way.
Eric Wong (4):
ipc: note failing sub name
lei up: infer rawstr from old searches via trailing "\n"
lei q: disallow "\n" in argv[] elements
lei q: make HTTP(S) query strings even less ugly
lib/PublicInbox/IPC.pm | 2 +-
lib/PublicInbox/LeiQuery.pm | 1 +
lib/PublicInbox/LeiUp.pm | 4 +++-
lib/PublicInbox/LeiXSearch.pm | 2 +-
t/lei.t | 3 +++
5 files changed, 9 insertions(+), 3 deletions(-)
^ permalink raw reply [relevance 71%]
* [PATCH 4/4] lei q: make HTTP(S) query strings even less ugly
2021-11-10 10:25 71% [PATCH 0/4] "lei q" rawstr fallout + fixes Eric Wong
2021-11-10 10:28 68% ` [PATCH 2/4] lei up: infer rawstr from old searches via trailing "\n" Eric Wong
2021-11-10 10:28 70% ` [PATCH 3/4] lei q: disallow "\n" in argv[] elements Eric Wong
@ 2021-11-10 10:28 71% ` Eric Wong
2 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-11-10 10:28 UTC (permalink / raw)
To: meta
Following commit 57fed2e4b78ed394 (lei: normalize whitespace in
remote queries, 2021-09-11), leaving the trailing `\n' from
stdin queries to be normalized to ` ' (SP) causes it to appear
as `+' in URLs, which Xapian ignores.
---
lib/PublicInbox/LeiXSearch.pm | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/lib/PublicInbox/LeiXSearch.pm b/lib/PublicInbox/LeiXSearch.pm
index 29df07e0c8a8..2958d3f910b0 100644
--- a/lib/PublicInbox/LeiXSearch.pm
+++ b/lib/PublicInbox/LeiXSearch.pm
@@ -342,7 +342,7 @@ sub query_remote_mboxrd {
local $SIG{TERM} = sub { exit(0) }; # for DESTROY (File::Temp, $reap)
my $lei = $self->{lei};
my $opt = $lei->{opt};
- my $qstr = $lei->{mset_opt}->{qstr};
+ chomp(my $qstr = $lei->{mset_opt}->{qstr});
$qstr =~ s/[ \n\t]+/ /sg; # make URLs less ugly
my @qform = (x => 'm');
push(@qform, t => 1) if $opt->{threads};
^ permalink raw reply related [relevance 71%]
* [PATCH 3/4] lei q: disallow "\n" in argv[] elements
2021-11-10 10:25 71% [PATCH 0/4] "lei q" rawstr fallout + fixes Eric Wong
2021-11-10 10:28 68% ` [PATCH 2/4] lei up: infer rawstr from old searches via trailing "\n" Eric Wong
@ 2021-11-10 10:28 70% ` Eric Wong
2021-11-10 10:28 71% ` [PATCH 4/4] lei q: make HTTP(S) query strings even less ugly Eric Wong
2 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-11-10 10:28 UTC (permalink / raw)
To: meta
I don't expect this to be hit in real-world use via normal
interactive shells. However, somebody could accidentally add
"\n" in languages (e.g. Perl, C) where it's easy to pass "\n"
in argv[].
---
lib/PublicInbox/LeiQuery.pm | 1 +
t/lei.t | 3 +++
2 files changed, 4 insertions(+)
diff --git a/lib/PublicInbox/LeiQuery.pm b/lib/PublicInbox/LeiQuery.pm
index 352ee60131aa..51ee3d9c83e4 100644
--- a/lib/PublicInbox/LeiQuery.pm
+++ b/lib/PublicInbox/LeiQuery.pm
@@ -141,6 +141,7 @@ no query allowed on command-line with --stdin
PublicInbox::InputPipe::consume($self->{0}, \&qstr_add, $self);
return;
}
+ chomp(@argv) and $self->qerr("# trailing `\\n' removed");
$mset_opt{q_raw} = [ @argv ]; # copy
$mset_opt{qstr} =
$self->{lse}->query_argv_to_string($self->{lse}->git, \@argv);
diff --git a/t/lei.t b/t/lei.t
index f7de1b711a83..b10c9b59c72b 100644
--- a/t/lei.t
+++ b/t/lei.t
@@ -143,6 +143,9 @@ my $test_fail = sub {
lei('-C', '/dev/null', 'q', 'whatever');
is($? >> 8, 1, 'chdir at beginning fails to /dev/null');
+ lei_ok('q', "foo\n");
+ like($lei_err, qr/trailing `\\n' removed/s, "noted `\\n' removal");
+
for my $lk (qw(ei inbox)) {
my $d = "$home/newline\n$lk";
mkdir $d;
^ permalink raw reply related [relevance 70%]
* [PATCH 2/4] lei up: infer rawstr from old searches via trailing "\n"
2021-11-10 10:25 71% [PATCH 0/4] "lei q" rawstr fallout + fixes Eric Wong
@ 2021-11-10 10:28 68% ` Eric Wong
2021-11-10 10:28 70% ` [PATCH 3/4] lei q: disallow "\n" in argv[] elements Eric Wong
2021-11-10 10:28 71% ` [PATCH 4/4] lei q: make HTTP(S) query strings even less ugly Eric Wong
2 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-11-10 10:28 UTC (permalink / raw)
To: meta
For --stdin searches created prior to commit 666dde69a3f6 (lei
q|up: fix saved searches for single-phrase search, 2021-11-08)
we still want to be able to run "lei up" on them without
regressions. So assume nobody manages to enter "\n" as an
argv[] element and consider the presence of "\n" as a previous
--stdin use.
This fixes errors from "lei up" such as:
lei_xsearch 2 wq_worker: Exception: Key too long: length was 840 bytes,
maximum length of a key is 255 bytes at ../PublicInbox/IPC.pm line 250.
Fixes: 666dde69a3f6 ("lei q|up: fix saved searches for single-phrase search")
---
lib/PublicInbox/LeiUp.pm | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/lib/PublicInbox/LeiUp.pm b/lib/PublicInbox/LeiUp.pm
index d7873a3f3469..b8a9836075ba 100644
--- a/lib/PublicInbox/LeiUp.pm
+++ b/lib/PublicInbox/LeiUp.pm
@@ -29,7 +29,9 @@ sub up1 ($$) {
my $q = $lss->{-cfg}->get_all('lei.q') //
die("lei.q unset in $f (out=$out)\n");
my $lse = $lei->{lse} // die 'BUG: {lse} missing';
- if ($lss->{-cfg}->{'lei.internal.rawstr'}) {
+ my $rawstr = $lss->{-cfg}->{'lei.internal.rawstr'} //
+ (scalar(@$q) == 1 && substr($q->[0], -1) eq "\n");
+ if ($rawstr) {
scalar(@$q) > 1 and
die "$f: lei.q has multiple values (@$q) (out=$out)\n";
$lse->query_approxidate($lse->git, $mset_opt->{qstr} = $q->[0]);
^ permalink raw reply related [relevance 68%]
* [PATCH] t/lei-watch: test with with higher sleep
@ 2021-11-10 10:33 71% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-11-10 10:33 UTC (permalink / raw)
To: meta
0.1s may not be enough for a task switch and inotify wakeup,
so try doubling it and see if it fixes test reliability, for
now. A future change may be to implement a watcher/tracer
for inotify -> lei/store events.
Link: https://public-inbox.org/meta/20211104134327.zrf5jijfz7dsvb7l@meerkat.local/
---
t/lei-watch.t | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/t/lei-watch.t b/t/lei-watch.t
index 9158571c..e6066033 100644
--- a/t/lei-watch.t
+++ b/t/lei-watch.t
@@ -52,7 +52,7 @@ test_lei(sub {
my @f = glob("$md/cur/*:2,");
is(scalar(@f), 1, 'got populated maildir with one result');
rename($f[0], "$f[0]S") or xbail "rename $!"; # set (S)een
- tick($have_fast_inotify ? 0.1 : 2.1); # always needed for 1 CPU systems
+ tick($have_fast_inotify ? 0.2 : 2.2); # always needed for 1 CPU systems
lei_ok qw(note-event done); # flushes immediately (instead of 5s)
lei_ok qw(q mid:testmessage@example.com -o), $md2, '-I', "$ro_home/t1";
^ permalink raw reply related [relevance 71%]
* [PATCH] lei forget-search: add help for --prune
@ 2021-11-12 11:08 71% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-11-12 11:08 UTC (permalink / raw)
To: meta
This enables tab-completion, since I'm using --prune quite a bit
and my fingers are about to fall off :<
---
lib/PublicInbox/LEI.pm | 3 +++
1 file changed, 3 insertions(+)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 887025de5ded..192f267ca1dd 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -411,6 +411,9 @@ my %OPTDESC = (
'url ls-mail-source' => 'show full URL of newsgroup or IMAP folder',
'format|f=s ls-external' => $ls_format,
+'prune:s forget-search' =>
+ ['TYPE|local|remote', 'prune all, remote or local folders' ],
+
'limit|n=i@' => ['NUM', 'limit on number of matches (default: 10000)' ],
'offset=i' => ['OFF', 'search result offset (default: 0)'],
^ permalink raw reply related [relevance 71%]
* lei spawns mua before results are written
@ 2021-11-14 20:41 71% Leah Neukirchen
2021-11-15 0:31 71% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Leah Neukirchen @ 2021-11-14 20:41 UTC (permalink / raw)
To: meta
Hi,
I tried building some tooling to use lei with mblaze and stumbled upon
the fact that lei seems to spawn the mua before all mails are written
into the -o maildir; at least, adding a "sleep 1" in front of my
script fixes it. Any way to wait for it? I guess I could just run lei
and my mua scripts afterwards.
Thanks,
--
Leah Neukirchen <leah@vuxu.org> https://leahneukirchen.org/
^ permalink raw reply [relevance 71%]
* Re: lei spawns mua before results are written
2021-11-14 20:41 71% lei spawns mua before results are written Leah Neukirchen
@ 2021-11-15 0:31 71% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-11-15 0:31 UTC (permalink / raw)
To: Leah Neukirchen; +Cc: meta
Leah Neukirchen <leah@vuxu.org> wrote:
> I tried building some tooling to use lei with mblaze and stumbled upon
> the fact that lei seems to spawn the mua before all mails are written
> into the -o maildir; at least, adding a "sleep 1" in front of my
> script fixes it. Any way to wait for it? I guess I could just run lei
> and my mua scripts afterwards.
Yes, it's intentional to allow Maildir and IMAP to allow MUAs to
work progressively while loading similar to how less(1) and
typical web browsers work. It's not really practical to do for
mbox* because of locking and potential incompatible locking
protocols.
You should be able to use --alert=COMMAND to run any arbitrary
command(s) (including an MUA, as an alternative to --mua).
I'm not sure if adding a --no-early-mua switch is needed. My
gut reaction is having yet another switch to document increases
the learning curve and support overhead... *shrug*
^ permalink raw reply [relevance 71%]
* RFC: should lei inject its own "Received:" header?
@ 2021-11-19 20:49 71% Konstantin Ryabitsev
2021-11-21 10:13 71% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Konstantin Ryabitsev @ 2021-11-19 20:49 UTC (permalink / raw)
To: meta
Hello:
I wonder if lei should inject its own "received"-like header on writing to a
maildir/imap target -- to indicate where the copy of the email came from. I
don't think it should use the actual Received: header, as this may cause some
weird SPF/DMARC issues, but perhaps something like:
X-Lei-Received: from https://lore.kernel.org/all/; Fri, 19 Nov 2021 14:32:44 -0500
Just a random thought.
-K
^ permalink raw reply [relevance 71%]
* lei q error after deleting cache (can't find mail_sync.sqlite3)
@ 2021-11-20 16:36 71% ` Johannes Altmanninger
0 siblings, 0 replies; 200+ results
From: Johannes Altmanninger @ 2021-11-20 16:36 UTC (permalink / raw)
To: meta
On Sat, Nov 20, 2021 at 04:30:57PM +0100, Johannes Altmanninger wrote:
> I've been using lei from the 1.7.0 release, which works great.
>
> I had some unrelated problems with one of the created maildirs so I deleted
> said maildir. Now I can't recreate the maildir. I tried deleting all
> persistent data but got an error:
>
> $ rm -rf the-maildir ~/.local/share/lei ~/.cache/{public-inbox,lei}
> $ lei q -I https://lore.kernel.org/git/ -o the-maildir rt:1.week.ago..
> failed to open /home/johannes/.cache/lei/all_locals_ever.git/lei_ale.state: No such file or directory
> at /usr/share/perl5/vendor_perl/PublicInbox/LeiQuery.pm line 117.
>
> A quick look at strace doesn't show access to any other files in $HOME
This is pretty weird: it works in a temporary home, but only for the first time.
rm -rf the-maildir someuniquename; HOME=$PWD/someuniquename lei q -I https://public-inbox.org/meta/ -o the-maildir rt:1.day.ago..
When I re-run this command, I get the same error
There seems to be no mention of someuniquename anywhere on my disk (except
in my shell history).
^ permalink raw reply [relevance 71%]
* Re: RFC: should lei inject its own "Received:" header?
2021-11-19 20:49 71% RFC: should lei inject its own "Received:" header? Konstantin Ryabitsev
@ 2021-11-21 10:13 71% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-11-21 10:13 UTC (permalink / raw)
To: Konstantin Ryabitsev; +Cc: meta
Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> Hello:
>
> I wonder if lei should inject its own "received"-like header on writing to a
> maildir/imap target -- to indicate where the copy of the email came from. I
> don't think it should use the actual Received: header, as this may cause some
> weird SPF/DMARC issues, but perhaps something like:
>
> X-Lei-Received: from https://lore.kernel.org/all/; Fri, 19 Nov 2021 14:32:44 -0500
>
> Just a random thought.
Probably not in the blob itself. Addition of extra headers
means slower and less-effective dedupe (not just for lei, but
also for deduplicating FSes (which operate at block-layer) and
potential FUSE implementation).
Perhaps the mail_sync.sqlite3 stuff could be taught to track the
origin of a blob, though...
^ permalink raw reply [relevance 71%]
* [PATCH] t/lei-mirror: skip lei comparisons if lei missing
@ 2021-11-22 7:42 96% ` Eric Wong
2021-11-22 13:47 99% ` Jörg Rödel
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2021-11-22 7:42 UTC (permalink / raw)
To: Jörg Rödel; +Cc: meta
Eric Wong <e@80x24.org> wrote:
> Will work on a patch to ensure lei ran successfully.
---------8<--------
Subject: [PATCH] t/lei-mirror: skip lei comparisons if lei missing
We can't compare created_at times with lei if lei tests are
skipped due to Inline::C or Socket::MsgHdr unavailability.
Reported-by: Jörg Rödel <joro@8bytes.org>
Link: https://public-inbox.org/meta/YZebmAxlFJy4lqAw@8bytes.org/
---
t/lei-mirror.t | 17 +++++++++++------
1 file changed, 11 insertions(+), 6 deletions(-)
diff --git a/t/lei-mirror.t b/t/lei-mirror.t
index 646ff2b19f98..d6fa6db5e3cb 100644
--- a/t/lei-mirror.t
+++ b/t/lei-mirror.t
@@ -170,12 +170,17 @@ SKIP: {
local $ENV{HOME} = $tmpdir;
ok(run_script([qw(-index -Lbasic), "$d/t1"]), 'index v1');
ok(run_script([qw(-index -Lbasic), "$d/t2"]), 'index v2');
- my $f = "$d/t1/public-inbox/msgmap.sqlite3";
- my $ca = PublicInbox::Msgmap->new_file($f)->created_at;
- is($ca, $created{v1}, 'clone + index v1 synced ->created_at');
- $f = "$d/t2/msgmap.sqlite3";
- $ca = PublicInbox::Msgmap->new_file($f)->created_at;
- is($ca, $created{v2}, 'clone + index v1 synced ->created_at');
+
+ SKIP: {
+ skip "lei didn't run", 2 if join(sort keys %created) ne 'v1v2';
+ my $f = "$d/t1/public-inbox/msgmap.sqlite3";
+ my $ca = PublicInbox::Msgmap->new_file($f)->created_at;
+ is($ca, $created{v1}, 'clone + index v1 synced ->created_at');
+
+ $f = "$d/t2/msgmap.sqlite3";
+ $ca = PublicInbox::Msgmap->new_file($f)->created_at;
+ is($ca, $created{v2}, 'clone + index v2 synced ->created_at');
+ }
test_lei(sub {
lei_ok qw(inspect num:1 --dir), "$d/t1";
ok(ref(json_utf8->decode($lei_out)), 'inspect num: on v1');
^ permalink raw reply related [relevance 96%]
* Re: [PATCH] t/lei-mirror: skip lei comparisons if lei missing
2021-11-22 7:42 96% ` [PATCH] t/lei-mirror: skip lei comparisons if lei missing Eric Wong
@ 2021-11-22 13:47 99% ` Jörg Rödel
2021-11-22 17:24 99% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Jörg Rödel @ 2021-11-22 13:47 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
On Mon, Nov 22, 2021 at 07:42:41AM +0000, Eric Wong wrote:
> t/lei-mirror.t | 17 +++++++++++------
> 1 file changed, 11 insertions(+), 6 deletions(-)
This seems to cause warnings:
[ 56s] t/lei-lcat.t ................. ok
[ 57s] Useless use of sort in scalar context at t/lei-mirror.t line 175.
[ 59s] Use of uninitialized value in join or string at t/lei-mirror.t line 175.
[ 59s] t/lei-mirror.t ............... ok
Regards,
Joerg
^ permalink raw reply [relevance 99%]
* Re: [PATCH] t/lei-mirror: skip lei comparisons if lei missing
2021-11-22 13:47 99% ` Jörg Rödel
@ 2021-11-22 17:24 99% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-11-22 17:24 UTC (permalink / raw)
To: Jörg Rödel; +Cc: meta
Jörg Rödel <joro@8bytes.org> wrote:
> This seems to cause warnings:
>
> [ 56s] t/lei-lcat.t ................. ok
> [ 57s] Useless use of sort in scalar context at t/lei-mirror.t line 175.
> [ 59s] Use of uninitialized value in join or string at t/lei-mirror.t line 175.
> [ 59s] t/lei-mirror.t ............... ok
Oops :x I'll squash this in before pushing:
diff --git a/t/lei-mirror.t b/t/lei-mirror.t
index d6fa6db5e3cb..32a5b0390714 100644
--- a/t/lei-mirror.t
+++ b/t/lei-mirror.t
@@ -172,7 +172,8 @@ SKIP: {
ok(run_script([qw(-index -Lbasic), "$d/t2"]), 'index v2');
SKIP: {
- skip "lei didn't run", 2 if join(sort keys %created) ne 'v1v2';
+ join('', sort(keys %created)) eq 'v1v2' or
+ skip "lei didn't run", 2;
my $f = "$d/t1/public-inbox/msgmap.sqlite3";
my $ca = PublicInbox::Msgmap->new_file($f)->created_at;
is($ca, $created{v1}, 'clone + index v1 synced ->created_at');
^ permalink raw reply related [relevance 99%]
* [PATCH] lei: always use 3-arg open perlop
@ 2021-11-22 18:38 71% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-11-22 18:38 UTC (permalink / raw)
To: meta
Future-proofing in case future versions of Perl warn on this, since
2-arg forms of open may be subject to injection vulnerabilities
with non-literal args.
---
lib/PublicInbox/LEI.pm | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 192f267ca1dd..4e0295fa4e8a 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -818,7 +818,8 @@ sub dispatch {
next if $d eq ''; # same as git(1)
chdir $d or return fail($self, "cd $d: $!");
}
- open $self->{3}, '.' or return fail($self, "open . $!");
+ open $self->{3}, '<', '.' or
+ return fail($self, "open . $!");
}
$cb->($self, @argv);
} elsif (grep(/\A-/, $cmd, @argv)) { # --help or -h only
^ permalink raw reply related [relevance 71%]
* lei q: importing messages when specifying '-f *json*'?
@ 2021-12-30 23:04 60% Kyle Meyer
0 siblings, 0 replies; 200+ results
From: Kyle Meyer @ 2021-12-30 23:04 UTC (permalink / raw)
To: meta; +Cc: piem
The lei-q interface I'm working on for piem [1] consumes JSON output to
display search results. From there, an individual message can be shown.
If the query is against a remote external, that involves two curl calls
to the remote, one for the search and one for the display.
I'd like to import hits from the first step into the local store so that
I can avoid the second curl call. However, I haven't been able to
figure out a way to do this when requesting JSON output.
For example [2], say I don't have meta's 20211124154539.350522-1-e@80x24.org
locally:
$ lei daemon-kill
$ export HOME=$(mktemp -d "${TMPDIR:-/tmp}"/pi-testing-XXXXXXX)
$ lei q m:20211124154539.350522-1-e@80x24.org
# /tmp/pi-testing-3JaSz2K/.config/lei/config created
[null]
If I search for that message against https://public-inbox.org/meta/ and
request JSON output
$ lei q -I https://public-inbox.org/meta/ -f ldjson \
m:20211124154539.350522-1-e@80x24.org
# /usr/bin/curl -Sf -s -d '' https://public-inbox.org/meta/?x=m&q=m%3A20211124154539.350522-1-e%4080x24.org
{"blob":"a8754283bd9e985d6e1156215071be59aa2b5a53",...}
then no message ends up in the local store:
$ lei q m:20211124154539.350522-1-e@80x24.org
[null]
In contrast, if I request mboxrd output
$ lei q -I https://public-inbox.org/meta/ -f mboxrd \
m:20211124154539.350522-1-e@80x24.org >/dev/null
# /usr/bin/curl -Sf -s -d '' https://public-inbox.org/meta/?x=m&q=m%3A20211124154539.350522-1-e%4080x24.org
the message is imported to the local store:
$ lei q -f ldjson m:20211124154539.350522-1-e@80x24.org
{"blob":"a8754283bd9e985d6e1156215071be59aa2b5a53",...}
$ git -C $HOME/.local/share/lei/store/local/0.git/ log --oneline
ca12a1b (HEAD -> master) [PATCH] eliminate some unused subs
I was hoping that --import-remote might do the trick, but that doesn't
seem to be the case:
$ lei daemon-kill
$ export HOME=$(mktemp -d "${TMPDIR:-/tmp}"/pi-testing-XXXXXXX)
$ lei q --import-remote -I https://public-inbox.org/meta/ \
-f ldjson m:20211124154539.350522-1-e@80x24.org
# /tmp/pi-testing-Ny7KDcB/.config/lei/config created
# /usr/bin/curl -Sf -s -d '' https://public-inbox.org/meta/?x=m&q=m%3A20211124154539.350522-1-e%4080x24.org
{"blob":"a8754283bd9e985d6e1156215071be59aa2b5a53",...}
$ lei q m:20211124154539.350522-1-e@80x24.org
[null]
Should --import-remote trigger an import in the case above?
[1] https://git.kyleam.com/piem/tree/piem-lei.el
[2] These are with public-inbox's current master (07cd8973baf).
^ permalink raw reply [relevance 60%]
* Using public-inbox+lei+Emacs+mu+mu4e (was: Large delays in mailing list delivery?)
@ 2022-02-02 9:34 48% ` Ævar Arnfjörð Bjarmason
2022-02-07 21:27 69% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Ævar Arnfjörð Bjarmason @ 2022-02-02 9:34 UTC (permalink / raw)
To: Eric Wong; +Cc: Konstantin Ryabitsev, Elijah Newren, git, Derrick Stolee, meta
On Mon, Dec 06 2021, Eric Wong wrote:
> Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:
>> On Fri, Dec 03 2021, Konstantin Ryabitsev wrote:
>>
>> > On Fri, Dec 03, 2021 at 09:02:48PM +0100, Ævar Arnfjörð Bjarmason wrote:
>> >> When I've experienced delays (sometimes of half a day or more) both
>> >> https://public-inbox.org/git/ and https://lore.kernel.org/git/ have been
>> >> updated.
>> >
>> > Btw, you can source lore.kernel.org straight into your gmail inbox. :)
>> >
>> > https://people.kernel.org/monsieuricon/lore-lei-part-1-getting-started
>> > https://people.kernel.org/monsieuricon/lore-lei-part-2-now-with-imap
>> >
>> > Or, you can read it via nntp://nntp.lore.kernel.org/.
>>
>> [CC'd meta@public-inbox.org, probably best to move this thread over
>> there sooner than later, but CC'ing git@ still in case this is
>> interesting to others]
>>
>> I poked a bit at setting this up but couldn't find from building
>> public-inbox.org & trying to page through the docs how I'd get from an
>> existing public-inbox.org/git/ checkout to a local Maildir.
>
> Existing, public-inboxes can be set as "externals" and managed
> via {add,forget,ls}-external sub-commands:
>
> # for locally-cloned inboxes:
> public-inbox-index /path/to/existing/inbox
> lei add-external /path/to/existing/inbox
>
> # relies on curl, memoizes data downloaded for each search:
> lei add-external https://yhbt.net/lore/git
>
> Local externals will be included by every "lei q" invocation;
> HTTP(S) ones require "lei q --remote"
>
> If you only want to use an external as a one-off without adding
> it, the -I/--include and -O/--only flags are useful:
>
> lei q -O https://yhbt.net/lore/git -o /tmp/results SEARCH_TERMS
>
>> If you could share some recipe or a pointer to the right docs for that
>> that would be much appreciated. Thanks!
>
> lei-overview(7) manpage documents some things, at least:
> https://public-inbox.org/lei-overview.html Patches welcome :>
>
> IMHO lei still kinda sucks, and I probably won't have time to
> work on it for a bit :<
Thanks. I finally got around to setting this up.
The above instructions didn't quite work for me, but here's what I did
(in the form of a script lifted from a screen(1) config I've
got). Indented with un-indented comments. The "stuff" is screen's way of
"run this command" (or well, input these characters):
I had a ~/g/git-ml clone already, but this makes one:
stuff "cd ~/g/git-ml || git clone https://public-inbox.org/git ~/g/git-ml^M"
The initial index:
## This will create a .git/publici-inbox in ~/g/git-ml. Takes a while
## the first time.
stuff "time public-inbox-index -v \$PWD^M"
I fiddled with this for a bit because it refused to work, turns out it
was missing the .git at the end, i.e. it expected a bare repo[1]:
## When we add the lei external it *must* have the ".git" part,
## because it'll try to find the "public-inbox" folder at wherever we
## point it.
stuff "test -d ~/.config/lei || lei add-external ~/g/git-ml/.git^M"
A bit of a UX wart not to be able to specify no --limit, or maybe I'm
missing a way:
## The one-off massive import of the Git ML. TODO: No way to specify
## an infinite limit? Not --no-limit or --limit=0.
stuff "test -d ~/Maildir/lei-q-git-ml || time lei q --limit=999999999 -v -o ~/Maildir/lei-q-git-ml l:git.vger.kernel.org^M"
The initial indexing:
## After the one-off import this will take forever *the first time*
## (or around 20m), but subsequent invocations will be fast:
stuff "time lei up ~/Maildir/lei-q-git-ml^M"
Runs an ad-hoc script to keep it up-to-date, which is quoted below:
## Run it in a loop
stuff "public-inbox-lei-pull-index^M"
That script (which I whipped up just now. Is there a better/more
standard way? to keep a public-inbox+lei pair up-to-date with
sleep/backoff etc?
#!/bin/sh
set -xe
repo=~/g/git-ml
while true
do
oid=$(git -C $repo rev-parse HEAD)
git -C $repo pull
noid=$(git -C $repo rev-parse HEAD)
if test "$oid" = "$noid"
then
echo Nothing to update
sleep 60
continue
fi
(
cd $repo &&
public-inbox-index -v "$PWD"
)
lei up ~/Maildir/lei-q-git-ml
sleep 1
done
I use Emacs+mu4e for my E-Mail. And since I index ~/Maildir having these
files dropped in there will be added to its index. Then I just changed
my saved search to also look through that maildir (I guess the entire
first condition could be dropped, but whatever):
"(maildir:/personal-gmail/* OR maildir:/lei-q-git-ml/*) AND list:git.vger.kernel.org OR recip:git@vger.kernel.org OR recip:git-packagers@googlegroups.com"
Because "mu" is generally good about de-duplicating stuff I've now got
an inbox with mixed messages I can sync from GMail (including my "Sent"
folder), and I get up-to-the-minute ML traffic now (it's bee 10hrs-4day
delayed for 3-4 months at least).
So new messages are generally from the "lei" directory, but when I send
one it'll be dropped in the personal-gmail.
I still need to check if it's doing the wrong thing with e.g. "read"
flags if I read a mail synced via lei that later arrives in GMail. But I
mostly don't use "read" statuses anyway...
1. Maybe this "I only tested if it complied" patch would make sense to catch that?
diff --git a/lib/PublicInbox/LeiXSearch.pm b/lib/PublicInbox/LeiXSearch.pm
index 2958d3f9..be49621f 100644
--- a/lib/PublicInbox/LeiXSearch.pm
+++ b/lib/PublicInbox/LeiXSearch.pm
@@ -613,7 +613,7 @@ sub add_uri {
}
}
-sub prepare_external {
+sub _prepare_external {
my ($self, $loc, $boost) = @_; # n.b. already ordered by boost
if (ref $loc) { # already a URI, or PublicInbox::Inbox-like object
return add_uri($self, $loc) if $loc->can('scheme');
@@ -638,6 +638,14 @@ sub prepare_external {
push @{$self->{locals}}, $loc;
}
+sub prepare_external {
+ my ($self, $loc, $boost) = @_;
+ my $ret = _prepare_external($self, $loc, $boost);
+ warn "W: we got nothing from $loc, did you mean $loc/.git?"
+ if !$ret && -e "$loc/.git";
+ return $ret;
+}
+
sub _lcat_i { # LeiMailSync->each_src iterator callback
my ($oidbin, $id, $each_smsg) = @_;
$each_smsg->({blob => unpack('H*', $oidbin), pct => 100});
^ permalink raw reply related [relevance 48%]
* Re: Using public-inbox+lei+Emacs+mu+mu4e (was: Large delays in mailing list delivery?)
2022-02-02 9:34 48% ` Using public-inbox+lei+Emacs+mu+mu4e (was: Large delays in mailing list delivery?) Ævar Arnfjörð Bjarmason
@ 2022-02-07 21:27 69% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2022-02-07 21:27 UTC (permalink / raw)
To: Ævar Arnfjörð Bjarmason
Cc: Konstantin Ryabitsev, Elijah Newren, git, Derrick Stolee, meta
Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:
> On Mon, Dec 06 2021, Eric Wong wrote:
> > Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:
> >> On Fri, Dec 03 2021, Konstantin Ryabitsev wrote:
> >>
> >> > On Fri, Dec 03, 2021 at 09:02:48PM +0100, Ævar Arnfjörð Bjarmason wrote:
> >> >> When I've experienced delays (sometimes of half a day or more) both
> >> >> https://public-inbox.org/git/ and https://lore.kernel.org/git/ have been
> >> >> updated.
> >> >
> >> > Btw, you can source lore.kernel.org straight into your gmail inbox. :)
> >> >
> >> > https://people.kernel.org/monsieuricon/lore-lei-part-1-getting-started
> >> > https://people.kernel.org/monsieuricon/lore-lei-part-2-now-with-imap
> >> >
> >> > Or, you can read it via nntp://nntp.lore.kernel.org/.
> >>
> >> [CC'd meta@public-inbox.org, probably best to move this thread over
> >> there sooner than later, but CC'ing git@ still in case this is
> >> interesting to others]
> >>
> >> I poked a bit at setting this up but couldn't find from building
> >> public-inbox.org & trying to page through the docs how I'd get from an
> >> existing public-inbox.org/git/ checkout to a local Maildir.
> >
> > Existing, public-inboxes can be set as "externals" and managed
> > via {add,forget,ls}-external sub-commands:
> >
> > # for locally-cloned inboxes:
> > public-inbox-index /path/to/existing/inbox
> > lei add-external /path/to/existing/inbox
> >
> > # relies on curl, memoizes data downloaded for each search:
> > lei add-external https://yhbt.net/lore/git
> >
> > Local externals will be included by every "lei q" invocation;
> > HTTP(S) ones require "lei q --remote"
> >
> > If you only want to use an external as a one-off without adding
> > it, the -I/--include and -O/--only flags are useful:
> >
> > lei q -O https://yhbt.net/lore/git -o /tmp/results SEARCH_TERMS
> >
> >> If you could share some recipe or a pointer to the right docs for that
> >> that would be much appreciated. Thanks!
> >
> > lei-overview(7) manpage documents some things, at least:
> > https://public-inbox.org/lei-overview.html Patches welcome :>
> >
> > IMHO lei still kinda sucks, and I probably won't have time to
> > work on it for a bit :<
>
> Thanks. I finally got around to setting this up.
>
> The above instructions didn't quite work for me, but here's what I did
> (in the form of a script lifted from a screen(1) config I've
> got). Indented with un-indented comments. The "stuff" is screen's way of
> "run this command" (or well, input these characters):
>
> I had a ~/g/git-ml clone already, but this makes one:
>
> stuff "cd ~/g/git-ml || git clone https://public-inbox.org/git ~/g/git-ml^M"
Wait, lack of --mirror or --bare for git-clone on any
public-inbox is a big mistake in terms of inode+disk use
> The initial index:
>
> ## This will create a .git/publici-inbox in ~/g/git-ml. Takes a while
> ## the first time.
> stuff "time public-inbox-index -v \$PWD^M"
>
> I fiddled with this for a bit because it refused to work, turns out it
> was missing the .git at the end, i.e. it expected a bare repo[1]:
>
> ## When we add the lei external it *must* have the ".git" part,
> ## because it'll try to find the "public-inbox" folder at wherever we
> ## point it.
> stuff "test -d ~/.config/lei || lei add-external ~/g/git-ml/.git^M"
Yeah, public-inbox has never supported non-bare usage.
> A bit of a UX wart not to be able to specify no --limit, or maybe I'm
> missing a way:
>
> ## The one-off massive import of the Git ML. TODO: No way to specify
> ## an infinite limit? Not --no-limit or --limit=0.
> stuff "test -d ~/Maildir/lei-q-git-ml || time lei q --limit=999999999 -v -o ~/Maildir/lei-q-git-ml l:git.vger.kernel.org^M"
Oops. --no-limit or --limit=0 or --limit=-1 support will be
added at some point...
> The initial indexing:
>
> ## After the one-off import this will take forever *the first time*
> ## (or around 20m), but subsequent invocations will be fast:
> stuff "time lei up ~/Maildir/lei-q-git-ml^M"
>
> Runs an ad-hoc script to keep it up-to-date, which is quoted below:
>
> ## Run it in a loop
> stuff "public-inbox-lei-pull-index^M"
>
> That script (which I whipped up just now. Is there a better/more
> standard way? to keep a public-inbox+lei pair up-to-date with
> sleep/backoff etc?
Not yet, unfortunately. I would like to have some sort of
long-polling support (similar to IDLE in IMAP) because I hate
sleep/backoff. But I don't think I'm physically nor mentally
capable of doing that or much of anything, anymore.
> #!/bin/sh
> set -xe
>
> repo=~/g/git-ml
> while true
> do
> oid=$(git -C $repo rev-parse HEAD)
> git -C $repo pull
> noid=$(git -C $repo rev-parse HEAD)
> if test "$oid" = "$noid"
> then
> echo Nothing to update
> sleep 60
> continue
> fi
> (
> cd $repo &&
> public-inbox-index -v "$PWD"
> )
> lei up ~/Maildir/lei-q-git-ml
> sleep 1
> done
>
> I use Emacs+mu4e for my E-Mail. And since I index ~/Maildir having these
> files dropped in there will be added to its index. Then I just changed
> my saved search to also look through that maildir (I guess the entire
> first condition could be dropped, but whatever):
>
> "(maildir:/personal-gmail/* OR maildir:/lei-q-git-ml/*) AND list:git.vger.kernel.org OR recip:git@vger.kernel.org OR recip:git-packagers@googlegroups.com"
>
> Because "mu" is generally good about de-duplicating stuff I've now got
> an inbox with mixed messages I can sync from GMail (including my "Sent"
> folder), and I get up-to-the-minute ML traffic now (it's bee 10hrs-4day
> delayed for 3-4 months at least).
>
> So new messages are generally from the "lei" directory, but when I send
> one it'll be dropped in the personal-gmail.
>
> I still need to check if it's doing the wrong thing with e.g. "read"
> flags if I read a mail synced via lei that later arrives in GMail. But I
> mostly don't use "read" statuses anyway...
lei can export flags from its internal DBs to IMAP via "lei export-kw"
see lei-mail-sync-overview(7) for some details, but usability is still
rough...
> 1. Maybe this "I only tested if it complied" patch would make sense to catch that?
>
> diff --git a/lib/PublicInbox/LeiXSearch.pm b/lib/PublicInbox/LeiXSearch.pm
> index 2958d3f9..be49621f 100644
> --- a/lib/PublicInbox/LeiXSearch.pm
> +++ b/lib/PublicInbox/LeiXSearch.pm
> @@ -613,7 +613,7 @@ sub add_uri {
> }
> }
>
> -sub prepare_external {
> +sub _prepare_external {
> my ($self, $loc, $boost) = @_; # n.b. already ordered by boost
> if (ref $loc) { # already a URI, or PublicInbox::Inbox-like object
> return add_uri($self, $loc) if $loc->can('scheme');
> @@ -638,6 +638,14 @@ sub prepare_external {
> push @{$self->{locals}}, $loc;
> }
>
> +sub prepare_external {
> + my ($self, $loc, $boost) = @_;
> + my $ret = _prepare_external($self, $loc, $boost);
> + warn "W: we got nothing from $loc, did you mean $loc/.git?"
> + if !$ret && -e "$loc/.git";
> + return $ret;
> +}
Probably add a note saying non-bare repos are a terrible idea,
anyways, especially for v1 public-inboxes like public-inbox.org/git
^ permalink raw reply [relevance 69%]
* [PATCH] t/lei-sigpipe: attempt to improve diagnostics for stuck test
@ 2022-02-17 21:02 63% ` Eric Wong
2022-02-20 1:38 51% ` Julien Moutinho
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2022-02-17 21:02 UTC (permalink / raw)
To: Julien Moutinho; +Cc: Dominique Martinet, meta
Julien Moutinho <julm+public-inbox@sourcephile.fr> wrote:
> With Inline::C, and with Nix's sandbox
> --------------------------------------
>
> Using (in pkgs/servers/mail/public-inbox/default.nix):
> > preCheck = ''
> > perl certs/create-certs.perl
> > export HOME=$(mktemp -d)
> > mkdir -p $HOME/.cache/public-inbox/inline-c
> > '';
>
> $ nix -L build -f . public-inbox
> > [...]
> > perl5.34.0-public-inbox> t/lei-refresh-mail-sync.t .... ok
> Hangs like forever after this line, while on t/lei-sigpipe.t:
>
> $ ps auxwwf
> > root 1755997 0.0 0.0 453172 2800 ? Ssl déc.04 0:03 nix-daemon --daemon
> > root 1811429 0.0 0.0 453652 1928 ? Ssl déc.04 0:00 \_ nix-daemon 1757028
> > root 1811455 0.0 0.0 453652 2004 ? Ssl déc.04 0:00 \_ nix-daemon 1757031
> > root 1839837 0.0 0.0 453652 1832 ? Ssl déc.04 0:00 \_ nix-daemon 1757027
> > root 1839841 0.0 0.0 453652 1996 ? Ssl déc.04 0:00 \_ nix-daemon 1757030
> > root 1839856 0.0 0.0 453652 1832 ? Ssl déc.04 0:00 \_ nix-daemon 1757029
> > root 3135741 0.0 0.2 603352 17080 ? Ssl 01:05 0:00 \_ nix-daemon 3135672
> > nixbld1 3137110 0.0 0.0 5200 4072 ? Ss 01:05 0:01 \_ bash -e /nix/store/qf3mzpvsmkrw963xchbivcci06078n13-builder.sh
> > nixbld1 3140619 0.0 0.0 4996 4012 ? S 01:05 0:00 \_ make SHELL=/nix/store/l0wlqpbsvh1pgvhcdhw7qkka3d31si7k-bash-5.1-p8/bin/bash VERBOSE=y test
> > nixbld1 3140633 0.0 0.1 17188 13352 ? S 01:05 0:01 | \_ /nix/store/vslsa0l17xjcrdgm2knwj0z5hlvf73m7-perl-5.34.0/bin/perl -MExtUtils::Command::MM -MTest::Harness -e undef *Test::Harness::Switches; test_harness(0, 'blib/lib', 'blib/arch') t/admin.t t/altid.t t/altid_v2.t t/cgi.t t/cmd_ipc.t t/content_hash.t t/dir_idle.t t/edit.t t/eml_content_disposition.t t/eml_content_type.t t/epoll.t t/extindex-psgi.t t/extsearch.t t/fake_inotify.t t/feed.t t/gcf2.t t/gcf2_client.t t/httpd.t t/idx_stack.t t/imap.t t/imap_searchqp.t t/imapd-tls.t t/imapd.t t/inbox_idle.t t/index-git-times.t t/indexlevels-mirror.t t/ipc.t t/kqnotify.t t/lei-auto-watch.t t/lei-convert.t t/lei-daemon.t t/lei-export-kw.t t/lei-externals.t t/lei-import-http.t t/lei-import-imap.t t/lei-import-maildir.t t/lei-import-nntp.t t/lei-import.t t/lei-index.t t/lei-inspect.t t/lei-lcat.t t/lei-p2q.t t/lei-q-kw.t t/lei-q-remote-import.t t/lei-q-save.t t/lei-q-thread.t t/lei-refresh-mail-sync.t t/lei-sigpipe.t t/lei-tag.t t/lei-up.t t/lei-watch.t t/lei.t t/lei_dedupe.t t/lei_external.t t/lei_lcat.t t/lei_mail_sync.t t/lei_overview.t t/lei_saved_search.t t/lei_store.t t/lei_to_mail.t t/lei_xsearch.t t/mbox_lock.t t/mbox_reader.t t/mdir_reader.t t/mime.t t/miscsearch.t t/net_reader-imap.t t/nntpd-tls.t t/nntpd.t t/nodatacow.t t/on_destroy.t t/plack.t t/psgi_attach.t t/psgi_bad_mids.t t/psgi_mount.t t/psgi_multipart_not.t t/psgi_scan_all.t t/psgi_search.t t/psgi_v2.t t/rename_noreplace.t t/reply.t t/search-thr-index.t t/shared_kv.t t/solver_git.t t/thread-index-gap.t t/uri_imap.t t/uri_nntps.t t/v2dupindex.t t/www_altid.t t/xcpdb-reshard.t t/www_static.t t/watch_multiple_headers.t t/watch_maildir_v2.t t/watch_maildir.t t/watch_imap.t t/watch_filter_rubylang.t t/view.t t/v2reindex.t t/v2mirror.t t/v2mda.t t/v2index-late-dupe.t t/v2-add-remove-add.t t/v1reindex.t t/v1-add-remove-add.t t/time.t t/thread-cycle.t t/spamcheck_spamc.t t/sigfd.t t/replace.t t/reindex-time-range.t t/qspawn.t t/purge.t t/psgi_text.t t/precheck.t t/over.t t/nulsubject.t t/nntpd-v2.t t/nntp.t t/multi-mid.t t/msgtime.t t/msgmap.t t/msg_iter.t t/mid.t t/mda_filter_rubylang.t t/mda.t t/linkify.t t/init.t t/indexlevels-mirror-v1.t t/inbox.t t/import.t t/imap_tracker.t t/hval.t t/httpd-unix.t t/httpd-https.t t/httpd-corner.t t/gzip_filter.t t/git.t t/filter_vger.t t/filter_subjecttag.t t/filter_rubylang.t t/filter_mirror.t t/filter_base.t t/emergency.t t/ds-poll.t t/ds-leak.t t/ds-kqxs.t t/config_limiter.t t/config.t t/address.t
> > nixbld1 3145719 0.0 0.4 44576 35640 ? S 01:09 0:01 | \_ t/lei-sigpipe.t
> > nixbld1 3145743 0.0 0.1 14388 10388 ? S 01:09 0:00 | \_ /nix/store/vslsa0l17xjcrdgm2knwj0z5hlvf73m7-perl-5.34.0/bin/perl -w -I/nix/store/vslsa0l17xjcrdgm2knwj0z5hlvf73m7-perl-5.34.0/lib/perl5/site_perl -I/nix/store/1hdx7bxjwqrfnalalkgbwi32l45h8z7b-perl5.34.0-Mail-IMAPClient-3.42/lib/perl5/site_perl -I/nix/st blib/script/lei q -q -t z:1..
> > nixbld1 3145724 0.0 0.5 50504 42160 ? S 01:09 0:00 \_ lei-daemon /build/pi-lei-sigpipe-7522-WoHO/lei-daemon/xdg_run/lei/5.seq.sock
> > nixbld1 3145730 0.0 0.4 50092 39736 ? S 01:09 0:00 \_ lei/store /build/pi-lei-sigpipe-7522-WoHO/lei-daemon/.local/share
>
> t/lei-sigpipe.t is on:
> > 01:30:29.508334 clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=1, tv_nsec=0}, 0x7fffffff5090) = 0 <1.000360>
> > 01:30:30.509837 clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=1, tv_nsec=0}, 0x7fffffff5090) = 0 <1.000186>
> > 01:30:31.510259 clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=1, tv_nsec=0}, 0x7fffffff5090) = 0 <1.000175>
> > [...]
Huh? but it's stuck in sysread() as noted below...
I'm not sure where clock_nanosleep(2) calls are coming from with
a 1s interval. I doubt I'd ever loop clock_nanosleep(CLOCK_REALTIME
no matter how messed up my brain gets :P
I do use nanosleep(2) with a 100ms interval on sendmsg
ENOBUFS|ENOMEM|ETOOMANYREFS in Spawn.pm and that's a different
syscall. From what I can tell, Mail::IMAPClient doesn't
clock_nanosleep, nor does Perl itself...
Sleeping on CLOCK_REALTIME seems wrong outside of a cron-like
scheduler...
> lei-daemon store is on:
> > 01:31:03.834708 epoll_wait(5,
OK, that's normal.
> Looks like it is waiting on sysread() in
> https://public-inbox.org/public-inbox.git/tree/t/lei-sigpipe.t#n35
OK, that makes more sense...
> This test does succeed outside Nix's sandbox:
> $ (cd public-inbox-1.7.0; export PERL_INLINE_DIRECTORY=$PWD/inline-c; rm -rf $PERL_INLINE_DIRECTORY; mkdir $PERL_INLINE_DIRECTORY; prove -bvw t/lei-sigpipe.t )
OK, <snip>
> More surprisingly, it even succeeds when run manually
> inside the hanging Nix sandbox:
OK, <snip>
> Even more strange, Dominique was able to reproduce
> the hang this morning, but no longer tonight..
It's been a while, and I can't reproduce it. Maybe this patch will help
with diagnosis:
------------8<---------
Subject: [PATCH] t/lei-sigpipe: attempt to improve diagnostics for stuck test
This may help diagnose a difficult-to-reproduce test failure on NixOS.
Link: https://public-inbox/meta/20211209013743.okzgim7bbrpahks7@sourcephile.fr/
---
t/lei-sigpipe.t | 8 +++++++-
1 file changed, 7 insertions(+), 1 deletion(-)
diff --git a/t/lei-sigpipe.t b/t/lei-sigpipe.t
index d9738b07..6b2772a6 100644
--- a/t/lei-sigpipe.t
+++ b/t/lei-sigpipe.t
@@ -27,11 +27,17 @@ EOM
}
lei_ok(qw(import), $f) if $imported++ == 0;
- open my $errfh, '+>', "$ENV{HOME}/stderr.log" or xbail $!;
+ open my $errfh, '+>>', "$ENV{HOME}/stderr.log" or xbail $!;
my $opt = { run_mode => 0, 2 => $errfh, 1 => $w };
my $cmd = [qw(lei q -q -t), @$out, 'z:1..'];
my $tp = start_script($cmd, undef, $opt);
close $w;
+ vec(my $rvec = '', fileno($r), 1) = 1;
+ if (!select($rvec, undef, undef, 30)) {
+ seek($errfh, 0, 0) or xbail $!;
+ my $s = do { local $/; <$errfh> };
+ xbail "lei q had no output after 30s, stderr=$s";
+ }
is(sysread($r, my $buf, 1), 1, 'read one byte');
close $r; # trigger SIGPIPE
$tp->join;
^ permalink raw reply related [relevance 63%]
* Re: [PATCH] t/lei-sigpipe: attempt to improve diagnostics for stuck test
2022-02-17 21:02 63% ` [PATCH] t/lei-sigpipe: attempt to improve diagnostics for stuck test Eric Wong
@ 2022-02-20 1:38 51% ` Julien Moutinho
2022-02-22 6:44 71% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Julien Moutinho @ 2022-02-20 1:38 UTC (permalink / raw)
To: Eric Wong; +Cc: Dominique Martinet, meta
[-- Attachment #1: Type: text/plain, Size: 3484 bytes --]
Hi Eric,
Thank you for following up on this
despite your having more important matters on your shoulders :\
Le jeu. 17 févr. 2022 21h02 +0000, Eric Wong a écrit :
> > Looks like it is waiting on sysread() in
> > https://public-inbox.org/public-inbox.git/tree/t/lei-sigpipe.t#n35
>
> OK, that makes more sense...
My tracing with print statements shows that sysread is passed
and that it is $tp->join which hangs:
[nixbld@localhost:/build]$ cat stderr.log
> lei-sigpipe: begin
> lei-sigpipe: start_script: done
> lei-sigpipe: close w: done
> lei-sigpipe: sysread
> lei-sigpipe: sysread: done
> lei-sigpipe: close r: done
[nixbld@localhost:/build]$ cat start_script.log
> start_script: begin
> start_script: fork
> start_script: pid!=0
> start_script: pid==0
> start_script: pid!=0: tail
> start_script: pid==0: if sub
> start_script: pid==0: exec lei
ps auxwwf | grep nixbld
> nixbld1 491489 0.1 0.0 5312 4036 ? Ss 02:12 0:01 \_ bash -e /nix/store/qf3mzpvsmkrw963xchbivcci06078n13-builder.sh
> nixbld1 494307 0.0 0.1 16672 12616 ? S 02:12 0:00 \_ perl -MExtUtils::Command::MM -MTest::Harness -e undef *Test::Harness::Switches; test_harness(0, 'blib/lib', 'blib/arch') t/lei-sigpipe.t
> nixbld1 494311 0.2 0.5 49320 40068 ? S 02:12 0:02 | \_ t/lei-sigpipe.t
> nixbld1 494445 0.0 0.0 0 0 ? Z 02:12 0:00 | \_ [/nix/store/176g] <defunct>
> nixbld1 494447 0.0 0.1 14388 10176 ? S 02:12 0:00 | \_ /nix/store/6wcdxh3h5jp9qgrccq3n71wnalca5mf5-perl-5.34.0/bin/perl -w -I/nix/store/6wcdxh3h5jp9qgrccq3n71wnalca5mf5-perl-5.34.0/lib/perl5/site_perl -I/nix/store/jibsc747pfzmyxp5lsqgzyvrs2lnkws6-perl5.34.0-Mail-IMAPClient-3.42/lib/perl5/site_perl -I/nix/st blib/script/lei q -q -t z:1..
> nixbld1 494427 0.1 0.5 50664 41892 ? S 02:12 0:00 \_ lei-daemon /build/pi-lei-sigpipe-2750-0bv8/lei-daemon/xdg_run/lei/5.seq.sock
> nixbld1 494433 0.0 0.4 50260 39580 ? S 02:12 0:00 \_ lei/store /build/pi-lei-sigpipe-2750-0bv8/lei-daemon/.local/share
The defunct process with truncated path "/nix/store/176g"
is points to /nix/store/176gh50y24c0lx2bnnmsvf9wazb73php-coreutils-9.0
It appears only when I export TAIL=${coreutils}/bin/tail
which does not workaround the hang.
This deadlock appears only when the test is run by nix,
it doesn't when I run it manually from a terminal
by entering the sandbox:
[nixbld@localhost:/]$ (cd /build/public-inbox-1.7.0; export PERL_INLINE_DIRECTORY=$PWD/inline-c; rm -rf $PERL_INLINE_DIRECTORY; mkdir $PERL_INLINE_DIRECTORY; prove -bvw t/lei-sigpipe.t )
> t/lei-sigpipe.t ..
> ok 1 - lei import $TMPDIR/big.eml
> ok 2 - read one byte
> ok 3 - signaled
> ok 4 - got SIGPIPE
> ok 5 - read one byte
> ok 6 - signaled -f mboxcl2
> ok 7 - got SIGPIPE -f mboxcl2
> ok 8 - read one byte
> ok 9 - signaled -f text
> ok 10 - got SIGPIPE -f text
> ok 11 - lei daemon-pid (daemon-pid after t/lei-sigpipe.t:58)
> ok 12 - daemon running after t/lei-sigpipe.t:58
> ok 13 - lei daemon-kill (daemon-kill after t/lei-sigpipe.t:58)
> ok 14 - t/lei-sigpipe.t:58 daemon stopped
> ok 15 - t/lei-sigpipe.t:58 daemon XDG_RUNTIME_DIR/lei/errors.log empty
> 1..15
> ok
> All tests successful.
> Files=1, Tests=15, 8 wallclock secs ( 0.06 usr 0.08 sys + 3.55 cusr 2.81 csys = 6.50 CPU)
> Result: PASS
[-- Attachment #2: 0001-trace-t-lei-sigpipe.t.patch --]
[-- Type: text/plain, Size: 3841 bytes --]
From d919b3441bdf61e8dfae0c08005b8a77662ddb91 Mon Sep 17 00:00:00 2001
From: Julien Moutinho <julm+public-inbox@sourcephile.fr>
Date: Sun, 20 Feb 2022 01:33:04 +0100
Subject: [PATCH] trace t/lei-sigpipe.t
---
lib/PublicInbox/TestCommon.pm | 10 ++++++++++
t/lei-sigpipe.t | 22 +++++++++++++++-------
2 files changed, 25 insertions(+), 7 deletions(-)
diff --git a/lib/PublicInbox/TestCommon.pm b/lib/PublicInbox/TestCommon.pm
index ca732811..d8416f90 100644
--- a/lib/PublicInbox/TestCommon.pm
+++ b/lib/PublicInbox/TestCommon.pm
@@ -439,6 +439,9 @@ sub start_script {
my $run_mode = $ENV{TEST_RUN_MODE} // $opt->{run_mode} // 2;
my $sub = $run_mode == 0 ? undef : key2sub($key);
my $tail;
+ open my $errfh, '+>>', "/build/start_script.log" or xbail $!;
+ $errfh->autoflush(1);
+ print $errfh "\nstart_script: begin\n";
if ($tail_cmd) {
my @paths;
for (@argv) {
@@ -459,8 +462,10 @@ sub start_script {
}
$tail = tail_f(@paths);
}
+ print $errfh "\nstart_script: fork\n";
my $pid = fork // die "fork: $!\n";
if ($pid == 0) {
+ print $errfh "\nstart_script: pid==0\n";
eval { PublicInbox::DS->Reset };
# pretend to be systemd (cf. sd_listen_fds(3))
# 3 == SD_LISTEN_FDS_START
@@ -484,18 +489,23 @@ sub start_script {
}
if ($opt->{-C}) { chdir($opt->{-C}) or die "chdir: $!" }
$0 = join(' ', @$cmd);
+ print $errfh "\nstart_script: pid==0: if sub\n";
if ($sub) {
+ print $errfh "\nstart_script: pid==0: sub\n";
eval { PublicInbox::DS->Reset };
_run_sub($sub, $key, \@argv);
POSIX::_exit($? >> 8);
} else {
+ print $errfh "\nstart_script: pid==0: exec $key\n";
exec(key2script($key), @argv);
die "FAIL: ",join(' ', $key, @argv), ": $!\n";
}
}
+ print $errfh "\nstart_script: pid!=0\n";
require PublicInbox::AutoReap;
my $td = PublicInbox::AutoReap->new($pid);
$td->{-extra} = $tail;
+ print $errfh "\nstart_script: pid!=0: tail\n";
$td;
}
diff --git a/t/lei-sigpipe.t b/t/lei-sigpipe.t
index 6b2772a6..e5e701dd 100644
--- a/t/lei-sigpipe.t
+++ b/t/lei-sigpipe.t
@@ -6,7 +6,7 @@ use v5.10.1;
use PublicInbox::TestCommon;
use POSIX qw(WTERMSIG WIFSIGNALED SIGPIPE);
test_lei(sub {
- my $f = "$ENV{HOME}/big.eml";
+ my $f = "/build/big.eml";
my $imported;
for my $out ([], [qw(-f mboxcl2)], [qw(-f text)]) {
pipe(my ($r, $w)) or BAIL_OUT $!;
@@ -27,25 +27,33 @@ EOM
}
lei_ok(qw(import), $f) if $imported++ == 0;
- open my $errfh, '+>>', "$ENV{HOME}/stderr.log" or xbail $!;
+ open my $errfh, '+>>', "/build/stderr.log" or xbail $!;
+ $errfh->autoflush(1);
+ print $errfh "\nlei-sigpipe: begin\n";
my $opt = { run_mode => 0, 2 => $errfh, 1 => $w };
my $cmd = [qw(lei q -q -t), @$out, 'z:1..'];
my $tp = start_script($cmd, undef, $opt);
+ print $errfh "\nlei-sigpipe: start_script: done\n";
close $w;
+ print $errfh "\nlei-sigpipe: close w: done\n";
vec(my $rvec = '', fileno($r), 1) = 1;
- if (!select($rvec, undef, undef, 30)) {
+ if (!select($rvec, undef, undef, 3)) {
seek($errfh, 0, 0) or xbail $!;
my $s = do { local $/; <$errfh> };
- xbail "lei q had no output after 30s, stderr=$s";
+ xbail "lei q had no output after 3s, stderr=$s";
}
+ print $errfh "\nlei-sigpipe: sysread\n";
is(sysread($r, my $buf, 1), 1, 'read one byte');
+ print $errfh "\nlei-sigpipe: sysread: done\n";
close $r; # trigger SIGPIPE
+ print $errfh "\nlei-sigpipe: close r: done\n";
$tp->join;
+ print $errfh "\nlei-sigpipe: join: done\n";
ok(WIFSIGNALED($?), "signaled @$out");
is(WTERMSIG($?), SIGPIPE, "got SIGPIPE @$out");
- seek($errfh, 0, 0) or xbail $!;
- my $s = do { local $/; <$errfh> };
- is($s, '', "quiet after sigpipe @$out");
+ #seek($errfh, 0, 0) or xbail $!;
+ #my $s = do { local $/; <$errfh> };
+ #is($s, '', "quiet after sigpipe @$out");
}
});
--
2.34.1
^ permalink raw reply related [relevance 51%]
* Re: [PATCH] t/lei-sigpipe: attempt to improve diagnostics for stuck test
2022-02-20 1:38 51% ` Julien Moutinho
@ 2022-02-22 6:44 71% ` Eric Wong
2022-02-27 4:15 71% ` Julien Moutinho
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2022-02-22 6:44 UTC (permalink / raw)
To: Julien Moutinho; +Cc: Dominique Martinet, meta
Julien Moutinho <julm+public-inbox@sourcephile.fr> wrote:
> This deadlock appears only when the test is run by nix,
> it doesn't when I run it manually from a terminal
> by entering the sandbox:
Oh, btw, is this with the FD_CLOEXEC fixes to nix shell itself?
https://public-inbox.org/meta/YfnoOVOG3TrbqZFs@codewreck.org/
^ permalink raw reply [relevance 71%]
* Re: [PATCH] t/lei-sigpipe: attempt to improve diagnostics for stuck test
2022-02-22 6:44 71% ` Eric Wong
@ 2022-02-27 4:15 71% ` Julien Moutinho
2022-02-27 6:41 71% ` Julien Moutinho
0 siblings, 1 reply; 200+ results
From: Julien Moutinho @ 2022-02-27 4:15 UTC (permalink / raw)
To: Eric Wong; +Cc: Dominique Martinet, meta
Le mar. 22 févr. 2022 06h44 +0000, Eric Wong a écrit :
> Julien Moutinho <julm+public-inbox@sourcephile.fr> wrote:
> > This deadlock appears only when the test is run by nix,
> > it doesn't when I run it manually from a terminal
> > by entering the sandbox:
>
> Oh, btw, is this with the FD_CLOEXEC fixes to nix shell itself?
> https://public-inbox.org/meta/YfnoOVOG3TrbqZFs@codewreck.org/
Yes, I'm using:
$ nix-daemon --version
> nix-daemon (Nix) 2.7.0pre20220127_558c4ee
Which contains the commit from https://github.com/NixOS/nix/pull/5683 :
$ git merge-base --is-ancestor 6e0cbc666b60515b5e201dd28855f5fe1de9a107 558c4ee
$ echo $?
> 0
^ permalink raw reply [relevance 71%]
* Re: [PATCH] t/lei-sigpipe: attempt to improve diagnostics for stuck test
2022-02-27 4:15 71% ` Julien Moutinho
@ 2022-02-27 6:41 71% ` Julien Moutinho
2022-02-27 7:23 71% ` Dominique Martinet
0 siblings, 1 reply; 200+ results
From: Julien Moutinho @ 2022-02-27 6:41 UTC (permalink / raw)
To: Eric Wong; +Cc: Dominique Martinet, meta
Le dim. 27 févr. 2022 05h15 +0100, Julien Moutinho a écrit :
> Yes, I'm using:
> $ nix-daemon --version
> > nix-daemon (Nix) 2.7.0pre20220127_558c4ee
Dominique reminded me that that fix was first intended for
`nix develop`, so I've asked whether it also applies to `nix build`:
https://github.com/NixOS/nix/pull/5683#issuecomment-1053239025
^ permalink raw reply [relevance 71%]
* Re: [PATCH] t/lei-sigpipe: attempt to improve diagnostics for stuck test
2022-02-27 6:41 71% ` Julien Moutinho
@ 2022-02-27 7:23 71% ` Dominique Martinet
2022-02-27 8:04 71% ` Julien Moutinho
0 siblings, 1 reply; 200+ results
From: Dominique Martinet @ 2022-02-27 7:23 UTC (permalink / raw)
To: Julien Moutinho; +Cc: Eric Wong, meta
Hello,
Julien Moutinho wrote on Sun, Feb 27, 2022 at 07:41:44AM +0100:
> Le dim. 27 févr. 2022 05h15 +0100, Julien Moutinho a écrit :
> > Yes, I'm using:
> > $ nix-daemon --version
> > > nix-daemon (Nix) 2.7.0pre20220127_558c4ee
> Dominique reminded me that that fix was first intended for
> `nix develop`, so I've asked whether it also applies to `nix build`:
> https://github.com/NixOS/nix/pull/5683#issuecomment-1053239025
I've replied over there, I don't think leaked fd is a problem.
However Julien's latest traces gave a good enough hint: the lei process
that should die from sigpipe just didn't die...
And surely enough, nix adds SIGPIPE to ignores.
This can be confirmed by looking at SigIgn in /proc/xxx/status of one of
the builder PIDs:
# grep SigIgn /proc/3124202/status
SigIgn: 0000000000001004
Where 0x1000 is the 13th bit:
$ kill -l 13
PIPE
Running tests manually won't inherit masked signals and thus won't
reproduce.
I guess we can now open a nix issue asking to reset signal handlers
after fork, before executing its builder?...
--
Dominique
^ permalink raw reply [relevance 71%]
* Re: [PATCH] t/lei-sigpipe: attempt to improve diagnostics for stuck test
2022-02-27 7:23 71% ` Dominique Martinet
@ 2022-02-27 8:04 71% ` Julien Moutinho
2022-02-27 11:17 62% ` [PATCH] t/lei-sigpipe: ensure SIGPIPE is unblocked for this test Eric Wong
0 siblings, 1 reply; 200+ results
From: Julien Moutinho @ 2022-02-27 8:04 UTC (permalink / raw)
To: Dominique Martinet; +Cc: Eric Wong, meta
Le dim. 27 févr. 2022 16h23 +0900, Dominique Martinet a écrit :
> I guess we can now open a nix issue asking to reset signal handlers
> after fork, before executing its builder?...
Well spotted Dominique, thanks!
Turns out this is actually systemd which ignores SIGPIPE.
I'm able to pass t/lei-sigpipe.t when my `nix build`
connects to a nix-daemon run manually outside systemd.
There is already a 3yo issue on nix's tracker for that:
https://github.com/NixOS/nix/issues/2803
suggesting to add to nix-daemon.service:
> [Service]
> IgnoreSIGPIPE=no
And that indeed also makes t/lei-sigpipe.t succeed.
In the meantime I'll disable t/lei-sigpipe.t
to finally get public-inbox added to Nixpkgs.
Thank you Dominique and Eric!
^ permalink raw reply [relevance 71%]
* [PATCH] t/lei-sigpipe: ensure SIGPIPE is unblocked for this test
2022-02-27 8:04 71% ` Julien Moutinho
@ 2022-02-27 11:17 62% ` Eric Wong
2022-03-11 10:42 69% ` [PATCH] t/lei-sigpipe.t: ensure SIGPIPE is not ignored instead of not blocked Julien Moutinho
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2022-02-27 11:17 UTC (permalink / raw)
To: Julien Moutinho; +Cc: Dominique Martinet, meta
Julien Moutinho <julm+public-inbox@sourcephile.fr> wrote:
> Le dim. 27 févr. 2022 16h23 +0900, Dominique Martinet a écrit :
> > I guess we can now open a nix issue asking to reset signal handlers
> > after fork, before executing its builder?...
> Well spotted Dominique, thanks!
>
> Turns out this is actually systemd which ignores SIGPIPE.
> I'm able to pass t/lei-sigpipe.t when my `nix build`
> connects to a nix-daemon run manually outside systemd.
> There is already a 3yo issue on nix's tracker for that:
> https://github.com/NixOS/nix/issues/2803
> suggesting to add to nix-daemon.service:
> > [Service]
> > IgnoreSIGPIPE=no
> And that indeed also makes t/lei-sigpipe.t succeed.
>
> In the meantime I'll disable t/lei-sigpipe.t
> to finally get public-inbox added to Nixpkgs.
>
> Thank you Dominique and Eric!
You're welcome and thank you for tracking this down.
Yes, blocking SIGPIPE makes sense for most daemons so I think
systemd is being reasonable, here. However, this lei test is
for simulating an interactive environment, so re-enabling
SIGPIPE under t/lei-sigpipe.t seems best... I initially
considered adding this for all test_lei(), but most lei tests
don't care and I didn't want to waste CPU time.
------8<-----
Subject: [PATCH] t/lei-sigpipe: ensure SIGPIPE is unblocked for this test
Tests run under systemd (and similar) have SIGPIPE blocked by
default. This was causing this SIGPIPE test to get stuck when
run by automated builders used by Nix. Thanks to Julien
Moutinho and Dominique Martinet for tracking down this failure.
Reported-by: Julien Moutinho <julm+public-inbox@sourcephile.fr>
Reported-by: Dominique Martinet <asmadeus@codewreck.org>
Link: https://public-inbox.org/meta/20220227080422.gyqowrxomzu6gyin@sourcephile.fr/
---
t/lei-sigpipe.t | 19 +++++++++++++++++--
1 file changed, 17 insertions(+), 2 deletions(-)
diff --git a/t/lei-sigpipe.t b/t/lei-sigpipe.t
index 6b2772a6..7fab9aeb 100644
--- a/t/lei-sigpipe.t
+++ b/t/lei-sigpipe.t
@@ -1,10 +1,25 @@
#!perl -w
-# Copyright (C) 2021 all contributors <meta@public-inbox.org>
+# Copyright (C) all contributors <meta@public-inbox.org>
# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
use strict;
use v5.10.1;
use PublicInbox::TestCommon;
-use POSIX qw(WTERMSIG WIFSIGNALED SIGPIPE);
+use POSIX qw(WTERMSIG WIFSIGNALED SIGPIPE SIG_UNBLOCK SIG_SETMASK sigprocmask);
+use PublicInbox::OnDestroy;
+
+# undo systemd (and similar) blocking SIGPIPE, since lei expects to be run
+# from an interactive terminal:
+# https://public-inbox.org/meta/20220227080422.gyqowrxomzu6gyin@sourcephile.fr/
+my $set = POSIX::SigSet->new;
+my $old = POSIX::SigSet->new;
+$set->emptyset or xbail "sigemptyset $!";
+$old->emptyset or xbail "sigemptyset $!";
+$set->addset(SIGPIPE);
+sigprocmask(SIG_UNBLOCK, $set, $old) or xbail "SIG_UNBLOCK: $!";
+my $cleanup = PublicInbox::OnDestroy->new($$, sub {
+ sigprocmask(SIG_SETMASK, $old);
+});
+
test_lei(sub {
my $f = "$ENV{HOME}/big.eml";
my $imported;
^ permalink raw reply related [relevance 62%]
* [PATCH] t/lei-sigpipe.t: ensure SIGPIPE is not ignored instead of not blocked
2022-02-27 11:17 62% ` [PATCH] t/lei-sigpipe: ensure SIGPIPE is unblocked for this test Eric Wong
@ 2022-03-11 10:42 69% ` Julien Moutinho
2022-03-14 22:14 71% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Julien Moutinho @ 2022-03-11 10:42 UTC (permalink / raw)
To: meta; +Cc: Dominique Martinet, Eric Wong, Julien Moutinho
Ignoring a signal is different than blocking a signal.
---
t/lei-sigpipe.t | 14 +++++---------
1 file changed, 5 insertions(+), 9 deletions(-)
diff --git a/t/lei-sigpipe.t b/t/lei-sigpipe.t
index 7fab9aeb..55c208e2 100644
--- a/t/lei-sigpipe.t
+++ b/t/lei-sigpipe.t
@@ -4,20 +4,16 @@
use strict;
use v5.10.1;
use PublicInbox::TestCommon;
-use POSIX qw(WTERMSIG WIFSIGNALED SIGPIPE SIG_UNBLOCK SIG_SETMASK sigprocmask);
+use POSIX qw(WTERMSIG WIFSIGNALED SIGPIPE);
use PublicInbox::OnDestroy;
-# undo systemd (and similar) blocking SIGPIPE, since lei expects to be run
+# undo systemd (and similar) ignoring SIGPIPE, since lei expects to be run
# from an interactive terminal:
# https://public-inbox.org/meta/20220227080422.gyqowrxomzu6gyin@sourcephile.fr/
-my $set = POSIX::SigSet->new;
-my $old = POSIX::SigSet->new;
-$set->emptyset or xbail "sigemptyset $!";
-$old->emptyset or xbail "sigemptyset $!";
-$set->addset(SIGPIPE);
-sigprocmask(SIG_UNBLOCK, $set, $old) or xbail "SIG_UNBLOCK: $!";
+my $oldSIGPIPE = $SIG{PIPE};
+$SIG{PIPE} = 'DEFAULT';
my $cleanup = PublicInbox::OnDestroy->new($$, sub {
- sigprocmask(SIG_SETMASK, $old);
+ $SIG{PIPE} = $oldSIGPIPE;
});
test_lei(sub {
--
2.35.1
^ permalink raw reply related [relevance 69%]
* Failed 'lei q' blocks 'lei init' from working
@ 2022-03-12 21:04 87% Nícolas F. R. A. Prado
2022-03-13 0:06 90% ` Kyle Meyer
0 siblings, 1 reply; 200+ results
From: Nícolas F. R. A. Prado @ 2022-03-12 21:04 UTC (permalink / raw)
To: meta
Hi,
Right after building lei from source, I get an error running the 'q' command:
[nfraprado@notapiano public-inbox]$ ./lei.sh q -I https://lore.kernel.org/all/ -o ~/.mail/lei/test --threads --dedupe=mid 'dfn:mt8192.dtsi rt:1.month.ago..'
open(/home/nfraprado/.local/share/lei/store/mail_sync.sqlite3): No such file or directory at /home/nfraprado/ext/git/public-inbox/lib/PublicInbox/LeiQuery.pm line 37.
That's somewhat reasonable since I haven't issued an init first. But if I then
try to init:
[nfraprado@notapiano public-inbox]$ ./lei.sh init
# leistore.dir=/home/nfraprado/.local/share/lei/store newly initialized
[nfraprado@notapiano public-inbox]$ ./lei.sh init
# leistore.dir=/home/nfraprado/.local/share/lei/store newly initialized
No store is ever created. If I reboot (or issue the 'e' sysrq to basically
restart userspace) and then run the init command right away, it does work.
So it seems to me that there's some state that is kept after running the q
command which blocks init from working, hence requiring the reboot to clear that
state.
Not quite sure if it's a Perl thing or a bug in public-inbox, but since it's
unusual behavior (and took me some time to find out) I figured I'd report it.
Making 'q' run 'init' if not already done so could also help to work around this
issue.
Thanks,
Nícolas
^ permalink raw reply [relevance 87%]
* Re: Failed 'lei q' blocks 'lei init' from working
2022-03-12 21:04 87% Failed 'lei q' blocks 'lei init' from working Nícolas F. R. A. Prado
@ 2022-03-13 0:06 90% ` Kyle Meyer
0 siblings, 0 replies; 200+ results
From: Kyle Meyer @ 2022-03-13 0:06 UTC (permalink / raw)
To: Nícolas F. R. A. Prado; +Cc: meta
Nícolas F. R. A. Prado writes:
> Hi,
>
> Right after building lei from source, I get an error running the 'q' command:
>
> [nfraprado@notapiano public-inbox]$ ./lei.sh q -I https://lore.kernel.org/all/ -o ~/.mail/lei/test --threads --dedupe=mid 'dfn:mt8192.dtsi rt:1.month.ago..'
> open(/home/nfraprado/.local/share/lei/store/mail_sync.sqlite3): No such file or directory at /home/nfraprado/ext/git/public-inbox/lib/PublicInbox/LeiQuery.pm line 37.
>
> That's somewhat reasonable since I haven't issued an init first. But if I then
> try to init [...]:
Hmm, an explicit lei-init call shouldn't be needed (i.e. lei-q should
already behave as you propose at the end of your message, initializing
the store/config if necessary).
Eric has even considered dropping lei-init entirely:
https://public-inbox.org/meta/20210325083207.GA30551@dcvr/
> Not quite sure if it's a Perl thing or a bug in public-inbox, but since it's
> unusual behavior (and took me some time to find out) I figured I'd report it.
This reminded me a bit of
https://public-inbox.org/meta/20211120153054.xq6jfad2m2buoic6@gmail.com/
Perhaps a lei-daemon process was running from some previous
experimentation?
^ permalink raw reply [relevance 90%]
* Re: [PATCH] t/lei-sigpipe.t: ensure SIGPIPE is not ignored instead of not blocked
2022-03-11 10:42 69% ` [PATCH] t/lei-sigpipe.t: ensure SIGPIPE is not ignored instead of not blocked Julien Moutinho
@ 2022-03-14 22:14 71% ` Eric Wong
2022-03-15 2:56 71% ` Julien Moutinho
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2022-03-14 22:14 UTC (permalink / raw)
To: Julien Moutinho; +Cc: meta, Dominique Martinet
Julien Moutinho <julm+public-inbox@sourcephile.fr> wrote:
> Ignoring a signal is different than blocking a signal.
Thanks, pushed as commit f1e4e14793d155ea7d6ed7a6858b668e97c7e5d8
I wasn't sure why this patch was necessary at first. Thus I
took take a closer look at systemd behavior to confirm, and
updated the message:
Ignoring a signal is different than blocking a signal, and the
"IgnoreSIGPIPE" option of systemd ignores.
[ew: note systemd behavior]
Acked-by: Eric Wong <e@80x24.org>
I (and any other project maintainer) would appreciate more
complete explanations in the future as to why a patch is needed.
Thanks again.
^ permalink raw reply [relevance 71%]
* Re: [PATCH] t/lei-sigpipe.t: ensure SIGPIPE is not ignored instead of not blocked
2022-03-14 22:14 71% ` Eric Wong
@ 2022-03-15 2:56 71% ` Julien Moutinho
0 siblings, 0 replies; 200+ results
From: Julien Moutinho @ 2022-03-15 2:56 UTC (permalink / raw)
To: Eric Wong; +Cc: meta, Dominique Martinet
Le lun. 14 mars 2022 22h14 +0000, Eric Wong a écrit :
> I (and any other project maintainer) would appreciate more
> complete explanations in the future as to why a patch is needed.
> Thanks again.
Oh sorry, being immersed into the testing I tought
that it was obvious. I should have clarified
that the unblocking fix was not working.
Thanks for the merge!
^ permalink raw reply [relevance 71%]
* [PATCH] lei: always open mail_sync.sqlite3 R/W
@ 2022-04-05 8:18 37% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2022-04-05 8:18 UTC (permalink / raw)
To: meta
This will make transparently upgrading from 1.7.0 -> 1.8.x
easier. Only a single user has access to mail_sync.sqlite3,
and R/W at the kernel-level is required for WAL, anyways.
---
lib/PublicInbox/LEI.pm | 6 +++---
lib/PublicInbox/LeiImport.pm | 8 ++++----
lib/PublicInbox/LeiImportKw.pm | 6 +++---
lib/PublicInbox/LeiLcat.pm | 10 +++++-----
lib/PublicInbox/LeiMailSync.pm | 34 +++++++++++++++++-----------------
lib/PublicInbox/LeiSearch.pm | 4 ++--
lib/PublicInbox/NetReader.pm | 4 ++--
7 files changed, 36 insertions(+), 36 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 4e0295fa..a7ddc21f 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -1,4 +1,4 @@
-# Copyright (C) 2020-2021 all contributors <meta@public-inbox.org>
+# Copyright (C) all contributors <meta@public-inbox.org>
# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
# Backend for `lei' (local email interface). Unlike the C10K-oriented
@@ -1502,11 +1502,11 @@ sub git_oid {
}
sub lms {
- my ($lei, $rw) = @_;
+ my ($lei, $creat) = @_;
my $sto = $lei->{sto} // _lei_store($lei) // return;
require PublicInbox::LeiMailSync;
my $f = "$sto->{priv_eidx}->{topdir}/mail_sync.sqlite3";
- (-f $f || $rw) ? PublicInbox::LeiMailSync->new($f) : undef;
+ (-f $f || $creat) ? PublicInbox::LeiMailSync->new($f) : undef;
}
sub sto_done_request {
diff --git a/lib/PublicInbox/LeiImport.pm b/lib/PublicInbox/LeiImport.pm
index bbc0634e..b9865829 100644
--- a/lib/PublicInbox/LeiImport.pm
+++ b/lib/PublicInbox/LeiImport.pm
@@ -1,4 +1,4 @@
-# Copyright (C) 2021 all contributors <meta@public-inbox.org>
+# Copyright (C) all contributors <meta@public-inbox.org>
# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
# front-end for the "lei import" sub-command
@@ -36,7 +36,7 @@ sub pmdir_cb { # called via wq_io_do from LeiPmdir->each_mdir_fn
my $kw = PublicInbox::MdirReader::flags2kw($fl);
substr($folder, 0, 0) = 'maildir:'; # add prefix
my $lse = $self->{lse} //= $self->{lei}->{sto}->search;
- my $lms = $self->{-lms_ro} //= $self->{lei}->lms; # may be 0 or undef
+ my $lms = $self->{-lms_rw} //= $self->{lei}->lms; # may be 0 or undef
my @oidbin = $lms ? $lms->name_oidbin($folder, $bn) : ();
@oidbin > 1 and warn("W: $folder/*/$$bn not unique:\n",
map { "\t".unpack('H*', $_)."\n" } @oidbin);
@@ -87,8 +87,8 @@ sub do_import_index ($$@) {
# $j = $net->net_concurrency($j); TODO
if ($lei->{opt}->{incremental} // 1) {
$net->{incremental} = 1;
- $net->{-lms_ro} = $lei->lms // 0;
- if ($self->{-import_kw} && $net->{-lms_ro} &&
+ $net->{-lms_rw} = $lei->lms // 0;
+ if ($self->{-import_kw} && $net->{-lms_rw} &&
!$lei->{opt}->{'new-only'} &&
$net->{imap_order}) {
require PublicInbox::LeiImportKw;
diff --git a/lib/PublicInbox/LeiImportKw.pm b/lib/PublicInbox/LeiImportKw.pm
index 54454511..52fd4043 100644
--- a/lib/PublicInbox/LeiImportKw.pm
+++ b/lib/PublicInbox/LeiImportKw.pm
@@ -1,4 +1,4 @@
-# Copyright (C) 2021 all contributors <meta@public-inbox.org>
+# Copyright (C) all contributors <meta@public-inbox.org>
# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
# WQ worker for dealing with LeiImport IMAP flags on already-imported messages
@@ -28,13 +28,13 @@ sub ipc_atfork_child {
$self->{verbose} = $lei->{opt}->{verbose};
$self->{lse} = $self->{sto}->search;
$self->{over} = $self->{lse}->over;
- $self->{-lms_ro} = $net->{-lms_ro} || die 'BUG: net->{-lms_ro} FALSE';
+ $self->{-lms_rw} = $net->{-lms_rw} || die 'BUG: net->{-lms_rw} FALSE';
$self->SUPER::ipc_atfork_child;
}
sub ck_update_kw { # via wq_io_do
my ($self, $url, $uid, $kw) = @_;
- my @oidbin = $self->{-lms_ro}->num_oidbin($url, $uid);
+ my @oidbin = $self->{-lms_rw}->num_oidbin($url, $uid);
my $uid_url = "$url/;UID=$uid";
@oidbin > 1 and warn("W: $uid_url not unique:\n",
map { "\t".unpack('H*', $_)."\n" } @oidbin);
diff --git a/lib/PublicInbox/LeiLcat.pm b/lib/PublicInbox/LeiLcat.pm
index 191f6f24..8d89cb73 100644
--- a/lib/PublicInbox/LeiLcat.pm
+++ b/lib/PublicInbox/LeiLcat.pm
@@ -1,4 +1,4 @@
-# Copyright (C) 2021 all contributors <meta@public-inbox.org>
+# Copyright (C) all contributors <meta@public-inbox.org>
# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
# lcat: local cat, display a local message by Message-ID or blob,
@@ -13,7 +13,7 @@ use PublicInbox::MID qw($MID_EXTRACT);
sub lcat_folder ($$;$$) {
my ($lei, $folder, $beg, $end) = @_;
- my $lms = $lei->{-lms_ro} //= $lei->lms // return;
+ my $lms = $lei->{-lms_rw} //= $lei->lms // return;
my $folders = [ $folder ];
eval { $lms->arg2folder($lei, $folders) };
return $lei->child_error(0, "# unknown folder: $folder") if $@;
@@ -31,7 +31,7 @@ sub lcat_folder ($$;$$) {
sub lcat_imap_uri ($$) {
my ($lei, $uri) = @_;
# cf. LeiXSearch->lcat_dump
- my $lms = $lei->{-lms_ro} //= $lei->lms // return;
+ my $lms = $lei->{-lms_rw} //= $lei->lms // return;
if (defined $uri->uid) {
push @{$lei->{lcat_todo}}, $lms->imap_oidhex($lei, $uri);
} elsif (defined(my $fid = $lms->fid_for($$uri))) {
@@ -45,7 +45,7 @@ sub lcat_nntp_uri ($$) {
my ($lei, $uri) = @_;
my $mid = $uri->message; # already unescaped by URI::news
return "mid:$mid" if defined($mid);
- my $lms = $lei->{-lms_ro} //= $lei->lms // return;
+ my $lms = $lei->{-lms_rw} //= $lei->lms // return;
my ($ng, $beg, $end) = $uri->group;
$uri->group($ng);
lcat_folder($lei, $$uri, $beg, $end);
@@ -118,7 +118,7 @@ could not extract Message-ID from $x
}
}
- delete $lei->{-lms_ro};
+ delete $lei->{-lms_rw};
@q ? join(' OR ', @q) : $lei->fail("no Message-ID in: @argv");
}
diff --git a/lib/PublicInbox/LeiMailSync.pm b/lib/PublicInbox/LeiMailSync.pm
index a9a65fd6..85480599 100644
--- a/lib/PublicInbox/LeiMailSync.pm
+++ b/lib/PublicInbox/LeiMailSync.pm
@@ -11,9 +11,9 @@ use PublicInbox::ContentHash qw(git_sha);
use Carp ();
sub dbh_new {
- my ($self, $rw) = @_;
+ my ($self) = @_;
my $f = $self->{filename};
- my $creat = $rw && !-s $f;
+ my $creat = !-s $f;
if ($creat) {
require PublicInbox::Syscall;
open my $fh, '+>>', $f or Carp::croak "open($f): $!";
@@ -23,11 +23,10 @@ sub dbh_new {
AutoCommit => 1,
RaiseError => 1,
PrintError => 0,
- ReadOnly => !$rw,
sqlite_use_immediate_transaction => 1,
});
# no sqlite_unicode, here, all strings are binary
- create_tables($self, $dbh) if $rw;
+ create_tables($self, $dbh);
$dbh->do('PRAGMA journal_mode = WAL') if $creat;
$dbh->do('PRAGMA case_sensitive_like = ON');
$dbh;
@@ -42,7 +41,7 @@ sub new {
}, $cls;
}
-sub lms_write_prepare { ($_[0]->{dbh} //= dbh_new($_[0], 1)); $_[0] }
+sub lms_write_prepare { ($_[0]->{dbh} //= dbh_new($_[0])); $_[0] }
sub lms_pause {
my ($self) = @_;
@@ -102,7 +101,7 @@ UPDATE folders SET loc = ? WHERE fid = ?
}
sub get_fid ($$$) {
- my ($sth, $folder, $dbh) = @_; # $dbh is set iff RW
+ my ($sth, $folder, $dbh) = @_;
$sth->bind_param(1, $folder, SQL_BLOB);
$sth->execute;
my ($fid) = $sth->fetchrow_array;
@@ -118,36 +117,37 @@ sub get_fid ($$$) {
}
sub fid_for {
- my ($self, $folder, $rw) = @_;
- my $dbh = $self->{dbh} //= dbh_new($self, $rw);
+ my ($self, $folder, $creat) = @_;
+ my $dbh = $self->{dbh} //= dbh_new($self);
my $sth = $dbh->prepare_cached(<<'', undef, 1);
SELECT fid FROM folders WHERE loc = ? LIMIT 1
- my $rw_dbh = $dbh->{ReadOnly} ? undef : $dbh;
- my $fid = get_fid($sth, $folder, $rw_dbh);
+ my $fid = get_fid($sth, $folder, $dbh);
return $fid if defined($fid);
# caller had trailing slash (LeiToMail)
if ($folder =~ s!\A((?:maildir|mh):.*?)/+\z!$1!i) {
- $fid = get_fid($sth, $folder, $rw_dbh);
+ $fid = get_fid($sth, $folder, $dbh);
if (defined $fid) {
- update_fid($dbh, $fid, $folder) if $rw;
+ update_fid($dbh, $fid, $folder);
return $fid;
}
# sometimes we stored trailing slash..
} elsif ($folder =~ m!\A(?:maildir|mh):!i) {
- $fid = get_fid($sth, $folder, $rw_dbh);
+ $fid = get_fid($sth, $folder, $dbh);
if (defined $fid) {
- update_fid($dbh, $fid, $folder) if $rw;
+ update_fid($dbh, $fid, $folder);
return $fid;
}
- } elsif ($rw && $folder =~ m!\Aimaps?://!i) {
+ } elsif ($creat && $folder =~ m!\Aimaps?://!i) {
require PublicInbox::URIimap;
- PublicInbox::URIimap->new($folder)->uidvalidity //
+ my $uri = PublicInbox::URIimap->new($folder);
+ $uri->uidvalidity //
Carp::croak("BUG: $folder has no UIDVALIDITY");
+ defined($uri->uid) and Carp::confess("BUG: $folder has UID");
}
- return unless $rw;
+ return unless $creat;
($fid) = $dbh->selectrow_array('SELECT MAX(fid) FROM folders');
$fid += 1;
diff --git a/lib/PublicInbox/LeiSearch.pm b/lib/PublicInbox/LeiSearch.pm
index d0ca13f0..936c2751 100644
--- a/lib/PublicInbox/LeiSearch.pm
+++ b/lib/PublicInbox/LeiSearch.pm
@@ -1,4 +1,4 @@
-# Copyright (C) 2020-2021 all contributors <meta@public-inbox.org>
+# Copyright (C) all contributors <meta@public-inbox.org>
# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
# read-only counterpart for PublicInbox::LeiStore
@@ -101,7 +101,7 @@ sub xoids_for {
my $git = $self->git;
my $xoids = {};
# no lms when used via {ale}:
- my $lms = $self->{-lms_ro} //= lms($self) if defined($self->{topdir});
+ my $lms = $self->{-lms_rw} //= lms($self) if defined($self->{topdir});
for my $mid (@$mids) {
for my $o (@overs) {
my ($id, $prev);
diff --git a/lib/PublicInbox/NetReader.pm b/lib/PublicInbox/NetReader.pm
index 032b4fda..c1af03a3 100644
--- a/lib/PublicInbox/NetReader.pm
+++ b/lib/PublicInbox/NetReader.pm
@@ -1,4 +1,4 @@
-# Copyright (C) 2021 all contributors <meta@public-inbox.org>
+# Copyright (C) all contributors <meta@public-inbox.org>
# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
# common reader code for IMAP and NNTP (and maybe JMAP)
@@ -481,7 +481,7 @@ sub itrk_last ($$;$$) {
my ($self, $uri, $r_uidval, $mic) = @_;
return (undef, undef, $r_uidval) unless $self->{incremental};
my ($itrk, $l_uid, $l_uidval);
- if (defined(my $lms = $self->{-lms_ro})) { # LeiMailSync or 0
+ if (defined(my $lms = $self->{-lms_rw})) { # LeiMailSync or 0
$uri->uidvalidity($r_uidval) if defined $r_uidval;
if ($mic) {
my $auth = $mic->Authmechanism // '';
^ permalink raw reply related [relevance 37%]
* [PATCH 0/4] lei: finish wiring up pure-Perl stuff for Linux
@ 2022-04-18 9:50 71% Eric Wong
2022-04-18 9:50 71% ` [PATCH 1/4] lei: clobber recvmsg buffer on errors Eric Wong
2022-04-18 9:50 53% ` [PATCH 3/4] lei: wire up pure Perl sendmsg/recvmsg for Linux users Eric Wong
0 siblings, 2 replies; 200+ results
From: Eric Wong @ 2022-04-18 9:50 UTC (permalink / raw)
To: meta
Some cleanups here, too; but this hopefully makes lei more
accessible to users who don't have space/bandwidth to use
Inline::C.
I thought I sent these out weeks ago, but Real-Life, power outages
and other lei problems got in the way, I guess :x
Eric Wong (4):
lei: clobber recvmsg buffer on errors
syscall: more idiomatic cmsghdr space allocation
lei: wire up pure Perl sendmsg/recvmsg for Linux users
syscall: golf + more idiomatic buffer initialization
lib/PublicInbox/IPC.pm | 4 ++++
lib/PublicInbox/LEI.pm | 4 ++++
lib/PublicInbox/LeiSelfSocket.pm | 1 +
lib/PublicInbox/Syscall.pm | 13 ++++++-------
lib/PublicInbox/TestCommon.pm | 16 ++++++++++++----
t/lei-daemon.t | 6 +++++-
t/lei-externals.t | 1 +
7 files changed, 33 insertions(+), 12 deletions(-)
^ permalink raw reply [relevance 71%]
* [PATCH 1/4] lei: clobber recvmsg buffer on errors
2022-04-18 9:50 71% [PATCH 0/4] lei: finish wiring up pure-Perl stuff for Linux Eric Wong
@ 2022-04-18 9:50 71% ` Eric Wong
2022-04-18 9:50 53% ` [PATCH 3/4] lei: wire up pure Perl sendmsg/recvmsg for Linux users Eric Wong
1 sibling, 0 replies; 200+ results
From: Eric Wong @ 2022-04-18 9:50 UTC (permalink / raw)
To: meta
It will be necessary when we drop the Inline::C requirement
since the pure Perl Linux syscall recvmsg implementation.
This likely would've caused errors for Socket::MsgHdr users
without Inline::C, but I haven't tested it since it's a rare
configuration.
---
lib/PublicInbox/LEI.pm | 1 +
lib/PublicInbox/LeiSelfSocket.pm | 1 +
2 files changed, 2 insertions(+)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index a7ddc21f..9ab91714 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -1149,6 +1149,7 @@ sub event_step {
if (scalar(@fds) == 1 && !defined($fds[0])) {
return if $! == EAGAIN;
die "recvmsg: $!" if $! != ECONNRESET;
+ $buf = '';
@fds = (); # for open loop below:
}
for (@fds) { open my $rfh, '+<&=', $_ }
diff --git a/lib/PublicInbox/LeiSelfSocket.pm b/lib/PublicInbox/LeiSelfSocket.pm
index dd64b6cf..860020cb 100644
--- a/lib/PublicInbox/LeiSelfSocket.pm
+++ b/lib/PublicInbox/LeiSelfSocket.pm
@@ -29,6 +29,7 @@ sub event_step {
if (scalar(@fds) == 1 && !defined($fds[0])) {
return if $!{EAGAIN};
die "recvmsg: $!" unless $!{ECONNRESET};
+ $buf = '';
} else { # just in case open so perl can auto-close them:
for (@fds) { open my $fh, '+<&=', $_ };
}
^ permalink raw reply related [relevance 71%]
* [PATCH 3/4] lei: wire up pure Perl sendmsg/recvmsg for Linux users
2022-04-18 9:50 71% [PATCH 0/4] lei: finish wiring up pure-Perl stuff for Linux Eric Wong
2022-04-18 9:50 71% ` [PATCH 1/4] lei: clobber recvmsg buffer on errors Eric Wong
@ 2022-04-18 9:50 53% ` Eric Wong
1 sibling, 0 replies; 200+ results
From: Eric Wong @ 2022-04-18 9:50 UTC (permalink / raw)
To: meta
This enables lei-daemon to work without Inline::C nor
Socket::MsgHdr installed. Prior to this, only the `lei' client
was using the pure Perl implementation. Either C implementation
is still marginally faster, however.
---
lib/PublicInbox/IPC.pm | 4 ++++
lib/PublicInbox/LEI.pm | 3 +++
lib/PublicInbox/TestCommon.pm | 16 ++++++++++++----
t/lei-daemon.t | 6 +++++-
t/lei-externals.t | 1 +
5 files changed, 25 insertions(+), 5 deletions(-)
diff --git a/lib/PublicInbox/IPC.pm b/lib/PublicInbox/IPC.pm
index 8376275e..67e86a43 100644
--- a/lib/PublicInbox/IPC.pm
+++ b/lib/PublicInbox/IPC.pm
@@ -47,6 +47,10 @@ my $send_cmd = PublicInbox::Spawn->can('send_cmd4') // do {
require PublicInbox::CmdIPC4;
$recv_cmd //= PublicInbox::CmdIPC4->can('recv_cmd4');
PublicInbox::CmdIPC4->can('send_cmd4');
+} // do {
+ require PublicInbox::Syscall;
+ $recv_cmd //= PublicInbox::Syscall->can('recv_cmd4');
+ PublicInbox::Syscall->can('send_cmd4');
};
sub _get_rec ($) {
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 9ab91714..4bd9183e 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -1277,6 +1277,9 @@ sub lazy_start {
require PublicInbox::CmdIPC4;
$send_cmd = PublicInbox::CmdIPC4->can('send_cmd4');
PublicInbox::CmdIPC4->can('recv_cmd4');
+ } // do {
+ $send_cmd = PublicInbox::Syscall->can('send_cmd4');
+ PublicInbox::Syscall->can('recv_cmd4');
};
}
$recv_cmd or die <<"";
diff --git a/lib/PublicInbox/TestCommon.pm b/lib/PublicInbox/TestCommon.pm
index ca732811..943dd2fa 100644
--- a/lib/PublicInbox/TestCommon.pm
+++ b/lib/PublicInbox/TestCommon.pm
@@ -1,4 +1,4 @@
-# Copyright (C) 2015-2021 all contributors <meta@public-inbox.org>
+# Copyright (C) all contributors <meta@public-inbox.org>
# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
# internal APIs used only for tests
@@ -19,7 +19,8 @@ BEGIN {
run_script start_script key2sub xsys xsys_e xqx eml_load tick
have_xapian_compact json_utf8 setup_public_inboxes create_inbox
tcp_host_port test_lei lei lei_ok $lei_out $lei_err $lei_opt
- test_httpd xbail require_cmd is_xdeeply tail_f);
+ test_httpd xbail require_cmd is_xdeeply tail_f
+ ignore_inline_c_missing);
require Test::More;
my @methods = grep(!/\W/, @Test::More::EXPORT);
eval(join('', map { "*$_=\\&Test::More::$_;" } @methods));
@@ -547,6 +548,11 @@ sub is_xdeeply ($$$) {
$ok;
}
+sub ignore_inline_c_missing {
+ $_[0] = join('', grep(/\S/, grep(!/compilation aborted/,
+ grep(!/\bInline\b/, split(/^/m, $_[0])))));
+}
+
sub test_lei {
SKIP: {
my ($cb) = pop @_;
@@ -571,8 +577,10 @@ SKIP: {
$ENV{LANG} = $ENV{LC_ALL} = 'C';
my (undef, $fn, $lineno) = caller(0);
my $t = "$fn:$lineno";
- state $lei_daemon = PublicInbox::Spawn->can('send_cmd4') ||
- eval { require Socket::MsgHdr; 1 };
+ state $lei_daemon = PublicInbox::Spawn->can('send_cmd4') || do {
+ require PublicInbox::Syscall;
+ PublicInbox::Syscall->can('send_cmd4');
+ } || eval { require Socket::MsgHdr; 1 };
unless ($lei_daemon) {
skip('Inline::C unconfigured/missing '.
'(mkdir -p ~/.cache/public-inbox/inline-c) OR Socket::MsgHdr missing',
diff --git a/t/lei-daemon.t b/t/lei-daemon.t
index b60c7ce6..e11105bc 100644
--- a/t/lei-daemon.t
+++ b/t/lei-daemon.t
@@ -1,5 +1,5 @@
#!perl -w
-# Copyright (C) 2020-2021 all contributors <meta@public-inbox.org>
+# Copyright (C) all contributors <meta@public-inbox.org>
# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
use strict; use v5.10.1; use PublicInbox::TestCommon;
use Socket qw(AF_UNIX SOCK_SEQPACKET MSG_EOR pack_sockaddr_un);
@@ -8,12 +8,16 @@ test_lei({ daemon_only => 1 }, sub {
my $send_cmd = PublicInbox::Spawn->can('send_cmd4') // do {
require PublicInbox::CmdIPC4;
PublicInbox::CmdIPC4->can('send_cmd4');
+ } // do {
+ require PublicInbox::Syscall;
+ PublicInbox::Syscall->can('send_cmd4');
};
$send_cmd or BAIL_OUT 'started testing lei-daemon w/o send_cmd4!';
my $sock = "$ENV{XDG_RUNTIME_DIR}/lei/5.seq.sock";
my $err_log = "$ENV{XDG_RUNTIME_DIR}/lei/errors.log";
lei_ok('daemon-pid');
+ ignore_inline_c_missing($lei_err);
is($lei_err, '', 'no error from daemon-pid');
like($lei_out, qr/\A[0-9]+\n\z/s, 'pid returned') or BAIL_OUT;
chomp(my $pid = $lei_out);
diff --git a/t/lei-externals.t b/t/lei-externals.t
index fed57789..284be1b9 100644
--- a/t/lei-externals.t
+++ b/t/lei-externals.t
@@ -76,6 +76,7 @@ test_lei(sub {
my $config_file = "$home/.config/lei/config";
my $store_dir = "$home/.local/share/lei";
lei_ok 'ls-external', \'ls-external on fresh install';
+ ignore_inline_c_missing($lei_err);
is($lei_out.$lei_err, '', 'ls-external no output, yet');
ok(!-e $config_file && !-e $store_dir,
'nothing created by ls-external');
^ permalink raw reply related [relevance 53%]
* [PATCH] lei: commit store on interrupted partial imports
@ 2022-04-21 11:59 58% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2022-04-21 11:59 UTC (permalink / raw)
To: meta
This change prevents lingering shard and git-fast-import
processes from remaining after interrupted "lei import" (and
similar). It also reduces the likelyhood of data-loss in case
of subsequent abnormal termination of the daemon.
I think this is the least surprising way to handle users
prematurely aborting imports or other similar operations which
write to lei/store and will result in reduced bandwidth waste
for users with intermittent connections. This is because the
lei/store processes may be shared by parallel "lei import"
callers, and commits done by any "lei import" caller will
inevitably trigger writes for all of them.
---
lib/PublicInbox/LEI.pm | 11 ++++++++---
lib/PublicInbox/LeiImportKw.pm | 4 +---
lib/PublicInbox/LeiNoteEvent.pm | 4 +---
lib/PublicInbox/LeiPmdir.pm | 6 ++----
4 files changed, 12 insertions(+), 13 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 4bd9183e..93b4ea03 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -660,6 +660,7 @@ sub wait_wq_events {
for my $wq (grep(defined, @$lei{qw(ikw pmd)})) { # auxiliary WQs
$wq->wq_close;
}
+ $wq1->{lei_sock} = $lei->{sock} if $wq1;
$op_c->{ops} = $ops;
}
@@ -1405,9 +1406,11 @@ sub fchdir {
}
sub wq_eof { # EOF callback for main daemon
- my ($lei) = @_;
+ my ($lei, $wq_fld) = @_;
local $current_lei = $lei;
- delete $lei->{wq1} // return $lei->fail; # already failed
+ my $wq = delete $lei->{$wq_fld // 'wq1'};
+ $lei->sto_done_request($wq);
+ $wq // $lei->fail; # already failed
}
sub watch_state_ok ($) {
@@ -1514,8 +1517,10 @@ sub lms {
}
sub sto_done_request {
- my ($lei, $sock) = @_;
+ my ($lei, $wq) = @_;
+ return unless $lei->{sto};
local $current_lei = $lei;
+ my $sock = $wq ? $wq->{lei_sock} : undef;
eval {
if ($sock //= $lei->{sock}) { # issue, async wait
$lei->{sto}->wq_io_do('done', [ $sock ]);
diff --git a/lib/PublicInbox/LeiImportKw.pm b/lib/PublicInbox/LeiImportKw.pm
index 52fd4043..4dd938f5 100644
--- a/lib/PublicInbox/LeiImportKw.pm
+++ b/lib/PublicInbox/LeiImportKw.pm
@@ -47,9 +47,7 @@ sub ck_update_kw { # via wq_io_do
}
sub _lei_wq_eof { # EOF callback for main lei daemon
- my ($lei) = @_;
- my $ikw = delete $lei->{ikw} or return $lei->fail;
- $lei->sto_done_request($ikw->{lei_sock});
+ $_[0]->wq_eof('ikw');
}
1;
diff --git a/lib/PublicInbox/LeiNoteEvent.pm b/lib/PublicInbox/LeiNoteEvent.pm
index 22d6ffac..db387633 100644
--- a/lib/PublicInbox/LeiNoteEvent.pm
+++ b/lib/PublicInbox/LeiNoteEvent.pm
@@ -114,9 +114,7 @@ sub ipc_atfork_child {
}
sub _lei_wq_eof { # EOF callback for main lei daemon
- my ($lei) = @_;
- my $lne = delete $lei->{lne} or return $lei->fail;
- $lei->sto_done_request($lne->{lei_sock});
+ $_[0]->wq_eof('lne');
}
1;
diff --git a/lib/PublicInbox/LeiPmdir.pm b/lib/PublicInbox/LeiPmdir.pm
index f9b68fc2..d4aa0212 100644
--- a/lib/PublicInbox/LeiPmdir.pm
+++ b/lib/PublicInbox/LeiPmdir.pm
@@ -1,4 +1,4 @@
-# Copyright (C) 2021 all contributors <meta@public-inbox.org>
+# Copyright (C) all contributors <meta@public-inbox.org>
# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
# WQ worker for dealing with parallel Maildir reads;
@@ -48,9 +48,7 @@ sub mdir_iter { # via wq_io_do
}
sub _lei_wq_eof { # EOF callback for main lei daemon
- my ($lei) = @_;
- my $pmd = delete $lei->{pmd} or return $lei->fail;
- $lei->sto_done_request($pmd->{lei_sock});
+ $_[0]->wq_eof('pmd');
}
1;
^ permalink raw reply related [relevance 58%]
* [PATCH 2/2] lei: move to v5.12 to avoid "use strict"
@ 2022-04-23 22:03 61% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2022-04-23 22:03 UTC (permalink / raw)
To: meta
Socket.pm still loads strict.pm, unfortunately, which hurts
startup time; but we'll save some LoC this way.
---
lib/PublicInbox/CmdIPC4.pm | 3 +--
lib/PublicInbox/LEI.pm | 3 +--
lib/PublicInbox/Spawn.pm | 2 +-
lib/PublicInbox/Syscall.pm | 3 +--
script/lei | 3 +--
5 files changed, 5 insertions(+), 9 deletions(-)
diff --git a/lib/PublicInbox/CmdIPC4.pm b/lib/PublicInbox/CmdIPC4.pm
index 76938b6d..e368d032 100644
--- a/lib/PublicInbox/CmdIPC4.pm
+++ b/lib/PublicInbox/CmdIPC4.pm
@@ -5,8 +5,7 @@
# first choice for script/lei front-end and 2nd choice for lei backend
# libsocket-msghdr-perl is in Debian but not many other distros as of 2021.
package PublicInbox::CmdIPC4;
-use strict;
-use v5.10.1;
+use v5.12;
use Socket qw(SOL_SOCKET SCM_RIGHTS);
BEGIN { eval {
require Socket::MsgHdr; # XS
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 93b4ea03..89aa4119 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -6,8 +6,7 @@
# local clients with read/write access to the FS and use as many
# system resources as the local user has access to.
package PublicInbox::LEI;
-use strict;
-use v5.10.1;
+use v5.12;
use parent qw(PublicInbox::DS PublicInbox::LeiExternal
PublicInbox::LeiQuery);
use Getopt::Long ();
diff --git a/lib/PublicInbox/Spawn.pm b/lib/PublicInbox/Spawn.pm
index 137b8087..3f69108a 100644
--- a/lib/PublicInbox/Spawn.pm
+++ b/lib/PublicInbox/Spawn.pm
@@ -15,7 +15,7 @@
# We don't want too many DSOs: https://udrepper.livejournal.com/8790.html
package PublicInbox::Spawn;
-use strict;
+use v5.12;
use parent qw(Exporter);
use Symbol qw(gensym);
use Fcntl qw(LOCK_EX SEEK_SET);
diff --git a/lib/PublicInbox/Syscall.pm b/lib/PublicInbox/Syscall.pm
index 22b779ad..777c44d0 100644
--- a/lib/PublicInbox/Syscall.pm
+++ b/lib/PublicInbox/Syscall.pm
@@ -16,8 +16,7 @@
# You may distribute under the terms of either the GNU General Public
# License or the Artistic License, as specified in the Perl README file.
package PublicInbox::Syscall;
-use strict;
-use v5.10.1;
+use v5.12;
use parent qw(Exporter);
use POSIX qw(ENOENT ENOSYS EINVAL O_NONBLOCK);
use Socket qw(SOL_SOCKET SCM_RIGHTS);
diff --git a/script/lei b/script/lei
index adef9944..5feb7751 100755
--- a/script/lei
+++ b/script/lei
@@ -1,8 +1,7 @@
#!perl -w
# Copyright (C) all contributors <meta@public-inbox.org>
# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
-use strict;
-use v5.10.1;
+use v5.12;
use Socket qw(AF_UNIX SOCK_SEQPACKET MSG_EOR pack_sockaddr_un);
use PublicInbox::CmdIPC4;
my $narg = 5;
^ permalink raw reply related [relevance 61%]
* [PATCH] lei: improve diagnosis of errors from children
@ 2022-04-30 21:04 71% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2022-04-30 21:04 UTC (permalink / raw)
To: meta
Not 100% sure what's going on, but maybe this helps.
---
Anybody else see this?
lib/PublicInbox/LEI.pm | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 89aa4119..d81ca296 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -1394,7 +1394,7 @@ sub wq_done_wait { # dwaitpid callback
local $current_lei = $lei;
my $err_type = $lei->{-err_type};
$? and $lei->child_error($?,
- $err_type ? "$err_type errors during $lei->{cmd}" : ());
+ $err_type ? "$err_type errors during $lei->{cmd} \$?=$?" : ());
$lei->dclose;
}
^ permalink raw reply related [relevance 71%]
* [PATCH] lei refresh-mail-sync: filter NNTP(S) from --all
@ 2022-04-30 21:29 71% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2022-04-30 21:29 UTC (permalink / raw)
To: meta
We currently do not support refresh from NNTP since deletes are
rare with public-inbox NNTP servers; but traditional Usenet
servers do delete/expire messages and we should probably support
that at some point.
---
lib/PublicInbox/LeiRefreshMailSync.pm | 2 ++
1 file changed, 2 insertions(+)
diff --git a/lib/PublicInbox/LeiRefreshMailSync.pm b/lib/PublicInbox/LeiRefreshMailSync.pm
index 7821008f..a60a9a5e 100644
--- a/lib/PublicInbox/LeiRefreshMailSync.pm
+++ b/lib/PublicInbox/LeiRefreshMailSync.pm
@@ -73,6 +73,8 @@ lei mail_sync.sqlite3 uninitialized, see lei-import(1)
EOM
if (defined(my $all = $lei->{opt}->{all})) {
$lms->group2folders($lei, $all, \@folders) or return;
+ # TODO: handle NNTP servers which delete messages
+ @folders = grep(!m!\Anntps?://!, @folders);
} else {
$lms->arg2folder($lei, \@folders); # may die
}
^ permalink raw reply related [relevance 71%]
* [PATCH] lei import: add label completions (+L:$LABEL)
@ 2022-05-02 18:10 71% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2022-05-02 18:10 UTC (permalink / raw)
To: meta
This can probably be added for "lei q", too, but we typically
import first. Labels can probably be made persistent on a
per-folder basis in the future.
---
lib/PublicInbox/LeiImport.pm | 2 ++
1 file changed, 2 insertions(+)
diff --git a/lib/PublicInbox/LeiImport.pm b/lib/PublicInbox/LeiImport.pm
index b9865829..2d91e4c4 100644
--- a/lib/PublicInbox/LeiImport.pm
+++ b/lib/PublicInbox/LeiImport.pm
@@ -117,6 +117,8 @@ sub _complete_import {
my ($lei, @argv) = @_;
my ($re, $cur, $match_cb) = $lei->complete_url_prepare(\@argv);
my @k = $lei->url_folder_cache->keys($argv[-1] // undef, 1);
+ my @L = eval { $lei->_lei_store->search->all_terms('L') };
+ push(@k, map { "+L:$_" } @L);
my @m = map { $match_cb->($_) } @k;
my %f = map { $_ => 1 } (@m ? @m : @k);
if (my $lms = $lei->lms) {
^ permalink raw reply related [relevance 71%]
* Trouble running lei
@ 2022-05-03 11:15 70% Filipe Manana
2022-05-03 11:37 71% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Filipe Manana @ 2022-05-03 11:15 UTC (permalink / raw)
To: meta
Hello,
I tried both 'master' branch and tag v1.8.0, I did the documented build steps:
$ git clone https://public-inbox.org/public-inbox.git/
$ cd public-inbox
$ perl Makefile.PL
$ make
$ echo $? # success, prints 0
$ cd certs
$ /usr/bin/perl ./create-certs.perl
$ cd ..
$ make test
(...)
All tests successful.
Files=157, Tests=6785, 491 wallclock secs ( 0.77 usr 0.13 sys + 39.65
cusr 14.90 csys = 55.45 CPU)
Result: PASS
$ make symlink-install
$HOME/bin is included in my $PATH, but when I run 'lei', I get an error:
$ lei q -o ~/Mail/overlay -I https://lore.kernel.org/all -t
'dfn:fs/btrfs/* AND rt:3.month.ago..'
Attempt to reload PublicInbox/LeiXSearch.pm aborted.
Compilation failed in require at
/home/fdmanana/git/hub/public-inbox/lib/PublicInbox/LeiQuery.pm line
74.
This is on a Ubuntu 20.04.3 LTS distro.
I have it working on a Debian SID box without any problems (it was set
up several months ago).
Any ideas about what's wrong?
Thank you.
^ permalink raw reply [relevance 70%]
* Re: Trouble running lei
2022-05-03 11:15 70% Trouble running lei Filipe Manana
@ 2022-05-03 11:37 71% ` Eric Wong
2022-05-03 12:50 71% ` Filipe Manana
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2022-05-03 11:37 UTC (permalink / raw)
To: Filipe Manana; +Cc: meta
Filipe Manana <fdmanana@kernel.org> wrote:
> Hello,
>
> I tried both 'master' branch and tag v1.8.0, I did the documented build steps:
>
> $ git clone https://public-inbox.org/public-inbox.git/
> $ cd public-inbox
>
> $ perl Makefile.PL
> $ make
> $ echo $? # success, prints 0
>
> $ cd certs
> $ /usr/bin/perl ./create-certs.perl
> $ cd ..
> $ make test
> (...)
Any messages about missing dependencies when running tests?
(e.g. Xapian)
Inline::C or Socket::Msghdr shouldn't be required with 1.8
on most arches...
> All tests successful.
> Files=157, Tests=6785, 491 wallclock secs ( 0.77 usr 0.13 sys + 39.65
> cusr 14.90 csys = 55.45 CPU)
Fwiw, "make check" or "make check-run" can be a lot faster on SMP.
> Result: PASS
>
> $ make symlink-install
>
> $HOME/bin is included in my $PATH, but when I run 'lei', I get an error:
>
> $ lei q -o ~/Mail/overlay -I https://lore.kernel.org/all -t
> 'dfn:fs/btrfs/* AND rt:3.month.ago..'
> Attempt to reload PublicInbox/LeiXSearch.pm aborted.
Odd, "Attempt to reload %s aborted" coming from perl itself means it
already tried and failed before. (man perldiag)
Were there previous errors from other commands?
Perhaps it's already running lei-daemon as an older version?
"lei daemon-kill" should kill it and it'll restart on the next
command, unless something else got wedged.
I've been meaning to make the daemon stuff "worth it"
w.r.t. automatic updates/flag/keyword sync (inotify/IDLE) but
haven't gotten around to it, yet :x
> Compilation failed in require at
> /home/fdmanana/git/hub/public-inbox/lib/PublicInbox/LeiQuery.pm line
> 74.
>
> This is on a Ubuntu 20.04.3 LTS distro.
> I have it working on a Debian SID box without any problems (it was set
> up several months ago).
>
> Any ideas about what's wrong?
Probably daemon-kill will get rid of an old version that's
already running (and verifying that it's actually dead).
^ permalink raw reply [relevance 71%]
* Re: Trouble running lei
2022-05-03 11:37 71% ` Eric Wong
@ 2022-05-03 12:50 71% ` Filipe Manana
2022-05-03 15:24 71% ` Konstantin Ryabitsev
0 siblings, 1 reply; 200+ results
From: Filipe Manana @ 2022-05-03 12:50 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
On Tue, May 3, 2022 at 12:37 PM Eric Wong <e@80x24.org> wrote:
>
> Filipe Manana <fdmanana@kernel.org> wrote:
> > Hello,
> >
> > I tried both 'master' branch and tag v1.8.0, I did the documented build steps:
> >
> > $ git clone https://public-inbox.org/public-inbox.git/
> > $ cd public-inbox
> >
> > $ perl Makefile.PL
> > $ make
> > $ echo $? # success, prints 0
> >
> > $ cd certs
> > $ /usr/bin/perl ./create-certs.perl
> > $ cd ..
> > $ make test
> > (...)
>
> Any messages about missing dependencies when running tests?
> (e.g. Xapian)
>
> Inline::C or Socket::Msghdr shouldn't be required with 1.8
> on most arches...
>
> > All tests successful.
> > Files=157, Tests=6785, 491 wallclock secs ( 0.77 usr 0.13 sys + 39.65
> > cusr 14.90 csys = 55.45 CPU)
>
> Fwiw, "make check" or "make check-run" can be a lot faster on SMP.
Oh, I didn't know about it. Good to know.
>
> > Result: PASS
> >
> > $ make symlink-install
> >
> > $HOME/bin is included in my $PATH, but when I run 'lei', I get an error:
> >
> > $ lei q -o ~/Mail/overlay -I https://lore.kernel.org/all -t
> > 'dfn:fs/btrfs/* AND rt:3.month.ago..'
> > Attempt to reload PublicInbox/LeiXSearch.pm aborted.
>
> Odd, "Attempt to reload %s aborted" coming from perl itself means it
> already tried and failed before. (man perldiag)
>
> Were there previous errors from other commands?
>
> Perhaps it's already running lei-daemon as an older version?
> "lei daemon-kill" should kill it and it'll restart on the next
> command, unless something else got wedged.
Ah, running "lei daemon-kill" fixed it.
I don't know if I did something wrong before, but after running that,
lei is now working fine.
Thanks!
>
> I've been meaning to make the daemon stuff "worth it"
> w.r.t. automatic updates/flag/keyword sync (inotify/IDLE) but
> haven't gotten around to it, yet :x
>
> > Compilation failed in require at
> > /home/fdmanana/git/hub/public-inbox/lib/PublicInbox/LeiQuery.pm line
> > 74.
> >
> > This is on a Ubuntu 20.04.3 LTS distro.
> > I have it working on a Debian SID box without any problems (it was set
> > up several months ago).
> >
> > Any ideas about what's wrong?
>
> Probably daemon-kill will get rid of an old version that's
> already running (and verifying that it's actually dead).
^ permalink raw reply [relevance 71%]
* Re: Trouble running lei
2022-05-03 12:50 71% ` Filipe Manana
@ 2022-05-03 15:24 71% ` Konstantin Ryabitsev
2022-05-03 20:32 71% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Konstantin Ryabitsev @ 2022-05-03 15:24 UTC (permalink / raw)
To: Filipe Manana; +Cc: Eric Wong, meta
On Tue, May 03, 2022 at 01:50:52PM +0100, Filipe Manana wrote:
> > Perhaps it's already running lei-daemon as an older version?
> > "lei daemon-kill" should kill it and it'll restart on the next
> > command, unless something else got wedged.
>
> Ah, running "lei daemon-kill" fixed it.
> I don't know if I did something wrong before, but after running that,
> lei is now working fine.
I think this is actually a common occurrence. Any way lei-daemon can recognize
when there's a version mismatch between itself and the binary talking to it?
Regards,
-K'
^ permalink raw reply [relevance 71%]
* Re: Trouble running lei
2022-05-03 15:24 71% ` Konstantin Ryabitsev
@ 2022-05-03 20:32 71% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2022-05-03 20:32 UTC (permalink / raw)
To: Konstantin Ryabitsev; +Cc: Filipe Manana, meta
Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> On Tue, May 03, 2022 at 01:50:52PM +0100, Filipe Manana wrote:
> > > Perhaps it's already running lei-daemon as an older version?
> > > "lei daemon-kill" should kill it and it'll restart on the next
> > > command, unless something else got wedged.
> >
> > Ah, running "lei daemon-kill" fixed it.
> > I don't know if I did something wrong before, but after running that,
> > lei is now working fine.
Good to know!
> I think this is actually a common occurrence. Any way lei-daemon can recognize
> when there's a version mismatch between itself and the binary talking to it?
Sorta, but even I manage to lose track of which commits cause
protocol or internal API changes since everything is lazy-loaded.
I think the best way would be to have inotify/kqueue watch the /PublicInbox/
directory and auto-restart iff no active commands are running...
^ permalink raw reply [relevance 71%]
* [PATCH 5/3] doc: lei-q: regenerate for patchid: help
@ 2022-06-22 7:50 58% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2022-06-22 7:50 UTC (permalink / raw)
To: meta; +Cc: Kyle Meyer
---
Oops, left this out of the original patch :x
Documentation/lei-q.pod | 49 +++++++++++++++++++++--------------------
1 file changed, 25 insertions(+), 24 deletions(-)
diff --git a/Documentation/lei-q.pod b/Documentation/lei-q.pod
index fd829655..1cbffba4 100644
--- a/Documentation/lei-q.pod
+++ b/Documentation/lei-q.pod
@@ -253,30 +253,31 @@ instances:
=for comment
AUTO-GENERATED-SEARCH-TERMS-BEGIN
- s: match within Subject e.g. s:"a quick brown fox"
- d: match date-time range, git "approxidate" formats supported
- Open-ended ranges such as `d:last.week..' and
- `d:..2.days.ago' are supported
- b: match within message body, including text attachments
- nq: match non-quoted text within message body
- q: match quoted text within message body
- n: match filename of attachment(s)
- t: match within the To header
- c: match within the Cc header
- f: match within the From header
- a: match within the To, Cc, and From headers
- tc: match within the To and Cc headers
- l: match contents of the List-Id header
- bs: match within the Subject and body
- dfn: match filename from diff
- dfa: match diff removed (-) lines
- dfb: match diff added (+) lines
- dfhh: match diff hunk header context (usually a function name)
- dfctx: match diff context lines
- dfpre: match pre-image git blob ID
- dfpost: match post-image git blob ID
- dfblob: match either pre or post-image git blob ID
- rt: match received time, like `d:' if sender's clock was correct
+ s: match within Subject e.g. s:"a quick brown fox"
+ d: match date-time range, git "approxidate" formats supported
+ Open-ended ranges such as `d:last.week..' and
+ `d:..2.days.ago' are supported
+ b: match within message body, including text attachments
+ nq: match non-quoted text within message body
+ q: match quoted text within message body
+ n: match filename of attachment(s)
+ t: match within the To header
+ c: match within the Cc header
+ f: match within the From header
+ a: match within the To, Cc, and From headers
+ tc: match within the To and Cc headers
+ l: match contents of the List-Id header
+ bs: match within the Subject and body
+ dfn: match filename from diff
+ dfa: match diff removed (-) lines
+ dfb: match diff added (+) lines
+ dfhh: match diff hunk header context (usually a function name)
+ dfctx: match diff context lines
+ dfpre: match pre-image git blob ID
+ dfpost: match post-image git blob ID
+ dfblob: match either pre or post-image git blob ID
+ patchid: match `git patch-id --stable' output
+ rt: match received time, like `d:' if sender's clock was correct
=for comment
AUTO-GENERATED-SEARCH-TERMS-END
^ permalink raw reply related [relevance 58%]
* lei missing mails
@ 2022-06-29 16:15 71% Rob Herring
2022-06-29 16:30 71% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Rob Herring @ 2022-06-29 16:15 UTC (permalink / raw)
To: meta
Hi,
I'm using lei with lore where I have 2 queries which overlap. Really,
one is a subset of the other. On those overlapping threads, I'm
finding that sometimes new messages are written to one mailbox and not
the other. (At least sometimes, the messages may be missing from all
mailboxes sometimes too. I'm not certain.) Using --remote-fudge-time
to force refetching seems to get the missing mails. I haven't found
anything strange in timestamps of the missing mails, but otherwise am
not sure how to debug this further. The queries are retrieving full
threads and the missing mails are in the threads, but not direct
matches to the queries. I realize that's not a lot of detail to go on.
Suggestions on debugging this further?
It might be helpful if lei could print out message-ids of messages
written to mailboxes.
Rob
^ permalink raw reply [relevance 71%]
* Re: lei missing mails
2022-06-29 16:15 71% lei missing mails Rob Herring
@ 2022-06-29 16:30 71% ` Eric Wong
2022-06-29 16:53 69% ` Rob Herring
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2022-06-29 16:30 UTC (permalink / raw)
To: Rob Herring; +Cc: meta
Rob Herring <robh@kernel.org> wrote:
> Hi,
>
> I'm using lei with lore where I have 2 queries which overlap. Really,
> one is a subset of the other. On those overlapping threads, I'm
> finding that sometimes new messages are written to one mailbox and not
> the other. (At least sometimes, the messages may be missing from all
> mailboxes sometimes too. I'm not certain.) Using --remote-fudge-time
> to force refetching seems to get the missing mails. I haven't found
> anything strange in timestamps of the missing mails, but otherwise am
> not sure how to debug this further. The queries are retrieving full
> threads and the missing mails are in the threads, but not direct
> matches to the queries. I realize that's not a lot of detail to go on.
> Suggestions on debugging this further?
Is this with 1.8 or 1.7?
I forgot to note in the release notes, but there were some
SQLite usage-related fixes which could avoid missing messages.
You'll need "lei daemon-kill" after upgrading to 1.8 to ensure
the new code is running.
What might be interesting is to use the URLs lei prints and
comparing the results w/o lei.
I'll have to double-check if overlapping affects things, but it
shouldn't; since the dedupe logic is per-output.
Is this exclusively with HTTPS endpoints and writing to Maildirs
(or something else?)
> It might be helpful if lei could print out message-ids of messages
> written to mailboxes.
That could get very noisy, especially as mailboxes are written
in parallel.
Thanks.
^ permalink raw reply [relevance 71%]
* Re: lei missing mails
2022-06-29 16:30 71% ` Eric Wong
@ 2022-06-29 16:53 69% ` Rob Herring
2022-06-29 17:27 71% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Rob Herring @ 2022-06-29 16:53 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
On Wed, Jun 29, 2022 at 10:30 AM Eric Wong <e@80x24.org> wrote:
>
> Rob Herring <robh@kernel.org> wrote:
> > Hi,
> >
> > I'm using lei with lore where I have 2 queries which overlap. Really,
> > one is a subset of the other. On those overlapping threads, I'm
> > finding that sometimes new messages are written to one mailbox and not
> > the other. (At least sometimes, the messages may be missing from all
> > mailboxes sometimes too. I'm not certain.) Using --remote-fudge-time
> > to force refetching seems to get the missing mails. I haven't found
> > anything strange in timestamps of the missing mails, but otherwise am
> > not sure how to debug this further. The queries are retrieving full
> > threads and the missing mails are in the threads, but not direct
> > matches to the queries. I realize that's not a lot of detail to go on.
> > Suggestions on debugging this further?
>
> Is this with 1.8 or 1.7?
Commit 68b53c888911 actually. So post 1.8.
> I forgot to note in the release notes, but there were some
> SQLite usage-related fixes which could avoid missing messages.
>
> You'll need "lei daemon-kill" after upgrading to 1.8 to ensure
> the new code is running.
It's possible I haven't done that since updating though I do vaguely
recall seeing something about needing to do that. Is there any way to
tell before I restart it?
> What might be interesting is to use the URLs lei prints and
> comparing the results w/o lei.
>
> I'll have to double-check if overlapping affects things, but it
> shouldn't; since the dedupe logic is per-output.
>
> Is this exclusively with HTTPS endpoints and writing to Maildirs
> (or something else?)
Yes. It's querying lore and writing to a maildir. Here's one of the queries:
[lei]
q = (dfn:drivers OR dfn:arch OR dfn:Documentation/* OR
dfn:include OR dfn:scripts) AND \
f:robh@kernel.org AND rt:6.month.ago..
[lei "q"]
include = https://lore.kernel.org/all/
external = 1
local = 1
remote = 1
threads = 1
dedupe = mid
output = maildir:/home/rob/Mail/my-patches
>
> > It might be helpful if lei could print out message-ids of messages
> > written to mailboxes.
>
> That could get very noisy, especially as mailboxes are written
> in parallel.
Verbose mode already is. Maybe specifying what info you want to be
verbose would help. The network side is mostly uninteresting in this
case for example.
Is there any tool to list new messages in a maildir? I could do that
before and after. I've done the clearing the new flag in mutt between
runs, but that's not really ideal.
Rob
^ permalink raw reply [relevance 69%]
* Re: lei missing mails
2022-06-29 16:53 69% ` Rob Herring
@ 2022-06-29 17:27 71% ` Eric Wong
2022-06-29 22:01 59% ` Rob Herring
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2022-06-29 17:27 UTC (permalink / raw)
To: Rob Herring; +Cc: meta
Rob Herring <robh@kernel.org> wrote:
> On Wed, Jun 29, 2022 at 10:30 AM Eric Wong <e@80x24.org> wrote:
> >
> > Rob Herring <robh@kernel.org> wrote:
> > > Hi,
> > >
> > > I'm using lei with lore where I have 2 queries which overlap. Really,
> > > one is a subset of the other. On those overlapping threads, I'm
> > > finding that sometimes new messages are written to one mailbox and not
> > > the other. (At least sometimes, the messages may be missing from all
> > > mailboxes sometimes too. I'm not certain.) Using --remote-fudge-time
> > > to force refetching seems to get the missing mails. I haven't found
> > > anything strange in timestamps of the missing mails, but otherwise am
> > > not sure how to debug this further. The queries are retrieving full
> > > threads and the missing mails are in the threads, but not direct
> > > matches to the queries. I realize that's not a lot of detail to go on.
> > > Suggestions on debugging this further?
> >
> > Is this with 1.8 or 1.7?
>
> Commit 68b53c888911 actually. So post 1.8.
OK, thanks for that info.
> > I forgot to note in the release notes, but there were some
> > SQLite usage-related fixes which could avoid missing messages.
> >
> > You'll need "lei daemon-kill" after upgrading to 1.8 to ensure
> > the new code is running.
>
> It's possible I haven't done that since updating though I do vaguely
> recall seeing something about needing to do that. Is there any way to
> tell before I restart it?
Not really, but it's pretty cheap to restart (assuming there's no
long-running jobs).
> > What might be interesting is to use the URLs lei prints and
> > comparing the results w/o lei.
> >
> > I'll have to double-check if overlapping affects things, but it
> > shouldn't; since the dedupe logic is per-output.
> >
> > Is this exclusively with HTTPS endpoints and writing to Maildirs
> > (or something else?)
>
> Yes. It's querying lore and writing to a maildir. Here's one of the queries:
>
> [lei]
> q = (dfn:drivers OR dfn:arch OR dfn:Documentation/* OR
> dfn:include OR dfn:scripts) AND \
> f:robh@kernel.org AND rt:6.month.ago..
> [lei "q"]
> include = https://lore.kernel.org/all/
> external = 1
> local = 1
> remote = 1
> threads = 1
> dedupe = mid
> output = maildir:/home/rob/Mail/my-patches
Fwiw, dedupe based on mid could be vulnerable to spoofing, which
is why `content' is the default. But yes, in the past, I've
noticed some messages to meta@public-inbox.org not showing up,
though not recently (I guess lack of activity here is a culprit :x)
I also just noticed an inotify-related bug deadlocking the whole
lei-deamon while looking into this :<
> > > It might be helpful if lei could print out message-ids of messages
> > > written to mailboxes.
> >
> > That could get very noisy, especially as mailboxes are written
> > in parallel.
>
> Verbose mode already is. Maybe specifying what info you want to be
> verbose would help. The network side is mostly uninteresting in this
> case for example.
Yes, I've been struggling with the verbosity, too; and many
other things :<
> Is there any tool to list new messages in a maildir? I could do that
> before and after. I've done the clearing the new flag in mutt between
> runs, but that's not really ideal.
I suppose `ls'. There are likely other tools more suited for Maildirs
but I'm not familiar with them off the top of my head.
Maybe lei could grow yet another command.
^ permalink raw reply [relevance 71%]
* Re: lei missing mails
2022-06-29 17:27 71% ` Eric Wong
@ 2022-06-29 22:01 59% ` Rob Herring
2022-06-30 8:55 71% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Rob Herring @ 2022-06-29 22:01 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
On Wed, Jun 29, 2022 at 11:27 AM Eric Wong <e@80x24.org> wrote:
>
> Rob Herring <robh@kernel.org> wrote:
> > On Wed, Jun 29, 2022 at 10:30 AM Eric Wong <e@80x24.org> wrote:
> > >
> > > Rob Herring <robh@kernel.org> wrote:
> > > > Hi,
> > > >
> > > > I'm using lei with lore where I have 2 queries which overlap. Really,
> > > > one is a subset of the other. On those overlapping threads, I'm
> > > > finding that sometimes new messages are written to one mailbox and not
> > > > the other. (At least sometimes, the messages may be missing from all
> > > > mailboxes sometimes too. I'm not certain.) Using --remote-fudge-time
> > > > to force refetching seems to get the missing mails. I haven't found
> > > > anything strange in timestamps of the missing mails, but otherwise am
> > > > not sure how to debug this further. The queries are retrieving full
> > > > threads and the missing mails are in the threads, but not direct
> > > > matches to the queries. I realize that's not a lot of detail to go on.
> > > > Suggestions on debugging this further?
> > >
> > > Is this with 1.8 or 1.7?
> >
> > Commit 68b53c888911 actually. So post 1.8.
>
> OK, thanks for that info.
>
> > > I forgot to note in the release notes, but there were some
> > > SQLite usage-related fixes which could avoid missing messages.
> > >
> > > You'll need "lei daemon-kill" after upgrading to 1.8 to ensure
> > > the new code is running.
> >
> > It's possible I haven't done that since updating though I do vaguely
> > recall seeing something about needing to do that. Is there any way to
> > tell before I restart it?
>
> Not really, but it's pretty cheap to restart (assuming there's no
> long-running jobs).
I've restarted and just hit this again.
> > > What might be interesting is to use the URLs lei prints and
> > > comparing the results w/o lei.
$ lei up --all
# updating /home/rob/Mail/from-me
# updating /home/rob/Mail/missing-cc
# updating /home/rob/Mail/my-patches
# updating /home/rob/Mail/pci
# https://lore.kernel.org/all/ limiting to 2022-06-27 12:42 -0600 and newer
# https://lore.kernel.org/all/ limiting to 2022-06-27 9:50 -0600 and newer
# https://lore.kernel.org/all/ limiting to 2022-06-27 12:42 -0600 and newer
# /usr/bin/curl -Sf -s -d ''
https://lore.kernel.org/all/?x=m&t=1&q=(dt%3A20220529211430..+AND+(f%3Arobh%40kernel.org+OR+f%3Arobh%2Bdt%40kernel.org))+AND+dt%3A20220627184226..
# /home/rob/.local/share/lei/store 144/144
# /home/rob/.local/share/lei/store 3/3
# /usr/bin/curl -Sf -s -d ''
https://lore.kernel.org/all/?x=m&t=1&q=((dfn%3Adrivers+OR+dfn%3Aarch+OR+dfn%3ADocumentation%2F*+OR+dfn%3Ainclude+OR+dfn%3Ascripts)+AND+f%3Arobh%40kernel.org+AND+rt%3A1640812470..)+AND+dt%3A20220627155025..
# /usr/bin/curl -Sf -s -d ''
https://lore.kernel.org/all/?x=m&t=1&q=(l%3Alinux-pci+dfn%3Adrivers%2Fpci%2Fcontroller+dt%3A20220529211430..)+AND+dt%3A20220627184226..
# /home/rob/.local/share/lei/store 0/0
# /home/rob/.local/share/lei/store 362/362
# 0 written to /home/rob/Mail/missing-cc/ (0 matches)
# https://lore.kernel.org/all/ 72/72
# https://lore.kernel.org/all/ 4/4
# https://lore.kernel.org/all/ 131/?
# https://lore.kernel.org/all/ 184/?
# https://lore.kernel.org/all/ 412/?
# https://lore.kernel.org/all/ 603/?
# https://lore.kernel.org/all/ 853/?
# https://lore.kernel.org/all/ 1069/?
# https://lore.kernel.org/all/ 1442/?
# https://lore.kernel.org/all/ 1443/1443
# 1 written to /home/rob/Mail/pci/ (75 matches)
# 2 written to /home/rob/Mail/my-patches/ (148 matches)
# 7 written to /home/rob/Mail/from-me/ (1805 matches)
What I expected was 3 messages written to 'my-patches'.
I think the problem is just simply that the new message missing
doesn't match the query, but is a reply to a match. So with a date
after the original match in the thread won't pick up anything. The 2nd
URL above indeed only has 2 results. I guess I just have to fetch a
wider window like a month every time? What's needed is a get any new
messages in existing threads. I don't suppose there's an efficient way
to do that?
> > >
> > > I'll have to double-check if overlapping affects things, but it
> > > shouldn't; since the dedupe logic is per-output.
> > >
> > > Is this exclusively with HTTPS endpoints and writing to Maildirs
> > > (or something else?)
> >
> > Yes. It's querying lore and writing to a maildir. Here's one of the queries:
> >
> > [lei]
> > q = (dfn:drivers OR dfn:arch OR dfn:Documentation/* OR
> > dfn:include OR dfn:scripts) AND \
> > f:robh@kernel.org AND rt:6.month.ago..
> > [lei "q"]
> > include = https://lore.kernel.org/all/
> > external = 1
> > local = 1
> > remote = 1
> > threads = 1
> > dedupe = mid
> > output = maildir:/home/rob/Mail/my-patches
>
> Fwiw, dedupe based on mid could be vulnerable to spoofing, which
> is why `content' is the default. But yes, in the past, I've
> noticed some messages to meta@public-inbox.org not showing up,
> though not recently (I guess lack of activity here is a culprit :x)
Does 'content' ignore trailers that mailman lists like to add? I think
I switched because of that.
Rob
^ permalink raw reply [relevance 59%]
* Re: lei missing mails
2022-06-29 22:01 59% ` Rob Herring
@ 2022-06-30 8:55 71% ` Eric Wong
2022-07-07 9:48 71% ` Eric Wong
2022-07-11 21:59 68% ` Rob Herring
0 siblings, 2 replies; 200+ results
From: Eric Wong @ 2022-06-30 8:55 UTC (permalink / raw)
To: Rob Herring; +Cc: meta
Rob Herring <robh@kernel.org> wrote:
> On Wed, Jun 29, 2022 at 11:27 AM Eric Wong <e@80x24.org> wrote:
> > Rob Herring <robh@kernel.org> wrote:
> > > On Wed, Jun 29, 2022 at 10:30 AM Eric Wong <e@80x24.org> wrote:
> > > > Rob Herring <robh@kernel.org> wrote:
> > > > > Hi,
> > > > >
> > > > > I'm using lei with lore where I have 2 queries which overlap. Really,
> > > > > one is a subset of the other. On those overlapping threads, I'm
> > > > > finding that sometimes new messages are written to one mailbox and not
> > > > > the other. (At least sometimes, the messages may be missing from all
> > > > > mailboxes sometimes too. I'm not certain.) Using --remote-fudge-time
> > > > > to force refetching seems to get the missing mails. I haven't found
> > > > > anything strange in timestamps of the missing mails, but otherwise am
> > > > > not sure how to debug this further. The queries are retrieving full
> > > > > threads and the missing mails are in the threads, but not direct
> > > > > matches to the queries. I realize that's not a lot of detail to go on.
> > > > > Suggestions on debugging this further?
> > > >
> > > > Is this with 1.8 or 1.7?
> > >
> > > Commit 68b53c888911 actually. So post 1.8.
> >
> > OK, thanks for that info.
> >
> > > > I forgot to note in the release notes, but there were some
> > > > SQLite usage-related fixes which could avoid missing messages.
> > > >
> > > > You'll need "lei daemon-kill" after upgrading to 1.8 to ensure
> > > > the new code is running.
> > >
> > > It's possible I haven't done that since updating though I do vaguely
> > > recall seeing something about needing to do that. Is there any way to
> > > tell before I restart it?
> >
> > Not really, but it's pretty cheap to restart (assuming there's no
> > long-running jobs).
>
> I've restarted and just hit this again.
Ugh, sorry to hear that :<
> > > > What might be interesting is to use the URLs lei prints and
> > > > comparing the results w/o lei.
>
> $ lei up --all
> # updating /home/rob/Mail/from-me
> # updating /home/rob/Mail/missing-cc
> # updating /home/rob/Mail/my-patches
> # updating /home/rob/Mail/pci
> # https://lore.kernel.org/all/ limiting to 2022-06-27 12:42 -0600 and newer
> # https://lore.kernel.org/all/ limiting to 2022-06-27 9:50 -0600 and newer
> # https://lore.kernel.org/all/ limiting to 2022-06-27 12:42 -0600 and newer
> # /usr/bin/curl -Sf -s -d ''
> https://lore.kernel.org/all/?x=m&t=1&q=(dt%3A20220529211430..+AND+(f%3Arobh%40kernel.org+OR+f%3Arobh%2Bdt%40kernel.org))+AND+dt%3A20220627184226..
> # /home/rob/.local/share/lei/store 144/144
> # /home/rob/.local/share/lei/store 3/3
> # /usr/bin/curl -Sf -s -d ''
> https://lore.kernel.org/all/?x=m&t=1&q=((dfn%3Adrivers+OR+dfn%3Aarch+OR+dfn%3ADocumentation%2F*+OR+dfn%3Ainclude+OR+dfn%3Ascripts)+AND+f%3Arobh%40kernel.org+AND+rt%3A1640812470..)+AND+dt%3A20220627155025..
> # /usr/bin/curl -Sf -s -d ''
> https://lore.kernel.org/all/?x=m&t=1&q=(l%3Alinux-pci+dfn%3Adrivers%2Fpci%2Fcontroller+dt%3A20220529211430..)+AND+dt%3A20220627184226..
> # /home/rob/.local/share/lei/store 0/0
> # /home/rob/.local/share/lei/store 362/362
> # 0 written to /home/rob/Mail/missing-cc/ (0 matches)
> # https://lore.kernel.org/all/ 72/72
> # https://lore.kernel.org/all/ 4/4
> # https://lore.kernel.org/all/ 131/?
> # https://lore.kernel.org/all/ 184/?
> # https://lore.kernel.org/all/ 412/?
> # https://lore.kernel.org/all/ 603/?
> # https://lore.kernel.org/all/ 853/?
> # https://lore.kernel.org/all/ 1069/?
> # https://lore.kernel.org/all/ 1442/?
> # https://lore.kernel.org/all/ 1443/1443
> # 1 written to /home/rob/Mail/pci/ (75 matches)
> # 2 written to /home/rob/Mail/my-patches/ (148 matches)
> # 7 written to /home/rob/Mail/from-me/ (1805 matches)
>
>
> What I expected was 3 messages written to 'my-patches'.
>
> I think the problem is just simply that the new message missing
> doesn't match the query, but is a reply to a match. So with a date
> after the original match in the thread won't pick up anything. The 2nd
> URL above indeed only has 2 results. I guess I just have to fetch a
> wider window like a month every time? What's needed is a get any new
> messages in existing threads. I don't suppose there's an efficient way
> to do that?
No, I don't think so. I think this is a separate issue in lei...
"t=1" in the remote query expands threads in a time-agnostic
way, so I don't think that's the problem (though I may be wrong...).
I'll have to check more closely this week (still stuck with POP3
user account/storage issues :<)
> > > >
> > > > I'll have to double-check if overlapping affects things, but it
> > > > shouldn't; since the dedupe logic is per-output.
> > > >
> > > > Is this exclusively with HTTPS endpoints and writing to Maildirs
> > > > (or something else?)
> > >
> > > Yes. It's querying lore and writing to a maildir. Here's one of the queries:
> > >
> > > [lei]
> > > q = (dfn:drivers OR dfn:arch OR dfn:Documentation/* OR
> > > dfn:include OR dfn:scripts) AND \
> > > f:robh@kernel.org AND rt:6.month.ago..
> > > [lei "q"]
> > > include = https://lore.kernel.org/all/
> > > external = 1
> > > local = 1
> > > remote = 1
> > > threads = 1
> > > dedupe = mid
> > > output = maildir:/home/rob/Mail/my-patches
> >
> > Fwiw, dedupe based on mid could be vulnerable to spoofing, which
> > is why `content' is the default. But yes, in the past, I've
> > noticed some messages to meta@public-inbox.org not showing up,
> > though not recently (I guess lack of activity here is a culprit :x)
>
> Does 'content' ignore trailers that mailman lists like to add? I think
> I switched because of that.
No, unfortunately not. Hopefully the admins can be convinced to
get rid of trailers (I'm happy vger did so a few years back).
But I'd rather deal with duplicates than miss messages (there
have been legitimate messages in the past which reused msgids,
unfortunately).
^ permalink raw reply [relevance 71%]
* [PATCH 0/2] lei: minor diagnostic improvement
@ 2022-07-07 9:40 71% Eric Wong
2022-07-07 9:40 71% ` [PATCH 1/2] lei_xsearch: simplify lei/store import check Eric Wong
2022-07-07 9:40 53% ` [PATCH 2/2] lei: track seen messages to note duplicates Eric Wong
0 siblings, 2 replies; 200+ results
From: Eric Wong @ 2022-07-07 9:40 UTC (permalink / raw)
To: meta
Still trying to consistently reproduce the source of missing
messages. It may be isolated to HTTP(S) remotes, or not, but
noting the number of seen vs. written messages ought to be a
reasonable start.
patch 1/2 is just a tiny simplification I noticed along the way
Eric Wong (2):
lei_xsearch: simplify lei/store import check
lei: track seen messages to note duplicates
lib/PublicInbox/LeiConvert.pm | 8 +++++---
lib/PublicInbox/LeiToMail.pm | 13 ++++++++++---
lib/PublicInbox/LeiXSearch.pm | 24 ++++++++++++++----------
3 files changed, 29 insertions(+), 16 deletions(-)
^ permalink raw reply [relevance 71%]
* [PATCH 1/2] lei_xsearch: simplify lei/store import check
2022-07-07 9:40 71% [PATCH 0/2] lei: minor diagnostic improvement Eric Wong
@ 2022-07-07 9:40 71% ` Eric Wong
2022-07-07 9:40 53% ` [PATCH 2/2] lei: track seen messages to note duplicates Eric Wong
1 sibling, 0 replies; 200+ results
From: Eric Wong @ 2022-07-07 9:40 UTC (permalink / raw)
To: meta
There's no need to check for two fields when one will suffice.
---
lib/PublicInbox/LeiXSearch.pm | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/lib/PublicInbox/LeiXSearch.pm b/lib/PublicInbox/LeiXSearch.pm
index 2958d3f9..41e79856 100644
--- a/lib/PublicInbox/LeiXSearch.pm
+++ b/lib/PublicInbox/LeiXSearch.pm
@@ -285,7 +285,7 @@ sub each_remote_eml { # callback for MboxReader->mboxrd
my ($res, $kw) = $self->{import_sto}->wq_do('add_eml', $eml);
if (ref($res) eq ref($smsg)) { # totally new message
$smsg = $res;
- $self->{-imported} = 1;
+ $self->{-sto_imported} = 1;
}
$smsg->{kw} = $kw; # short-circuit xsmsg_vmd
}
@@ -376,7 +376,7 @@ sub query_remote_mboxrd {
$fh = IO::Uncompress::Gunzip->new($fh, MultiStream => 1);
PublicInbox::MboxReader->mboxrd($fh, \&each_remote_eml, $self,
$lei, $each_smsg);
- if ($self->{import_sto} && delete($self->{-imported})) {
+ if (delete($self->{-sto_imported})) {
my $wait = $self->{import_sto}->wq_do('done');
}
$reap_curl->join;
^ permalink raw reply related [relevance 71%]
* [PATCH 2/2] lei: track seen messages to note duplicates
2022-07-07 9:40 71% [PATCH 0/2] lei: minor diagnostic improvement Eric Wong
2022-07-07 9:40 71% ` [PATCH 1/2] lei_xsearch: simplify lei/store import check Eric Wong
@ 2022-07-07 9:40 53% ` Eric Wong
1 sibling, 0 replies; 200+ results
From: Eric Wong @ 2022-07-07 9:40 UTC (permalink / raw)
To: meta
This may help track down deduplication or other bugs in lei
which lead to occasionally missing messages.
Link: https://public-inbox.org/meta/CAL_JsqJH8xx_2NyZffNsRXbGXiv3kjmCETvKXt3Yfb0uToLm9Q@mail.gmail.com/
---
lib/PublicInbox/LeiConvert.pm | 8 +++++---
lib/PublicInbox/LeiToMail.pm | 13 ++++++++++---
lib/PublicInbox/LeiXSearch.pm | 20 ++++++++++++--------
3 files changed, 27 insertions(+), 14 deletions(-)
diff --git a/lib/PublicInbox/LeiConvert.pm b/lib/PublicInbox/LeiConvert.pm
index 906f3026..59af40de 100644
--- a/lib/PublicInbox/LeiConvert.pm
+++ b/lib/PublicInbox/LeiConvert.pm
@@ -1,4 +1,4 @@
-# Copyright (C) 2021 all contributors <meta@public-inbox.org>
+# Copyright (C) all contributors <meta@public-inbox.org>
# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
# front-end for the "lei convert" sub-command
@@ -35,8 +35,10 @@ sub process_inputs { # via wq_do
my $lei = $self->{lei};
delete $lei->{1};
delete $self->{wcb}; # commit
- my $nr = delete($lei->{-nr_write}) // 0;
- $lei->qerr("# converted $nr messages");
+ my $nr_w = delete($lei->{-nr_write}) // 0;
+ my $d = (delete($lei->{-nr_seen}) // 0) - $nr_w;
+ $d = $d ? " ($d duplicates)" : '';
+ $lei->qerr("# converted $nr_w messages$d");
}
sub lei_convert { # the main "lei convert" method
diff --git a/lib/PublicInbox/LeiToMail.pm b/lib/PublicInbox/LeiToMail.pm
index 3c5e7e59..2aa3977e 100644
--- a/lib/PublicInbox/LeiToMail.pm
+++ b/lib/PublicInbox/LeiToMail.pm
@@ -1,4 +1,4 @@
-# Copyright (C) 2020-2021 all contributors <meta@public-inbox.org>
+# Copyright (C) all contributors <meta@public-inbox.org>
# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
# Writes PublicInbox::Eml objects atomically to a mbox variant or Maildir
@@ -197,6 +197,7 @@ sub _mbox_write_cb ($$) {
sub { # for git_to_mail
my ($buf, $smsg, $eml) = @_;
$eml //= PublicInbox::Eml->new($buf);
+ ++$lei->{-nr_seen};
return if $dedupe->is_dup($eml, $smsg);
$lse->xsmsg_vmd($smsg) if $lse;
$smsg->{-recent} = 1 if $set_recent;
@@ -291,6 +292,8 @@ sub _maildir_write_cb ($$) {
sub { # for git_to_mail
my ($bref, $smsg, $eml) = @_;
$dst // return $lei->fail; # dst may be undef-ed in last run
+
+ ++$lei->{-nr_seen};
return if $dedupe && $dedupe->is_dup($eml //
PublicInbox::Eml->new($$bref),
$smsg);
@@ -317,6 +320,8 @@ sub _imap_write_cb ($$) {
sub { # for git_to_mail
my ($bref, $smsg, $eml) = @_;
$mic // return $lei->fail; # mic may be undef-ed in last run
+
+ ++$lei->{-nr_seen};
return if $dedupe && $dedupe->is_dup($eml //
PublicInbox::Eml->new($$bref),
$smsg);
@@ -360,6 +365,7 @@ sub _v2_write_cb ($$) {
sub { # for git_to_mail
my ($bref, $smsg, $eml) = @_;
$eml //= PublicInbox::Eml->new($bref);
+ ++$lei->{-nr_seen};
return if $dedupe && $dedupe->is_dup($eml, $smsg);
$lei->{v2w}->wq_do('add', $eml); # V2Writable->add
++$lei->{-nr_write};
@@ -792,9 +798,10 @@ sub wq_atexit_child {
my $lei = $self->{lei};
delete $self->{wcb};
$lei->{ale}->git->async_wait_all;
- my $nr = delete($lei->{-nr_write}) or return;
+ my ($nr_w, $nr_s) = delete(@$lei{qw(-nr_write -nr_seen)});
+ $nr_s or return;
return if $lei->{early_mua} || !$lei->{-progress} || !$lei->{pkt_op_p};
- $lei->{pkt_op_p}->pkt_do('l2m_progress', $nr);
+ $lei->{pkt_op_p}->pkt_do('l2m_progress', $nr_w, $nr_s);
}
# runs on a 1s timer in lei-daemon
diff --git a/lib/PublicInbox/LeiXSearch.pm b/lib/PublicInbox/LeiXSearch.pm
index 41e79856..6f877019 100644
--- a/lib/PublicInbox/LeiXSearch.pm
+++ b/lib/PublicInbox/LeiXSearch.pm
@@ -1,4 +1,4 @@
-# Copyright (C) 2020-2021 all contributors <meta@public-inbox.org>
+# Copyright (C) all contributors <meta@public-inbox.org>
# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
# Combine any combination of PublicInbox::Search,
@@ -163,8 +163,9 @@ sub mset_progress {
}
sub l2m_progress {
- my ($lei, $nr) = @_;
- $lei->{-nr_write} += $nr;
+ my ($lei, $nr_write, $nr_seen) = @_;
+ $lei->{-nr_write} += $nr_write;
+ $lei->{-nr_seen} += $nr_seen;
}
sub query_one_mset { # for --threads and l2m w/o sort
@@ -447,13 +448,16 @@ Error closing $lei->{ovv}->{dst}: \$!=$! \$?=$?
}
if ($lei->{-progress}) {
my $tot = $lei->{-mset_total} // 0;
- my $nr = $lei->{-nr_write} // 0;
+ my $nr_w = $lei->{-nr_write} // 0;
+ my $d = ($lei->{-nr_seen} // 0) - $nr_w;
+ my $x = "$tot matches";
+ $x .= ", $d duplicates" if $d;
if ($l2m) {
- my $m = "# $nr written to " .
- "$lei->{ovv}->{dst} ($tot matches)";
- $nr ? $lei->qfin($m) : $lei->qerr($m);
+ my $m = "# $nr_w written to " .
+ "$lei->{ovv}->{dst} ($x)";
+ $nr_w ? $lei->qfin($m) : $lei->qerr($m);
} else {
- $lei->qerr("# $tot matches");
+ $lei->qerr("# $x");
}
}
$lei->start_mua if $l2m && !$l2m->lock_free;
^ permalink raw reply related [relevance 53%]
* Re: lei missing mails
2022-06-30 8:55 71% ` Eric Wong
@ 2022-07-07 9:48 71% ` Eric Wong
2022-07-11 21:17 71% ` Rob Herring
2022-07-11 21:59 68% ` Rob Herring
1 sibling, 1 reply; 200+ results
From: Eric Wong @ 2022-07-07 9:48 UTC (permalink / raw)
To: Rob Herring; +Cc: meta
Eric Wong <e@80x24.org> wrote:
> Rob Herring <robh@kernel.org> wrote:
> > I've restarted and just hit this again.
>
> Ugh, sorry to hear that :<
Btw, is this only with HTTP(S) endpoints? (not local mirrors?)
I wonder if it's something wrong on the HTTP end (server or
client), but did more digging the other day and couldn't
reproduce the problem...
I just posted a very minor diagnostic change which might help
(forgot to Cc you :x)
https://public-inbox.org/meta/20220707094030.1185793-1-e@80x24.org/
As usual, daemon-kill is necessary after upgrading.
^ permalink raw reply [relevance 71%]
* Re: lei missing mails
2022-07-07 9:48 71% ` Eric Wong
@ 2022-07-11 21:17 71% ` Rob Herring
0 siblings, 0 replies; 200+ results
From: Rob Herring @ 2022-07-11 21:17 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
On Thu, Jul 7, 2022 at 3:48 AM Eric Wong <e@80x24.org> wrote:
>
> Eric Wong <e@80x24.org> wrote:
> > Rob Herring <robh@kernel.org> wrote:
> > > I've restarted and just hit this again.
> >
> > Ugh, sorry to hear that :<
>
> Btw, is this only with HTTP(S) endpoints? (not local mirrors?)
Yes. Only in that https is the only case I've tried. Don't know about
local mirrors because I don't know how to set that up...
Rob
^ permalink raw reply [relevance 71%]
* Re: lei missing mails
2022-06-30 8:55 71% ` Eric Wong
2022-07-07 9:48 71% ` Eric Wong
@ 2022-07-11 21:59 68% ` Rob Herring
2022-07-18 23:41 71% ` Eric Wong
1 sibling, 1 reply; 200+ results
From: Rob Herring @ 2022-07-11 21:59 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
On Thu, Jun 30, 2022 at 2:55 AM Eric Wong <e@80x24.org> wrote:
>
> Rob Herring <robh@kernel.org> wrote:
> > On Wed, Jun 29, 2022 at 11:27 AM Eric Wong <e@80x24.org> wrote:
> > > Rob Herring <robh@kernel.org> wrote:
> > > > On Wed, Jun 29, 2022 at 10:30 AM Eric Wong <e@80x24.org> wrote:
> > > > > Rob Herring <robh@kernel.org> wrote:
> > > > > > Hi,
> > > > > >
> > > > > > I'm using lei with lore where I have 2 queries which overlap. Really,
> > > > > > one is a subset of the other. On those overlapping threads, I'm
> > > > > > finding that sometimes new messages are written to one mailbox and not
> > > > > > the other. (At least sometimes, the messages may be missing from all
> > > > > > mailboxes sometimes too. I'm not certain.) Using --remote-fudge-time
> > > > > > to force refetching seems to get the missing mails. I haven't found
> > > > > > anything strange in timestamps of the missing mails, but otherwise am
> > > > > > not sure how to debug this further. The queries are retrieving full
> > > > > > threads and the missing mails are in the threads, but not direct
> > > > > > matches to the queries. I realize that's not a lot of detail to go on.
> > > > > > Suggestions on debugging this further?
> > > > >
> > > > > Is this with 1.8 or 1.7?
> > > >
> > > > Commit 68b53c888911 actually. So post 1.8.
> > >
> > > OK, thanks for that info.
> > >
> > > > > I forgot to note in the release notes, but there were some
> > > > > SQLite usage-related fixes which could avoid missing messages.
> > > > >
> > > > > You'll need "lei daemon-kill" after upgrading to 1.8 to ensure
> > > > > the new code is running.
> > > >
> > > > It's possible I haven't done that since updating though I do vaguely
> > > > recall seeing something about needing to do that. Is there any way to
> > > > tell before I restart it?
> > >
> > > Not really, but it's pretty cheap to restart (assuming there's no
> > > long-running jobs).
> >
> > I've restarted and just hit this again.
>
> Ugh, sorry to hear that :<
>
> > > > > What might be interesting is to use the URLs lei prints and
> > > > > comparing the results w/o lei.
> >
> > $ lei up --all
> > # updating /home/rob/Mail/from-me
> > # updating /home/rob/Mail/missing-cc
> > # updating /home/rob/Mail/my-patches
> > # updating /home/rob/Mail/pci
> > # https://lore.kernel.org/all/ limiting to 2022-06-27 12:42 -0600 and newer
> > # https://lore.kernel.org/all/ limiting to 2022-06-27 9:50 -0600 and newer
> > # https://lore.kernel.org/all/ limiting to 2022-06-27 12:42 -0600 and newer
> > # /usr/bin/curl -Sf -s -d ''
> > https://lore.kernel.org/all/?x=m&t=1&q=(dt%3A20220529211430..+AND+(f%3Arobh%40kernel.org+OR+f%3Arobh%2Bdt%40kernel.org))+AND+dt%3A20220627184226..
> > # /home/rob/.local/share/lei/store 144/144
> > # /home/rob/.local/share/lei/store 3/3
> > # /usr/bin/curl -Sf -s -d ''
> > https://lore.kernel.org/all/?x=m&t=1&q=((dfn%3Adrivers+OR+dfn%3Aarch+OR+dfn%3ADocumentation%2F*+OR+dfn%3Ainclude+OR+dfn%3Ascripts)+AND+f%3Arobh%40kernel.org+AND+rt%3A1640812470..)+AND+dt%3A20220627155025..
> > # /usr/bin/curl -Sf -s -d ''
> > https://lore.kernel.org/all/?x=m&t=1&q=(l%3Alinux-pci+dfn%3Adrivers%2Fpci%2Fcontroller+dt%3A20220529211430..)+AND+dt%3A20220627184226..
> > # /home/rob/.local/share/lei/store 0/0
> > # /home/rob/.local/share/lei/store 362/362
> > # 0 written to /home/rob/Mail/missing-cc/ (0 matches)
> > # https://lore.kernel.org/all/ 72/72
> > # https://lore.kernel.org/all/ 4/4
> > # https://lore.kernel.org/all/ 131/?
> > # https://lore.kernel.org/all/ 184/?
> > # https://lore.kernel.org/all/ 412/?
> > # https://lore.kernel.org/all/ 603/?
> > # https://lore.kernel.org/all/ 853/?
> > # https://lore.kernel.org/all/ 1069/?
> > # https://lore.kernel.org/all/ 1442/?
> > # https://lore.kernel.org/all/ 1443/1443
> > # 1 written to /home/rob/Mail/pci/ (75 matches)
> > # 2 written to /home/rob/Mail/my-patches/ (148 matches)
> > # 7 written to /home/rob/Mail/from-me/ (1805 matches)
> >
> >
> > What I expected was 3 messages written to 'my-patches'.
> >
> > I think the problem is just simply that the new message missing
> > doesn't match the query, but is a reply to a match. So with a date
> > after the original match in the thread won't pick up anything. The 2nd
> > URL above indeed only has 2 results. I guess I just have to fetch a
> > wider window like a month every time? What's needed is a get any new
> > messages in existing threads. I don't suppose there's an efficient way
> > to do that?
>
> No, I don't think so. I think this is a separate issue in lei...
> "t=1" in the remote query expands threads in a time-agnostic
> way, so I don't think that's the problem (though I may be wrong...).
Based on what the web interface presents, it sure seems like 't=1' is
independent of the query. The results listed are only those that match
the query and date range on the match.
For example, this query returns 3 matches:
https://lore.kernel.org/all/?x=m&t=1&q=((dfn%3Adrivers+OR+dfn%3Aarch+OR+dfn%3ADocumentation%2F*+OR+dfn%3Ainclude+OR+dfn%3Ascripts)+AND+f%3Arobh%40kernel.org+AND+rt%3A1641934905..)+AND+dt%3A20220630203819..
If I change 'dt' to 1 day earlier, I get 1 more match:
https://lore.kernel.org/all/?x=m&t=1&q=((dfn%3Adrivers+OR+dfn%3Aarch+OR+dfn%3ADocumentation%2F*+OR+dfn%3Ainclude+OR+dfn%3Ascripts)+AND+f%3Arobh%40kernel.org+AND+rt%3A1641934905..)+AND+dt%3A20220629203819..
That 4th match has a reply after 6/30, but the 1st query will not get
the reply. This is all reproducible without lei involved at all.
What seems to be needed is a 'thread date' which is the latest time
for any message in a thread that matches. Or perhaps some way to
separate the query from what's transferred. IOW, query for X, but only
send results newer than some date.
Rob
^ permalink raw reply [relevance 68%]
* Re: lei missing mails
2022-07-11 21:59 68% ` Rob Herring
@ 2022-07-18 23:41 71% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2022-07-18 23:41 UTC (permalink / raw)
To: Rob Herring; +Cc: meta
Rob Herring <robh@kernel.org> wrote:
> On Thu, Jun 30, 2022 at 2:55 AM Eric Wong <e@80x24.org> wrote:
> >
> > Rob Herring <robh@kernel.org> wrote:
> > > On Wed, Jun 29, 2022 at 11:27 AM Eric Wong <e@80x24.org> wrote:
> > > > Rob Herring <robh@kernel.org> wrote:
> > > > > On Wed, Jun 29, 2022 at 10:30 AM Eric Wong <e@80x24.org> wrote:
> > > > > > Rob Herring <robh@kernel.org> wrote:
> > > > > > > Hi,
> > > > > > >
> > > > > > > I'm using lei with lore where I have 2 queries which overlap. Really,
> > > > > > > one is a subset of the other. On those overlapping threads, I'm
> > > > > > > finding that sometimes new messages are written to one mailbox and not
> > > > > > > the other. (At least sometimes, the messages may be missing from all
> > > > > > > mailboxes sometimes too. I'm not certain.) Using --remote-fudge-time
> > > > > > > to force refetching seems to get the missing mails. I haven't found
> > > > > > > anything strange in timestamps of the missing mails, but otherwise am
> > > > > > > not sure how to debug this further. The queries are retrieving full
> > > > > > > threads and the missing mails are in the threads, but not direct
> > > > > > > matches to the queries. I realize that's not a lot of detail to go on.
> > > > > > > Suggestions on debugging this further?
> > > > > >
> > > > > > Is this with 1.8 or 1.7?
> > > > >
> > > > > Commit 68b53c888911 actually. So post 1.8.
> > > >
> > > > OK, thanks for that info.
> > > >
> > > > > > I forgot to note in the release notes, but there were some
> > > > > > SQLite usage-related fixes which could avoid missing messages.
> > > > > >
> > > > > > You'll need "lei daemon-kill" after upgrading to 1.8 to ensure
> > > > > > the new code is running.
> > > > >
> > > > > It's possible I haven't done that since updating though I do vaguely
> > > > > recall seeing something about needing to do that. Is there any way to
> > > > > tell before I restart it?
> > > >
> > > > Not really, but it's pretty cheap to restart (assuming there's no
> > > > long-running jobs).
> > >
> > > I've restarted and just hit this again.
> >
> > Ugh, sorry to hear that :<
> >
> > > > > > What might be interesting is to use the URLs lei prints and
> > > > > > comparing the results w/o lei.
> > >
> > > $ lei up --all
> > > # updating /home/rob/Mail/from-me
> > > # updating /home/rob/Mail/missing-cc
> > > # updating /home/rob/Mail/my-patches
> > > # updating /home/rob/Mail/pci
> > > # https://lore.kernel.org/all/ limiting to 2022-06-27 12:42 -0600 and newer
> > > # https://lore.kernel.org/all/ limiting to 2022-06-27 9:50 -0600 and newer
> > > # https://lore.kernel.org/all/ limiting to 2022-06-27 12:42 -0600 and newer
> > > # /usr/bin/curl -Sf -s -d ''
> > > https://lore.kernel.org/all/?x=m&t=1&q=(dt%3A20220529211430..+AND+(f%3Arobh%40kernel.org+OR+f%3Arobh%2Bdt%40kernel.org))+AND+dt%3A20220627184226..
> > > # /home/rob/.local/share/lei/store 144/144
> > > # /home/rob/.local/share/lei/store 3/3
> > > # /usr/bin/curl -Sf -s -d ''
> > > https://lore.kernel.org/all/?x=m&t=1&q=((dfn%3Adrivers+OR+dfn%3Aarch+OR+dfn%3ADocumentation%2F*+OR+dfn%3Ainclude+OR+dfn%3Ascripts)+AND+f%3Arobh%40kernel.org+AND+rt%3A1640812470..)+AND+dt%3A20220627155025..
> > > # /usr/bin/curl -Sf -s -d ''
> > > https://lore.kernel.org/all/?x=m&t=1&q=(l%3Alinux-pci+dfn%3Adrivers%2Fpci%2Fcontroller+dt%3A20220529211430..)+AND+dt%3A20220627184226..
> > > # /home/rob/.local/share/lei/store 0/0
> > > # /home/rob/.local/share/lei/store 362/362
> > > # 0 written to /home/rob/Mail/missing-cc/ (0 matches)
> > > # https://lore.kernel.org/all/ 72/72
> > > # https://lore.kernel.org/all/ 4/4
> > > # https://lore.kernel.org/all/ 131/?
> > > # https://lore.kernel.org/all/ 184/?
> > > # https://lore.kernel.org/all/ 412/?
> > > # https://lore.kernel.org/all/ 603/?
> > > # https://lore.kernel.org/all/ 853/?
> > > # https://lore.kernel.org/all/ 1069/?
> > > # https://lore.kernel.org/all/ 1442/?
> > > # https://lore.kernel.org/all/ 1443/1443
> > > # 1 written to /home/rob/Mail/pci/ (75 matches)
> > > # 2 written to /home/rob/Mail/my-patches/ (148 matches)
> > > # 7 written to /home/rob/Mail/from-me/ (1805 matches)
> > >
> > >
> > > What I expected was 3 messages written to 'my-patches'.
> > >
> > > I think the problem is just simply that the new message missing
> > > doesn't match the query, but is a reply to a match. So with a date
> > > after the original match in the thread won't pick up anything. The 2nd
> > > URL above indeed only has 2 results. I guess I just have to fetch a
> > > wider window like a month every time? What's needed is a get any new
> > > messages in existing threads. I don't suppose there's an efficient way
> > > to do that?
> >
> > No, I don't think so. I think this is a separate issue in lei...
> > "t=1" in the remote query expands threads in a time-agnostic
> > way, so I don't think that's the problem (though I may be wrong...).
>
> Based on what the web interface presents, it sure seems like 't=1' is
> independent of the query. The results listed are only those that match
> the query and date range on the match.
Actually, for the HTML results, t=1 is ignored right now...
it's only for mboxrd downloads (via POST) at the moment...
That should be clarified/changed.
> For example, this query returns 3 matches:
>
> https://lore.kernel.org/all/?x=m&t=1&q=((dfn%3Adrivers+OR+dfn%3Aarch+OR+dfn%3ADocumentation%2F*+OR+dfn%3Ainclude+OR+dfn%3Ascripts)+AND+f%3Arobh%40kernel.org+AND+rt%3A1641934905..)+AND+dt%3A20220630203819..
>
> If I change 'dt' to 1 day earlier, I get 1 more match:
>
> https://lore.kernel.org/all/?x=m&t=1&q=((dfn%3Adrivers+OR+dfn%3Aarch+OR+dfn%3ADocumentation%2F*+OR+dfn%3Ainclude+OR+dfn%3Ascripts)+AND+f%3Arobh%40kernel.org+AND+rt%3A1641934905..)+AND+dt%3A20220629203819..
>
> That 4th match has a reply after 6/30, but the 1st query will not get
> the reply. This is all reproducible without lei involved at all.
Right. t=1 only expands threads if they're linked via
References/In-Reply-To or (loosely) via matching Subjects.
> What seems to be needed is a 'thread date' which is the latest time
> for any message in a thread that matches. Or perhaps some way to
> separate the query from what's transferred. IOW, query for X, but only
> send results newer than some date.
Yeah, the thread ID info is stored independently of the search
index documents, though; so searching (Xapian) vs filtering
(SQLite) gets a bit tricky
I'll look into JMAP further before making more changes to the
data models to accomodate this.
Apologies for the delays; been stressed over other things :<
^ permalink raw reply [relevance 71%]
* [PATCH 0/2] lei inotify/EVFILT_VNODE deadlock fix
@ 2022-07-19 22:42 71% Eric Wong
2022-07-19 22:42 51% ` [PATCH 1/2] lei: avoid deadlock on inotify/EVFILT_VNODE wakeups Eric Wong
2022-07-19 22:42 71% ` [PATCH 2/2] lei note-event: inline note_event_arm_done Eric Wong
0 siblings, 2 replies; 200+ results
From: Eric Wong @ 2022-07-19 22:42 UTC (permalink / raw)
To: meta
For every bug I fix, I feel dumber for introducing it :<
Eric Wong (2):
lei: avoid deadlock on inotify/EVFILT_VNODE wakeups
lei note-event: inline note_event_arm_done
MANIFEST | 1 +
lib/PublicInbox/IPC.pm | 26 +++++++++++++++--
lib/PublicInbox/LeiNoteEvent.pm | 15 ++++------
lib/PublicInbox/WQBlocked.pm | 49 +++++++++++++++++++++++++++++++++
4 files changed, 79 insertions(+), 12 deletions(-)
create mode 100644 lib/PublicInbox/WQBlocked.pm
^ permalink raw reply [relevance 71%]
* [PATCH 2/2] lei note-event: inline note_event_arm_done
2022-07-19 22:42 71% [PATCH 0/2] lei inotify/EVFILT_VNODE deadlock fix Eric Wong
2022-07-19 22:42 51% ` [PATCH 1/2] lei: avoid deadlock on inotify/EVFILT_VNODE wakeups Eric Wong
@ 2022-07-19 22:42 71% ` Eric Wong
1 sibling, 0 replies; 200+ results
From: Eric Wong @ 2022-07-19 22:42 UTC (permalink / raw)
To: meta
This was a single-caller sub since 47d4e53734820b4e
(lei_mail_sync: rely on flock(2), avoid IPC, 2021-09-18)
and unlikely to be used further, so inline it and save
a few KB of memory.
---
lib/PublicInbox/LeiNoteEvent.pm | 10 ++--------
1 file changed, 2 insertions(+), 8 deletions(-)
diff --git a/lib/PublicInbox/LeiNoteEvent.pm b/lib/PublicInbox/LeiNoteEvent.pm
index 93f80116..8581bd9a 100644
--- a/lib/PublicInbox/LeiNoteEvent.pm
+++ b/lib/PublicInbox/LeiNoteEvent.pm
@@ -27,13 +27,6 @@ sub flush_task { # PublicInbox::DS timer callback
for my $lei (values %$todo) { flush_lei($lei) }
}
-# sets a timer to flush
-sub note_event_arm_done ($) {
- my ($lei) = @_;
- PublicInbox::DS::add_uniq_timer('flush_timer', 5, \&flush_task);
- $to_flush->{$lei->{cfg}->{'-f'}} //= $lei;
-}
-
sub eml_event ($$$$) {
my ($self, $eml, $vmd, $state) = @_;
my $sto = $self->{lei}->{sto};
@@ -92,7 +85,8 @@ sub lei_note_event {
$jobs = 4 if $jobs > 4; # same default as V2Writable
my ($op_c, $ops) = $lei->workers_start($wq, $jobs);
$lei->wait_wq_events($op_c, $ops);
- note_event_arm_done($lei);
+ PublicInbox::DS::add_uniq_timer('flush_timer', 5, \&flush_task);
+ $to_flush->{$lei->{cfg}->{'-f'}} //= $lei;
$wq->prepare_nonblock;
$lei->{lne} = $wq;
};
^ permalink raw reply related [relevance 71%]
* [PATCH 1/2] lei: avoid deadlock on inotify/EVFILT_VNODE wakeups
2022-07-19 22:42 71% [PATCH 0/2] lei inotify/EVFILT_VNODE deadlock fix Eric Wong
@ 2022-07-19 22:42 51% ` Eric Wong
2022-07-19 22:42 71% ` [PATCH 2/2] lei note-event: inline note_event_arm_done Eric Wong
1 sibling, 0 replies; 200+ results
From: Eric Wong @ 2022-07-19 22:42 UTC (permalink / raw)
To: meta
Enqueuing "note-event" requests from the DS event loop must
not wait on workers being able to drain the queue quickly
enough. Thus we make the SOCK_SEQPACKET writes nonblocking
and rely on the lei-daemon event loop to enqueue writes.
This is a unique problem for "note-event" since it reuses
workers in between commands, while most lei commands currently
fork off new workers.
---
MANIFEST | 1 +
lib/PublicInbox/IPC.pm | 26 +++++++++++++++--
lib/PublicInbox/LeiNoteEvent.pm | 5 ++--
lib/PublicInbox/WQBlocked.pm | 49 +++++++++++++++++++++++++++++++++
4 files changed, 77 insertions(+), 4 deletions(-)
create mode 100644 lib/PublicInbox/WQBlocked.pm
diff --git a/MANIFEST b/MANIFEST
index 2cbe66b7..923f5147 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -337,6 +337,7 @@ lib/PublicInbox/V2Writable.pm
lib/PublicInbox/View.pm
lib/PublicInbox/ViewDiff.pm
lib/PublicInbox/ViewVCS.pm
+lib/PublicInbox/WQBlocked.pm
lib/PublicInbox/WQWorker.pm
lib/PublicInbox/WWW.pm
lib/PublicInbox/WWW.pod
diff --git a/lib/PublicInbox/IPC.pm b/lib/PublicInbox/IPC.pm
index 67e86a43..74862673 100644
--- a/lib/PublicInbox/IPC.pm
+++ b/lib/PublicInbox/IPC.pm
@@ -1,4 +1,4 @@
-# Copyright (C) 2020-2021 all contributors <meta@public-inbox.org>
+# Copyright (C) all contributors <meta@public-inbox.org>
# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
# base class for remote IPC calls and workqueues, requires Storable or Sereal
@@ -43,7 +43,7 @@ if ($enc && $dec) { # should be custom ops
}
my $recv_cmd = PublicInbox::Spawn->can('recv_cmd4');
-my $send_cmd = PublicInbox::Spawn->can('send_cmd4') // do {
+our $send_cmd = PublicInbox::Spawn->can('send_cmd4') // do {
require PublicInbox::CmdIPC4;
$recv_cmd //= PublicInbox::CmdIPC4->can('recv_cmd4');
PublicInbox::CmdIPC4->can('send_cmd4');
@@ -348,6 +348,24 @@ sub wq_do {
}
}
+sub prepare_nonblock {
+ ($_[0]->{-wq_s1} // die 'BUG: no {-wq_s1}')->blocking(0);
+ $_[0]->{-reap_async} or die 'BUG: {-reap_async} needed for nonblock';
+ require PublicInbox::WQBlocked;
+}
+
+sub wq_nonblock_do { # always async
+ my ($self, $sub, @args) = @_;
+ my $buf = ipc_freeze([$sub, @args]);
+ if ($self->{wqb}) { # saturated once, assume saturated forever
+ $self->{wqb}->flush_send($buf);
+ } else {
+ $send_cmd->($self->{-wq_s1}, [], $buf, MSG_EOR) //
+ ($!{EAGAIN} ? PublicInbox::WQBlocked->new($self, $buf)
+ : croak("sendmsg: $!"));
+ }
+}
+
sub _wq_worker_start ($$$$) {
my ($self, $oldset, $fields, $one) = @_;
my ($bcast1, $bcast2);
@@ -405,6 +423,10 @@ sub wq_workers_start {
sub wq_close {
my ($self) = @_;
+ if (my $wqb = delete $self->{wqb}) {
+ $self->{-reap_async} or die 'BUG: {-reap_async} unset';
+ $wqb->enq_close;
+ }
delete @$self{qw(-wq_s1 -wq_s2)} or return;
return if $self->{-reap_async};
my @pids = keys %{$self->{-wq_workers}};
diff --git a/lib/PublicInbox/LeiNoteEvent.pm b/lib/PublicInbox/LeiNoteEvent.pm
index db387633..93f80116 100644
--- a/lib/PublicInbox/LeiNoteEvent.pm
+++ b/lib/PublicInbox/LeiNoteEvent.pm
@@ -58,7 +58,7 @@ sub eml_event ($$$$) {
}
}
-sub maildir_event { # via wq_io_do
+sub maildir_event { # via wq_nonblock_do
my ($self, $fn, $vmd, $state) = @_;
if (my $eml = PublicInbox::InboxWritable::eml_from_path($fn)) {
eml_event($self, $eml, $vmd, $state);
@@ -93,6 +93,7 @@ sub lei_note_event {
my ($op_c, $ops) = $lei->workers_start($wq, $jobs);
$lei->wait_wq_events($op_c, $ops);
note_event_arm_done($lei);
+ $wq->prepare_nonblock;
$lei->{lne} = $wq;
};
if ($folder =~ /\Amaildir:/i) {
@@ -101,7 +102,7 @@ sub lei_note_event {
return if index($fl, 'T') >= 0;
my $kw = PublicInbox::MdirReader::flags2kw($fl);
my $vmd = { kw => $kw, sync_info => [ $folder, \$bn ] };
- $self->wq_do('maildir_event', $fn, $vmd, $state);
+ $self->wq_nonblock_do('maildir_event', $fn, $vmd, $state);
} # else: TODO: imap
}
diff --git a/lib/PublicInbox/WQBlocked.pm b/lib/PublicInbox/WQBlocked.pm
new file mode 100644
index 00000000..fbb43600
--- /dev/null
+++ b/lib/PublicInbox/WQBlocked.pm
@@ -0,0 +1,49 @@
+# Copyright (C) all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+
+# non-blocking workqueues, currently used by LeiNoteEvent to track renames
+package PublicInbox::WQBlocked;
+use v5.12;
+use parent qw(PublicInbox::DS);
+use PublicInbox::Syscall qw(EPOLLOUT EPOLLONESHOT);
+use PublicInbox::IPC;
+use Carp ();
+use Socket qw(MSG_EOR);
+
+sub new {
+ my ($cls, $wq, $buf) = @_;
+ my $self = bless { msgq => [$buf], }, $cls;
+ $wq->{wqb} = $self->SUPER::new($wq->{-wq_s1}, EPOLLOUT|EPOLLONESHOT);
+}
+
+sub flush_send {
+ my ($self) = @_;
+ push(@{$self->{msgq}}, $_[1]) if defined($_[1]);
+ while (defined(my $buf = shift @{$self->{msgq}})) {
+ if (ref($buf) eq 'CODE') {
+ $buf->($self); # could be \&PublicInbox::DS::close
+ } else {
+ my $wq_s1 = $self->{sock};
+ my $n = $PublicInbox::IPC::send_cmd->($wq_s1, [], $buf,
+ MSG_EOR);
+ next if defined($n);
+ Carp::croak("sendmsg: $!") unless $!{EAGAIN};
+ PublicInbox::DS::epwait($wq_s1, EPOLLOUT|EPOLLONESHOT);
+ unshift @{$self->{msgq}}, $buf;
+ last; # wait for ->event_step
+ }
+ }
+}
+
+sub enq_close { flush_send($_[0], $_[0]->can('close')) }
+
+sub event_step { # called on EPOLLOUT wakeup
+ my ($self) = @_;
+ eval { flush_send($self) } if $self->{sock};
+ if ($@) {
+ warn $@;
+ $self->close;
+ }
+}
+
+1;
^ permalink raw reply related [relevance 51%]
* [PATCH] lei: do not wait for sto->done on disconnected EOF
@ 2022-08-16 3:44 71% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2022-08-16 3:44 UTC (permalink / raw)
To: meta
lei-daemon (the top-level daemon process) should not have
synchronous waits, and this was causing a deadlock with
interrupted commands. There may still be a bug lurking in
lei/store despite this fix, though. I originally thought commit
fd261b9e65674505 (lei_store_err: use level-trigger for error pipe, 2022-08-15)
was sufficient, but at least this change is needed, as well.
---
lib/PublicInbox/LEI.pm | 11 ++++-------
1 file changed, 4 insertions(+), 7 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index d81ca296..595b3fa9 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -1520,13 +1520,10 @@ sub sto_done_request {
return unless $lei->{sto};
local $current_lei = $lei;
my $sock = $wq ? $wq->{lei_sock} : undef;
- eval {
- if ($sock //= $lei->{sock}) { # issue, async wait
- $lei->{sto}->wq_io_do('done', [ $sock ]);
- } else { # forcibly wait
- my $wait = $lei->{sto}->wq_do('done');
- }
- };
+ $sock //= $lei->{sock};
+ my @io;
+ push(@io, $sock) if $sock; # async wait iff possible
+ eval { $lei->{sto}->wq_io_do('done', \@io) };
warn($@) if $@;
}
^ permalink raw reply related [relevance 71%]
* [PATCH 0/4] lei reindex, minor tweaks
@ 2022-08-17 9:33 71% Eric Wong
2022-08-17 9:33 71% ` [PATCH 2/4] lei inspect: less scary exception for invalid "docid:" inspect Eric Wong
` (2 more replies)
0 siblings, 3 replies; 200+ results
From: Eric Wong @ 2022-08-17 9:33 UTC (permalink / raw)
To: meta
Reindex is far from complete, and probably needs a compact, and
some other fixups for old data + rethread support.
But avoiding false positives from base-85 is nice.
Eric Wong (4):
searchidx: fix spelling error in comment
lei inspect: less scary exception for invalid "docid:" inspect
lei/store: reduce work when accessing mail_sync.sqlite3
lei reindex: new command to reindex lei/store
Documentation/lei-reindex.pod | 47 +++++++++++++++++++++++++++++++++
MANIFEST | 2 ++
lib/PublicInbox/LEI.pm | 2 ++
lib/PublicInbox/LeiInspect.pm | 5 ++--
lib/PublicInbox/LeiReindex.pm | 49 +++++++++++++++++++++++++++++++++++
lib/PublicInbox/LeiStore.pm | 38 ++++++++++++++++++++++++---
lib/PublicInbox/SearchIdx.pm | 2 +-
7 files changed, 138 insertions(+), 7 deletions(-)
create mode 100644 Documentation/lei-reindex.pod
create mode 100644 lib/PublicInbox/LeiReindex.pm
^ permalink raw reply [relevance 71%]
* [PATCH 2/4] lei inspect: less scary exception for invalid "docid:" inspect
2022-08-17 9:33 71% [PATCH 0/4] lei reindex, minor tweaks Eric Wong
@ 2022-08-17 9:33 71% ` Eric Wong
2022-08-17 9:33 71% ` [PATCH 3/4] lei/store: reduce work when accessing mail_sync.sqlite3 Eric Wong
2022-08-17 9:33 69% ` [PATCH 4/4] lei reindex: new command to reindex lei/store Eric Wong
2 siblings, 0 replies; 200+ results
From: Eric Wong @ 2022-08-17 9:33 UTC (permalink / raw)
To: meta
It still says "Exception:", but doesn't pointlessly print out
the line number and file of the exception when it's a data/input
problem, and not a code problem on our end.
---
lib/PublicInbox/LeiInspect.pm | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/lib/PublicInbox/LeiInspect.pm b/lib/PublicInbox/LeiInspect.pm
index d7775d4b..d1dca4ef 100644
--- a/lib/PublicInbox/LeiInspect.pm
+++ b/lib/PublicInbox/LeiInspect.pm
@@ -1,4 +1,4 @@
-# Copyright (C) 2021 all contributors <meta@public-inbox.org>
+# Copyright (C) all contributors <meta@public-inbox.org>
# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
# "lei inspect" general purpose inspector for stuff in SQLite and
@@ -235,7 +235,8 @@ sub inspect_argv { # via wq_do
$lei->{1}->autoflush(0);
$lei->out('[') if $multi;
while (defined(my $x = shift @$argv)) {
- inspect1($lei, $x, scalar(@$argv)) or return;
+ eval { inspect1($lei, $x, scalar(@$argv)) or return };
+ warn "E: $@\n" if $@;
}
$lei->out(']') if $multi;
}
^ permalink raw reply related [relevance 71%]
* [PATCH 3/4] lei/store: reduce work when accessing mail_sync.sqlite3
2022-08-17 9:33 71% [PATCH 0/4] lei reindex, minor tweaks Eric Wong
2022-08-17 9:33 71% ` [PATCH 2/4] lei inspect: less scary exception for invalid "docid:" inspect Eric Wong
@ 2022-08-17 9:33 71% ` Eric Wong
2022-08-17 9:33 69% ` [PATCH 4/4] lei reindex: new command to reindex lei/store Eric Wong
2 siblings, 0 replies; 200+ results
From: Eric Wong @ 2022-08-17 9:33 UTC (permalink / raw)
To: meta
There's no need to initialize eidx if we already have an open
handle for mail_sync.sqlite3
---
lib/PublicInbox/LeiStore.pm | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/lib/PublicInbox/LeiStore.pm b/lib/PublicInbox/LeiStore.pm
index 66049dfe..d49746cb 100644
--- a/lib/PublicInbox/LeiStore.pm
+++ b/lib/PublicInbox/LeiStore.pm
@@ -255,13 +255,13 @@ sub remove_eml_vmd { # remove just the VMD
sub _lms_rw ($) { # it is important to have eidx processes open before lms
my ($self) = @_;
- my ($eidx, $tl) = eidx_init($self);
- $self->{lms} //= do {
+ $self->{lms} // do {
require PublicInbox::LeiMailSync;
+ my ($eidx, $tl) = eidx_init($self);
my $f = "$self->{priv_eidx}->{topdir}/mail_sync.sqlite3";
my $lms = PublicInbox::LeiMailSync->new($f);
$lms->lms_write_prepare;
- $lms;
+ $self->{lms} = $lms;
};
}
^ permalink raw reply related [relevance 71%]
* [PATCH 4/4] lei reindex: new command to reindex lei/store
2022-08-17 9:33 71% [PATCH 0/4] lei reindex, minor tweaks Eric Wong
2022-08-17 9:33 71% ` [PATCH 2/4] lei inspect: less scary exception for invalid "docid:" inspect Eric Wong
2022-08-17 9:33 71% ` [PATCH 3/4] lei/store: reduce work when accessing mail_sync.sqlite3 Eric Wong
@ 2022-08-17 9:33 69% ` Eric Wong
2022-08-18 7:22 90% ` Eric Wong
2 siblings, 1 reply; 200+ results
From: Eric Wong @ 2022-08-17 9:33 UTC (permalink / raw)
To: meta
---
Documentation/lei-reindex.pod | 47 +++++++++++++++++++++++++++++++++
MANIFEST | 2 ++
lib/PublicInbox/LEI.pm | 2 ++
lib/PublicInbox/LeiReindex.pm | 49 +++++++++++++++++++++++++++++++++++
lib/PublicInbox/LeiStore.pm | 32 ++++++++++++++++++++++-
5 files changed, 131 insertions(+), 1 deletion(-)
create mode 100644 Documentation/lei-reindex.pod
create mode 100644 lib/PublicInbox/LeiReindex.pm
diff --git a/Documentation/lei-reindex.pod b/Documentation/lei-reindex.pod
new file mode 100644
index 00000000..3a5861c4
--- /dev/null
+++ b/Documentation/lei-reindex.pod
@@ -0,0 +1,47 @@
+=head1 NAME
+
+lei-reindex - reindex messages already in lei/store
+
+=head1 SYNOPSIS
+
+lei reindex [OPTIONS]
+
+=head1 DESCRIPTION
+
+Forces a re-index of all messages previously-indexed by L<lei-import(1)>
+or L<lei-index(1)>. This can be used for in-place upgrades and bugfixes
+while other processes are querying the store. Keep in mind this roughly
+doubles the size of the already-large Xapian database.
+
+It does not re-index messages in externals, using the C<--reindex>
+switch of L<public-inbox-index(1)> or L<public-inbox-extindex(1)> is
+needed for that.
+
+=head1 OPTIONS
+
+=over
+
+=item -q
+
+=item --quiet
+
+Suppress feedback messages.
+
+=back
+
+=head1 CONTACT
+
+Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org>
+
+The mail archives are hosted at L<https://public-inbox.org/meta/> and
+L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/>
+
+=head1 COPYRIGHT
+
+Copyright all contributors L<mailto:meta@public-inbox.org>
+
+License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>
+
+=head1 SEE ALSO
+
+L<lei-index(1)>, L<lei-import(1)>
diff --git a/MANIFEST b/MANIFEST
index cc0a9a4c..27e4c4e0 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -56,6 +56,7 @@ Documentation/lei-p2q.pod
Documentation/lei-q.pod
Documentation/lei-rediff.pod
Documentation/lei-refresh-mail-sync.pod
+Documentation/lei-reindex.pod
Documentation/lei-rm-watch.pod
Documentation/lei-rm.pod
Documentation/lei-security.pod
@@ -256,6 +257,7 @@ lib/PublicInbox/LeiPmdir.pm
lib/PublicInbox/LeiQuery.pm
lib/PublicInbox/LeiRediff.pm
lib/PublicInbox/LeiRefreshMailSync.pm
+lib/PublicInbox/LeiReindex.pm
lib/PublicInbox/LeiRemote.pm
lib/PublicInbox/LeiRm.pm
lib/PublicInbox/LeiRmWatch.pm
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 595b3fa9..8a3a3ab6 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -253,6 +253,8 @@ our %CMD = ( # sorted in order of importance/use:
'forget-watch' => [ '{WATCH_NUMBER|--prune}', 'stop and forget a watch',
qw(prune), @c_opt ],
+'reindex' => [ '', 'reindex all locally-indexed messages', @c_opt ],
+
'index' => [ 'LOCATION...', 'one-time index from URL or filesystem',
qw(in-format|F=s kw! offset=i recursive|r exclude=s include|I=s
verbose|v+ incremental!), @net_opt, # mainly for --proxy=
diff --git a/lib/PublicInbox/LeiReindex.pm b/lib/PublicInbox/LeiReindex.pm
new file mode 100644
index 00000000..3f109f33
--- /dev/null
+++ b/lib/PublicInbox/LeiReindex.pm
@@ -0,0 +1,49 @@
+# Copyright all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+
+# "lei reindex" command to reindex everything in lei/store
+package PublicInbox::LeiReindex;
+use v5.12;
+use parent qw(PublicInbox::IPC);
+
+sub reindex_full {
+ my ($lei) = @_;
+ my $sto = $lei->{sto};
+ my $max = $sto->search->over(1)->max;
+ $lei->qerr("# reindexing 1..$max");
+ $sto->wq_do('reindex_art', $_) for (1..$max);
+}
+
+sub reindex_store { # via wq_do
+ my ($self) = @_;
+ my ($lei, $argv) = delete @$self{qw(lei argv)};
+ if (!@$argv) {
+ reindex_full($lei);
+ }
+}
+
+sub lei_reindex {
+ my ($lei, @argv) = @_;
+ my $sto = $lei->_lei_store or return $lei->fail('nothing indexed');
+ $sto->write_prepare($lei);
+ my $self = bless { lei => $lei, argv => \@argv }, __PACKAGE__;
+ my ($op_c, $ops) = $lei->workers_start($self, 1);
+ $lei->{wq1} = $self;
+ $lei->wait_wq_events($op_c, $ops);
+ $self->wq_do('reindex_store');
+ $self->wq_close;
+}
+
+sub _lei_wq_eof { # EOF callback for main lei daemon
+ my ($lei) = @_;
+ $lei->{sto}->wq_do('reindex_done');
+ $lei->wq_eof;
+}
+
+sub ipc_atfork_child {
+ my ($self) = @_;
+ $self->{lei}->_lei_atfork_child;
+ $self->SUPER::ipc_atfork_child;
+}
+
+1;
diff --git a/lib/PublicInbox/LeiStore.pm b/lib/PublicInbox/LeiStore.pm
index d49746cb..277ed6bd 100644
--- a/lib/PublicInbox/LeiStore.pm
+++ b/lib/PublicInbox/LeiStore.pm
@@ -1,4 +1,4 @@
-# Copyright (C) 2020-2021 all contributors <meta@public-inbox.org>
+# Copyright (C) all contributors <meta@public-inbox.org>
# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
#
# Local storage (cache/memo) for lei(1), suitable for personal/private
@@ -335,6 +335,36 @@ sub _docids_and_maybe_kw ($$) {
($docids, [ sort keys %$kw ]);
}
+sub _reindex_1 { # git->cat_async callback
+ my ($bref, $hex, $type, $size, $smsg) = @_;
+ my ($self, $eidx, $tl) = delete @$smsg{qw(-self -eidx -tl)};
+ $bref //= _lms_rw($self)->local_blob($hex, 1);
+ if ($bref) {
+ my $eml = PublicInbox::Eml->new($bref);
+ $smsg->{-merge_vmd} = 1; # preserve existing keywords
+ $eidx->idx_shard($smsg->{num})->index_eml($eml, $smsg);
+ } else {
+ warn("E: $type $hex\n");
+ }
+}
+
+sub reindex_art {
+ my ($self, $art) = @_;
+ my ($eidx, $tl) = eidx_init($self);
+ my $smsg = $eidx->{oidx}->get_art($art) // return;
+ return if $smsg->{bytes} == 0; # external-only message
+ @$smsg{qw(-self -eidx -tl)} = ($self, $eidx, $tl);
+ $eidx->git->cat_async($smsg->{blob} // die("no blob (#$art)"),
+ \&_reindex_1, $smsg);
+}
+
+sub reindex_done {
+ my ($self) = @_;
+ my ($eidx, $tl) = eidx_init($self);
+ $eidx->git->async_wait_all;
+ # ->done to be called via sto_done_request
+}
+
sub add_eml {
my ($self, $eml, $vmd, $xoids) = @_;
my $im = $self->{-fake_im} // $self->importer; # may create new epoch
^ permalink raw reply related [relevance 69%]
* Re: [PATCH 4/4] lei reindex: new command to reindex lei/store
2022-08-17 9:33 69% ` [PATCH 4/4] lei reindex: new command to reindex lei/store Eric Wong
@ 2022-08-18 7:22 90% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2022-08-18 7:22 UTC (permalink / raw)
To: meta
Eric Wong <e@80x24.org> wrote:
> index d49746cb..277ed6bd 100644
> --- a/lib/PublicInbox/LeiStore.pm
> +++ b/lib/PublicInbox/LeiStore.pm
> @@ -335,6 +335,36 @@ sub _docids_and_maybe_kw ($$) {
> ($docids, [ sort keys %$kw ]);
> }
>
> +sub _reindex_1 { # git->cat_async callback
> + my ($bref, $hex, $type, $size, $smsg) = @_;
> + my ($self, $eidx, $tl) = delete @$smsg{qw(-self -eidx -tl)};
> + $bref //= _lms_rw($self)->local_blob($hex, 1);
> + if ($bref) {
> + my $eml = PublicInbox::Eml->new($bref);
> + $smsg->{-merge_vmd} = 1; # preserve existing keywords
> + $eidx->idx_shard($smsg->{num})->index_eml($eml, $smsg);
> + } else {
> + warn("E: $type $hex\n");
This path has been worrying me a bit, I hit it quite a bit on
one of my systems since there was a time when external-only
messages were fully-indexed inside lei/store. Nowadays,
duplicate indexing is avoided...
^ permalink raw reply [relevance 90%]
* [PATCH 1/4] lei reindex: account for parallel lei/store users
2022-08-19 9:07 71% [PATCH 0/4] lei reindex-related stuff Eric Wong
@ 2022-08-19 9:07 90% ` Eric Wong
2022-08-19 9:07 62% ` [PATCH 2/4] tests: add some basic "lei reindex" tests Eric Wong
2022-08-19 9:07 71% ` [PATCH 4/4] lei/store: reindex culls over-indexed messages Eric Wong
2 siblings, 0 replies; 200+ results
From: Eric Wong @ 2022-08-19 9:07 UTC (permalink / raw)
To: meta
We need to call eidx_init in each git->cat_async callback
since another requestor may've stopped the shard processes.
---
lib/PublicInbox/LeiStore.pm | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/lib/PublicInbox/LeiStore.pm b/lib/PublicInbox/LeiStore.pm
index 277ed6bd..8e710540 100644
--- a/lib/PublicInbox/LeiStore.pm
+++ b/lib/PublicInbox/LeiStore.pm
@@ -337,7 +337,8 @@ sub _docids_and_maybe_kw ($$) {
sub _reindex_1 { # git->cat_async callback
my ($bref, $hex, $type, $size, $smsg) = @_;
- my ($self, $eidx, $tl) = delete @$smsg{qw(-self -eidx -tl)};
+ my $self = delete $smsg->{-sto};
+ my ($eidx, $tl) = eidx_init($self);
$bref //= _lms_rw($self)->local_blob($hex, 1);
if ($bref) {
my $eml = PublicInbox::Eml->new($bref);
@@ -353,7 +354,7 @@ sub reindex_art {
my ($eidx, $tl) = eidx_init($self);
my $smsg = $eidx->{oidx}->get_art($art) // return;
return if $smsg->{bytes} == 0; # external-only message
- @$smsg{qw(-self -eidx -tl)} = ($self, $eidx, $tl);
+ $smsg->{-sto} = $self;
$eidx->git->cat_async($smsg->{blob} // die("no blob (#$art)"),
\&_reindex_1, $smsg);
}
^ permalink raw reply related [relevance 90%]
* [PATCH 0/4] lei reindex-related stuff
@ 2022-08-19 9:07 71% Eric Wong
2022-08-19 9:07 90% ` [PATCH 1/4] lei reindex: account for parallel lei/store users Eric Wong
` (2 more replies)
0 siblings, 3 replies; 200+ results
From: Eric Wong @ 2022-08-19 9:07 UTC (permalink / raw)
To: meta
1/4 is an important fix
And I'm still unusually stressed out about how to deal with
ancient, pre-release stuff from lei. --rethread may happen
another time, since AFAIK it's not necessary, right now as there
were no threading fixes required since lei...
Eric Wong (4):
lei reindex: account for parallel lei/store users
tests: add some basic "lei reindex" tests
smsg: ->populate falls back to old {ds}/{ts} values
lei/store: reindex culls over-indexed messages
MANIFEST | 1 +
lib/PublicInbox/LeiStore.pm | 14 ++++++++++++--
lib/PublicInbox/Smsg.pm | 6 ++++--
t/lei-index.t | 12 +++++++++++-
t/lei-reindex.t | 12 ++++++++++++
5 files changed, 40 insertions(+), 5 deletions(-)
create mode 100644 t/lei-reindex.t
^ permalink raw reply [relevance 71%]
* [PATCH 2/4] tests: add some basic "lei reindex" tests
2022-08-19 9:07 71% [PATCH 0/4] lei reindex-related stuff Eric Wong
2022-08-19 9:07 90% ` [PATCH 1/4] lei reindex: account for parallel lei/store users Eric Wong
@ 2022-08-19 9:07 62% ` Eric Wong
2022-08-19 9:07 71% ` [PATCH 4/4] lei/store: reindex culls over-indexed messages Eric Wong
2 siblings, 0 replies; 200+ results
From: Eric Wong @ 2022-08-19 9:07 UTC (permalink / raw)
To: meta
This is a bit hard-to-test, but at least we must ensure
volatile-metadata is preserved.
---
MANIFEST | 1 +
t/lei-index.t | 12 +++++++++++-
t/lei-reindex.t | 12 ++++++++++++
3 files changed, 24 insertions(+), 1 deletion(-)
create mode 100644 t/lei-reindex.t
diff --git a/MANIFEST b/MANIFEST
index 27e4c4e0..43128382 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -477,6 +477,7 @@ t/lei-q-remote-import.t
t/lei-q-save.t
t/lei-q-thread.t
t/lei-refresh-mail-sync.t
+t/lei-reindex.t
t/lei-sigpipe.t
t/lei-tag.t
t/lei-up.t
diff --git a/t/lei-index.t b/t/lei-index.t
index aab8f7e6..c31b1c3c 100644
--- a/t/lei-index.t
+++ b/t/lei-index.t
@@ -1,5 +1,5 @@
#!perl -w
-# Copyright (C) 2021 all contributors <meta@public-inbox.org>
+# Copyright (C) all contributors <meta@public-inbox.org>
# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
use strict; use v5.10.1; use PublicInbox::TestCommon;
use File::Spec;
@@ -85,6 +85,10 @@ test_lei({ tmpdir => $tmpdir }, sub {
lei_ok qw(q m:multipart-html-sucks@11);
is_deeply(json_utf8->decode($lei_out)->[0]->{'kw'},
['seen'], 'keyword set');
+ lei_ok 'reindex';
+ lei_ok qw(q m:multipart-html-sucks@11);
+ is_deeply(json_utf8->decode($lei_out)->[0]->{'kw'},
+ ['seen'], 'keyword still set after reindex');
$srv->{nntpd} and
lei_ok('index', "nntp://$srv->{nntp_host_port}/t.v2");
@@ -104,6 +108,12 @@ test_lei({ tmpdir => $tmpdir }, sub {
my $t = xqx(['git', "--git-dir=$store_path/ALL.git",
qw(cat-file -t), $res_b->{blob}]);
is($t, "blob\n", 'got blob');
+
+ lei_ok('reindex');
+ lei_ok qw(q m:multipart-html-sucks@11);
+ $res_a = json_utf8->decode($lei_out)->[0];
+ is_deeply($res_a->{'kw'}, ['seen'],
+ 'keywords still set after reindex');
});
done_testing;
diff --git a/t/lei-reindex.t b/t/lei-reindex.t
new file mode 100644
index 00000000..73346ee8
--- /dev/null
+++ b/t/lei-reindex.t
@@ -0,0 +1,12 @@
+#!perl -w
+# Copyright (C) all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+use v5.12; use PublicInbox::TestCommon;
+require_mods(qw(lei));
+my ($tmpdir, $for_destroy) = tmpdir;
+test_lei(sub {
+ ok(!lei('reindex'), 'reindex fails w/o store');
+ like $lei_err, qr/nothing indexed/, "`nothing indexed' noted";
+});
+
+done_testing;
^ permalink raw reply related [relevance 62%]
* [PATCH 4/4] lei/store: reindex culls over-indexed messages
2022-08-19 9:07 71% [PATCH 0/4] lei reindex-related stuff Eric Wong
2022-08-19 9:07 90% ` [PATCH 1/4] lei reindex: account for parallel lei/store users Eric Wong
2022-08-19 9:07 62% ` [PATCH 2/4] tests: add some basic "lei reindex" tests Eric Wong
@ 2022-08-19 9:07 71% ` Eric Wong
2 siblings, 0 replies; 200+ results
From: Eric Wong @ 2022-08-19 9:07 UTC (permalink / raw)
To: meta
I may be the only lei user who has redundantly-indexed messages
needing this, though...
---
lib/PublicInbox/LeiStore.pm | 9 +++++++++
1 file changed, 9 insertions(+)
diff --git a/lib/PublicInbox/LeiStore.pm b/lib/PublicInbox/LeiStore.pm
index 8e710540..57f0e013 100644
--- a/lib/PublicInbox/LeiStore.pm
+++ b/lib/PublicInbox/LeiStore.pm
@@ -344,6 +344,15 @@ sub _reindex_1 { # git->cat_async callback
my $eml = PublicInbox::Eml->new($bref);
$smsg->{-merge_vmd} = 1; # preserve existing keywords
$eidx->idx_shard($smsg->{num})->index_eml($eml, $smsg);
+ } elsif ($type eq 'missing') {
+ # pre-release/buggy lei may've indexed external-only msgs,
+ # try to correct that, here
+ warn("E: missing $hex, culling (ancient lei artifact?)\n");
+ $smsg->{to} = $smsg->{cc} = $smsg->{from} = '';
+ $smsg->{bytes} = 0;
+ $eidx->{oidx}->update_blob($smsg, '');
+ my $eml = PublicInbox::Eml->new("\r\n\r\n");
+ $eidx->idx_shard($smsg->{num})->index_eml($eml, $smsg);
} else {
warn("E: $type $hex\n");
}
^ permalink raw reply related [relevance 71%]
* [PATCH 1/3] Makefile.PL: add lei-reindex manpage
@ 2022-08-30 9:10 71% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2022-08-30 9:10 UTC (permalink / raw)
To: meta
I forgot to add this when I added the new command :x
---
Makefile.PL | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/Makefile.PL b/Makefile.PL
index 67012d3e..ff03b615 100644
--- a/Makefile.PL
+++ b/Makefile.PL
@@ -53,7 +53,8 @@ $v->{-m1} = [ map {
lei-import lei-index lei-init lei-inspect lei-lcat
lei-ls-external lei-ls-label lei-ls-mail-source lei-ls-mail-sync
lei-ls-search lei-ls-watch lei-mail-diff lei-p2q lei-q
- lei-rediff lei-refresh-mail-sync lei-rm lei-rm-watch lei-tag
+ lei-rediff lei-refresh-mail-sync lei-reindex
+ lei-rm lei-rm-watch lei-tag
lei-up)];
$v->{-m5} = [ qw(public-inbox-config public-inbox-v1-format
public-inbox-v2-format public-inbox-extindex-format
^ permalink raw reply related [relevance 71%]
* [PATCH 0/2] lei-related import tweaks
@ 2022-09-02 18:26 71% Eric Wong
2022-09-02 18:26 69% ` [PATCH 1/2] lei/store: do not write info/refs file Eric Wong
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2022-09-02 18:26 UTC (permalink / raw)
To: meta
Just some minor nits I noticed in lei-land...
Eric Wong (2):
lei/store: do not write info/refs file
import: pass --quiet to `git gc' if STDERR isn't a tty
lib/PublicInbox/Import.pm | 7 +++++--
t/lei_store.t | 5 ++++-
2 files changed, 9 insertions(+), 3 deletions(-)
^ permalink raw reply [relevance 71%]
* [PATCH 1/2] lei/store: do not write info/refs file
2022-09-02 18:26 71% [PATCH 0/2] lei-related import tweaks Eric Wong
@ 2022-09-02 18:26 69% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2022-09-02 18:26 UTC (permalink / raw)
To: meta
That file is meant for dumb HTTP servers, so avoid wasting two
inodes on something that should never be served for private
email.
---
lib/PublicInbox/Import.pm | 2 +-
t/lei_store.t | 5 ++++-
2 files changed, 5 insertions(+), 2 deletions(-)
diff --git a/lib/PublicInbox/Import.pm b/lib/PublicInbox/Import.pm
index aef49033..2c8f310a 100644
--- a/lib/PublicInbox/Import.pm
+++ b/lib/PublicInbox/Import.pm
@@ -182,8 +182,8 @@ sub _update_git_info ($$) {
my $env = { GIT_INDEX_FILE => $index };
run_die([@cmd, qw(read-tree -m -v -i), $self->{ref}], $env);
}
- eval { run_die([@cmd, 'update-server-info']) };
my $ibx = $self->{ibx};
+ eval { run_die([@cmd, 'update-server-info']) } if $ibx;
if ($ibx && $ibx->version == 1 && -d "$ibx->{inboxdir}/public-inbox" &&
eval { require PublicInbox::SearchIdx }) {
eval {
diff --git a/t/lei_store.t b/t/lei_store.t
index 40ad7800..5a5e5de0 100644
--- a/t/lei_store.t
+++ b/t/lei_store.t
@@ -1,5 +1,5 @@
#!perl -w
-# Copyright (C) 2020-2021 all contributors <meta@public-inbox.org>
+# Copyright (C) all contributors <meta@public-inbox.org>
# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
use strict;
use v5.10.1;
@@ -149,4 +149,7 @@ EOM
is($mset->size, 1, 'rt:1.hour.ago.. works w/ local time');
}
+is_deeply([glob("$store_dir/local/*.git/info/refs")], [],
+ 'no info/refs in private lei/store');
+
done_testing;
^ permalink raw reply related [relevance 69%]
* [PATCH] lei: add diagnostics for IMAP writer failures
@ 2022-09-09 17:44 64% ` Eric Wong
2022-09-09 18:00 86% ` [PATCH] doc: document --jobs for `lei q' and `lei up' Eric Wong
2022-09-09 20:35 71% ` [PATCH] lei: add diagnostics for IMAP writer failures Ricardo Ribalda
0 siblings, 2 replies; 200+ results
From: Eric Wong @ 2022-09-09 17:44 UTC (permalink / raw)
To: Ricardo Ribalda; +Cc: meta
Ricardo Ribalda <ribalda@chromium.org> wrote:
> Hi
>
> I am getting a lot of those messages when using lei in imap mode.
>
> Nonetheless the mail seems to arrive fine to its destination (on the
> first run, some of the mail was lost, but for small batches of mails
> it seems to work fine).
>
> It is a valid message or just some red-herring.
>
> I am using debian testing/
>
> Regards!
>
> # https://lore.kernel.org/all/ limiting to 2022-09-07 10:03 +0000 and newer
> # /usr/local/google/home/ribalda/.local/share/lei/store 13/13
> # /usr/bin/curl -Sf -s -d ''
> https://lore.kernel.org/all/?x=m&t=1&q=((ribalda)+AND+rt%3A1660039651..)+AND+dt%3A20220907100321..
> 1285740 lei2mail 6 wq_worker: do_post_auth: Can't call method
> "uidvalidity" on an undefined value at
> /usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
<snip>
> 1285735 lei2mail 1 wq_worker: write_mail: Can't use an undefined value
> as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
> line 783.
> # https://lore.kernel.org/all/ 18/18
> # 14 written to imaps://imap.gmail.com/lei/me (31 matches)
I wonder if it's excessive parallelism for gmail's IMAP.
I haven't tested IMAP destinations, much...
Can you try the patch at the bottom?
There's also another patch coming to document the `--jobs|-j' CLI
switch for `lei up' and `lei q', but trying `-j ,1' may help you
if it's parallelism. Note the comma before `1', it accepts
`-j $Q,$W' since $Q is the number of query processes and $W is
the number of LeiToMail writers.
-------8<-------
From: Eric Wong <e@80x24.org>
Subject: [PATCH] lei: add diagnostics for IMAP writer failures
This may help diagnose the problem with IMAP destinations
encountered at:
https://public-inbox.org/meta/CANiDSCsDfutAUMBLPZbxdyka+_jnhv+4YNYdL9QPRoC=wNUGCQ@mail.gmail.com/
---
lib/PublicInbox/LeiToMail.pm | 4 ++--
lib/PublicInbox/NetReader.pm | 8 +++++++-
2 files changed, 9 insertions(+), 3 deletions(-)
diff --git a/lib/PublicInbox/LeiToMail.pm b/lib/PublicInbox/LeiToMail.pm
index 2aa3977e..bc00b96a 100644
--- a/lib/PublicInbox/LeiToMail.pm
+++ b/lib/PublicInbox/LeiToMail.pm
@@ -310,8 +310,8 @@ sub _imap_write_cb ($$) {
my $dedupe = $lei->{dedupe};
$dedupe->prepare_dedupe if $dedupe;
my $append = $lei->{net}->can('imap_append');
- my $uri = $self->{uri};
- my $mic = $lei->{net}->mic_get($uri);
+ my $uri = $self->{uri} // die 'BUG: no {uri}';
+ my $mic = $lei->{net}->mic_get($uri) // die 'BUG: no $mic';
my $folder = $uri->mailbox;
$uri->uidvalidity($mic->uidvalidity($folder));
my $lse = $lei->{lse}; # may be undef
diff --git a/lib/PublicInbox/NetReader.pm b/lib/PublicInbox/NetReader.pm
index c1af03a3..4de2583e 100644
--- a/lib/PublicInbox/NetReader.pm
+++ b/lib/PublicInbox/NetReader.pm
@@ -685,7 +685,13 @@ sub mic_get {
}
my $mic = mic_new($self, $mic_arg, $sec, $uri);
$cached //= {}; # invalid placeholder if no cache enabled
- $mic && $mic->IsConnected ? ($cached->{$sec} = $mic) : undef;
+ if ($mic && $mic->IsConnected) {
+ $cached->{$sec} = $mic;
+ } else {
+ warn 'IMAP LastError: ',$mic->LastError, "\n" if $mic;
+ warn "IMAP errno: $!\n" if $!;
+ undef;
+ }
}
sub imap_each {
^ permalink raw reply related [relevance 64%]
* [PATCH] doc: document --jobs for `lei q' and `lei up'
2022-09-09 17:44 64% ` [PATCH] lei: add diagnostics for IMAP writer failures Eric Wong
@ 2022-09-09 18:00 86% ` Eric Wong
2022-09-09 20:35 71% ` [PATCH] lei: add diagnostics for IMAP writer failures Ricardo Ribalda
1 sibling, 0 replies; 200+ results
From: Eric Wong @ 2022-09-09 18:00 UTC (permalink / raw)
To: meta; +Cc: Ricardo Ribalda
Eric Wong <e@80x24.org> wrote:
> There's also another patch coming to document the `--jobs|-j' CLI
> switch for `lei up' and `lei q', but trying `-j ,1' may help you
> if it's parallelism. Note the comma before `1', it accepts
> `-j $Q,$W' since $Q is the number of query processes and $W is
> the number of LeiToMail writers.
-------8<-----
From: Eric Wong <e@80x24.org>
Subject: [PATCH] doc: document --jobs for `lei q' and `lei up'
These may be helpful for users on slow disks or limited IMAP
connections.
---
Documentation/lei-q.pod | 17 +++++++++++++++++
Documentation/lei-up.pod | 4 +++-
2 files changed, 20 insertions(+), 1 deletion(-)
diff --git a/Documentation/lei-q.pod b/Documentation/lei-q.pod
index 2f0c3bc6..8134223e 100644
--- a/Documentation/lei-q.pod
+++ b/Documentation/lei-q.pod
@@ -124,6 +124,23 @@ of the same thread.
TODO: Warning: this flag may become persistent and saved in
lei/store unless an MUA unflags it! (Behavior undecided)
+=item --jobs=QUERY_WORKERS[,WRITE_WORKERS]
+=item --jobs=,WRITE_WORKERS
+
+=item -j QUERY_WORKERS[,WRITE_WORKERS]
+=item -j ,WRITE_WORKERS
+
+Set the number of query and write worker processes for parallelism.
+
+C<QUERY_WORKERS> defaults to the number of CPUs available, but 4 per
+remote (HTTP/HTTPS) host.
+
+C<WRITE_WORKERS> defaults to the number of CPUs available for Maildir,
+IMAP/IMAPS, and mbox* destinations.
+
+Omitting C<QUERY_WORKERS> but leaving the comma (C<,>) allows
+one to only set C<WRITE_WORKERS>
+
=item --dedupe=STRATEGY
=item -d STRATEGY
diff --git a/Documentation/lei-up.pod b/Documentation/lei-up.pod
index ac644a96..3b7c6f46 100644
--- a/Documentation/lei-up.pod
+++ b/Documentation/lei-up.pod
@@ -64,7 +64,9 @@ specified via C<lei q --only>.
=item --mua=CMD
-C<--lock>, C<--alert>, and C<--mua> are all supported and
+=item --jobs QUERY_WORKERS[,WRITE_WORKERS]
+
+C<--lock>, C<--alert>, C<--mua>, and C<--jobs> are all supported and
documented in L<lei-q(1)>.
C<--mua> is incompatible with C<--all>.
^ permalink raw reply related [relevance 86%]
* Re: [PATCH] lei: add diagnostics for IMAP writer failures
2022-09-09 17:44 64% ` [PATCH] lei: add diagnostics for IMAP writer failures Eric Wong
2022-09-09 18:00 86% ` [PATCH] doc: document --jobs for `lei q' and `lei up' Eric Wong
@ 2022-09-09 20:35 71% ` Ricardo Ribalda
2022-09-10 1:18 59% ` [PATCH v2] lei: bail out earlier on " Eric Wong
1 sibling, 1 reply; 200+ results
From: Ricardo Ribalda @ 2022-09-09 20:35 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
Hi Eric
On Fri, 9 Sept 2022 at 19:45, Eric Wong <e@80x24.org> wrote:
>
> Ricardo Ribalda <ribalda@chromium.org> wrote:
> > Hi
> >
> > I am getting a lot of those messages when using lei in imap mode.
> >
> > Nonetheless the mail seems to arrive fine to its destination (on the
> > first run, some of the mail was lost, but for small batches of mails
> > it seems to work fine).
> >
> > It is a valid message or just some red-herring.
> >
> > I am using debian testing/
> >
> > Regards!
> >
> > # https://lore.kernel.org/all/ limiting to 2022-09-07 10:03 +0000 and newer
> > # /usr/local/google/home/ribalda/.local/share/lei/store 13/13
> > # /usr/bin/curl -Sf -s -d ''
> > https://lore.kernel.org/all/?x=m&t=1&q=((ribalda)+AND+rt%3A1660039651..)+AND+dt%3A20220907100321..
> > 1285740 lei2mail 6 wq_worker: do_post_auth: Can't call method
> > "uidvalidity" on an undefined value at
> > /usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
>
> <snip>
>
> > 1285735 lei2mail 1 wq_worker: write_mail: Can't use an undefined value
> > as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
> > line 783.
> > # https://lore.kernel.org/all/ 18/18
> > # 14 written to imaps://imap.gmail.com/lei/me (31 matches)
>
> I wonder if it's excessive parallelism for gmail's IMAP.
> I haven't tested IMAP destinations, much...
>
> Can you try the patch at the bottom?
>
> There's also another patch coming to document the `--jobs|-j' CLI
> switch for `lei up' and `lei q', but trying `-j ,1' may help you
> if it's parallelism. Note the comma before `1', it accepts
> `-j $Q,$W' since $Q is the number of query processes and $W is
> the number of LeiToMail writers.
The patch did not seem to have any effect :(, I never get a "IMAP
LastError: " message
On the other hand, the -j worked! I can go up to -j ,15 without any error.
>
> -------8<-------
> From: Eric Wong <e@80x24.org>
> Subject: [PATCH] lei: add diagnostics for IMAP writer failures
>
> This may help diagnose the problem with IMAP destinations
> encountered at:
> https://public-inbox.org/meta/CANiDSCsDfutAUMBLPZbxdyka+_jnhv+4YNYdL9QPRoC=wNUGCQ@mail.gmail.com/
> ---
> lib/PublicInbox/LeiToMail.pm | 4 ++--
> lib/PublicInbox/NetReader.pm | 8 +++++++-
> 2 files changed, 9 insertions(+), 3 deletions(-)
>
> diff --git a/lib/PublicInbox/LeiToMail.pm b/lib/PublicInbox/LeiToMail.pm
> index 2aa3977e..bc00b96a 100644
> --- a/lib/PublicInbox/LeiToMail.pm
> +++ b/lib/PublicInbox/LeiToMail.pm
> @@ -310,8 +310,8 @@ sub _imap_write_cb ($$) {
> my $dedupe = $lei->{dedupe};
> $dedupe->prepare_dedupe if $dedupe;
> my $append = $lei->{net}->can('imap_append');
> - my $uri = $self->{uri};
> - my $mic = $lei->{net}->mic_get($uri);
> + my $uri = $self->{uri} // die 'BUG: no {uri}';
> + my $mic = $lei->{net}->mic_get($uri) // die 'BUG: no $mic';
> my $folder = $uri->mailbox;
> $uri->uidvalidity($mic->uidvalidity($folder));
> my $lse = $lei->{lse}; # may be undef
> diff --git a/lib/PublicInbox/NetReader.pm b/lib/PublicInbox/NetReader.pm
> index c1af03a3..4de2583e 100644
> --- a/lib/PublicInbox/NetReader.pm
> +++ b/lib/PublicInbox/NetReader.pm
> @@ -685,7 +685,13 @@ sub mic_get {
> }
> my $mic = mic_new($self, $mic_arg, $sec, $uri);
> $cached //= {}; # invalid placeholder if no cache enabled
> - $mic && $mic->IsConnected ? ($cached->{$sec} = $mic) : undef;
> + if ($mic && $mic->IsConnected) {
> + $cached->{$sec} = $mic;
> + } else {
> + warn 'IMAP LastError: ',$mic->LastError, "\n" if $mic;
> + warn "IMAP errno: $!\n" if $!;
> + undef;
> + }
> }
>
> sub imap_each {
--
Ricardo Ribalda
^ permalink raw reply [relevance 71%]
* [PATCH v2] lei: bail out earlier on IMAP writer failures
2022-09-09 20:35 71% ` [PATCH] lei: add diagnostics for IMAP writer failures Ricardo Ribalda
@ 2022-09-10 1:18 59% ` Eric Wong
2022-09-10 19:34 37% ` Ricardo Ribalda
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2022-09-10 1:18 UTC (permalink / raw)
To: Ricardo Ribalda; +Cc: meta
Ricardo Ribalda <ribalda@chromium.org> wrote:
> The patch did not seem to have any effect :(, I never get a "IMAP
> LastError: " message
Yeah, I guess IMAP servers will just shutdown the socket w/o
saying anything. At least I didn't get anything from dovecot...
The below patch is a refinement of what I posted originally
and should stop the process instead of attempting to continue
and spew.
> On the other hand, the -j worked! I can go up to -j ,15 without any error.
Good to know.
I wonder if making the default `-j ,4' for IMAP is reasonable if
unspecified. That's the default limit for HTTP(S) hosts, and I
seem to recall 4 being a reasonable limit for browsers.
Thanks for the report and followup!
------8<------
From: Eric Wong <e@80x24.org>
Subject: [PATCH] lei: bail out earlier on IMAP writer failures
Excessive IMAP connections can overload IMAP servers and cause
clients to be disconnected without diagnostic messages.
Use $lei->fail on these exceptions to propagate errors to the
CLI ASAP to avoid further errors down the line.
This ought to make problems more apparent for users using IMAP
destinations.
Reported-by: Ricardo Ribalda <ribalda@chromium.org>
Link: https://public-inbox.org/meta/CANiDSCsDfutAUMBLPZbxdyka+_jnhv+4YNYdL9QPRoC=wNUGCQ@mail.gmail.com/
---
lib/PublicInbox/LeiToMail.pm | 10 +++++++---
lib/PublicInbox/NetReader.pm | 8 +++++++-
2 files changed, 14 insertions(+), 4 deletions(-)
diff --git a/lib/PublicInbox/LeiToMail.pm b/lib/PublicInbox/LeiToMail.pm
index 2aa3977e..03cbde3b 100644
--- a/lib/PublicInbox/LeiToMail.pm
+++ b/lib/PublicInbox/LeiToMail.pm
@@ -310,8 +310,11 @@ sub _imap_write_cb ($$) {
my $dedupe = $lei->{dedupe};
$dedupe->prepare_dedupe if $dedupe;
my $append = $lei->{net}->can('imap_append');
- my $uri = $self->{uri};
- my $mic = $lei->{net}->mic_get($uri);
+ my $uri = $self->{uri} // die 'BUG: no {uri}';
+ my $mic = $lei->{net}->mic_get($uri) // die <<EOM;
+E: $uri connection failed.
+E: Consider using `--jobs ,1' to limit IMAP connections
+EOM
my $folder = $uri->mailbox;
$uri->uidvalidity($mic->uidvalidity($folder));
my $lse = $lei->{lse}; # may be undef
@@ -749,7 +752,8 @@ sub do_post_auth {
$au_peers->[1] = undef;
sysread($au_peers->[0], my $barrier1, 1);
}
- $self->{wcb} = $self->write_cb($lei);
+ eval { $self->{wcb} = $self->write_cb($lei) };
+ $lei->fail($@) if $@;
if ($au_peers) { # wait for peer l2m to set write_cb
$au_peers->[3] = undef;
sysread($au_peers->[2], my $barrier2, 1);
diff --git a/lib/PublicInbox/NetReader.pm b/lib/PublicInbox/NetReader.pm
index c1af03a3..4de2583e 100644
--- a/lib/PublicInbox/NetReader.pm
+++ b/lib/PublicInbox/NetReader.pm
@@ -685,7 +685,13 @@ sub mic_get {
}
my $mic = mic_new($self, $mic_arg, $sec, $uri);
$cached //= {}; # invalid placeholder if no cache enabled
- $mic && $mic->IsConnected ? ($cached->{$sec} = $mic) : undef;
+ if ($mic && $mic->IsConnected) {
+ $cached->{$sec} = $mic;
+ } else {
+ warn 'IMAP LastError: ',$mic->LastError, "\n" if $mic;
+ warn "IMAP errno: $!\n" if $!;
+ undef;
+ }
}
sub imap_each {
^ permalink raw reply related [relevance 59%]
* [PATCH] lei: fix --help for --jobs with `up' and `q'
@ 2022-09-10 1:35 71% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2022-09-10 1:35 UTC (permalink / raw)
To: meta
The help needs to match on the short option, too, and that
`lei q' option is (like most options) shared with `lei up'.
---
lib/PublicInbox/LEI.pm | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 8a3a3ab6..f3e80113 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -399,8 +399,10 @@ my %OPTDESC = (
'include specified external(s) in search' ],
'only|O=s@ q' => [ 'LOCATION',
'only use specified external(s) for search' ],
-'jobs=s q' => [ '[SEARCH_JOBS][,WRITER_JOBS]',
- 'control number of search and writer jobs' ],
+'jobs|j=s' => [ 'JOBSPEC',
+ 'control number of query and writer jobs' .
+ "integers delimited by `,', either of which may be omitted"
+ ],
'jobs|j=i add-external' => 'set parallelism when indexing after --mirror',
'in-format|F=s' => $stdin_formats,
^ permalink raw reply related [relevance 71%]
* Re: [PATCH v2] lei: bail out earlier on IMAP writer failures
2022-09-10 1:18 59% ` [PATCH v2] lei: bail out earlier on " Eric Wong
@ 2022-09-10 19:34 37% ` Ricardo Ribalda
2022-09-10 19:50 71% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Ricardo Ribalda @ 2022-09-10 19:34 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
Hi Eric
On Sat, 10 Sept 2022 at 03:19, Eric Wong <e@80x24.org> wrote:
>
> Ricardo Ribalda <ribalda@chromium.org> wrote:
> > The patch did not seem to have any effect :(, I never get a "IMAP
> > LastError: " message
>
> Yeah, I guess IMAP servers will just shutdown the socket w/o
> saying anything. At least I didn't get anything from dovecot...
>
> The below patch is a refinement of what I posted originally
> and should stop the process instead of attempting to continue
> and spew.
>
> > On the other hand, the -j worked! I can go up to -j ,15 without any error.
>
> Good to know.
>
> I wonder if making the default `-j ,4' for IMAP is reasonable if
> unspecified. That's the default limit for HTTP(S) hosts, and I
> seem to recall 4 being a reasonable limit for browsers.
>
> Thanks for the report and followup!
Similar output:
ribalda@denia:/tmp/public-inbox$ lei up imaps://imap.gmail.com/lei/me
# https://lore.kernel.org/all/ limiting to 2022-09-08 19:33 +0000 and newer
# /usr/local/google/home/ribalda/.local/share/lei/store 12/12
# /usr/bin/curl -Sf -s -d ''
https://lore.kernel.org/all/?x=m&t=1&q=((ribalda)+AND+rt%3A1660159987..)+AND+dt%3A20220908193300..
1486767 lei2mail 17 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486762 lei2mail 12 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486782 lei2mail 32 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486771 lei2mail 21 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486766 lei2mail 16 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486788 lei2mail 38 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486761 lei2mail 11 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486791 lei2mail 41 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486775 lei2mail 25 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486765 lei2mail 15 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486755 lei2mail 5 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486793 lei2mail 43 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486795 lei2mail 45 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486763 lei2mail 13 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486779 lei2mail 29 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486773 lei2mail 23 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486751 lei2mail 1 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486787 lei2mail 37 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486796 lei2mail 46 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486776 lei2mail 26 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486752 lei2mail 2 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486750 lei2mail 0 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486786 lei2mail 36 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486777 lei2mail 27 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486764 lei2mail 14 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486794 lei2mail 44 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486756 lei2mail 6 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486790 lei2mail 40 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486784 lei2mail 34 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486792 lei2mail 42 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486768 lei2mail 18 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486774 lei2mail 24 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486757 lei2mail 7 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
# https://lore.kernel.org/all/ 20/20
# 0 written to imaps://imap.gmail.com/lei/me (32 matches)
ribalda@denia:/tmp/public-inbox$ lei up imaps://imap.gmail.com/lei/me -j ,15
# https://lore.kernel.org/all/ limiting to 2022-09-08 19:33 +0000 and newer
# /usr/local/google/home/ribalda/.local/share/lei/store 12/12
# /usr/bin/curl -Sf -s -d ''
https://lore.kernel.org/all/?x=m&t=1&q=((ribalda)+AND+rt%3A1660160023..)+AND+dt%3A20220908193307..
# https://lore.kernel.org/all/ 20/20
# 0 written to imaps://imap.gmail.com/lei/me (32 matches)
Thanks!
>
> ------8<------
> From: Eric Wong <e@80x24.org>
> Subject: [PATCH] lei: bail out earlier on IMAP writer failures
>
> Excessive IMAP connections can overload IMAP servers and cause
> clients to be disconnected without diagnostic messages.
> Use $lei->fail on these exceptions to propagate errors to the
> CLI ASAP to avoid further errors down the line.
>
> This ought to make problems more apparent for users using IMAP
> destinations.
>
> Reported-by: Ricardo Ribalda <ribalda@chromium.org>
> Link: https://public-inbox.org/meta/CANiDSCsDfutAUMBLPZbxdyka+_jnhv+4YNYdL9QPRoC=wNUGCQ@mail.gmail.com/
> ---
> lib/PublicInbox/LeiToMail.pm | 10 +++++++---
> lib/PublicInbox/NetReader.pm | 8 +++++++-
> 2 files changed, 14 insertions(+), 4 deletions(-)
>
> diff --git a/lib/PublicInbox/LeiToMail.pm b/lib/PublicInbox/LeiToMail.pm
> index 2aa3977e..03cbde3b 100644
> --- a/lib/PublicInbox/LeiToMail.pm
> +++ b/lib/PublicInbox/LeiToMail.pm
> @@ -310,8 +310,11 @@ sub _imap_write_cb ($$) {
> my $dedupe = $lei->{dedupe};
> $dedupe->prepare_dedupe if $dedupe;
> my $append = $lei->{net}->can('imap_append');
> - my $uri = $self->{uri};
> - my $mic = $lei->{net}->mic_get($uri);
> + my $uri = $self->{uri} // die 'BUG: no {uri}';
> + my $mic = $lei->{net}->mic_get($uri) // die <<EOM;
> +E: $uri connection failed.
> +E: Consider using `--jobs ,1' to limit IMAP connections
> +EOM
> my $folder = $uri->mailbox;
> $uri->uidvalidity($mic->uidvalidity($folder));
> my $lse = $lei->{lse}; # may be undef
> @@ -749,7 +752,8 @@ sub do_post_auth {
> $au_peers->[1] = undef;
> sysread($au_peers->[0], my $barrier1, 1);
> }
> - $self->{wcb} = $self->write_cb($lei);
> + eval { $self->{wcb} = $self->write_cb($lei) };
> + $lei->fail($@) if $@;
> if ($au_peers) { # wait for peer l2m to set write_cb
> $au_peers->[3] = undef;
> sysread($au_peers->[2], my $barrier2, 1);
> diff --git a/lib/PublicInbox/NetReader.pm b/lib/PublicInbox/NetReader.pm
> index c1af03a3..4de2583e 100644
> --- a/lib/PublicInbox/NetReader.pm
> +++ b/lib/PublicInbox/NetReader.pm
> @@ -685,7 +685,13 @@ sub mic_get {
> }
> my $mic = mic_new($self, $mic_arg, $sec, $uri);
> $cached //= {}; # invalid placeholder if no cache enabled
> - $mic && $mic->IsConnected ? ($cached->{$sec} = $mic) : undef;
> + if ($mic && $mic->IsConnected) {
> + $cached->{$sec} = $mic;
> + } else {
> + warn 'IMAP LastError: ',$mic->LastError, "\n" if $mic;
> + warn "IMAP errno: $!\n" if $!;
> + undef;
> + }
> }
>
> sub imap_each {
--
Ricardo Ribalda
^ permalink raw reply [relevance 37%]
* Re: [PATCH v2] lei: bail out earlier on IMAP writer failures
2022-09-10 19:34 37% ` Ricardo Ribalda
@ 2022-09-10 19:50 71% ` Eric Wong
2022-09-10 19:53 70% ` Ricardo Ribalda
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2022-09-10 19:50 UTC (permalink / raw)
To: Ricardo Ribalda; +Cc: meta
Ricardo Ribalda <ribalda@chromium.org> wrote:
> Similar output:
Oh, wait, did you run `lei daemon-kill' after applying the
patch? That's needed to reload the code, I'll try to make it
either auto-reload or warn users in the future.
Thanks.
^ permalink raw reply [relevance 71%]
* Re: [PATCH v2] lei: bail out earlier on IMAP writer failures
2022-09-10 19:50 71% ` Eric Wong
@ 2022-09-10 19:53 70% ` Ricardo Ribalda
2022-09-10 20:19 71% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Ricardo Ribalda @ 2022-09-10 19:53 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
On Sat, 10 Sept 2022 at 21:51, Eric Wong <e@80x24.org> wrote:
>
> Ricardo Ribalda <ribalda@chromium.org> wrote:
> > Similar output:
>
> Oh, wait, did you run `lei daemon-kill' after applying the
> patch? That's needed to reload the code, I'll try to make it
> either auto-reload or warn users in the future.
>
> Thanks.
Ups :S
ribalda@denia:/tmp/public-inbox$ lei up imaps://imap.gmail.com/lei/me
# https://lore.kernel.org/all/ limiting to 2022-09-08 19:52 +0000 and newer
# /usr/local/google/home/ribalda/.local/share/lei/store 12/12
# /usr/bin/curl -Sf -s -d ''
https://lore.kernel.org/all/?x=m&t=1&q=((ribalda)+AND+rt%3A1660161176..)+AND+dt%3A20220908195223..
E: imaps://imap.gmail.com/lei/me connection failed.
E: Consider using `--jobs ,1' to limit IMAP connections
ribalda@denia:/tmp/public-inbox$ lei up imaps://imap.gmail.com/lei/me --jobs ,15
# https://lore.kernel.org/all/ limiting to 2022-09-08 19:52 +0000 and newer
# /usr/local/google/home/ribalda/.local/share/lei/store 12/12
# /usr/bin/curl -Sf -s -d ''
https://lore.kernel.org/all/?x=m&t=1&q=((ribalda)+AND+rt%3A1660161182..)+AND+dt%3A20220908195223..
# https://lore.kernel.org/all/ 20/20
# 0 written to imaps://imap.gmail.com/lei/me (32 matches, 20 duplicates)
--
Ricardo Ribalda
^ permalink raw reply [relevance 70%]
* Re: [PATCH v2] lei: bail out earlier on IMAP writer failures
2022-09-10 19:53 70% ` Ricardo Ribalda
@ 2022-09-10 20:19 71% ` Eric Wong
2022-11-14 8:07 64% ` [PATCH] lei q|up: limit default write --jobs for IMAP(S) Eric Wong
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2022-09-10 20:19 UTC (permalink / raw)
To: Ricardo Ribalda; +Cc: meta
Ricardo Ribalda <ribalda@chromium.org> wrote:
> ribalda@denia:/tmp/public-inbox$ lei up imaps://imap.gmail.com/lei/me
> # https://lore.kernel.org/all/ limiting to 2022-09-08 19:52 +0000 and newer
> # /usr/local/google/home/ribalda/.local/share/lei/store 12/12
> # /usr/bin/curl -Sf -s -d ''
> https://lore.kernel.org/all/?x=m&t=1&q=((ribalda)+AND+rt%3A1660161176..)+AND+dt%3A20220908195223..
> E: imaps://imap.gmail.com/lei/me connection failed.
> E: Consider using `--jobs ,1' to limit IMAP connections
Thanks for confirming things work as intended. I think the
default should be clamped, though... 15 seems a bit high for
smaller IMAP servers *shrug*
> ribalda@denia:/tmp/public-inbox$ lei up imaps://imap.gmail.com/lei/me --jobs ,15
^ permalink raw reply [relevance 71%]
* [PATCH 0/4] fixes noticed while diagnosing t/lei-up.t
@ 2022-09-30 9:21 71% Eric Wong
2022-09-30 9:21 56% ` [PATCH 2/4] t/lei-up: improve diagnostics for this test Eric Wong
` (2 more replies)
0 siblings, 3 replies; 200+ results
From: Eric Wong @ 2022-09-30 9:21 UTC (permalink / raw)
To: meta
I'm still trying to figure out why OverIdx->adj_counter (via
next_tid) in the LeiSavedSearch dedupe check occasionally fails
under CentOS 7.x (but not other systems).
Meanwhile, some improvements noticed along the way.
The underlying problem remains...
Disabling WAL didn't help t/lei-up.t on CentOS 7.x, so the
obvious newish feature we use is unlikely the culprit...
Eric Wong (4):
tests: favor 3 argument `open' with interopolation
t/lei-up: improve diagnostics for this test
lei_to_mail: propagate errors to script/lei
t/altid_v2: improve test style
lib/PublicInbox/LeiToMail.pm | 34 ++++++++++++++++------------
t/altid_v2.t | 10 ++++-----
t/hl_mod.t | 4 ++--
t/lei-up.t | 43 ++++++++++++++++++++++--------------
t/lei_to_mail.t | 10 ++++-----
5 files changed, 59 insertions(+), 42 deletions(-)
^ permalink raw reply [relevance 71%]
* [PATCH 3/4] lei_to_mail: propagate errors to script/lei
2022-09-30 9:21 71% [PATCH 0/4] fixes noticed while diagnosing t/lei-up.t Eric Wong
2022-09-30 9:21 56% ` [PATCH 2/4] t/lei-up: improve diagnostics for this test Eric Wong
@ 2022-09-30 9:21 61% ` Eric Wong
2022-09-30 17:20 65% ` SQLite <3.8.3 was broken on fork (was: fixes noticed while diagnosing t/lei-up.t) Eric Wong
2 siblings, 0 replies; 200+ results
From: Eric Wong @ 2022-09-30 9:21 UTC (permalink / raw)
To: meta
We need to rely on lei->fail to propagate errors in lei workers
to the script/lei client, otherwise tests and other scripts can
stumble forward with incomplete/incorrect/broken outputs.
This helps me focus on occasional t/lei-up.t failures I see on
CentOS 7.x where OverIdx->adj_counter fails on "lei up --all"...
---
lib/PublicInbox/LeiToMail.pm | 34 ++++++++++++++++++++--------------
1 file changed, 20 insertions(+), 14 deletions(-)
diff --git a/lib/PublicInbox/LeiToMail.pm b/lib/PublicInbox/LeiToMail.pm
index 03cbde3b..b58e2652 100644
--- a/lib/PublicInbox/LeiToMail.pm
+++ b/lib/PublicInbox/LeiToMail.pm
@@ -132,19 +132,22 @@ sub eml2mboxcl2 {
}
sub git_to_mail { # git->cat_async callback
- my ($bref, $oid, $type, $size, $arg) = @_;
+ my ($bref, $oid, $type, $size, $smsg) = @_;
+ my $self = delete $smsg->{l2m} // die "BUG: no l2m";
$type // return; # called by git->async_abort
- my ($write_cb, $smsg) = @$arg;
- if ($type eq 'missing' && $smsg->{-lms_rw}) {
- if ($bref = $smsg->{-lms_rw}->local_blob($oid, 1)) {
+ eval {
+ if ($type eq 'missing' &&
+ ($bref = $self->{-lms_rw}->local_blob($oid, 1))) {
$type = 'blob';
$size = length($$bref);
}
- }
- return warn("W: $oid is $type (!= blob)\n") if $type ne 'blob';
- return warn("E: $oid is empty\n") unless $size;
- die "BUG: expected=$smsg->{blob} got=$oid" if $smsg->{blob} ne $oid;
- $write_cb->($bref, $smsg);
+ $type eq 'blob' or return $self->{lei}->child_error(1,
+ "W: $oid is $type (!= blob)");
+ $size or return $self->{lei}->child_error(1,"E: $oid is empty");
+ $smsg->{blob} eq $oid or die "BUG: expected=$smsg->{blob}";
+ $self->{wcb}->($bref, $smsg);
+ };
+ $self->{lei}->fail("$@ (oid=$oid)") if $@;
}
sub reap_compress { # dwaitpid callback
@@ -790,19 +793,22 @@ sub poke_dst {
sub write_mail { # via ->wq_io_do
my ($self, $smsg, $eml) = @_;
- return $self->{wcb}->(undef, $smsg, $eml) if $eml;
- $smsg->{-lms_rw} = $self->{-lms_rw};
- $self->{git}->cat_async($smsg->{blob}, \&git_to_mail,
- [$self->{wcb}, $smsg]);
+ if ($eml) {
+ eval { $self->{wcb}->(undef, $smsg, $eml) };
+ $self->{lei}->fail("blob=$smsg->{blob} $@") if $@;
+ } else {
+ $smsg->{l2m} = $self;
+ $self->{git}->cat_async($smsg->{blob}, \&git_to_mail, $smsg);
+ }
}
sub wq_atexit_child {
my ($self) = @_;
local $PublicInbox::DS::in_loop = 0; # waitpid synchronously
my $lei = $self->{lei};
- delete $self->{wcb};
$lei->{ale}->git->async_wait_all;
my ($nr_w, $nr_s) = delete(@$lei{qw(-nr_write -nr_seen)});
+ delete $self->{wcb};
$nr_s or return;
return if $lei->{early_mua} || !$lei->{-progress} || !$lei->{pkt_op_p};
$lei->{pkt_op_p}->pkt_do('l2m_progress', $nr_w, $nr_s);
^ permalink raw reply related [relevance 61%]
* [PATCH 2/4] t/lei-up: improve diagnostics for this test
2022-09-30 9:21 71% [PATCH 0/4] fixes noticed while diagnosing t/lei-up.t Eric Wong
@ 2022-09-30 9:21 56% ` Eric Wong
2022-09-30 9:21 61% ` [PATCH 3/4] lei_to_mail: propagate errors to script/lei Eric Wong
2022-09-30 17:20 65% ` SQLite <3.8.3 was broken on fork (was: fixes noticed while diagnosing t/lei-up.t) Eric Wong
2 siblings, 0 replies; 200+ results
From: Eric Wong @ 2022-09-30 9:21 UTC (permalink / raw)
To: meta
I'm getting occasional failures for this test on CentOS 7.x (but
not on FreeBSD nor Debian 10/11). I'm not why, yet, so just
improve diagnostics for now.
---
t/lei-up.t | 43 +++++++++++++++++++++++++++----------------
1 file changed, 27 insertions(+), 16 deletions(-)
diff --git a/t/lei-up.t b/t/lei-up.t
index 022ebc05..baed6507 100644
--- a/t/lei-up.t
+++ b/t/lei-up.t
@@ -5,39 +5,50 @@ use strict; use v5.10.1; use PublicInbox::TestCommon;
use IO::Uncompress::Gunzip qw(gunzip $GunzipError);
test_lei(sub {
my ($ro_home, $cfg_path) = setup_public_inboxes;
- my $s = eml_load('t/plack-qp.eml')->as_string;
+ my $home = $ENV{HOME};
+ my $qp = eml_load('t/plack-qp.eml');
+ my $s = $qp->as_string;
lei_ok [qw(import -q -F eml -)], undef, { 0 => \$s, %$lei_opt };
- lei_ok qw(q z:0.. -f mboxcl2 -o), "$ENV{HOME}/a.mbox.gz";
- lei_ok qw(q z:0.. -f mboxcl2 -o), "$ENV{HOME}/b.mbox.gz";
- lei_ok qw(q z:0.. -f mboxcl2 -o), "$ENV{HOME}/a";
- lei_ok qw(q z:0.. -f mboxcl2 -o), "$ENV{HOME}/b";
+ lei_ok qw(q z:0.. -f mboxcl2 -o), "$home/a.mbox.gz";
+ lei_ok qw(q z:0.. -f mboxcl2 -o), "$home/b.mbox.gz";
+ lei_ok qw(q z:0.. -f mboxcl2 -o), "$home/a";
+ lei_ok qw(q z:0.. -f mboxcl2 -o), "$home/b";
+ my $uc;
+ for my $x (qw(a b)) {
+ gunzip("$home/$x.mbox.gz" => \$uc, MultiStream => 1) or
+ xbail "gunzip $GunzipError";
+ ok(index($uc, $qp->body_raw) >= 0,
+ "original mail in $x.mbox.gz");
+ open my $fh, '<', "$home/$x" or xbail $!;
+ $uc = do { local $/; <$fh> } // xbail $!;
+ ok(index($uc, $qp->body_raw) >= 0,
+ "original mail in uncompressed $x");
+ }
lei_ok qw(ls-search);
$s = eml_load('t/utf8.eml')->as_string;
lei_ok [qw(import -q -F eml -)], undef, { 0 => \$s, %$lei_opt };
lei_ok qw(up --all=local);
- open my $fh, '<', "$ENV{HOME}/a.mbox.gz" or xbail "open: $!";
- my $gz = do { local $/; <$fh> };
- my $uc;
- gunzip(\$gz => \$uc, MultiStream => 1) or xbail "gunzip $GunzipError";
- open $fh, '<', "$ENV{HOME}/a" or xbail "open: $!";
+ gunzip("$home/a.mbox.gz" => \$uc, MultiStream => 1) or
+ xbail "gunzip $GunzipError";
+
+ open my $fh, '<', "$home/a" or xbail "open: $!";
my $exp = do { local $/; <$fh> };
is($uc, $exp, 'compressed and uncompressed match (a.gz)');
like($exp, qr/testmessage\@example.com/, '2nd message added');
- open $fh, '<', "$ENV{HOME}/b.mbox.gz" or xbail "open: $!";
- $gz = do { local $/; <$fh> };
undef $uc;
- gunzip(\$gz => \$uc, MultiStream => 1) or xbail "gunzip $GunzipError";
+ gunzip("$home/b.mbox.gz" => \$uc, MultiStream => 1) or
+ xbail "gunzip $GunzipError";
is($uc, $exp, 'compressed and uncompressed match (b.gz)');
- open $fh, '<', "$ENV{HOME}/b" or xbail "open: $!";
+ open $fh, '<', "$home/b" or xbail "open: $!";
$uc = do { local $/; <$fh> };
is($uc, $exp, 'uncompressed both match');
- lei_ok [ qw(up -q), "$ENV{HOME}/b", "--mua=touch $ENV{HOME}/c" ],
+ lei_ok [ qw(up -q), "$home/b", "--mua=touch $home/c" ],
undef, { run_mode => 0 };
- ok(-f "$ENV{HOME}/c", '--mua works with single output');
+ ok(-f "$home/c", '--mua works with single output');
});
done_testing;
^ permalink raw reply related [relevance 56%]
* SQLite <3.8.3 was broken on fork (was: fixes noticed while diagnosing t/lei-up.t)
2022-09-30 9:21 71% [PATCH 0/4] fixes noticed while diagnosing t/lei-up.t Eric Wong
2022-09-30 9:21 56% ` [PATCH 2/4] t/lei-up: improve diagnostics for this test Eric Wong
2022-09-30 9:21 61% ` [PATCH 3/4] lei_to_mail: propagate errors to script/lei Eric Wong
@ 2022-09-30 17:20 65% ` Eric Wong
2 siblings, 0 replies; 200+ results
From: Eric Wong @ 2022-09-30 17:20 UTC (permalink / raw)
To: meta
Eric Wong <e@80x24.org> wrote:
> I'm still trying to figure out why OverIdx->adj_counter (via
> next_tid) in the LeiSavedSearch dedupe check occasionally fails
> under CentOS 7.x (but not other systems).
CentOS 7.x ships SQLite 3.7.17, and SQLite prior to 3.8.3 (2014-02-03)
didn't reset its PRNG for generating temporary filenames upon fork.
I can't find a way reset that PRNG via DBD::SQLite, either...
One possible nasty thing I could do is reset TMPDIR upon fork,
but that may leave bugs lingering, too :<
Here's the debugging patch I used to strace what was going on:
(I may just make lei_ok bail out on error...)
diff --git a/lib/PublicInbox/OverIdx.pm b/lib/PublicInbox/OverIdx.pm
index e7c96e14..e1af910c 100644
--- a/lib/PublicInbox/OverIdx.pm
+++ b/lib/PublicInbox/OverIdx.pm
@@ -51,10 +51,22 @@ SELECT val FROM counter WHERE key = ? LIMIT 1
sub adj_counter ($$$) {
my ($self, $key, $op) = @_;
my $dbh = $self->{dbh};
+use PublicInbox::Spawn qw(spawn);
+my $trace = "/tmp/$$.strace";
+open my $err, '>>', $trace;
+my $pid = spawn([qw(strace -f -s4096 -v -p), $$], undef, { 2 => $err });
+select undef, undef, undef, 0.1;
+
my $sth = $dbh->prepare_cached(<<"");
UPDATE counter SET val = val $op 1 WHERE key = ?
- $sth->execute($key);
+ eval { $sth->execute($key) };
+my $err = $@;
+syswrite(STDERR, "$$ $err\n") if $err;
+kill('TERM', $pid);
+waitpid($pid, 0);
+die $err if $err;
+#unlink($trace);
get_counter($dbh, $key);
}
diff --git a/t/lei-up.t b/t/lei-up.t
index baed6507..9c65a243 100644
--- a/t/lei-up.t
+++ b/t/lei-up.t
@@ -27,7 +27,7 @@ test_lei(sub {
lei_ok qw(ls-search);
$s = eml_load('t/utf8.eml')->as_string;
lei_ok [qw(import -q -F eml -)], undef, { 0 => \$s, %$lei_opt };
- lei_ok qw(up --all=local);
+ lei_ok qw(up --all=local) or xbail "lei up --all=local failed $?";
gunzip("$home/a.mbox.gz" => \$uc, MultiStream => 1) or
xbail "gunzip $GunzipError";
^ permalink raw reply related [relevance 65%]
* [PATCH] lei: force --jobs=1,1 for SQLite < 3.8.3
@ 2022-10-01 0:33 60% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2022-10-01 0:33 UTC (permalink / raw)
To: meta
SQLite prior to 3.8.3 did not reset its PRNG for generating
unique temporary file names, so it would barf on t/lei-up.t
occasionally due to O_EXCL -> EEXIST conflicts.
This fixes occasional test failures under CentOS 7.x which ships
SQLite 3.7.17.
---
lib/PublicInbox/LeiQuery.pm | 4 +++-
lib/PublicInbox/LeiUp.pm | 5 +++--
lib/PublicInbox/OverIdx.pm | 12 ++++++++++++
3 files changed, 18 insertions(+), 3 deletions(-)
diff --git a/lib/PublicInbox/LeiQuery.pm b/lib/PublicInbox/LeiQuery.pm
index c998e5c0..df9c32b3 100644
--- a/lib/PublicInbox/LeiQuery.pm
+++ b/lib/PublicInbox/LeiQuery.pm
@@ -1,4 +1,4 @@
-# Copyright (C) 2021 all contributors <meta@public-inbox.org>
+# Copyright (C) all contributors <meta@public-inbox.org>
# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
# handles "lei q" command and provides internals for
@@ -6,6 +6,7 @@
package PublicInbox::LeiQuery;
use strict;
use v5.10.1;
+use PublicInbox::OverIdx;
sub prep_ext { # externals_each callback
my ($lxs, $exclude, $loc) = @_;
@@ -17,6 +18,7 @@ sub _start_query { # used by "lei q" and "lei up"
require PublicInbox::LeiOverview;
PublicInbox::LeiOverview->new($self) or return;
my $opt = $self->{opt};
+ PublicInbox::OverIdx::fork_ok($opt);
my ($xj, $mj) = split(/,/, $opt->{jobs} // '');
(defined($xj) && $xj ne '' && $xj !~ /\A[1-9][0-9]*\z/) and
die "`$xj' search jobs must be >= 1\n";
diff --git a/lib/PublicInbox/LeiUp.pm b/lib/PublicInbox/LeiUp.pm
index b8a98360..5ad21451 100644
--- a/lib/PublicInbox/LeiUp.pm
+++ b/lib/PublicInbox/LeiUp.pm
@@ -1,4 +1,4 @@
-# Copyright (C) 2021 all contributors <meta@public-inbox.org>
+# Copyright (C) all contributors <meta@public-inbox.org>
# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
# "lei up" - updates the result of "lei q --save"
@@ -7,7 +7,7 @@ use strict;
use v5.10.1;
# n.b. we use LeiInput to setup IMAP auth
use parent qw(PublicInbox::IPC PublicInbox::LeiInput);
-use PublicInbox::LeiSavedSearch;
+use PublicInbox::LeiSavedSearch; # OverIdx
use PublicInbox::DS;
use PublicInbox::PktOp;
use PublicInbox::LeiFinmsg;
@@ -75,6 +75,7 @@ sub redispatch_all ($$) {
my $upq = [ (@{$self->{o_local} // []}, @{$self->{o_remote} // []}) ];
return up1($lei, $upq->[0]) if @$upq == 1; # just one, may start MUA
+ PublicInbox::OverIdx::fork_ok($lei->{opt});
# FIXME: this is also used per-query, see lei->_start_query
my $j = $lei->{opt}->{jobs} || do {
my $n = $self->detect_nproc // 1;
diff --git a/lib/PublicInbox/OverIdx.pm b/lib/PublicInbox/OverIdx.pm
index e7c96e14..a49ca6db 100644
--- a/lib/PublicInbox/OverIdx.pm
+++ b/lib/PublicInbox/OverIdx.pm
@@ -670,4 +670,16 @@ sub vivify_xvmd {
$smsg->{-vivify_xvmd} = \@vivify_xvmd;
}
+sub fork_ok {
+ return 1 if $DBD::SQLite::sqlite_version >= 3008003;
+ my ($opt) = @_;
+ my @j = split(/,/, $opt->{jobs} // '');
+ state $warned;
+ grep { $_ > 1 } @j and $warned //= warn('DBD::SQLite version is ',
+ $DBD::SQLite::sqlite_version,
+ ", need >= 3008003 (3.8.3) for --jobs > 1\n");
+ $opt->{jobs} = '1,1';
+ undef;
+}
+
1;
--
2.33.0
^ permalink raw reply related [relevance 60%]
* [Need Help] lei add quotes at the search
@ 2022-10-30 4:03 70% Hangbin Liu
2022-10-30 5:13 71% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Hangbin Liu @ 2022-10-30 4:03 UTC (permalink / raw)
To: meta
Hi,
I used to use a search like
lei q -I https://lore.kernel.org/all/ -o ~/Mail/liuhangbin --threads --dedupe=mid '((tc:liuhangbin AND rt:6.month.ago..) NOT (tc:stable@vger.kernel.org OR f:sfr@canb.auug.org.au)'
It works on fc35. But after I update to fc36 with lei-1.9.0-1.fc36. It start to
add quotes in the search link and make the search never works. e.g.
$ lei q -I https://lore.kernel.org/all/ -o ~/Mail/liuhangbin --threads --dedupe=mid '((tc:liuhangbin AND rt:6.month.ago..) NOT (tc:stable@vger.kernel.org OR f:sfr@canb.auug.org.au)'
# /home/Liu/.local/share/lei/store 0/0
# /usr/bin/curl -Sf -s -d '' https://lore.kernel.org/all/?x=m&t=1&q=((tc%3A%22liuhangbin+AND+rt%3A6.month.ago..)+NOT+(tc%3Astable%40vger.kernel.org+OR+f%3Asfr%40canb.auug.org.au)%22
# 0 written to /home/Liu/Mail/liuhangbin/ (0 matches)
Do you think if this is a bug, or I should update my search.
Thanks
Hangbin
^ permalink raw reply [relevance 70%]
* Re: [Need Help] lei add quotes at the search
2022-10-30 4:03 70% [Need Help] lei add quotes at the search Hangbin Liu
@ 2022-10-30 5:13 71% ` Eric Wong
2022-10-30 7:08 59% ` Hangbin Liu
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2022-10-30 5:13 UTC (permalink / raw)
To: Hangbin Liu; +Cc: meta
Hangbin Liu <liuhangbin@gmail.com> wrote:
> Hi,
>
> I used to use a search like
>
> lei q -I https://lore.kernel.org/all/ -o ~/Mail/liuhangbin --threads --dedupe=mid '((tc:liuhangbin AND rt:6.month.ago..) NOT (tc:stable@vger.kernel.org OR f:sfr@canb.auug.org.au)'
>
> It works on fc35. But after I update to fc36 with lei-1.9.0-1.fc36. It start to
> add quotes in the search link and make the search never works. e.g.
Are you able to show the curl CLI from fc35?
Which public-inbox/lei version was it?
I'm actually curious fc35 worked at all, since the quoting would've
been broken, I think...
> $ lei q -I https://lore.kernel.org/all/ -o ~/Mail/liuhangbin --threads --dedupe=mid '((tc:liuhangbin AND rt:6.month.ago..) NOT (tc:stable@vger.kernel.org OR f:sfr@canb.auug.org.au)'
> # /home/Liu/.local/share/lei/store 0/0
> # /usr/bin/curl -Sf -s -d '' https://lore.kernel.org/all/?x=m&t=1&q=((tc%3A%22liuhangbin+AND+rt%3A6.month.ago..)+NOT+(tc%3Astable%40vger.kernel.org+OR+f%3Asfr%40canb.auug.org.au)%22
> # 0 written to /home/Liu/Mail/liuhangbin/ (0 matches)
>
> Do you think if this is a bug, or I should update my search.
The %22 in fc36 is because your entire query is treated as one
element in argv and matches expected behavior.
Since '(' and ')' in the shell CLI is special, I suggest either:
a) using --stdin to enter queries containing '(' and ')'
b) quoting (or escaping) only the '(' and ')':
'('tc:liuhangbin AND rt:6.month.ago..')' NOT ...
or
\(tc:liuhangbin AND rt:6.month.ago..\) NOT ...
Which makes your argv something like:
[ "(tc:liuhangbin", "AND", "rt:6.month.ago..)", "NOT", ... ]
^ permalink raw reply [relevance 71%]
* Re: [Need Help] lei add quotes at the search
2022-10-30 5:13 71% ` Eric Wong
@ 2022-10-30 7:08 59% ` Hangbin Liu
2022-10-30 23:06 63% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Hangbin Liu @ 2022-10-30 7:08 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
Hi Eric,
Thanks for the help.
On Sun, Oct 30, 2022 at 05:13:33AM +0000, Eric Wong wrote:
> Hangbin Liu <liuhangbin@gmail.com> wrote:
> > Hi,
> >
> > I used to use a search like
> >
> > lei q -I https://lore.kernel.org/all/ -o ~/Mail/liuhangbin --threads --dedupe=mid '((tc:liuhangbin AND rt:6.month.ago..) NOT (tc:stable@vger.kernel.org OR f:sfr@canb.auug.org.au)'
> >
> > It works on fc35. But after I update to fc36 with lei-1.9.0-1.fc36. It start to
> > add quotes in the search link and make the search never works. e.g.
>
> Are you able to show the curl CLI from fc35?
> Which public-inbox/lei version was it?
>
> I'm actually curious fc35 worked at all, since the quoting would've
> been broken, I think...
Sorry, I don't have the fc35 environment now.
>
> > $ lei q -I https://lore.kernel.org/all/ -o ~/Mail/liuhangbin --threads --dedupe=mid '((tc:liuhangbin AND rt:6.month.ago..) NOT (tc:stable@vger.kernel.org OR f:sfr@canb.auug.org.au)'
> > # /home/Liu/.local/share/lei/store 0/0
> > # /usr/bin/curl -Sf -s -d '' https://lore.kernel.org/all/?x=m&t=1&q=((tc%3A%22liuhangbin+AND+rt%3A6.month.ago..)+NOT+(tc%3Astable%40vger.kernel.org+OR+f%3Asfr%40canb.auug.org.au)%22
> > # 0 written to /home/Liu/Mail/liuhangbin/ (0 matches)
> >
> > Do you think if this is a bug, or I should update my search.
>
> The %22 in fc36 is because your entire query is treated as one
> element in argv and matches expected behavior.
>
> Since '(' and ')' in the shell CLI is special, I suggest either:
I'm curious about why the quote(%22) is added after "tc", not after "("
>
> a) using --stdin to enter queries containing '(' and ')'
>
> b) quoting (or escaping) only the '(' and ')':
>
> '('tc:liuhangbin AND rt:6.month.ago..')' NOT ...
>
> or
>
> \(tc:liuhangbin AND rt:6.month.ago..\) NOT ...
with this way, the cmd line works. And in config file, it would looks like
[lei]
q = ((tc:liuhangbin
q = AND
q = rt:6.month.ago..)
q = NOT
q = (tc:stable@vger.kernel.org
q = OR
q = f:sfr@canb.auug.org.au)
But if I have a long search line. This will breaks too much and hard to edit.
e.g. My real previous search is like
[lei]
q = (tc:liuhangbin OR \
(dfn:drivers/net/wireguard/ AND rt:6.month.ago..) OR \
(dfn:tools/testing/selftests/net/ AND rt:1.month.ago..) OR \
(dfn:drivers/net/team/ AND rt:6.month.ago..) OR \
(dfn:net/ipv4/igmp.c AND rt:6.month.ago..) OR \
(dfn:net/ipv6/mcast.c AND rt:6.month.ago..)) \
NOT (tc:stable@vger.kernel.org OR f:sfr@canb.auug.org.au)
If I add "\" on each "(", this will break to a very long config search.
I tried to adjust it to
[lei]
q = (tc:liuhangbin OR \
(dfn:drivers/net/wireguard/ AND rt:6.month.ago..) OR \
(dfn:tools/testing/selftests/net/ AND rt:1.month.ago..) OR \
(dfn:drivers/net/team/ AND rt:6.month.ago..) OR \
(dfn:net/ipv4/igmp.c AND rt:6.month.ago..) OR \
(dfn:net/ipv6/mcast.c AND rt:6.month.ago..))
q = NOT
q = (tc:stable@vger.kernel.org
q = OR
q = f:sfr@canb.auug.org.au)
And now it works...
$ lei up /home/Liu/Mail/gmail/Linux_Kernel
# https://lore.kernel.org/all/ limiting to 2022-09-30 17:00 +0800 and newer
60927 lei_xsearch 0 wq_worker: query_one_mset: Exception: Unknown range operation at /usr/share/perl5/vendor_perl/PublicInbox/IPC.pm line 254.
# /usr/bin/curl -Sf -s -d '' https://lore.kernel.org/all/?x=m&t=1&q=((tc%3Aliuhangbin+OR+(dfn%3Adrivers%2Fnet%2Fwireguard%2F+AND+rt%3A6.month.ago..)+OR+(dfn%3Atools%2Ftesting%2Fselftests%2Fnet%2F+AND+rt%3A1.month.ago..)+OR+(dfn%3Adrivers%2Fnet%2Fteam%2F+AND+rt%3A6.month.ago..)+OR+(dfn%3Anet%2Fipv4%2Figmp.c+AND+rt%3A6.month.ago..)+OR+(dfn%3Anet%2Fipv6%2Fmcast.c+AND+rt%3A1651301673..))+NOT+(tc%3Astable%40vger.kernel.org+OR+f%3Asfr%40canb.auug.org.au))+AND+dt%3A20220930090001..
# https://lore.kernel.org/all/ 43/?
So I want to know when/why *lei* add the quotes.
Thanks
Hangbin
^ permalink raw reply [relevance 59%]
* Re: [Need Help] lei add quotes at the search
2022-10-30 7:08 59% ` Hangbin Liu
@ 2022-10-30 23:06 63% ` Eric Wong
2022-10-31 7:36 71% ` Hangbin Liu
2022-10-31 7:47 71% ` Hangbin Liu
0 siblings, 2 replies; 200+ results
From: Eric Wong @ 2022-10-30 23:06 UTC (permalink / raw)
To: Hangbin Liu; +Cc: meta
Hangbin Liu <liuhangbin@gmail.com> wrote:
> Hi Eric,
>
> Thanks for the help.
>
> On Sun, Oct 30, 2022 at 05:13:33AM +0000, Eric Wong wrote:
> > Hangbin Liu <liuhangbin@gmail.com> wrote:
> > > Hi,
> > >
> > > I used to use a search like
> > >
> > > lei q -I https://lore.kernel.org/all/ -o ~/Mail/liuhangbin --threads --dedupe=mid '((tc:liuhangbin AND rt:6.month.ago..) NOT (tc:stable@vger.kernel.org OR f:sfr@canb.auug.org.au)'
> > >
> > > It works on fc35. But after I update to fc36 with lei-1.9.0-1.fc36. It start to
> > > add quotes in the search link and make the search never works. e.g.
> >
> > Are you able to show the curl CLI from fc35?
> > Which public-inbox/lei version was it?
> >
> > I'm actually curious fc35 worked at all, since the quoting would've
> > been broken, I think...
>
> Sorry, I don't have the fc35 environment now.
No worries, I dont think fc35 is really a culprit. Were you
running a pre-release version of public-inbox or lei before?
> > > $ lei q -I https://lore.kernel.org/all/ -o ~/Mail/liuhangbin --threads --dedupe=mid '((tc:liuhangbin AND rt:6.month.ago..) NOT (tc:stable@vger.kernel.org OR f:sfr@canb.auug.org.au)'
> > > # /home/Liu/.local/share/lei/store 0/0
> > > # /usr/bin/curl -Sf -s -d '' https://lore.kernel.org/all/?x=m&t=1&q=((tc%3A%22liuhangbin+AND+rt%3A6.month.ago..)+NOT+(tc%3Astable%40vger.kernel.org+OR+f%3Asfr%40canb.auug.org.au)%22
> > > # 0 written to /home/Liu/Mail/liuhangbin/ (0 matches)
> > >
> > > Do you think if this is a bug, or I should update my search.
> >
> > The %22 in fc36 is because your entire query is treated as one
> > element in argv and matches expected behavior.
> >
> > Since '(' and ')' in the shell CLI is special, I suggest either:
>
> I'm curious about why the quote(%22) is added after "tc", not after "("
It's because Xapian can only handle a phrase after the `tc:' prefix.
thus: tc:"foo bar" actually parses `tc:' as a prefix for To/Cc;
while: "tc:foo bar" looks for the phrase "tc:foo bar" anywhere
in the message, and won't limit to To/Cc headers.
This happens in the query_argv_to_string sub:
https://public-inbox.org/meta/2feb3e13b49d222bc7bd28430a9cf159692a933f/s/?b=lib/PublicInbox/Search.pm#n358
From the CLI: lei q "tc:foo bar" is indistinguishable
from lei q tc:"foo bar" , so it gets treated as the latter.
> > a) using --stdin to enter queries containing '(' and ')'
> >
> > b) quoting (or escaping) only the '(' and ')':
> >
> > '('tc:liuhangbin AND rt:6.month.ago..')' NOT ...
> >
> > or
> >
> > \(tc:liuhangbin AND rt:6.month.ago..\) NOT ...
>
> with this way, the cmd line works. And in config file, it would looks like
>
> [lei]
> q = ((tc:liuhangbin
> q = AND
> q = rt:6.month.ago..)
> q = NOT
> q = (tc:stable@vger.kernel.org
> q = OR
> q = f:sfr@canb.auug.org.au)
>
> But if I have a long search line. This will breaks too much and hard to edit.
> e.g. My real previous search is like
>
> [lei]
> q = (tc:liuhangbin OR \
> (dfn:drivers/net/wireguard/ AND rt:6.month.ago..) OR \
> (dfn:tools/testing/selftests/net/ AND rt:1.month.ago..) OR \
> (dfn:drivers/net/team/ AND rt:6.month.ago..) OR \
> (dfn:net/ipv4/igmp.c AND rt:6.month.ago..) OR \
> (dfn:net/ipv6/mcast.c AND rt:6.month.ago..)) \
> NOT (tc:stable@vger.kernel.org OR f:sfr@canb.auug.org.au)
>
> If I add "\" on each "(", this will break to a very long config search.
> I tried to adjust it to
I think that can work if lei.internal.rawstr is set in the
config to indicate stdin was used (It's auto-set by --stdin).
I guess it also works if it's the only lei.q config entry
and the lei.q entry contains "\n"
cf. https://public-inbox.org/meta/20211110102837.41721-1-e@80x24.org/
> [lei]
> q = (tc:liuhangbin OR \
> (dfn:drivers/net/wireguard/ AND rt:6.month.ago..) OR \
> (dfn:tools/testing/selftests/net/ AND rt:1.month.ago..) OR \
> (dfn:drivers/net/team/ AND rt:6.month.ago..) OR \
> (dfn:net/ipv4/igmp.c AND rt:6.month.ago..) OR \
> (dfn:net/ipv6/mcast.c AND rt:6.month.ago..))
> q = NOT
> q = (tc:stable@vger.kernel.org
> q = OR
> q = f:sfr@canb.auug.org.au)
>
> And now it works...
Sorta... at least for remotes it does:
> $ lei up /home/Liu/Mail/gmail/Linux_Kernel
> # https://lore.kernel.org/all/ limiting to 2022-09-30 17:00 +0800 and newer
> 60927 lei_xsearch 0 wq_worker: query_one_mset: Exception: Unknown range operation at /usr/share/perl5/vendor_perl/PublicInbox/IPC.pm line 254.
Note that Exception means it's not handling the first part of
the query when hitting the local Xapian DB. It's not doing the
approxidate ($X.month.ago) substitution for the local Xapian DB,
thus you got the "Unknown range operation".
> # /usr/bin/curl -Sf -s -d '' https://lore.kernel.org/all/?x=m&t=1&q=((tc%3Aliuhangbin+OR+(dfn%3Adrivers%2Fnet%2Fwireguard%2F+AND+rt%3A6.month.ago..)+OR+(dfn%3Atools%2Ftesting%2Fselftests%2Fnet%2F+AND+rt%3A1.month.ago..)+OR+(dfn%3Adrivers%2Fnet%2Fteam%2F+AND+rt%3A6.month.ago..)+OR+(dfn%3Anet%2Fipv4%2Figmp.c+AND+rt%3A6.month.ago..)+OR+(dfn%3Anet%2Fipv6%2Fmcast.c+AND+rt%3A1651301673..))+NOT+(tc%3Astable%40vger.kernel.org+OR+f%3Asfr%40canb.auug.org.au))+AND+dt%3A20220930090001..
> # https://lore.kernel.org/all/ 43/?
Of course, the lack of approxidate parsing there inside lei is
fine, since the lore.kernel.org instance will do it remotely...
> So I want to know when/why *lei* add the quotes.
lei adds quotes since it can't distinguish if the shell user
used single or double quotes. Xapian uses double quotes for
phrase search, and I wanted: lei q "this is a phrase"
to work naturally, which means: lei q 'this is a phrase'
(with single quotes) works the same way as with double quotes
because the difference is handled by the shell and lei never
sees it.
^ permalink raw reply [relevance 63%]
* Re: [Need Help] lei add quotes at the search
2022-10-30 23:06 63% ` Eric Wong
@ 2022-10-31 7:36 71% ` Hangbin Liu
2022-10-31 7:47 71% ` Hangbin Liu
1 sibling, 0 replies; 200+ results
From: Hangbin Liu @ 2022-10-31 7:36 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
On Sun, Oct 30, 2022 at 11:06:31PM +0000, Eric Wong wrote:
> Hangbin Liu <liuhangbin@gmail.com> wrote:
> > Sorry, I don't have the fc35 environment now.
>
> No worries, I dont think fc35 is really a culprit. Were you
> running a pre-release version of public-inbox or lei before?
Sorry, I forgot. Maybe I installed via `dnf copr enable icon/b4`
because I start using lei after reading blog
https://people.kernel.org/monsieuricon/lore-lei-part-1-getting-started
> > I'm curious about why the quote(%22) is added after "tc", not after "("
>
> It's because Xapian can only handle a phrase after the `tc:' prefix.
> thus: tc:"foo bar" actually parses `tc:' as a prefix for To/Cc;
> while: "tc:foo bar" looks for the phrase "tc:foo bar" anywhere
> in the message, and won't limit to To/Cc headers.
>
> This happens in the query_argv_to_string sub:
>
> https://public-inbox.org/meta/2feb3e13b49d222bc7bd28430a9cf159692a933f/s/?b=lib/PublicInbox/Search.pm#n358
>
> From the CLI: lei q "tc:foo bar" is indistinguishable
> from lei q tc:"foo bar" , so it gets treated as the latter.
Thanks for the explanation.
> > But if I have a long search line. This will breaks too much and hard to edit.
> > e.g. My real previous search is like
> >
> > [lei]
> > q = (tc:liuhangbin OR \
> > (dfn:drivers/net/wireguard/ AND rt:6.month.ago..) OR \
> > (dfn:tools/testing/selftests/net/ AND rt:1.month.ago..) OR \
> > (dfn:drivers/net/team/ AND rt:6.month.ago..) OR \
> > (dfn:net/ipv4/igmp.c AND rt:6.month.ago..) OR \
> > (dfn:net/ipv6/mcast.c AND rt:6.month.ago..)) \
> > NOT (tc:stable@vger.kernel.org OR f:sfr@canb.auug.org.au)
> >
> > If I add "\" on each "(", this will break to a very long config search.
> > I tried to adjust it to
>
> I think that can work if lei.internal.rawstr is set in the
> config to indicate stdin was used (It's auto-set by --stdin).
> I guess it also works if it's the only lei.q config entry
> and the lei.q entry contains "\n"
>
> cf. https://public-inbox.org/meta/20211110102837.41721-1-e@80x24.org/
>
> > [lei]
> > q = (tc:liuhangbin OR \
> > (dfn:drivers/net/wireguard/ AND rt:6.month.ago..) OR \
> > (dfn:tools/testing/selftests/net/ AND rt:1.month.ago..) OR \
> > (dfn:drivers/net/team/ AND rt:6.month.ago..) OR \
> > (dfn:net/ipv4/igmp.c AND rt:6.month.ago..) OR \
> > (dfn:net/ipv6/mcast.c AND rt:6.month.ago..))
> > q = NOT
> > q = (tc:stable@vger.kernel.org
> > q = OR
> > q = f:sfr@canb.auug.org.au)
> >
> > And now it works...
>
> Sorta... at least for remotes it does:
>
> > $ lei up /home/Liu/Mail/gmail/Linux_Kernel
> > # https://lore.kernel.org/all/ limiting to 2022-09-30 17:00 +0800 and newer
> > 60927 lei_xsearch 0 wq_worker: query_one_mset: Exception: Unknown range operation at /usr/share/perl5/vendor_perl/PublicInbox/IPC.pm line 254.
>
> Note that Exception means it's not handling the first part of
> the query when hitting the local Xapian DB. It's not doing the
> approxidate ($X.month.ago) substitution for the local Xapian DB,
> thus you got the "Unknown range operation".
>
> > # /usr/bin/curl -Sf -s -d '' https://lore.kernel.org/all/?x=m&t=1&q=((tc%3Aliuhangbin+OR+(dfn%3Adrivers%2Fnet%2Fwireguard%2F+AND+rt%3A6.month.ago..)+OR+(dfn%3Atools%2Ftesting%2Fselftests%2Fnet%2F+AND+rt%3A1.month.ago..)+OR+(dfn%3Adrivers%2Fnet%2Fteam%2F+AND+rt%3A6.month.ago..)+OR+(dfn%3Anet%2Fipv4%2Figmp.c+AND+rt%3A6.month.ago..)+OR+(dfn%3Anet%2Fipv6%2Fmcast.c+AND+rt%3A1651301673..))+NOT+(tc%3Astable%40vger.kernel.org+OR+f%3Asfr%40canb.auug.org.au))+AND+dt%3A20220930090001..
> > # https://lore.kernel.org/all/ 43/?
>
> Of course, the lack of approxidate parsing there inside lei is
> fine, since the lore.kernel.org instance will do it remotely...
>
> > So I want to know when/why *lei* add the quotes.
>
> lei adds quotes since it can't distinguish if the shell user
> used single or double quotes. Xapian uses double quotes for
> phrase search, and I wanted: lei q "this is a phrase"
> to work naturally, which means: lei q 'this is a phrase'
> (with single quotes) works the same way as with double quotes
> because the difference is handled by the shell and lei never
> sees it.
Thanks for the help.
Hangbin
^ permalink raw reply [relevance 71%]
* Re: [Need Help] lei add quotes at the search
2022-10-30 23:06 63% ` Eric Wong
2022-10-31 7:36 71% ` Hangbin Liu
@ 2022-10-31 7:47 71% ` Hangbin Liu
1 sibling, 0 replies; 200+ results
From: Hangbin Liu @ 2022-10-31 7:47 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
On Sun, Oct 30, 2022 at 11:06:31PM +0000, Eric Wong wrote:
> > If I add "\" on each "(", this will break to a very long config search.
> > I tried to adjust it to
>
> I think that can work if lei.internal.rawstr is set in the
> config to indicate stdin was used (It's auto-set by --stdin).
OH, BTW, I will get error
fatal: bad config line 4 in file [..snip..]/lei.saved-search
if adding '\' in the config file.
> I guess it also works if it's the only lei.q config entry
> and the lei.q entry contains "\n"
But with "\n" in the config file. My previous config will works fine.
>
> cf. https://public-inbox.org/meta/20211110102837.41721-1-e@80x24.org/
Thanks
Hangbin
^ permalink raw reply [relevance 71%]
* [PATCH] lei up: improve error for multiple lei.q values
@ 2022-10-31 21:52 90% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2022-10-31 21:52 UTC (permalink / raw)
To: meta
Point users towards the lei.internal.rawstr variable which
may be tripping up handling of lei.q after `lei edit-search'.
---
lib/PublicInbox/LeiUp.pm | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/lib/PublicInbox/LeiUp.pm b/lib/PublicInbox/LeiUp.pm
index 5ad21451..49917339 100644
--- a/lib/PublicInbox/LeiUp.pm
+++ b/lib/PublicInbox/LeiUp.pm
@@ -32,8 +32,10 @@ sub up1 ($$) {
my $rawstr = $lss->{-cfg}->{'lei.internal.rawstr'} //
(scalar(@$q) == 1 && substr($q->[0], -1) eq "\n");
if ($rawstr) {
- scalar(@$q) > 1 and
- die "$f: lei.q has multiple values (@$q) (out=$out)\n";
+ die <<EOM if scalar(@$q) > 1;
+$f: lei.q has multiple values (@$q) (out=$out)
+$f: while lei.internal.rawstr is set
+EOM
$lse->query_approxidate($lse->git, $mset_opt->{qstr} = $q->[0]);
} else {
$mset_opt->{qstr} = $lse->query_argv_to_string($lse->git, $q);
^ permalink raw reply related [relevance 90%]
* [PATCH] lei: fix globbing semantics to match end-of-filename
@ 2022-11-01 9:36 60% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2022-11-01 9:36 UTC (permalink / raw)
To: meta
Globs such as `*/foo' should not match `*/foobar'. I noticed
this while adding glob support to public-inbox-clone.
This may subtly break some existing cases, but there aren't many
lei users, yet, and globbing semantics should match what most
other glob-using programs, do...
We'll also make `lei ls-mail-sync' behave more consistently with
`lei ls-external', as far as the basename matching fallback
goes.
---
lib/PublicInbox/LeiExternal.pm | 4 ++--
lib/PublicInbox/LeiLsExternal.pm | 1 +
lib/PublicInbox/LeiLsMailSync.pm | 7 +++++--
3 files changed, 8 insertions(+), 4 deletions(-)
diff --git a/lib/PublicInbox/LeiExternal.pm b/lib/PublicInbox/LeiExternal.pm
index 30bb1a45..a6562e7f 100644
--- a/lib/PublicInbox/LeiExternal.pm
+++ b/lib/PublicInbox/LeiExternal.pm
@@ -1,4 +1,4 @@
-# Copyright (C) 2020-2021 all contributors <meta@public-inbox.org>
+# Copyright (C) all contributors <meta@public-inbox.org>
# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
# *-external commands of lei
@@ -88,7 +88,7 @@ sub get_externals {
my @cur = externals_each($self);
my $do_glob = !$self->{opt}->{globoff}; # glob by default
if ($do_glob && (my $re = glob2re($loc))) {
- @m = grep(m!$re!, @cur);
+ @m = grep(m!$re/?\z!, @cur);
return @m if scalar(@m);
} elsif (index($loc, '/') < 0) { # exact basename match:
@m = grep(m!/\Q$loc\E/?\z!, @cur);
diff --git a/lib/PublicInbox/LeiLsExternal.pm b/lib/PublicInbox/LeiLsExternal.pm
index dd2eb2e7..e624cbd4 100644
--- a/lib/PublicInbox/LeiLsExternal.pm
+++ b/lib/PublicInbox/LeiLsExternal.pm
@@ -13,6 +13,7 @@ sub lei_ls_external {
my ($OFS, $ORS) = $lei->{opt}->{z} ? ("\0", "\0\0") : (" ", "\n");
$filter //= '*';
my $re = $do_glob ? $lei->glob2re($filter) : undef;
+ $re .= '/?\\z' if defined $re;
$re //= index($filter, '/') < 0 ?
qr!/\Q$filter\E/?\z! : # exact basename match
qr/\Q$filter\E/; # grep -F semantics
diff --git a/lib/PublicInbox/LeiLsMailSync.pm b/lib/PublicInbox/LeiLsMailSync.pm
index 2b167b1d..8da0c284 100644
--- a/lib/PublicInbox/LeiLsMailSync.pm
+++ b/lib/PublicInbox/LeiLsMailSync.pm
@@ -1,4 +1,4 @@
-# Copyright (C) 2021 all contributors <meta@public-inbox.org>
+# Copyright (C) all contributors <meta@public-inbox.org>
# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
# front-end for the "lei ls-mail-sync" sub-command
@@ -12,7 +12,10 @@ sub lei_ls_mail_sync {
my $lms = $lei->lms or return;
my $opt = $lei->{opt};
my $re = $opt->{globoff} ? undef : $lei->glob2re($filter // '*');
- $re //= qr/\Q$filter\E/;
+ $re .= '/?\\z' if defined $re;
+ $re //= index($filter, '/') < 0 ?
+ qr!/\Q$filter\E/?\z! : # exact basename match
+ qr/\Q$filter\E/; # grep -F semantics
my @f = $lms->folders;
@f = $opt->{'invert-match'} ? grep(!/$re/, @f) : grep(/$re/, @f);
if ($opt->{'local'} && !$opt->{remote}) {
^ permalink raw reply related [relevance 60%]
* [PATCH 3/6] doc: txt2pre: linkify "lei COMMAND" form
2022-11-03 0:48 62% ` [PATCH 2/6] doc: lei: improve description of *-search commands Eric Wong
@ 2022-11-03 0:48 71% ` Eric Wong
2022-11-03 0:48 90% ` [PATCH 5/6] doc: lei-import: link to lei-store-format(5) Eric Wong
2022-11-03 0:48 90% ` [PATCH 6/6] txt2pre: linkify lei/store => lei-store-format.html Eric Wong
3 siblings, 0 replies; 200+ results
From: Eric Wong @ 2022-11-03 0:48 UTC (permalink / raw)
To: meta
While manpages are named `L<lei-COMMAND(1)>', `lei COMMAND'
can be worth linkifying for ease-of-navigation, too.
---
Documentation/txt2pre | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/Documentation/txt2pre b/Documentation/txt2pre
index c8dbd2ba..82573a30 100755
--- a/Documentation/txt2pre
+++ b/Documentation/txt2pre
@@ -9,7 +9,7 @@ use strict;
use warnings;
use PublicInbox::Linkify;
use PublicInbox::Hval qw(ascii_html);
-my %xurls;
+my (%xurls, %lei);
for (qw[lei(1)
lei-add-external(1)
lei-add-watch(1)
@@ -77,6 +77,7 @@ for (qw[lei(1)
my ($n) = (/([\w\-\.]+)/);
$xurls{$_} = "$n.html";
$xurls{$n} = "$n.html";
+ /\Alei-(.+?)\(1\)\z/ and $xurls{"lei $1"} = "$n.html";
}
for (qw[make(1) flock(2) setrlimit(2) vfork(2) tmpfs(5) inotify(7) unix(7)
@@ -161,6 +162,9 @@ if ($str =~ /^NAME\n\s+([^\n]+)/sm) {
if ($title =~ /([\w\.\-]+)/) {
delete $xurls{$1};
}
+ if ($title =~ /\blei-([\w\-]+)\b/) {
+ delete $xurls{"lei $1"};
+ }
}
$title = ascii_html($title);
my $l = PublicInbox::Linkify->new;
^ permalink raw reply related [relevance 71%]
* [PATCH 2/6] doc: lei: improve description of *-search commands
@ 2022-11-03 0:48 62% ` Eric Wong
2022-11-03 0:48 71% ` [PATCH 3/6] doc: txt2pre: linkify "lei COMMAND" form Eric Wong
` (2 subsequent siblings)
3 siblings, 0 replies; 200+ results
From: Eric Wong @ 2022-11-03 0:48 UTC (permalink / raw)
To: meta
The `OUTPUT' use may not be immediately apparent, clarify
that it's from `lei q'.
---
Documentation/lei-edit-search.pod | 6 ++++--
Documentation/lei-forget-search.pod | 4 +++-
Documentation/lei-ls-search.pod | 5 +++--
3 files changed, 10 insertions(+), 5 deletions(-)
diff --git a/Documentation/lei-edit-search.pod b/Documentation/lei-edit-search.pod
index 21cb11aa..7f447ca2 100644
--- a/Documentation/lei-edit-search.pod
+++ b/Documentation/lei-edit-search.pod
@@ -8,7 +8,9 @@ lei edit-search [OPTIONS] OUTPUT
=head1 DESCRIPTION
-Invoke C<git config --edit> to edit the saved search at C<OUTPUT>.
+Invoke C<git config --edit> to edit the saved search at C<OUTPUT>,
+where C<OUTPUT> was supplied for argument of C<lei q -o OUTPUT ...>
+A listing of outputs is available via C<lei ls-search>.
=head1 CONTACT
@@ -19,7 +21,7 @@ and L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta
=head1 COPYRIGHT
-Copyright 2021 all contributors L<mailto:meta@public-inbox.org>
+Copyright all contributors L<mailto:meta@public-inbox.org>
License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>
diff --git a/Documentation/lei-forget-search.pod b/Documentation/lei-forget-search.pod
index adbe7638..5ff526f1 100644
--- a/Documentation/lei-forget-search.pod
+++ b/Documentation/lei-forget-search.pod
@@ -8,7 +8,9 @@ lei forget-search [OPTIONS] OUTPUT
=head1 DESCRIPTION
-Forget a saved search at C<OUTPUT>.
+Forget a saved search at C<OUTPUT>,
+where C<OUTPUT> was supplied for argument of C<lei q -o OUTPUT ...>
+A listing of outputs is available via C<lei ls-search>.
=head1 OPTIONS
diff --git a/Documentation/lei-ls-search.pod b/Documentation/lei-ls-search.pod
index a56611bf..0fe4b759 100644
--- a/Documentation/lei-ls-search.pod
+++ b/Documentation/lei-ls-search.pod
@@ -8,7 +8,8 @@ lei ls-search [OPTIONS] [PREFIX]
=head1 DESCRIPTION
-List saved search queries. If C<PREFIX> is given, restrict the output
+List saved search queries (generated from C<lei q -o OUTPUT>).
+If C<PREFIX> is given, restrict the output
to entries that start with the specified value.
=head1 OPTIONS
@@ -55,7 +56,7 @@ and L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta
=head1 COPYRIGHT
-Copyright 2021 all contributors L<mailto:meta@public-inbox.org>
+Copyright all contributors L<mailto:meta@public-inbox.org>
License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>
^ permalink raw reply related [relevance 62%]
* [PATCH 5/6] doc: lei-import: link to lei-store-format(5)
2022-11-03 0:48 62% ` [PATCH 2/6] doc: lei: improve description of *-search commands Eric Wong
2022-11-03 0:48 71% ` [PATCH 3/6] doc: txt2pre: linkify "lei COMMAND" form Eric Wong
@ 2022-11-03 0:48 90% ` Eric Wong
2022-11-03 2:03 90% ` Eric Wong
2022-11-03 0:48 90% ` [PATCH 6/6] txt2pre: linkify lei/store => lei-store-format.html Eric Wong
3 siblings, 1 reply; 200+ results
From: Eric Wong @ 2022-11-03 0:48 UTC (permalink / raw)
To: meta
Users should know where `lei import' writes to.
---
Documentation/lei-import.pod | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/Documentation/lei-import.pod b/Documentation/lei-import.pod
index 4ac7dccd..25ef75c3 100644
--- a/Documentation/lei-import.pod
+++ b/Documentation/lei-import.pod
@@ -10,7 +10,8 @@ lei import [OPTIONS] (--stdin|-)
=head1 DESCRIPTION
-Import messages into the local storage of L<lei(1)>. C<LOCATION> is a
+Import messages into the local storage of L<lei(1)>
+(aka L<lei/store|lei-store-format(5)>). C<LOCATION> is a
source of messages: a directory (Maildir), a file, or a URL
(C<imap://>, C<imaps://>, C<nntp://>, or C<nntps://>). URLs requiring
authentication use L<git-credential(1)> to
@@ -102,4 +103,4 @@ License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>
=head1 SEE ALSO
-L<lei-index(1)>
+L<lei-index(1)>, L<lei-store-format(5)>
^ permalink raw reply related [relevance 90%]
* [PATCH 6/6] txt2pre: linkify lei/store => lei-store-format.html
` (2 preceding siblings ...)
2022-11-03 0:48 90% ` [PATCH 5/6] doc: lei-import: link to lei-store-format(5) Eric Wong
@ 2022-11-03 0:48 90% ` Eric Wong
3 siblings, 0 replies; 200+ results
From: Eric Wong @ 2022-11-03 0:48 UTC (permalink / raw)
To: meta
Linking to the manpage probably helps clarify what `lei/store'
refers to without too much clutter in the raw POD source.
---
Documentation/txt2pre | 2 ++
1 file changed, 2 insertions(+)
diff --git a/Documentation/txt2pre b/Documentation/txt2pre
index b9d74fb7..62175f34 100755
--- a/Documentation/txt2pre
+++ b/Documentation/txt2pre
@@ -80,6 +80,8 @@ for (qw[lei(1)
/\Alei-(.+?)\(1\)\z/ and $xurls{"lei $1"} = "$n.html";
}
+$xurls{'lei/store'} = 'lei-store-format.html';
+
for (qw[make(1) flock(2) setrlimit(2) vfork(2) tmpfs(5) inotify(7) unix(7)
syslog(3)]) {
my ($n, $s) = (/([\w\-]+)\((\d)\)/);
^ permalink raw reply related [relevance 90%]
* Re: [PATCH 5/6] doc: lei-import: link to lei-store-format(5)
2022-11-03 0:48 90% ` [PATCH 5/6] doc: lei-import: link to lei-store-format(5) Eric Wong
@ 2022-11-03 2:03 90% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2022-11-03 2:03 UTC (permalink / raw)
To: meta
Eric Wong <e@80x24.org> wrote:
> +(aka L<lei/store|lei-store-format(5)>). C<LOCATION> is a
Can't have unescaped `/' like that, will squash this in:
diff --git a/Documentation/lei-import.pod b/Documentation/lei-import.pod
index 25ef75c3..69ec6497 100644
--- a/Documentation/lei-import.pod
+++ b/Documentation/lei-import.pod
@@ -11,7 +11,7 @@ lei import [OPTIONS] (--stdin|-)
=head1 DESCRIPTION
Import messages into the local storage of L<lei(1)>
-(aka L<lei/store|lei-store-format(5)>). C<LOCATION> is a
+(aka L<leiE<sol>store|lei-store-format(5)>). C<LOCATION> is a
source of messages: a directory (Maildir), a file, or a URL
(C<imap://>, C<imaps://>, C<nntp://>, or C<nntps://>). URLs requiring
authentication use L<git-credential(1)> to
^ permalink raw reply related [relevance 90%]
* [PATCH] lei q|up: limit default write --jobs for IMAP(S)
2022-09-10 20:19 71% ` Eric Wong
@ 2022-11-14 8:07 64% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2022-11-14 8:07 UTC (permalink / raw)
To: meta; +Cc: Ricardo Ribalda
Eric Wong <e@80x24.org> wrote:
> Thanks for confirming things work as intended. I think the
> default should be clamped, though... 15 seems a bit high for
> smaller IMAP servers *shrug*
--------8<-------
Subject: [PATCH] lei q|up: limit default write --jobs for IMAP(S)
IMAP(S) servers often limit per-user connections, so avoid
bumping into limits to improve the out-of-the-box experience.
4 seems like a conservative default, since we already chose
that number for remote HTTP(S) endpoints.
Link: https://public-inbox.org/meta/20220910201958.GA12212@dcvr/
---
/me having git-repack OOM due to excessive default pack.threads
reminded me of this issue :x
Documentation/lei-q.pod | 4 ++--
lib/PublicInbox/LeiQuery.pm | 7 +++++--
2 files changed, 7 insertions(+), 4 deletions(-)
diff --git a/Documentation/lei-q.pod b/Documentation/lei-q.pod
index 8134223e..d52c5b04 100644
--- a/Documentation/lei-q.pod
+++ b/Documentation/lei-q.pod
@@ -135,8 +135,8 @@ Set the number of query and write worker processes for parallelism.
C<QUERY_WORKERS> defaults to the number of CPUs available, but 4 per
remote (HTTP/HTTPS) host.
-C<WRITE_WORKERS> defaults to the number of CPUs available for Maildir,
-IMAP/IMAPS, and mbox* destinations.
+C<WRITE_WORKERS> defaults to 75% of the number of CPUs available for
+Maildir and mbox* destinations, but 4 per IMAP/IMAPS host.
Omitting C<QUERY_WORKERS> but leaving the comma (C<,>) allows
one to only set C<WRITE_WORKERS>
diff --git a/lib/PublicInbox/LeiQuery.pm b/lib/PublicInbox/LeiQuery.pm
index df9c32b3..0f839236 100644
--- a/lib/PublicInbox/LeiQuery.pm
+++ b/lib/PublicInbox/LeiQuery.pm
@@ -39,8 +39,11 @@ sub _start_query { # used by "lei q" and "lei up"
$lms->lms_write_prepare->lms_pause; # just create
}
}
- $l2m and $l2m->{-wq_nr_workers} //= $mj //
- int($nproc * 0.75 + 0.5); # keep some CPU for git
+ $l2m and $l2m->{-wq_nr_workers} //= $mj // do {
+ # keep some CPU for git, and don't overload IMAP destinations
+ my $n = int($nproc * 0.75 + 0.5);
+ $self->{net} && $n > 4 ? 4 : $n;
+ };
# descending docid order is cheapest, MUA controls sorting order
$self->{mset_opt}->{relevance} //= -2 if $l2m || $opt->{threads};
^ permalink raw reply related [relevance 64%]
* [PATCH 0/2] lei - expanding relative paths for `lei up'
@ 2022-12-01 11:21 90% Eric Wong
2022-12-01 11:21 67% ` [PATCH 1/2] lei: stricter external checks for valid $GIT_DIR/objects Eric Wong
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2022-12-01 11:21 UTC (permalink / raw)
To: meta
I ran `lei q --only ./ -o $MAILDIR $QUERY' at some point from
inside an inboxdir.
Then I got confused when `lei up --all' running from $HOME
started complaining about $HOME/objects being missing from ALE.
It turned out $HOME/public-inbox (one of my worktrees) was
causing $HOME to false-positive as a v1 public-inbox for lei :x.
So this is a two-pronged fix to prevent some weird stuff from
happening.
Eric Wong (2):
lei: stricter external checks for valid $GIT_DIR/objects
lei_saved_search: expand only/include/exclude to absolute paths
lib/PublicInbox/LeiQuery.pm | 23 ++++++++++++++++++++---
lib/PublicInbox/LeiXSearch.pm | 14 ++++++++++----
t/lei-q-save.t | 13 +++++++++----
t/lei.t | 3 ++-
4 files changed, 41 insertions(+), 12 deletions(-)
^ permalink raw reply [relevance 90%]
* [PATCH 1/2] lei: stricter external checks for valid $GIT_DIR/objects
2022-12-01 11:21 90% [PATCH 0/2] lei - expanding relative paths for `lei up' Eric Wong
@ 2022-12-01 11:21 67% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2022-12-01 11:21 UTC (permalink / raw)
To: meta
I ended up with my $HOME in
~/.cache/lei/all_locals_ever.git/objects/info/alterntes
and am trying to avoid that in the future.
---
lib/PublicInbox/LeiXSearch.pm | 5 +++--
t/lei.t | 3 ++-
2 files changed, 5 insertions(+), 3 deletions(-)
diff --git a/lib/PublicInbox/LeiXSearch.pm b/lib/PublicInbox/LeiXSearch.pm
index 90cb83b9..8e195c4c 100644
--- a/lib/PublicInbox/LeiXSearch.pm
+++ b/lib/PublicInbox/LeiXSearch.pm
@@ -617,11 +617,12 @@ sub prepare_external {
} elsif ($loc =~ m!\Ahttps?://!) {
require URI;
return add_uri($self, URI->new($loc));
- } elsif (-f "$loc/ei.lock") {
+ } elsif (-f "$loc/ei.lock" && -d "$loc/ALL.git/objects") {
require PublicInbox::ExtSearch;
die "`\\n' not allowed in `$loc'\n" if index($loc, "\n") >= 0;
$loc = PublicInbox::ExtSearch->new($loc);
- } elsif (-f "$loc/inbox.lock" || -d "$loc/public-inbox") {
+ } elsif ((-f "$loc/inbox.lock" && -d "$loc/all.git/objects") ||
+ (-d "$loc/public-inbox" && -d "$loc/objects")) {
die "`\\n' not allowed in `$loc'\n" if index($loc, "\n") >= 0;
require PublicInbox::Inbox; # v2, v1
$loc = bless { inboxdir => $loc }, 'PublicInbox::Inbox';
diff --git a/t/lei.t b/t/lei.t
index b10c9b59..a80143ef 100644
--- a/t/lei.t
+++ b/t/lei.t
@@ -148,7 +148,8 @@ my $test_fail = sub {
for my $lk (qw(ei inbox)) {
my $d = "$home/newline\n$lk";
- mkdir $d;
+ my $all = $lk eq 'ei' ? 'ALL' : 'all';
+ File::Path::mkpath("$d/$all.git/objects");
open my $fh, '>', "$d/$lk.lock" or BAIL_OUT "open $d/$lk.lock";
for my $fl (qw(-I --only)) {
ok(!lei('q', $fl, $d, 'whatever'),
^ permalink raw reply related [relevance 67%]
* FUSE3 vs read-write IMAP for lei
@ 2022-12-09 1:41 65% ` Eric Wong
2023-02-20 19:27 71% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2022-12-09 1:41 UTC (permalink / raw)
To: meta
Eric Wong <e@80x24.org> wrote:
> I don't think lei+FUSE will be as portable or useful as a
> local IMAP server (and maybe JMAP, eventually); but r/w IMAP
> support would be nice..
One thing about lei which really bothers me is that it needs to
write out mail already in local git repos to the FS to be read
by regular MUAs.
This is wasteful, of course; so there's 2 ways I can imagine
exposing mail to ordinary MUAs without excessive disk
traffic/wear:
== read-write IMAP for lei
* connect via localhost (127.0.0.1, [::1]) (don't think local sockets
are supported by any MUAs)
* May be extended to JMAP; but it's hard to be motivated on JMAP
since my favorite MUA doesn't do JMAP, yet.
+ portable (can be done in pure Perl + DBD::SQLite + Xapian)
+ we already have a read-only IMAP server which can be extended
for read/write
- still needs login w/ username+password due to multi-user systems
- may still susceptible to abuse and from multi-user systems
- IMAP gets pretty complex, and MUAs sometimes don't do it well
== FUSE3 - Maildir-oriented FS
+ will require C compiler since old FUSE XS modules don't do
FUSE3 (for readdirplus); and going w/o readdirplus is
unimaginable for Maildir.
+ I already have existing (unreleased) AGPL-3 work based on
FUSE3 + URCU + Perl5 with just-ahead-of-time (JAOT) compilation
* I've seen the light w/ URCU, and can't go back to C without it :P
- likely Linux-only (not sure how good FUSE support will be if
depending on FUSE3 features)
- kernel caches still incur nasty memory overhead w/ Maildir
- readdir(3) userspace API still sucks
Of course, doing both is an option, too, given enough time...
^ permalink raw reply [relevance 65%]
* [1/2 PATCH] hoist MailDiff and ContentDigestDbg out of lei
@ 2023-01-11 11:00 42% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2023-01-11 11:00 UTC (permalink / raw)
To: meta
These will be reused in the web UI, too.
---
<20230111105539.302803-1-e@80x24.org> was actually [2/2] of
this series. My mind drifted and I thought it was just one
patch :x
MANIFEST | 3 ++
lib/PublicInbox/ContentDigestDbg.pm | 17 +++++++
lib/PublicInbox/LeiMailDiff.pm | 71 +++--------------------------
lib/PublicInbox/MailDiff.pm | 50 ++++++++++++++++++++
t/lei-mail-diff.t | 14 ++++++
5 files changed, 91 insertions(+), 64 deletions(-)
create mode 100644 lib/PublicInbox/ContentDigestDbg.pm
create mode 100644 lib/PublicInbox/MailDiff.pm
create mode 100644 t/lei-mail-diff.t
diff --git a/MANIFEST b/MANIFEST
index 565317ce..3626e4d2 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -163,6 +163,7 @@ lib/PublicInbox/CmdIPC4.pm
lib/PublicInbox/CompressNoop.pm
lib/PublicInbox/Config.pm
lib/PublicInbox/ConfigIter.pm
+lib/PublicInbox/ContentDigestDbg.pm
lib/PublicInbox/ContentHash.pm
lib/PublicInbox/DS.pm
lib/PublicInbox/DSKQXS.pm
@@ -280,6 +281,7 @@ lib/PublicInbox/Lock.pm
lib/PublicInbox/MDA.pm
lib/PublicInbox/MID.pm
lib/PublicInbox/MIME.pm
+lib/PublicInbox/MailDiff.pm
lib/PublicInbox/ManifestJsGz.pm
lib/PublicInbox/Mbox.pm
lib/PublicInbox/MboxGz.pm
@@ -478,6 +480,7 @@ t/lei-import.t
t/lei-index.t
t/lei-inspect.t
t/lei-lcat.t
+t/lei-mail-diff.t
t/lei-mirror.psgi
t/lei-mirror.t
t/lei-p2q.t
diff --git a/lib/PublicInbox/ContentDigestDbg.pm b/lib/PublicInbox/ContentDigestDbg.pm
new file mode 100644
index 00000000..425e8589
--- /dev/null
+++ b/lib/PublicInbox/ContentDigestDbg.pm
@@ -0,0 +1,17 @@
+# Copyright (C) all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+package PublicInbox::ContentDigestDbg; # cf. PublicInbox::ContentDigest
+use v5.12;
+use Data::Dumper;
+use Digest::SHA;
+
+sub new { bless { dig => Digest::SHA->new(256), fh => $_[1] }, __PACKAGE__ }
+
+sub add {
+ $_[0]->{dig}->add($_[1]);
+ print { $_[0]->{fh} } Dumper([split(/^/sm, $_[1])]) or die "print $!";
+}
+
+sub hexdigest { $_[0]->{dig}->hexdigest; }
+
+1;
diff --git a/lib/PublicInbox/LeiMailDiff.pm b/lib/PublicInbox/LeiMailDiff.pm
index 2b4cfd9e..c813144f 100644
--- a/lib/PublicInbox/LeiMailDiff.pm
+++ b/lib/PublicInbox/LeiMailDiff.pm
@@ -4,59 +4,16 @@
# The "lei mail-diff" sub-command, diffs input contents against
# the first message of input
package PublicInbox::LeiMailDiff;
-use strict;
-use v5.10.1;
-use parent qw(PublicInbox::IPC PublicInbox::LeiInput);
-use File::Temp 0.19 (); # 0.19 for ->newdir
+use v5.12;
+use parent qw(PublicInbox::IPC PublicInbox::LeiInput PublicInbox::MailDiff);
use PublicInbox::Spawn qw(spawn which);
-use PublicInbox::MsgIter qw(msg_part_text);
-use File::Path qw(remove_tree);
-use PublicInbox::ContentHash qw(content_digest);
+use File::Path ();
require PublicInbox::LeiRediff;
-use Data::Dumper ();
-
-sub write_part { # Eml->each_part callback
- my ($ary, $self) = @_;
- my ($part, $depth, $idx) = @$ary;
- if ($idx ne '1' || $self->{lei}->{opt}->{'raw-header'}) {
- open my $fh, '>', "$self->{curdir}/$idx.hdr" or die "open: $!";
- print $fh ${$part->{hdr}} or die "print $!";
- close $fh or die "close $!";
- }
- my $ct = $part->content_type || 'text/plain';
- my ($s, $err) = msg_part_text($part, $ct);
- my $sfx = defined($s) ? 'txt' : 'bin';
- open my $fh, '>', "$self->{curdir}/$idx.$sfx" or die "open: $!";
- print $fh ($s // $part->body) or die "print $!";
- close $fh or die "close $!";
-}
-
-sub dump_eml ($$$) {
- my ($self, $dir, $eml) = @_;
- local $self->{curdir} = $dir;
- mkdir $dir or die "mkdir($dir): $!";
- $eml->each_part(\&write_part, $self);
-
- open my $fh, '>', "$dir/content_digest" or die "open: $!";
- my $dig = PublicInbox::ContentDigestDbg->new($fh);
- local $Data::Dumper::Useqq = 1;
- local $Data::Dumper::Terse = 1;
- content_digest($eml, $dig);
- print $fh "\n", $dig->hexdigest, "\n" or die "print $!";
- close $fh or die "close: $!";
-}
-
-sub prep_a ($$) {
- my ($self, $eml) = @_;
- $self->{tmp} = File::Temp->newdir('lei-mail-diff-XXXX', TMPDIR => 1);
- dump_eml($self, "$self->{tmp}/a", $eml);
-}
sub diff_a ($$) {
my ($self, $eml) = @_;
- ++$self->{nr};
- my $dir = "$self->{tmp}/N$self->{nr}";
- dump_eml($self, $dir, $eml);
+ my $dir = "$self->{tmp}/N".(++$self->{nr});
+ $self->dump_eml($dir, $eml);
my $cmd = [ qw(git diff --no-index) ];
my $lei = $self->{lei};
PublicInbox::LeiRediff::_lei_diff_prepare($lei, $cmd);
@@ -71,7 +28,7 @@ sub diff_a ($$) {
sub input_eml_cb { # used by PublicInbox::LeiInput::input_fh
my ($self, $eml) = @_;
- $self->{tmp} ? diff_a($self, $eml) : prep_a($self, $eml);
+ $self->{tmp} ? diff_a($self, $eml) : $self->prep_a($eml);
}
sub lei_mail_diff {
@@ -82,24 +39,10 @@ sub lei_mail_diff {
$lei->{opt}->{color} //= $isatty;
$lei->start_pager if $isatty;
$lei->{-err_type} = 'non-fatal';
+ $self->{-raw_hdr} = $lei->{opt}->{'raw-header'};
$lei->wq1_start($self);
}
no warnings 'once';
*net_merge_all_done = \&PublicInbox::LeiInput::input_only_net_merge_all_done;
-
-package PublicInbox::ContentDigestDbg; # cf. PublicInbox::ContentDigest
-use strict;
-use v5.10.1;
-use Data::Dumper;
-
-sub new { bless { dig => Digest::SHA->new(256), fh => $_[1] }, __PACKAGE__ }
-
-sub add {
- $_[0]->{dig}->add($_[1]);
- print { $_[0]->{fh} } Dumper([split(/^/sm, $_[1])]) or die "print $!";
-}
-
-sub hexdigest { $_[0]->{dig}->hexdigest; }
-
1;
diff --git a/lib/PublicInbox/MailDiff.pm b/lib/PublicInbox/MailDiff.pm
new file mode 100644
index 00000000..06eb3a0d
--- /dev/null
+++ b/lib/PublicInbox/MailDiff.pm
@@ -0,0 +1,50 @@
+# Copyright (C) all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+package PublicInbox::MailDiff;
+use v5.12;
+use File::Temp 0.19 (); # 0.19 for ->newdir
+use PublicInbox::ContentHash qw(content_digest);
+use PublicInbox::ContentDigestDbg;
+use Data::Dumper ();
+use PublicInbox::MsgIter qw(msg_part_text);
+
+sub write_part { # Eml->each_part callback
+ my ($ary, $self) = @_;
+ my ($part, $depth, $idx) = @$ary;
+ if ($idx ne '1' || $self->{-raw_hdr}) {
+ open my $fh, '>', "$self->{curdir}/$idx.hdr" or die "open: $!";
+ print $fh ${$part->{hdr}} or die "print $!";
+ close $fh or die "close $!";
+ }
+ my $ct = $part->content_type || 'text/plain';
+ my ($s, $err) = msg_part_text($part, $ct);
+ my $sfx = defined($s) ? 'txt' : 'bin';
+ open my $fh, '>', "$self->{curdir}/$idx.$sfx" or die "open: $!";
+ print $fh ($s // $part->body) or die "print $!";
+ close $fh or die "close $!";
+}
+
+# public
+sub dump_eml ($$$) {
+ my ($self, $dir, $eml) = @_;
+ local $self->{curdir} = $dir;
+ mkdir $dir or die "mkdir($dir): $!";
+ $eml->each_part(\&write_part, $self);
+
+ open my $fh, '>', "$dir/content_digest" or die "open: $!";
+ my $dig = PublicInbox::ContentDigestDbg->new($fh);
+ local $Data::Dumper::Useqq = 1;
+ local $Data::Dumper::Terse = 1;
+ content_digest($eml, $dig);
+ print $fh "\n", $dig->hexdigest, "\n" or die "print $!";
+ close $fh or die "close: $!";
+}
+
+# public
+sub prep_a ($$) {
+ my ($self, $eml) = @_;
+ $self->{tmp} = File::Temp->newdir('mail-diff-XXXX', TMPDIR => 1);
+ dump_eml($self, "$self->{tmp}/a", $eml);
+}
+
+1;
diff --git a/t/lei-mail-diff.t b/t/lei-mail-diff.t
new file mode 100644
index 00000000..9398596a
--- /dev/null
+++ b/t/lei-mail-diff.t
@@ -0,0 +1,14 @@
+#!perl -w
+# Copyright (C) all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+use v5.12; use PublicInbox::TestCommon;
+
+test_lei(sub {
+ ok(!lei('mail-diff', 't/data/0001.patch', 't/data/binary.patch'),
+ 'different messages are different');
+ like($lei_out, qr/^\+/m, 'diff shown');
+ lei_ok('mail-diff', 't/data/0001.patch', 't/data/0001.patch');
+ is($lei_out, '', 'no output if identical');
+});
+
+done_testing;
^ permalink raw reply related [relevance 42%]
* [PATCH 11/12] ipc+lei: switch to awaitpid
@ 2023-01-17 7:19 39% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2023-01-17 7:19 UTC (permalink / raw)
To: meta
This avoids awkwardly stuffing an arrayref into callbacks
which expect multiple arguments. IPC->awaitpid_init now
allows pre-registering callbacks before spawning workers.
---
lib/PublicInbox/IPC.pm | 30 ++++++++++++++----------------
lib/PublicInbox/LEI.pm | 8 +++-----
lib/PublicInbox/LeiConvert.pm | 2 +-
lib/PublicInbox/LeiInput.pm | 2 +-
lib/PublicInbox/LeiMirror.pm | 7 +++----
lib/PublicInbox/LeiStore.pm | 7 +++----
lib/PublicInbox/LeiToMail.pm | 7 +++----
lib/PublicInbox/LeiUp.pm | 5 ++---
lib/PublicInbox/LeiXSearch.pm | 9 ++++-----
script/public-inbox-clone | 2 +-
10 files changed, 35 insertions(+), 44 deletions(-)
diff --git a/lib/PublicInbox/IPC.pm b/lib/PublicInbox/IPC.pm
index 34e40118..edc5ba64 100644
--- a/lib/PublicInbox/IPC.pm
+++ b/lib/PublicInbox/IPC.pm
@@ -12,7 +12,7 @@ use strict;
use v5.10.1;
use parent qw(Exporter);
use Carp qw(croak);
-use PublicInbox::DS qw(dwaitpid);
+use PublicInbox::DS qw(awaitpid);
use PublicInbox::Spawn;
use PublicInbox::OnDestroy;
use PublicInbox::WQWorker;
@@ -133,26 +133,26 @@ sub ipc_worker_spawn {
$self->{-ipc_req} = $w_req;
$self->{-ipc_res} = $r_res;
$self->{-ipc_ppid} = $$;
+ awaitpid($pid, \&ipc_worker_reap, $self);
$self->{-ipc_pid} = $pid;
}
-sub ipc_worker_reap { # dwaitpid callback
- my ($args, $pid) = @_;
- my ($self, @uargs) = @$args;
+sub ipc_worker_reap { # awaitpid callback
+ my ($pid, $self) = @_;
delete $self->{-wq_workers}->{$pid};
- return $self->{-reap_do}->($args, $pid) if $self->{-reap_do};
+ if (my $cb_args = $self->{-reap_do}) {
+ return $cb_args->[0]->($pid, $self, @$cb_args[1..$#$cb_args]);
+ }
return if !$?;
my $s = $? & 127;
# TERM(15) is our default exit signal, PIPE(13) is likely w/ pager
warn "$self->{-wq_ident} PID:$pid died \$?=$?\n" if $s != 15 && $s != 13
}
-sub wq_wait_async {
- my ($self, $cb, @uargs) = @_;
- local $PublicInbox::DS::in_loop = 1;
- $self->{-reap_do} = $cb;
- my @pids = keys %{$self->{-wq_workers}};
- dwaitpid($_, \&ipc_worker_reap, [ $self, @uargs ]) for @pids;
+# register wait workers
+sub awaitpid_init {
+ my ($self, @cb_args) = @_;
+ $self->{-reap_do} = \@cb_args;
}
# for base class, override in sub classes
@@ -178,9 +178,7 @@ sub ipc_worker_stop {
}
die 'no PID with IPC pipes' unless $pid;
$w_req = $r_res = undef;
-
- return if $$ != $ppid;
- dwaitpid($pid, \&ipc_worker_reap, [$self]);
+ awaitpid($pid) if $$ == $ppid; # for non-event loop
}
# use this if we have multiple readers reading curl or "pigz -dc"
@@ -397,6 +395,7 @@ sub _wq_worker_start ($$$$) {
undef $end; # trigger exit
} else {
$self->{-wq_workers}->{$pid} = $bcast1;
+ awaitpid($pid, \&ipc_worker_reap, $self);
}
}
@@ -428,8 +427,7 @@ sub wq_close {
}
delete @$self{qw(-wq_s1 -wq_s2)} or return;
return if $self->{-reap_do};
- my @pids = keys %{$self->{-wq_workers}};
- dwaitpid($_, \&ipc_worker_reap, [ $self ]) for @pids;
+ awaitpid($_) for keys %{$self->{-wq_workers}};
}
sub wq_kill {
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index b78d70de..6ad42111 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -18,7 +18,6 @@ use IO::Handle ();
use Fcntl qw(SEEK_SET);
use PublicInbox::Config;
use PublicInbox::Syscall qw(EPOLLIN);
-use PublicInbox::DS qw(dwaitpid);
use PublicInbox::Spawn qw(spawn popen_rd);
use PublicInbox::Lock;
use PublicInbox::Eml;
@@ -644,12 +643,12 @@ sub workers_start {
my $end = $lei->pkt_op_pair;
my $ident = $wq->{-wq_ident} // "lei-$lei->{cmd} worker";
$flds->{lei} = $lei;
+ $wq->awaitpid_init($wq->can('_wq_done_wait') // \&wq_done_wait, $lei);
$wq->wq_workers_start($ident, $jobs, $lei->oldset, $flds);
delete $lei->{pkt_op_p};
my $op_c = delete $lei->{pkt_op_c};
@$end = ();
$lei->event_step_init;
- $wq->wq_wait_async($wq->can('_wq_done_wait') // \&wq_done_wait, $lei);
($op_c, $ops);
}
@@ -1391,9 +1390,8 @@ sub DESTROY {
# preserve $? for ->fail or ->x_it code
}
-sub wq_done_wait { # dwaitpid callback
- my ($arg, $pid) = @_;
- my ($wq, $lei) = @$arg;
+sub wq_done_wait { # awaitpid cb (via wq_eof / IPC->awaitpid_init)
+ my ($pid, $wq, $lei) = @_;
local $current_lei = $lei;
my $err_type = $lei->{-err_type};
$? and $lei->child_error($?,
diff --git a/lib/PublicInbox/LeiConvert.pm b/lib/PublicInbox/LeiConvert.pm
index 59af40de..1acd4558 100644
--- a/lib/PublicInbox/LeiConvert.pm
+++ b/lib/PublicInbox/LeiConvert.pm
@@ -30,7 +30,7 @@ sub input_maildir_cb {
sub process_inputs { # via wq_do
my ($self) = @_;
- local $PublicInbox::DS::in_loop = 0; # force synchronous dwaitpid
+ local $PublicInbox::DS::in_loop = 0; # force synchronous awaitpid
$self->SUPER::process_inputs;
my $lei = $self->{lei};
delete $lei->{1};
diff --git a/lib/PublicInbox/LeiInput.pm b/lib/PublicInbox/LeiInput.pm
index a1dcc907..c258f824 100644
--- a/lib/PublicInbox/LeiInput.pm
+++ b/lib/PublicInbox/LeiInput.pm
@@ -177,7 +177,7 @@ sub input_path_url {
$mbl->{fh} =
PublicInbox::MboxReader::zsfxcat($in, $zsfx, $lei);
}
- local $PublicInbox::DS::in_loop = 0 if $zsfx; # dwaitpid
+ local $PublicInbox::DS::in_loop = 0 if $zsfx; # awaitpid
$self->input_fh($ifmt, $mbl->{fh}, $input, @args);
} elsif (-d _ && (-d "$input/cur" || -d "$input/new")) {
return $lei->fail(<<EOM) if $ifmt && $ifmt ne 'maildir';
diff --git a/lib/PublicInbox/LeiMirror.pm b/lib/PublicInbox/LeiMirror.pm
index 87abf88c..abf66315 100644
--- a/lib/PublicInbox/LeiMirror.pm
+++ b/lib/PublicInbox/LeiMirror.pm
@@ -31,9 +31,8 @@ sub keep_going ($) {
$_[0]->{lei}->{opt}->{'keep-going'});
}
-sub _wq_done_wait { # dwaitpid callback (via wq_eof)
- my ($arg, $pid) = @_;
- my ($mrr, $lei) = @$arg;
+sub _wq_done_wait { # awaitpid cb (via wq_eof / IPC->awaitpid_init)
+ my ($pid, $mrr, $lei) = @_;
if ($?) {
$lei->child_error($?);
} elsif (!$lei->{child_error}) {
@@ -236,7 +235,7 @@ sub index_cloned_inbox {
my ($k) = ($sw =~ /\A([\w-]+)/);
$opt->{$k} = $lei->{opt}->{$k};
}
- # force synchronous dwaitpid for v2:
+ # force synchronous awaitpid for v2:
local $PublicInbox::DS::in_loop = 0;
my $cfg = PublicInbox::Config->new(undef, $lei->{2});
my $env = PublicInbox::Admin::index_prepare($opt, $cfg);
diff --git a/lib/PublicInbox/LeiStore.pm b/lib/PublicInbox/LeiStore.pm
index 57f0e013..0ecf1388 100644
--- a/lib/PublicInbox/LeiStore.pm
+++ b/lib/PublicInbox/LeiStore.pm
@@ -604,9 +604,8 @@ sub recv_and_run {
$self->SUPER::recv_and_run(@args);
}
-sub _sto_atexit { # dwaitpid callback
- my ($args, $pid) = @_;
- my $self = $args->[0];
+sub _sto_atexit { # awaitpid cb (via awaitpid_init)
+ my ($pid, $sto) = @_;
warn "lei/store PID:$pid died \$?=$?\n" if $?;
}
@@ -621,12 +620,12 @@ sub write_prepare {
# Mail we import into lei are private, so headers filtered out
# by -mda for public mail are not appropriate
local @PublicInbox::MDA::BAD_HEADERS = ();
+ $self->awaitpid_init(\&_sto_atexit); # outlives $lei
$self->wq_workers_start("lei/store $dir", 1, $lei->oldset, {
lei => $lei,
-err_wr => $w,
to_close => [ $r ],
});
- $self->wq_wait_async(\&_sto_atexit); # outlives $lei
require PublicInbox::LeiStoreErr;
PublicInbox::LeiStoreErr->new($r, $lei);
}
diff --git a/lib/PublicInbox/LeiToMail.pm b/lib/PublicInbox/LeiToMail.pm
index 1528165a..6a4554e7 100644
--- a/lib/PublicInbox/LeiToMail.pm
+++ b/lib/PublicInbox/LeiToMail.pm
@@ -652,9 +652,8 @@ sub _do_augment_mbox {
$dedupe->pause_dedupe if $dedupe;
}
-sub v2w_done_wait { # dwaitpid callback
- my ($arg, $pid) = @_;
- my ($v2w, $lei) = @$arg;
+sub v2w_done_wait { # awaitpid cb (via awaitpid_init)
+ my ($pid, $v2w, $lei) = @_;
$lei->child_error($?, "error for $v2w->{ibx}->{inboxdir}") if $?;
}
@@ -680,8 +679,8 @@ sub _pre_augment_v2 {
PublicInbox::InboxWritable->new($ibx, @creat);
$ibx->init_inbox if @creat;
my $v2w = $ibx->importer;
+ $v2w->awaitpid_init(\&v2w_done_wait, $lei);
$v2w->wq_workers_start("lei/v2w $dir", 1, $lei->oldset, {lei => $lei});
- $v2w->wq_wait_async(\&v2w_done_wait, $lei);
$lei->{v2w} = $v2w;
return if !$lei->{opt}->{shared};
my $d = "$lei->{ale}->{git}->{git_dir}/objects";
diff --git a/lib/PublicInbox/LeiUp.pm b/lib/PublicInbox/LeiUp.pm
index 49917339..3e92242e 100644
--- a/lib/PublicInbox/LeiUp.pm
+++ b/lib/PublicInbox/LeiUp.pm
@@ -165,9 +165,8 @@ sub _complete_up { # lei__complete hook
map { $match_cb->($_) } PublicInbox::LeiSavedSearch::list($lei);
}
-sub _wq_done_wait { # dwaitpid callback
- my ($arg, $pid) = @_;
- my ($wq, $lei) = @$arg;
+sub _wq_done_wait { # awaitpid cb (via awaitpid_init)
+ my ($pid, $wq, $lei) = @_;
$lei->child_error($?, 'auth failure') if $?
}
diff --git a/lib/PublicInbox/LeiXSearch.pm b/lib/PublicInbox/LeiXSearch.pm
index 730df1f7..f9aa870e 100644
--- a/lib/PublicInbox/LeiXSearch.pm
+++ b/lib/PublicInbox/LeiXSearch.pm
@@ -400,9 +400,8 @@ sub query_remote_mboxrd {
sub git { $_[0]->{git} // die 'BUG: git uninitialized' }
-sub xsearch_done_wait { # dwaitpid callback
- my ($arg, $pid) = @_;
- my ($wq, $lei) = @$arg;
+sub xsearch_done_wait { # awaitpid cb (via awaitpid_init)
+ my ($pid, $wq, $lei) = @_;
return if !$?;
my $s = $? & 127;
return $lei->child_error($?) if $s == 13 || $s == 15;
@@ -573,16 +572,16 @@ sub do_query {
fcntl($b_r, $F_SETPIPE_SZ, 4096) if $F_SETPIPE_SZ;
$l2m->{au_peers} = [ $a_r, $a_w, $b_r, $b_w ];
}
+ $l2m->awaitpid_init(\&xsearch_done_wait, $lei);
$l2m->wq_workers_start('lei2mail', undef,
$lei->oldset, { lei => $lei });
- $l2m->wq_wait_async(\&xsearch_done_wait, $lei);
pipe($lei->{startq}, $lei->{au_done}) or die "pipe: $!";
fcntl($lei->{startq}, $F_SETPIPE_SZ, 4096) if $F_SETPIPE_SZ;
delete $l2m->{au_peers};
}
+ $self->awaitpid_init(\&xsearch_done_wait, $lei);
$self->wq_workers_start('lei_xsearch', undef,
$lei->oldset, { lei => $lei });
- $self->wq_wait_async(\&xsearch_done_wait, $lei);
my $op_c = delete $lei->{pkt_op_c};
delete $lei->{pkt_op_p};
@$end = ();
diff --git a/script/public-inbox-clone b/script/public-inbox-clone
index e93ac37b..598979bc 100755
--- a/script/public-inbox-clone
+++ b/script/public-inbox-clone
@@ -62,5 +62,5 @@ my $mrr = bless {
$? = 0;
$mrr->do_mirror;
-$mrr->can('_wq_done_wait')->([$mrr, $lei], $$);
+$mrr->can('_wq_done_wait')->($$, $mrr, $lei);
exit(($lei->{child_error} // 0) >> 8);
^ permalink raw reply related [relevance 39%]
* [PATCH 2/2] content_digest_dbg: convert to arrayref and limit to lei
@ 2023-01-29 10:30 63% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2023-01-29 10:30 UTC (permalink / raw)
To: meta
Since it's an extremely small class and not subclassed or
anything, we'll make it even smaller as an arrayref.
We also don't load this for PublicInbox::WWW or anything that
runs in public-facing daemons.
---
lib/PublicInbox/ContentDigestDbg.pm | 10 ++++++----
lib/PublicInbox/MailDiff.pm | 5 +----
2 files changed, 7 insertions(+), 8 deletions(-)
diff --git a/lib/PublicInbox/ContentDigestDbg.pm b/lib/PublicInbox/ContentDigestDbg.pm
index 899afbbe..5de0ee8a 100644
--- a/lib/PublicInbox/ContentDigestDbg.pm
+++ b/lib/PublicInbox/ContentDigestDbg.pm
@@ -1,17 +1,19 @@
# Copyright (C) all contributors <meta@public-inbox.org>
# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+# only loaded in lei
package PublicInbox::ContentDigestDbg; # cf. PublicInbox::ContentDigest
use v5.12;
use Data::Dumper;
use PublicInbox::SHA;
+$Data::Dumper::Useqq = $Data::Dumper::Terse = 1;
-sub new { bless { dig => PublicInbox::SHA->new(256), fh => $_[1] }, __PACKAGE__ }
+sub new { bless [ PublicInbox::SHA->new(256), $_[1] ], __PACKAGE__ }
sub add {
- $_[0]->{dig}->add($_[1]);
- print { $_[0]->{fh} } Dumper([split(/^/sm, $_[1])]) or die "print $!";
+ $_[0]->[0]->add($_[1]);
+ print { $_[0]->[1] } Dumper([split(/^/sm, $_[1])]) or die "print $!";
}
-sub hexdigest { $_[0]->{dig}->hexdigest; }
+sub hexdigest { $_[0]->[0]->hexdigest }
1;
diff --git a/lib/PublicInbox/MailDiff.pm b/lib/PublicInbox/MailDiff.pm
index 0ed06f9a..a0ecef9f 100644
--- a/lib/PublicInbox/MailDiff.pm
+++ b/lib/PublicInbox/MailDiff.pm
@@ -4,8 +4,6 @@ package PublicInbox::MailDiff;
use v5.12;
use File::Temp 0.19 (); # 0.19 for ->newdir
use PublicInbox::ContentHash qw(content_digest);
-use PublicInbox::ContentDigestDbg;
-use Data::Dumper ();
use PublicInbox::MsgIter qw(msg_part_text);
use PublicInbox::ViewDiff qw(flush_diff);
use PublicInbox::GitAsyncCat;
@@ -34,12 +32,11 @@ sub dump_eml ($$$) {
$eml->each_part(\&write_part, $self);
return if $self->{ctx}; # don't need content_digest noise in WWW UI
+ require PublicInbox::ContentDigestDbg;
# XXX is this even useful? perhaps hide it behind a CLI switch
open my $fh, '>', "$dir/content_digest" or die "open: $!";
my $dig = PublicInbox::ContentDigestDbg->new($fh);
- local $Data::Dumper::Useqq = 1;
- local $Data::Dumper::Terse = 1;
content_digest($eml, $dig);
print $fh "\n", $dig->hexdigest, "\n" or die "print $!";
close $fh or die "close: $!";
^ permalink raw reply related [relevance 63%]
* [PATCH 0/2] fix xt/lei-auth-fail.t
@ 2023-01-29 22:58 71% Eric Wong
2023-01-29 22:58 69% ` [PATCH 2/2] xt/lei-auth-fail: use valid label name Eric Wong
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2023-01-29 22:58 UTC (permalink / raw)
To: meta
I need a failing mock IMAP server for this, or remember xt/ exists :<
Eric Wong (2):
lei_input: give a hint for upper-case in labels
xt/lei-auth-fail: use valid label name
lib/PublicInbox/LeiInput.pm | 2 ++
xt/lei-auth-fail.t | 7 ++++---
2 files changed, 6 insertions(+), 3 deletions(-)
^ permalink raw reply [relevance 71%]
* [PATCH 2/2] xt/lei-auth-fail: use valid label name
2023-01-29 22:58 71% [PATCH 0/2] fix xt/lei-auth-fail.t Eric Wong
@ 2023-01-29 22:58 69% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2023-01-29 22:58 UTC (permalink / raw)
To: meta
Uppercase characters aren't allowed for labels due to Xapian
boolean limitations, so we need to use lowercase labels.
Fixes: 27015c3365fd0690 (lei_input: disallow uppercase characters for labels, 2021-10-31)
---
xt/lei-auth-fail.t | 7 ++++---
1 file changed, 4 insertions(+), 3 deletions(-)
diff --git a/xt/lei-auth-fail.t b/xt/lei-auth-fail.t
index 06cb8533..1ccc2ab2 100644
--- a/xt/lei-auth-fail.t
+++ b/xt/lei-auth-fail.t
@@ -1,7 +1,8 @@
#!perl -w
-# Copyright (C) 2021 all contributors <meta@public-inbox.org>
+# Copyright (C) all contributors <meta@public-inbox.org>
# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
-use strict; use v5.10.1; use PublicInbox::TestCommon;
+use v5.12;
+use PublicInbox::TestCommon;
require_mods(qw(Mail::IMAPClient lei));
# TODO: mock IMAP server which fails at authentication so we don't
@@ -13,7 +14,7 @@ test_lei(sub {
for my $pfx ([qw(q z:0.. --only), "$ro_home/t1", '-o'],
[qw(convert -o mboxrd:/dev/stdout)],
[qw(convert t/utf8.eml -o), $imap_fail],
- ['import'], [qw(tag +L:INBOX)]) {
+ ['import'], [qw(tag +L:inbox)]) {
ok(!lei(@$pfx, $imap_fail), "IMAP auth failure on @$pfx");
like($lei_err, qr!\bE:.*?imaps?://.*?!sm, 'error shown');
unlike($lei_err, qr!Hunter2!s, 'password not shown');
^ permalink raw reply related [relevance 69%]
* [PATCH] lei: drop -watches and -lei_note_event from workers
@ 2023-01-31 0:05 70% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2023-01-31 0:05 UTC (permalink / raw)
To: meta
I noticed these while tracking down circular refs for commit
7b654d175cf2e31b (ipc: drop awaitpid_init to avoid circular refs, 2023-01-30).
While they're not the cause of circular refs, they're still
a waste of memory in worker processes.
---
lib/PublicInbox/LEI.pm | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index ffd50db5..d05b20de 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -561,17 +561,17 @@ sub note_sigpipe { # triggers sigpipe_handler
sub _lei_atfork_child {
my ($self, $persist) = @_;
# we need to explicitly close things which are on stack
+ my $cfg = $self->{cfg};
if ($persist) {
open $self->{3}, '<', '/' or die "open(/) $!";
fchdir($self);
close($_) for (grep(defined, delete @$self{qw(0 1 2 sock)}));
- if (my $cfg = $self->{cfg}) {
- delete @$cfg{qw(-lei_store -watches -lei_note_event)};
- }
+ delete @$cfg{qw(-lei_store -watches -lei_note_event)};
} else { # worker, Net::NNTP (Net::Cmd) uses STDERR directly
open STDERR, '+>&='.fileno($self->{2}) or warn "open $!";
STDERR->autoflush(1);
POSIX::setpgid(0, $$) // die "setpgid(0, $$): $!";
+ delete @$cfg{qw(-watches -lei_note_event)};
}
close($_) for (grep(defined, delete @$self{qw(old_1 au_done)}));
delete $self->{-socks};
^ permalink raw reply related [relevance 70%]
* [PATCH] t/lei-refresh-mail-sync: avoid kill+sleep loop
@ 2023-02-12 3:12 61% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2023-02-12 3:12 UTC (permalink / raw)
To: meta
While we can't waitpid() on daemonized process, we can abuse the
lack of FD_CLOEXEC to detect a process death. This saves
roughly 400ms for this slow test.
---
lib/PublicInbox/TestCommon.pm | 3 +++
t/lei-refresh-mail-sync.t | 20 ++++++++++++--------
2 files changed, 15 insertions(+), 8 deletions(-)
diff --git a/lib/PublicInbox/TestCommon.pm b/lib/PublicInbox/TestCommon.pm
index 1fe7931e..8a34e45a 100644
--- a/lib/PublicInbox/TestCommon.pm
+++ b/lib/PublicInbox/TestCommon.pm
@@ -480,6 +480,9 @@ sub start_script {
my $pid = fork // die "fork: $!\n";
if ($pid == 0) {
eval { PublicInbox::DS->Reset };
+ for (@{delete($opt->{-CLOFORK}) // []}) {
+ close($_) or die "close $!";
+ }
# pretend to be systemd (cf. sd_listen_fds(3))
# 3 == SD_LISTEN_FDS_START
my $fd;
diff --git a/t/lei-refresh-mail-sync.t b/t/lei-refresh-mail-sync.t
index ea83a513..0498a0c4 100644
--- a/t/lei-refresh-mail-sync.t
+++ b/t/lei-refresh-mail-sync.t
@@ -5,17 +5,20 @@ use strict; use v5.10.1; use PublicInbox::TestCommon;
require_mods(qw(lei));
use File::Path qw(remove_tree);
require Socket;
+use Fcntl qw(F_SETFD);
+
+pipe(my ($stop_r, $stop_w)) or xbail "pipe: $!";
+fcntl($stop_w, F_SETFD, 0) or xbail "F_SETFD: $!";
my $stop_daemon = sub { # needed since we don't have inotify
+ close $stop_w or xbail "close \$stop_w: $!";
lei_ok qw(daemon-pid);
chomp(my $pid = $lei_out);
$pid > 0 or xbail "bad pid: $pid";
kill('TERM', $pid) or xbail "kill: $!";
- for (0..10) {
- tick;
- kill(0, $pid) or last;
- }
- kill(0, $pid) and xbail "daemon still running (PID:$pid)";
+ is(sysread($stop_r, my $buf, 1), 0, 'daemon stop pipe read EOF');
+ pipe($stop_r, $stop_w) or xbail "pipe: $!";
+ fcntl($stop_w, F_SETFD, 0) or xbail "F_SETFD: $!";
};
test_lei({ daemon_only => 1 }, sub {
@@ -88,7 +91,8 @@ SKIP: {
$sock_cls //= ref($s);
my $cmd = [ "-$x", '-W0', "--stdout=$home/$x.out",
"--stderr=$home/$x.err" ];
- my $td = start_script($cmd, $env, { 3 => $s }) or xbail("-$x");
+ my $opt = { 3 => $s, -CLOFORK => [ $stop_w ] };
+ my $td = start_script($cmd, $env, $opt) or xbail("-$x");
my $addr = tcp_host_port($s);
$srv->{$x} = { addr => $addr, td => $td, cmd => $cmd, s => $s };
}
@@ -139,8 +143,8 @@ SKIP: {
my $cmd = $srv->{imapd}->{cmd};
my $s = $srv->{imapd}->{s};
$s->blocking(0);
- $srv->{imapd}->{td} = start_script($cmd, $env, { 3 => $s }) or
- xbail "@$cmd";
+ my $opt = { 3 => $s, -CLOFORK => [ $stop_w ] };
+ $srv->{imapd}->{td} = start_script($cmd, $env, $opt) or xbail "@$cmd";
lei_ok 'refresh-mail-sync', '--all';
lei_ok 'inspect', "blob:$oid";
is($lei_out, $before, 'no changes when server was down');
^ permalink raw reply related [relevance 61%]
* lei q -tt doesn't work properly?
@ 2023-02-13 16:06 63% Maxim Mikityanskiy
2023-02-14 2:42 66% ` [PATCH] lei q: do not collapse threads with `-tt' Eric Wong
0 siblings, 1 reply; 200+ results
From: Maxim Mikityanskiy @ 2023-02-13 16:06 UTC (permalink / raw)
To: meta; +Cc: Eric Wong, Kyle Meyer
Hello,
I'm trying to use the -tt flag to download the whole thread, but mark
the actual matching emails as important. I'm not sure if I'm doing it
incorrectly, or maybe there is a bug in lei.
According to the man page:
--cut--
-t Return all messages in the same thread as the actual match(es).
Using this twice ("-tt") sets the "flagged" (AKA "important") on
messages which were actual matches. This is useful to distinguish
messages which were direct hits from messages which were merely
part of the same thread.
--cut--
I'm using this command, for example:
lei q --no-save -a -o /tmp/lei-test -I 'https://lore.kernel.org/all' \
-tt 'a:syzbot AND rt:2023-01-01..2023-01-07'
What I expect to see is at least one flagged email in each thread
(otherwise why would this thread by downloaded), however, instead, most
of the emails are not flagged, that is, the whole threads don't have any
flagged email (although they clearly have emails from syzbot, which
caused the match). Occasionally, some emails are flagged, for example,
these two:
https://lore.kernel.org/all/87wn621hmp.fsf@toke.dk/
https://lore.kernel.org/all/20230103081308.942805751@linuxfoundation.org/
It looks as if the match works correctly, but the -tt option fails to
mark most of the matched emails as important, except a few that actually
got marked (I couldn't find a pattern here). It's also not consistent,
for example, after I removed /tmp/lei-test and restarted the lei q
command, I got many more important emails, almost in each thread, but
there were still threads without flagged emails.
I'm checking the flags with mutt.
Does anyone know what could be the reason for such behavior?
Thanks,
Max
^ permalink raw reply [relevance 63%]
* [PATCH] lei q: do not collapse threads with `-tt'
2023-02-13 16:06 63% lei q -tt doesn't work properly? Maxim Mikityanskiy
@ 2023-02-14 2:42 66% ` Eric Wong
2023-02-26 12:17 71% ` Maxim Mikityanskiy
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2023-02-14 2:42 UTC (permalink / raw)
To: Maxim Mikityanskiy; +Cc: meta, Kyle Meyer
Maxim Mikityanskiy <maxtram95@gmail.com> wrote:
> lei q --no-save -a -o /tmp/lei-test -I 'https://lore.kernel.org/all' \
> -tt 'a:syzbot AND rt:2023-01-01..2023-01-07'
At first, I thought -a (--augment) was causing it...
Sidenote: you also don't need to quote the query (I forget the exact
rules, but I tried to keep quotes easier for phrase searches).
> It looks as if the match works correctly, but the -tt option fails to
> mark most of the matched emails as important, except a few that actually
> got marked (I couldn't find a pattern here). It's also not consistent,
> for example, after I removed /tmp/lei-test and restarted the lei q
> command, I got many more important emails, almost in each thread, but
> there were still threads without flagged emails.
Yes, now it seems it's the collapsing optimization.
> I'm checking the flags with mutt.
>
> Does anyone know what could be the reason for such behavior?
I think the following patch fixes it.
(I accidentally sent you a private copy with invalid blobs since
I had other unpublished changes)
-----8<-------
Subject: [PATCH] lei q: do not collapse threads with `-tt'
While having Xapian collapse threads is an easy way to reduce
the amount of deduplication work we need to do when writing
out threads; we can't rely on it when using `lei q -tt` since
that needs to flag all hits.
Reported-by: Maxim Mikityanskiy <maxtram95@gmail.com>
Link: https://public-inbox.org/git/Y+pgBmj0jxR+cVkD@mail.gmail.com/
---
lib/PublicInbox/Search.pm | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/lib/PublicInbox/Search.pm b/lib/PublicInbox/Search.pm
index 2feb3e13..273cc57c 100644
--- a/lib/PublicInbox/Search.pm
+++ b/lib/PublicInbox/Search.pm
@@ -460,8 +460,9 @@ sub _enquire_once { # retry_reopen callback
$enquire->set_sort_by_relevance_then_value(TS, !$opts->{asc});
}
- # `mairix -t / --threads' or JMAP collapseThreads
- if ($opts->{threads} && has_threadid($self)) {
+ # `lei q -t / --threads' or JMAP collapseThreads; but don't collapse
+ # on `-tt' ({threads} > 1) which sets the Flagged|Important keyword
+ if (($opts->{threads} // 0) == 1 && has_threadid($self)) {
$enquire->set_collapse_key(THREADID);
}
$enquire->get_mset($opts->{offset} || 0, $opts->{limit} || 50);
^ permalink raw reply related [relevance 66%]
* Re: FUSE3 vs read-write IMAP for lei
2022-12-09 1:41 65% ` FUSE3 vs read-write IMAP for lei Eric Wong
@ 2023-02-20 19:27 71% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2023-02-20 19:27 UTC (permalink / raw)
To: meta
Eric Wong <e@80x24.org> wrote:
> == FUSE3 - Maildir-oriented FS
>
> + will require C compiler since old FUSE XS modules don't do
> FUSE3 (for readdirplus); and going w/o readdirplus is
> unimaginable for Maildir.
>
> + I already have existing (unreleased) AGPL-3 work based on
> FUSE3 + URCU + Perl5 with just-ahead-of-time (JAOT) compilation
Fwiw, I've pushed out a new "fuse3" branch to public-inbox.git:
https://80x24.org/public-inbox.git/80ce906027eeb7b4cc5cc7d3858294927951988a/s/
I think I need some C + URCU in my life to keep my brain working
> * I've seen the light w/ URCU, and can't go back to C without it :P
>
> - likely Linux-only (not sure how good FUSE support will be if
> depending on FUSE3 features)
Well, I got rid of the futex requirement from the original...
> - kernel caches still incur nasty memory overhead w/ Maildir
>
> - readdir(3) userspace API still sucks
^ permalink raw reply [relevance 71%]
* Re: [PATCH] lei q: do not collapse threads with `-tt'
2023-02-14 2:42 66% ` [PATCH] lei q: do not collapse threads with `-tt' Eric Wong
@ 2023-02-26 12:17 71% ` Maxim Mikityanskiy
2023-02-26 17:09 69% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Maxim Mikityanskiy @ 2023-02-26 12:17 UTC (permalink / raw)
To: Eric Wong; +Cc: meta, Kyle Meyer
On Tue, Feb 14, 2023 at 02:42:32AM +0000, Eric Wong wrote:
> Maxim Mikityanskiy <maxtram95@gmail.com> wrote:
> > lei q --no-save -a -o /tmp/lei-test -I 'https://lore.kernel.org/all' \
> > -tt 'a:syzbot AND rt:2023-01-01..2023-01-07'
>
> At first, I thought -a (--augment) was causing it...
>
> Sidenote: you also don't need to quote the query (I forget the exact
> rules, but I tried to keep quotes easier for phrase searches).
>
> > It looks as if the match works correctly, but the -tt option fails to
> > mark most of the matched emails as important, except a few that actually
> > got marked (I couldn't find a pattern here). It's also not consistent,
> > for example, after I removed /tmp/lei-test and restarted the lei q
> > command, I got many more important emails, almost in each thread, but
> > there were still threads without flagged emails.
>
> Yes, now it seems it's the collapsing optimization.
>
> > I'm checking the flags with mutt.
> >
> > Does anyone know what could be the reason for such behavior?
>
> I think the following patch fixes it.
Sorry for taking too long, I finally found a minute to test it, and
unfortunately I didn't see a difference. I queried for:
a:syzbot AND rt:2023-02-01..2023-02-07
and I still saw I lot of threads without a single flag.
I double-checked that the patch was actually applied, killed lei-daemon,
and removed the mailbox directory, but it didn't help.
> (I accidentally sent you a private copy with invalid blobs since
> I had other unpublished changes)
>
> -----8<-------
> Subject: [PATCH] lei q: do not collapse threads with `-tt'
>
> While having Xapian collapse threads is an easy way to reduce
> the amount of deduplication work we need to do when writing
> out threads; we can't rely on it when using `lei q -tt` since
> that needs to flag all hits.
>
> Reported-by: Maxim Mikityanskiy <maxtram95@gmail.com>
> Link: https://public-inbox.org/git/Y+pgBmj0jxR+cVkD@mail.gmail.com/
> ---
> lib/PublicInbox/Search.pm | 5 +++--
> 1 file changed, 3 insertions(+), 2 deletions(-)
>
> diff --git a/lib/PublicInbox/Search.pm b/lib/PublicInbox/Search.pm
> index 2feb3e13..273cc57c 100644
> --- a/lib/PublicInbox/Search.pm
> +++ b/lib/PublicInbox/Search.pm
> @@ -460,8 +460,9 @@ sub _enquire_once { # retry_reopen callback
> $enquire->set_sort_by_relevance_then_value(TS, !$opts->{asc});
> }
>
> - # `mairix -t / --threads' or JMAP collapseThreads
> - if ($opts->{threads} && has_threadid($self)) {
> + # `lei q -t / --threads' or JMAP collapseThreads; but don't collapse
> + # on `-tt' ({threads} > 1) which sets the Flagged|Important keyword
> + if (($opts->{threads} // 0) == 1 && has_threadid($self)) {
> $enquire->set_collapse_key(THREADID);
> }
> $enquire->get_mset($opts->{offset} || 0, $opts->{limit} || 50);
^ permalink raw reply [relevance 71%]
* Re: [PATCH] lei q: do not collapse threads with `-tt'
2023-02-26 12:17 71% ` Maxim Mikityanskiy
@ 2023-02-26 17:09 69% ` Eric Wong
2023-02-26 17:15 71% ` [PATCH] doc: note "lei q -tt" is broken with HTTP(S) remotes Eric Wong
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2023-02-26 17:09 UTC (permalink / raw)
To: Maxim Mikityanskiy; +Cc: meta, Kyle Meyer
Maxim Mikityanskiy <maxtram95@gmail.com> wrote:
> On Tue, Feb 14, 2023 at 02:42:32AM +0000, Eric Wong wrote:
> > Maxim Mikityanskiy <maxtram95@gmail.com> wrote:
> > > lei q --no-save -a -o /tmp/lei-test -I 'https://lore.kernel.org/all' \
> > > -tt 'a:syzbot AND rt:2023-01-01..2023-01-07'
<snip>
> > Yes, now it seems it's the collapsing optimization.
<snip>
> Sorry for taking too long, I finally found a minute to test it, and
> unfortunately I didn't see a difference. I queried for:
>
> a:syzbot AND rt:2023-02-01..2023-02-07
>
> and I still saw I lot of threads without a single flag.
>
> I double-checked that the patch was actually applied, killed lei-daemon,
> and removed the mailbox directory, but it didn't help.
Ah, oops. My original fix only works for locally-cloned inboxes;
but not remote (http/https) inboxes...
I think some inconsistency on the client side is also introduced
by using -I/--include vs --only; since -I/--include will use
previously-indexed messages in ~/.local/share/lei/store
Getting -tt to work on remote inboxes will take more effort.
I'm not sure which option is better:
1) Support t=2 natively in the WWW interface. This requires
both the server and client to be updated. It may require
extra dedupe step on the server, making it more expensive.
Thinking out loud, I think the dedupe step can be avoided
by sorting on THREADID...
2) use t=1 in the client as-is, but index the streamed mbox
locally, first. This requires a temporary Xapian DB to
ensure there's no overlap if using --only.
This only requires a client update, but likely adds more
complexity. It also delays updates to the Maildir,
meaning all messages need to be downloaded before the MUA
sees it...
I'm leaning towards 1...
^ permalink raw reply [relevance 69%]
* [PATCH] doc: note "lei q -tt" is broken with HTTP(S) remotes
2023-02-26 17:09 69% ` Eric Wong
@ 2023-02-26 17:15 71% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2023-02-26 17:15 UTC (permalink / raw)
To: Maxim Mikityanskiy; +Cc: meta, Kyle Meyer
Eric Wong <e@80x24.org> wrote:
> Getting -tt to work on remote inboxes will take more effort.
> I'm not sure which option is better:
I suppose documenting the current breakage first is important:
--------- 8< --------
Subject: [PATCH] doc: note "lei q -tt" is broken with HTTP(S) remotes
I'm still trying to decide how to handle HTTP(S) remotes
properly...
Link: https://public-inbox.org/meta/20230226170931.M947721@dcvr/
---
Documentation/lei-q.pod | 3 +++
1 file changed, 3 insertions(+)
diff --git a/Documentation/lei-q.pod b/Documentation/lei-q.pod
index d52c5b04..5e9a5658 100644
--- a/Documentation/lei-q.pod
+++ b/Documentation/lei-q.pod
@@ -124,6 +124,9 @@ of the same thread.
TODO: Warning: this flag may become persistent and saved in
lei/store unless an MUA unflags it! (Behavior undecided)
+Caveat: C<-tt> only works on locally-indexed messages at the
+moment, and not on remote (HTTP(S)) endpoints.
+
=item --jobs=QUERY_WORKERS[,WRITE_WORKERS]
=item --jobs=,WRITE_WORKERS
^ permalink raw reply related [relevance 71%]
* [PATCH 4/6] doc: lei import: add hints about nntp.* and imap.* config options
@ 2023-03-09 19:28 71% ` Eric Wong
2023-03-09 19:28 68% ` [PATCH 5/6] doc: lei config: update with --edit and --list examples Eric Wong
1 sibling, 0 replies; 200+ results
From: Eric Wong @ 2023-03-09 19:28 UTC (permalink / raw)
To: meta
I'm setting up more imports and forgot about them :x
---
Documentation/lei-import.pod | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/Documentation/lei-import.pod b/Documentation/lei-import.pod
index 69ec6497..31d6db13 100644
--- a/Documentation/lei-import.pod
+++ b/Documentation/lei-import.pod
@@ -86,8 +86,13 @@ Default: C<auto>
Use the specified proxy (e.g., C<socks5h://0:9050>).
+Consider L<imap.proxy> and L<nntp.proxy> which can be persistently
+configured on a per-host basis in L<lei-config(1)>.
+
=back
+See L<lei-config(1)> for various C<imap.*> and C<nntp.*> options.
+
=head1 CONTACT
Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org>
@@ -103,4 +108,4 @@ License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>
=head1 SEE ALSO
-L<lei-index(1)>, L<lei-store-format(5)>
+L<lei-config(1)>, L<lei-index(1)>, L<lei-store-format(5)>
^ permalink raw reply related [relevance 71%]
* [PATCH 5/6] doc: lei config: update with --edit and --list examples
2023-03-09 19:28 71% ` [PATCH 4/6] doc: lei import: add hints about nntp.* and imap.* config options Eric Wong
@ 2023-03-09 19:28 68% ` Eric Wong
1 sibling, 0 replies; 200+ results
From: Eric Wong @ 2023-03-09 19:28 UTC (permalink / raw)
To: meta
I typically use --edit/-e to make changes and --list/-l with
git; and same with lei.
---
Documentation/lei-config.pod | 29 +++++++++++++++++++++++++++--
1 file changed, 27 insertions(+), 2 deletions(-)
diff --git a/Documentation/lei-config.pod b/Documentation/lei-config.pod
index 663404fe..23a60c8a 100644
--- a/Documentation/lei-config.pod
+++ b/Documentation/lei-config.pod
@@ -4,7 +4,11 @@ lei-config - git-config wrapper for lei configuration file
=head1 SYNOPSIS
-lei config [OPTIONS]
+lei config <name> [[<value>] [<value-pattern>]]
+
+lei config -l | --list
+
+lei config -e | --edit
=head1 DESCRIPTION
@@ -97,6 +101,27 @@ C<frag>, C<func>, and C<context>.
=back
+=head1 OPTIONS
+
+Most L<git-config(1)> command-line switches are accepted by C<lei config>
+as-is. The most-frequently-used options are expected to be:
+
+=over 4
+
+=item -e
+
+=item --edit
+
+Opens an editor to edit the lei config file
+
+=item -l
+
+=item --list
+
+List all variables set in config file, along with their values.
+
+=back
+
=head1 CONTACT
Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org>
@@ -106,6 +131,6 @@ L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/>
=head1 COPYRIGHT
-Copyright 2021 all contributors L<mailto:meta@public-inbox.org>
+Copyright all contributors L<mailto:meta@public-inbox.org>
License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>
^ permalink raw reply related [relevance 68%]
* [PATCH] lei: improve bash completion involving colons
@ 2023-03-23 21:45 46% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2023-03-23 21:45 UTC (permalink / raw)
To: meta
This fixes completions of labels (`+L:' for `lei import' and
`L:' for `lei q') so they can appear anywhere in the
command-line.
I mainly wanted this for `lei import $URL +L:label', but
this also fixes `lei forget-external' completions for URLs
(which involve colons).
---
contrib/completion/lei-completion.bash | 15 ++++++++-----
lib/PublicInbox/LeiExternal.pm | 31 +++++++++++---------------
lib/PublicInbox/LeiForgetExternal.pm | 8 ++-----
lib/PublicInbox/LeiImport.pm | 22 +++++++++++-------
lib/PublicInbox/LeiQuery.pm | 2 ++
5 files changed, 40 insertions(+), 38 deletions(-)
diff --git a/contrib/completion/lei-completion.bash b/contrib/completion/lei-completion.bash
index 5c137e68..b86afa2c 100644
--- a/contrib/completion/lei-completion.bash
+++ b/contrib/completion/lei-completion.bash
@@ -1,16 +1,19 @@
-# Copyright (C) 2020-2021 all contributors <meta@public-inbox.org>
+# Copyright (C) all contributors <meta@public-inbox.org>
# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
# preliminary bash completion support for lei (Local Email Interface)
# Needs a lot of work, see `lei__complete' in lib/PublicInbox::LEI.pm
_lei() {
local wordlist="$(lei _complete ${COMP_WORDS[@]})"
- case $wordlist in
- *':'* | *'='* | '//'*) compopt -o nospace ;;
- *) compopt +o nospace ;; # the default
- esac
wordlist="${wordlist//;/\\\\;}" # escape ';' for ';UIDVALIDITY' and such
- COMPREPLY=($(compgen -W "$wordlist" -- "${COMP_WORDS[COMP_CWORD]}"))
+
+ local word="${COMP_WORDS[COMP_CWORD]}"
+ if test "$word" = ':' && test $COMP_CWORD -ge 1
+ then
+ COMPREPLY=($(compgen -W "$wordlist" --))
+ else
+ COMPREPLY=($(compgen -W "$wordlist" -- "$word"))
+ fi
return 0
}
complete -o default -o bashdefault -F _lei lei
diff --git a/lib/PublicInbox/LeiExternal.pm b/lib/PublicInbox/LeiExternal.pm
index 3e2a2288..31b9bd1e 100644
--- a/lib/PublicInbox/LeiExternal.pm
+++ b/lib/PublicInbox/LeiExternal.pm
@@ -86,39 +86,34 @@ sub canonicalize_excludes {
# returns an anonymous sub which returns an array of potential results
sub complete_url_prepare {
my $argv = $_[-1]; # $_[0] may be $lei
- # Workaround bash word-splitting URLs to ['https', ':', '//' ...]
- # Maybe there's a better way to go about this in
- # contrib/completion/lei-completion.bash
- my $re = '';
- my $cur = pop(@$argv) // '';
+ # Workaround bash default COMP_WORDBREAKS splitting URLs to
+ # ['https', ':', '//', ...]. COMP_WORDBREAKS is global for all
+ # completions loaded, not just ours, so we can't change it.
+ # cf. contrib/completion/lei-completion.bash
+ my ($pfx, $cur) = ('', pop(@$argv) // '');
if (@$argv) {
my @x = @$argv;
- if ($cur eq ':' && @x) {
+ if ($cur =~ /\A[:;=]\z/) { # COMP_WORDBREAKS + URL union
push @x, $cur;
$cur = '';
}
- while (@x > 2 && $x[0] !~ /\A(?:http|nntp|imap)s?\z/i &&
- $x[1] ne ':') {
- shift @x;
+ while (@x && $pfx !~ m!\A(?: (?:[\+\-]?(?:L|kw):) |
+ (?:(?:imap|nntp|http)s?:) |
+ (?:--\w?\z)|(?:-\w?\z) )!x) {
+ $pfx = pop(@x).$pfx;
}
- if (@x >= 2) { # qw(https : hostname : 443) or qw(http :)
- $re = join('', @x);
- } else { # just filter out the flags and hope for the best
- $re = join('', grep(!/^-/, @$argv));
- }
- $re = quotemeta($re);
}
+ my $re = qr!\A\Q$pfx\E(\Q$cur\E.*)!;
my $match_cb = sub {
# the "//;" here (for AUTH=ANONYMOUS) interacts badly with
# bash tab completion, strip it out for now since our commands
# work w/o it. Not sure if there's a better solution...
$_[0] =~ s!//;AUTH=ANONYMOUS\@!//!i;
- $_[0] =~ s!;!\\;!g;
# only return the part specified on the CLI
# don't duplicate if already 100% completed
- $_[0] =~ /\A$re(\Q$cur\E.*)/ ? ($cur eq $1 ? () : $1) : ()
+ $_[0] =~ $re ? ($cur eq $1 ? () : $1) : ()
};
- wantarray ? ($re, $cur, $match_cb) : $match_cb;
+ wantarray ? ($pfx, $cur, $match_cb) : $match_cb;
}
1;
diff --git a/lib/PublicInbox/LeiForgetExternal.pm b/lib/PublicInbox/LeiForgetExternal.pm
index 07f0ac80..39bfc60b 100644
--- a/lib/PublicInbox/LeiForgetExternal.pm
+++ b/lib/PublicInbox/LeiForgetExternal.pm
@@ -32,14 +32,10 @@ sub lei_forget_external {
sub _complete_forget_external {
my ($lei, @argv) = @_;
my $cfg = $lei->_lei_cfg or return ();
- my ($cur, $re, $match_cb) = $lei->complete_url_prepare(\@argv);
- # FIXME: bash completion off "http:" or "https:" when the last
- # character is a colon doesn't work properly even if we're
- # returning "//$HTTP_HOST/$PATH_INFO/", not sure why, could
- # be a bash issue.
+ my ($pfx, $cur, $match_cb) = $lei->complete_url_prepare(\@argv);
map {
$match_cb->(substr($_, length('external.')));
- } grep(/\Aexternal\.$re\Q$cur/, @{$cfg->{-section_order}});
+ } grep(/\Aexternal\.\Q$pfx$cur/, @{$cfg->{-section_order}});
}
1;
diff --git a/lib/PublicInbox/LeiImport.pm b/lib/PublicInbox/LeiImport.pm
index 2d91e4c4..9053048a 100644
--- a/lib/PublicInbox/LeiImport.pm
+++ b/lib/PublicInbox/LeiImport.pm
@@ -115,18 +115,24 @@ sub lei_import { # the main "lei import" method
sub _complete_import {
my ($lei, @argv) = @_;
- my ($re, $cur, $match_cb) = $lei->complete_url_prepare(\@argv);
- my @k = $lei->url_folder_cache->keys($argv[-1] // undef, 1);
+ my $has_arg = @argv;
+ my ($pfx, $cur, $match_cb) = $lei->complete_url_prepare(\@argv);
+ my @try = $has_arg ? ($pfx.$cur, $argv[-1]) : ($argv[-1]);
+ push(@try, undef) if defined $try[-1];
+ my (@f, @k);
+ for (@try) {
+ @k = $lei->url_folder_cache->keys($_, 1) and last;
+ }
my @L = eval { $lei->_lei_store->search->all_terms('L') };
push(@k, map { "+L:$_" } @L);
- my @m = map { $match_cb->($_) } @k;
- my %f = map { $_ => 1 } (@m ? @m : @k);
if (my $lms = $lei->lms) {
- @k = $lms->folders($argv[-1] // undef, 1);
- @m = map { $match_cb->($_) } @k;
- if (@m) { @f{@m} = @m } else { @f{@k} = @k }
+ for (@try) {
+ @f = $lms->folders($_, 1) and last;
+ }
+ push @k, @f;
}
- keys %f;
+ my @m = map { $match_cb->($_) } @k;
+ @m ? @m : @k;
}
no warnings 'once';
diff --git a/lib/PublicInbox/LeiQuery.pm b/lib/PublicInbox/LeiQuery.pm
index 358574ea..3337e5d4 100644
--- a/lib/PublicInbox/LeiQuery.pm
+++ b/lib/PublicInbox/LeiQuery.pm
@@ -173,6 +173,8 @@ no query allowed on command-line with --stdin
# shell completion helper called by lei__complete
sub _complete_q {
my ($self, @argv) = @_;
+ join('', @argv) =~ /\bL:\S*\z/ and
+ return eval { $self->_lei_store->search->all_terms('L') };
my @cur;
my $cb = $self->lazy_cb(qw(forget-external _complete_));
while (@argv) {
^ permalink raw reply related [relevance 46%]
* repeat `lei import' users?
@ 2023-03-23 22:05 71% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2023-03-23 22:05 UTC (permalink / raw)
To: meta
Just wondering if there's lei-mail-sync-overview(7) followers,
yet... It could be a bit less clunky :x
^ permalink raw reply [relevance 71%]
* Issues with `lei` as non-root
@ 2023-03-28 1:00 71% Louis DeLosSantos
2023-03-28 1:32 71% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Louis DeLosSantos @ 2023-03-28 1:00 UTC (permalink / raw)
To: meta
Hello,
I'm experimenting with `lei` as a nice search tool for `lore.kernel.org`
Everything works fine with the caveat that it seems to break if I'm not root.
When using `lei` as non-root we get this error:
```
E: Linux::Inotify2->new: Too many open files at
/usr/share/perl5/vendor_perl/PublicInbox/DirIdle.pm line 40.
connect(/run/user/1000/lei/5.seq.sock): Connection refused (after
attempted daemon start)
```
Any ideas why this may occur? Is `lei` designed to only be ran as root
or is Fedora installing perl in an odd fashion which results in root
needing to be used?
^ permalink raw reply [relevance 71%]
* Re: Issues with `lei` as non-root
2023-03-28 1:00 71% Issues with `lei` as non-root Louis DeLosSantos
@ 2023-03-28 1:32 71% ` Eric Wong
2023-03-28 2:30 45% ` Louis DeLosSantos
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2023-03-28 1:32 UTC (permalink / raw)
To: Louis DeLosSantos; +Cc: meta
Louis DeLosSantos <louis.delos@gmail.com> wrote:
> Hello,
>
> I'm experimenting with `lei` as a nice search tool for `lore.kernel.org`
>
> Everything works fine with the caveat that it seems to break if I'm not root.
>
> When using `lei` as non-root we get this error:
I've never used lei as root nor has any part of public-inbox
ever been intended to run as root.
> ```
> E: Linux::Inotify2->new: Too many open files at
> /usr/share/perl5/vendor_perl/PublicInbox/DirIdle.pm line 40.
> connect(/run/user/1000/lei/5.seq.sock): Connection refused (after
> attempted daemon start)
> ```
>
> Any ideas why this may occur? Is `lei` designed to only be ran as root
> or is Fedora installing perl in an odd fashion which results in root
> needing to be used?
What's the output of `ulimit -n` and `lsof -p $(lei daemon-pid)`?
(you may need to use `ps -ef |grep lei-daemon` to get the PID
if lei is broken and using too many FDs, though)
`ulimit -n' is the open file limit, typically 1024 or higher.
If `lsof -p $PID` may reveal a bug in lei which leaves too many
files open. lei (especially with inotify on Linux) should use
far less than 1024.
(FreeBSD may end up using far more open files, but that's a
different story)
^ permalink raw reply [relevance 71%]
* Re: Issues with `lei` as non-root
2023-03-28 1:32 71% ` Eric Wong
@ 2023-03-28 2:30 45% ` Louis DeLosSantos
2023-03-28 2:52 71% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Louis DeLosSantos @ 2023-03-28 2:30 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
> What's the output of `ulimit -n` and `lsof -p $(lei daemon-pid)`?
🖳 ulimit -n
1024
🖳 ps -ef |grep lei-daemon
root 861005 1 0 19:24 ? 00:00:00 lei-daemon
/tmp/lei-0/5.seq.sock
louis 1025477 1015489 0 22:22 pts/6 00:00:00 grep --color=auto lei-daemon
~
🖳 sudo lsof -p 861005
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
lei-daemo 861005 root cwd DIR 0,34 246
2902791 /home/louis/Mail/linux-bpf
lei-daemo 861005 root rtd DIR 0,34 158 256 /
lei-daemo 861005 root txt REG 0,34 15984
746278 /usr/bin/perl
lei-daemo 861005 root mem REG 0,32
746278 /usr/bin/perl (path dev=0,34)
lei-daemo 861005 root mem REG 0,32
669987 /usr/lib64/libstdc++.so.6.0.30 (path dev=0,34)
lei-daemo 861005 root mem REG 0,32
654072 /usr/lib64/libgcc_s-12-20221121.so.1 (path dev=0,34)
lei-daemo 861005 root mem REG 0,32
707780 /usr/lib64/libxapian.so.30.12.1 (path dev=0,34)
lei-daemo 861005 root mem REG 0,32
744976 /usr/lib64/perl5/vendor_perl/auto/Data/Dumper/Dumper.so (path
dev=0,34)
lei-daemo 861005 root mem REG 0,32
2847838 /usr/lib64/perl5/vendor_perl/auto/Search/Xapian/Xapian.so
(path dev=0,34)
lei-daemo 861005 root mem REG 0,32
670001 /usr/lib64/libz.so.1.2.12 (path dev=0,34)
lei-daemo 861005 root mem REG 0,32
670059 /usr/lib64/libsqlite3.so.0.8.6 (path dev=0,34)
lei-daemo 861005 root mem REG 0,32
1142922 /usr/lib/locale/locale-archive (path dev=0,34)
lei-daemo 861005 root mem REG 0,32
746285 /usr/lib64/perl5/auto/Sys/Hostname/Hostname.so (path dev=0,34)
lei-daemo 861005 root mem REG 0,32
670032 /usr/lib64/libuuid.so.1.3.0 (path dev=0,34)
lei-daemo 861005 root mem REG 0,32
2846948 /usr/lib64/perl5/vendor_perl/auto/Compress/Raw/Zlib/Zlib.so
(path dev=0,34)
lei-daemo 861005 root mem REG 0,32
2848056 /usr/lib64/perl5/vendor_perl/auto/DBD/SQLite/SQLite.so (path
dev=0,34)
lei-daemo 861005 root mem REG 0,32
748851 /usr/lib64/perl5/vendor_perl/auto/DBI/DBI.so (path dev=0,34)
lei-daemo 861005 root mem REG 0,32
745669 /usr/lib64/perl5/auto/attributes/attributes.so (path dev=0,34)
lei-daemo 861005 root mem REG 0,32
2847712 /usr/lib64/perl5/vendor_perl/auto/Cpanel/JSON/XS/XS.so (path
dev=0,34)
lei-daemo 861005 root mem REG 0,32
2848586 /usr/lib64/perl5/vendor_perl/auto/Linux/Inotify2/Inotify2.so
(path dev=0,34)
lei-daemo 861005 root mem REG 0,32
2849070 /usr/lib64/perl5/vendor_perl/auto/Socket/MsgHdr/MsgHdr.so
(path dev=0,34)
lei-daemo 861005 root mem REG 0,32
2847193 /usr/lib64/perl5/vendor_perl/auto/Sys/Syslog/Syslog.so (path
dev=0,34)
lei-daemo 861005 root mem REG 0,32
745481 /usr/lib64/perl5/vendor_perl/auto/MIME/Base64/Base64.so (path
dev=0,34)
lei-daemo 861005 root mem REG 0,32
745670 /usr/lib64/perl5/auto/re/re.so (path dev=0,34)
lei-daemo 861005 root mem REG 0,32
745506 /usr/lib64/perl5/vendor_perl/auto/Storable/Storable.so (path
dev=0,34)
lei-daemo 861005 root mem REG 0,32
745611 /usr/lib64/perl5/vendor_perl/auto/Encode/Encode.so (path
dev=0,34)
lei-daemo 861005 root mem REG 0,32
2849930 /usr/lib64/perl5/vendor_perl/auto/Email/Address/XS/XS.so (path
dev=0,34)
lei-daemo 861005 root mem REG 0,32
1143694 /usr/lib64/libm.so.6 (path dev=0,34)
lei-daemo 861005 root mem REG 0,32
1143691 /usr/lib64/libc.so.6 (path dev=0,34)
lei-daemo 861005 root mem REG 0,32
745650 /usr/lib64/libperl.so.5.36.0 (path dev=0,34)
lei-daemo 861005 root mem REG 0,32
2846914 /usr/lib64/perl5/vendor_perl/auto/Digest/SHA/SHA.so (path
dev=0,34)
lei-daemo 861005 root mem REG 0,32
745663 /usr/lib64/perl5/auto/File/Glob/Glob.so (path dev=0,34)
lei-daemo 861005 root mem REG 0,32
745449 /usr/lib64/perl5/auto/IO/IO.so (path dev=0,34)
lei-daemo 861005 root mem REG 0,32
745355 /usr/lib64/perl5/vendor_perl/auto/Socket/Socket.so (path
dev=0,34)
lei-daemo 861005 root mem REG 0,32
745492 /usr/lib64/perl5/vendor_perl/auto/List/Util/Util.so (path
dev=0,34)
lei-daemo 861005 root mem REG 0,32
745259 /usr/lib64/perl5/auto/POSIX/POSIX.so (path dev=0,34)
lei-daemo 861005 root mem REG 0,32
1019649 /usr/lib64/libcrypt.so.2.0.0 (path dev=0,34)
lei-daemo 861005 root mem REG 0,32
745556 /usr/lib64/perl5/vendor_perl/auto/Cwd/Cwd.so (path dev=0,34)
lei-daemo 861005 root mem REG 0,32
2846928 /usr/lib64/perl5/vendor_perl/auto/Time/HiRes/HiRes.so (path
dev=0,34)
lei-daemo 861005 root mem REG 0,32
745427 /usr/lib64/perl5/auto/Fcntl/Fcntl.so (path dev=0,34)
lei-daemo 861005 root mem REG 0,32
1143688 /usr/lib64/ld-linux-x86-64.so.2 (path dev=0,34)
lei-daemo 861005 root 0u REG 0,38 0
3448 /tmp/lei-0/errors.log
lei-daemo 861005 root 1u REG 0,38 0
3448 /tmp/lei-0/errors.log
lei-daemo 861005 root 2u REG 0,38 0
3448 /tmp/lei-0/errors.log
lei-daemo 861005 root 3u a_inode 0,14 0
15364 [eventpoll:4,5,7,8,14]
lei-daemo 861005 root 4u unix 0x0000000033b859c6 0t0
4095008 /tmp/lei-0/5.seq.sock type=SEQPACKET (LISTEN)
lei-daemo 861005 root 5u unix 0x000000007a56c532 0t0
4085549 type=SEQPACKET (CONNECTED)
lei-daemo 861005 root 6u unix 0x00000000c6d0e27b 0t0
4085550 type=SEQPACKET (CONNECTED)
lei-daemo 861005 root 7r a_inode 0,14 0 15364 inotify
lei-daemo 861005 root 8u a_inode 0,14 0
15364 [signalfd]
lei-daemo 861005 root 14r FIFO 0,13 0t0 4098533 pipe
lei-daemo 861005 root 16u unix 0x00000000557447d0 0t0
4098534 type=SEQPACKET (CONNECTED)
lei-daemo 861005 root 17u unix 0x00000000f60edd15 0t0
4098535 type=SEQPACKET (CONNECTED)
Above is 54 open sockets. Which seems fine.
Should daemon be running as root, if I intend to only use lei as user?
On Mon, Mar 27, 2023 at 9:32 PM Eric Wong <e@80x24.org> wrote:
>
> Louis DeLosSantos <louis.delos@gmail.com> wrote:
> > Hello,
> >
> > I'm experimenting with `lei` as a nice search tool for `lore.kernel.org`
> >
> > Everything works fine with the caveat that it seems to break if I'm not root.
> >
> > When using `lei` as non-root we get this error:
>
> I've never used lei as root nor has any part of public-inbox
> ever been intended to run as root.
>
> > ```
> > E: Linux::Inotify2->new: Too many open files at
> > /usr/share/perl5/vendor_perl/PublicInbox/DirIdle.pm line 40.
> > connect(/run/user/1000/lei/5.seq.sock): Connection refused (after
> > attempted daemon start)
> > ```
> >
> > Any ideas why this may occur? Is `lei` designed to only be ran as root
> > or is Fedora installing perl in an odd fashion which results in root
> > needing to be used?
>
> What's the output of `ulimit -n` and `lsof -p $(lei daemon-pid)`?
>
> (you may need to use `ps -ef |grep lei-daemon` to get the PID
> if lei is broken and using too many FDs, though)
>
> `ulimit -n' is the open file limit, typically 1024 or higher.
>
> If `lsof -p $PID` may reveal a bug in lei which leaves too many
> files open. lei (especially with inotify on Linux) should use
> far less than 1024.
>
> (FreeBSD may end up using far more open files, but that's a
> different story)
^ permalink raw reply [relevance 45%]
* Re: Issues with `lei` as non-root
2023-03-28 2:30 45% ` Louis DeLosSantos
@ 2023-03-28 2:52 71% ` Eric Wong
2023-03-28 3:05 42% ` Louis DeLosSantos
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2023-03-28 2:52 UTC (permalink / raw)
To: Louis DeLosSantos; +Cc: meta
Louis DeLosSantos <louis.delos@gmail.com> wrote:
<snip>
> Above is 54 open sockets. Which seems fine.
Actually, only 18 (0..17). The `mem' stuff is mmap-ed and
doesn't count against `ulimit -n` (RLIMIT_NOFILE).
> Should daemon be running as root, if I intend to only use lei as user?
Definitely not; the lei-daemon is per-user.
I also forgot, inotify has its own per-user limits; perhaps
you're hitting those?
# show system-wide limits
$ head /proc/sys/fs/inotify/max_*
# show per-user inotify FDs (-nP speeds up lsof by avoiding lookups)
$ lsof -nP -u $USER |grep inotify
^ permalink raw reply [relevance 71%]
* Re: Issues with `lei` as non-root
2023-03-28 2:52 71% ` Eric Wong
@ 2023-03-28 3:05 42% ` Louis DeLosSantos
2023-03-28 3:38 71% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Louis DeLosSantos @ 2023-03-28 3:05 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
> Actually, only 18 (0..17). The `mem' stuff is mmap-ed and
doesn't count against `ulimit -n` (RLIMIT_NOFILE).
Yup, you are right, did a quick "wc -l" without thinking about it.
> Definitely not; the lei-daemon is per-user.
Okay, maybe this is the issue to begin with? I installed lei from dnf.
I'm not sure what launches the daemon, is it launched on first run?
If that is the case, it was probably launched when I restored to `sudo
lei q ....` command.
But, if its running as systemd service, I could move it to user service.
# show system-wide limits
~
🖳 head /proc/sys/fs/inotify/max_*
==> /proc/sys/fs/inotify/max_queued_events <==
16384
==> /proc/sys/fs/inotify/max_user_instances <==
128
==> /proc/sys/fs/inotify/max_user_watches <==
524288
# show per-user inotify FDs (-nP speeds up lsof by avoiding lookups)
systemd 1686 louis 6r a_inode 0,14 0
15364 inotify
systemd 1686 louis 11r a_inode 0,14 0
15364 inotify
systemd 1686 louis 13r a_inode 0,14 0
15364 inotify
dbus-brok 1727 louis 7r a_inode 0,14 0
15364 inotify
swaync 1868 louis 15r a_inode 0,14 0
15364 inotify
dbus-brok 1898 louis 7r a_inode 0,14 0
15364 inotify
xdg-deskt 1914 louis 10r a_inode 0,14 0
15364 inotify
xdg-deskt 1938 louis 12r a_inode 0,14 0
15364 inotify
xdg-deskt 1968 louis 14r a_inode 0,14 0
15364 inotify
wireplumb 1979 louis 19r a_inode 0,14 0
15364 inotify
wireplumb 1979 louis 28r a_inode 0,14 0
15364 inotify
wireplumb 1979 louis 30r a_inode 0,14 0
15364 inotify
code 2362 louis 63r a_inode 0,14 0
15364 inotify
code 2362 louis 86r a_inode 0,14 0
15364 inotify
code 2362 louis 109r a_inode 0,14 0
15364 inotify
code 2456 louis 21r a_inode 0,14 0
15364 inotify
code 2493 louis 51r a_inode 0,14 0
15364 inotify
code 2547 louis 49r a_inode 0,14 0
15364 inotify
code 2547 louis 54r a_inode 0,14 0
15364 inotify
firefox 4168 louis 80r a_inode 0,14 0
15364 inotify
cgroupify 4347 louis 6r a_inode 0,14 0
15364 inotify
flatpak-s 53051 louis 7r a_inode 0,14 0
15364 inotify
obsidian 53073 louis 69r a_inode 0,14 0
15364 inotify
flatpak-p 53080 louis 7r a_inode 0,14 0
15364 inotify
obsidian 53135 louis 21r a_inode 0,14 0
15364 inotify
obsidian 53141 louis 56r a_inode 0,14 0
15364 inotify
tail 235359 louis 4r a_inode 0,14 0
15364 inotify
tail 251716 louis 4r a_inode 0,14 0
15364 inotify
tail 251821 louis 4r a_inode 0,14 0
15364 inotify
tail 252069 louis 4r a_inode 0,14 0
15364 inotify
tail 252266 louis 4r a_inode 0,14 0
15364 inotify
tail 252399 louis 4r a_inode 0,14 0
15364 inotify
tail 252434 louis 4r a_inode 0,14 0
15364 inotify
tail 252749 louis 4r a_inode 0,14 0
15364 inotify
tail 252953 louis 4r a_inode 0,14 0
15364 inotify
tail 253611 louis 4r a_inode 0,14 0
15364 inotify
tail 253772 louis 4r a_inode 0,14 0
15364 inotify
tail 253858 louis 4r a_inode 0,14 0
15364 inotify
tail 253932 louis 4r a_inode 0,14 0
15364 inotify
tail 254159 louis 4r a_inode 0,14 0
15364 inotify
tail 254323 louis 4r a_inode 0,14 0
15364 inotify
tail 254425 louis 4r a_inode 0,14 0
15364 inotify
tail 255070 louis 4r a_inode 0,14 0
15364 inotify
tail 255592 louis 4r a_inode 0,14 0
15364 inotify
tail 256816 louis 4r a_inode 0,14 0
15364 inotify
tail 256939 louis 4r a_inode 0,14 0
15364 inotify
tail 257302 louis 4r a_inode 0,14 0
15364 inotify
tail 257435 louis 4r a_inode 0,14 0
15364 inotify
tail 257746 louis 4r a_inode 0,14 0
15364 inotify
tail 258071 louis 4r a_inode 0,14 0
15364 inotify
tail 258169 louis 4r a_inode 0,14 0
15364 inotify
tail 258279 louis 4r a_inode 0,14 0
15364 inotify
tail 258413 louis 4r a_inode 0,14 0
15364 inotify
tail 258715 louis 4r a_inode 0,14 0
15364 inotify
tail 259167 louis 4r a_inode 0,14 0
15364 inotify
tail 259289 louis 4r a_inode 0,14 0
15364 inotify
tail 259484 louis 4r a_inode 0,14 0
15364 inotify
tail 259623 louis 4r a_inode 0,14 0
15364 inotify
tail 259876 louis 4r a_inode 0,14 0
15364 inotify
tail 260275 louis 4r a_inode 0,14 0
15364 inotify
tail 260412 louis 4r a_inode 0,14 0
15364 inotify
tail 260518 louis 4r a_inode 0,14 0
15364 inotify
tail 260650 louis 4r a_inode 0,14 0
15364 inotify
tail 260818 louis 4r a_inode 0,14 0
15364 inotify
tail 261665 louis 4r a_inode 0,14 0
15364 inotify
tail 262636 louis 4r a_inode 0,14 0
15364 inotify
tail 262963 louis 4r a_inode 0,14 0
15364 inotify
tail 263781 louis 4r a_inode 0,14 0
15364 inotify
tail 264183 louis 4r a_inode 0,14 0
15364 inotify
tail 264242 louis 4r a_inode 0,14 0
15364 inotify
tail 264484 louis 4r a_inode 0,14 0
15364 inotify
tail 264769 louis 4r a_inode 0,14 0
15364 inotify
tail 264970 louis 4r a_inode 0,14 0
15364 inotify
tail 265488 louis 4r a_inode 0,14 0
15364 inotify
tail 265939 louis 4r a_inode 0,14 0
15364 inotify
tail 266145 louis 4r a_inode 0,14 0
15364 inotify
tail 267183 louis 4r a_inode 0,14 0
15364 inotify
tail 267352 louis 4r a_inode 0,14 0
15364 inotify
tail 267951 louis 4r a_inode 0,14 0
15364 inotify
tail 268135 louis 4r a_inode 0,14 0
15364 inotify
tail 268320 louis 4r a_inode 0,14 0
15364 inotify
tail 268892 louis 4r a_inode 0,14 0
15364 inotify
tail 269418 louis 4r a_inode 0,14 0
15364 inotify
tail 269631 louis 4r a_inode 0,14 0
15364 inotify
tail 339541 louis 4r a_inode 0,14 0
15364 inotify
tail 339854 louis 4r a_inode 0,14 0
15364 inotify
tail 340349 louis 4r a_inode 0,14 0
15364 inotify
tail 340566 louis 4r a_inode 0,14 0
15364 inotify
tail 340712 louis 4r a_inode 0,14 0
15364 inotify
tail 340858 louis 4r a_inode 0,14 0
15364 inotify
tail 341122 louis 4r a_inode 0,14 0
15364 inotify
tail 341356 louis 4r a_inode 0,14 0
15364 inotify
tail 341499 louis 4r a_inode 0,14 0
15364 inotify
tail 341643 louis 4r a_inode 0,14 0
15364 inotify
tail 341858 louis 4r a_inode 0,14 0
15364 inotify
tail 341948 louis 4r a_inode 0,14 0
15364 inotify
tail 342271 louis 4r a_inode 0,14 0
15364 inotify
tail 342410 louis 4r a_inode 0,14 0
15364 inotify
tail 342557 louis 4r a_inode 0,14 0
15364 inotify
tail 342802 louis 4r a_inode 0,14 0
15364 inotify
tail 342944 louis 4r a_inode 0,14 0
15364 inotify
tail 343409 louis 4r a_inode 0,14 0
15364 inotify
tail 343690 louis 4r a_inode 0,14 0
15364 inotify
tail 343864 louis 4r a_inode 0,14 0
15364 inotify
tail 344049 louis 4r a_inode 0,14 0
15364 inotify
tail 344285 louis 4r a_inode 0,14 0
15364 inotify
tail 344429 louis 4r a_inode 0,14 0
15364 inotify
tail 345251 louis 4r a_inode 0,14 0
15364 inotify
tail 345541 louis 4r a_inode 0,14 0
15364 inotify
tail 345972 louis 4r a_inode 0,14 0
15364 inotify
tail 346163 louis 4r a_inode 0,14 0
15364 inotify
tail 346755 louis 4r a_inode 0,14 0
15364 inotify
tail 347063 louis 4r a_inode 0,14 0
15364 inotify
tail 350273 louis 4r a_inode 0,14 0
15364 inotify
tail 350316 louis 4r a_inode 0,14 0
15364 inotify
tail 350781 louis 4r a_inode 0,14 0
15364 inotify
tail 350942 louis 4r a_inode 0,14 0
15364 inotify
tail 351111 louis 4r a_inode 0,14 0
15364 inotify
tail 351746 louis 4r a_inode 0,14 0
15364 inotify
tail 353489 louis 4r a_inode 0,14 0
15364 inotify
tail 353801 louis 4r a_inode 0,14 0
15364 inotify
tail 354352 louis 4r a_inode 0,14 0
15364 inotify
tail 354638 louis 4r a_inode 0,14 0
15364 inotify
tail 354923 louis 4r a_inode 0,14 0
15364 inotify
tail 362029 louis 4r a_inode 0,14 0
15364 inotify
tail 367093 louis 4r a_inode 0,14 0
15364 inotify
waybar 811099 louis 37r a_inode 0,14 0
15364 inotify
waybar 811099 louis 38r a_inode 0,14 0
15364 inotify
On Mon, Mar 27, 2023 at 10:52 PM Eric Wong <e@80x24.org> wrote:
>
> Louis DeLosSantos <louis.delos@gmail.com> wrote:
>
> <snip>
>
> > Above is 54 open sockets. Which seems fine.
>
> Actually, only 18 (0..17). The `mem' stuff is mmap-ed and
> doesn't count against `ulimit -n` (RLIMIT_NOFILE).
>
> > Should daemon be running as root, if I intend to only use lei as user?
>
> Definitely not; the lei-daemon is per-user.
>
> I also forgot, inotify has its own per-user limits; perhaps
> you're hitting those?
>
> # show system-wide limits
> $ head /proc/sys/fs/inotify/max_*
>
> # show per-user inotify FDs (-nP speeds up lsof by avoiding lookups)
> $ lsof -nP -u $USER |grep inotify
^ permalink raw reply [relevance 42%]
* Re: Issues with `lei` as non-root
2023-03-28 3:05 42% ` Louis DeLosSantos
@ 2023-03-28 3:38 71% ` Eric Wong
2023-03-28 4:08 71% ` Louis DeLosSantos
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2023-03-28 3:38 UTC (permalink / raw)
To: Louis DeLosSantos; +Cc: meta
Louis DeLosSantos <louis.delos@gmail.com> wrote:
> > Definitely not; the lei-daemon is per-user.
>
> Okay, maybe this is the issue to begin with? I installed lei from dnf.
> I'm not sure what launches the daemon, is it launched on first run?
>
> If that is the case, it was probably launched when I restored to `sudo
> lei q ....` command.
> But, if its running as systemd service, I could move it to user service.
You shouldn't need to manage it as a service; it's auto-started
and killing it is harmless in most cases. I'm considering it
have it auto-exit if it stays idle for a long time and there's
no active inotify watches.
lei-daemon doesn't start until any other lei command is invoked;
so it shouldn't be started on installation.
> # show system-wide limits
> ==> /proc/sys/fs/inotify/max_user_instances <==
> 128
<snip>
> tail 367093 louis 4r a_inode 0,14 0
Thats a lot of tail processes....
I wonder if they were spawned by `lei q -v' for emitting curl stderr?
They should be auto-killed.
(or if you have some other reason for running tail on your system).
^ permalink raw reply [relevance 71%]
* Re: Issues with `lei` as non-root
2023-03-28 3:38 71% ` Eric Wong
@ 2023-03-28 4:08 71% ` Louis DeLosSantos
0 siblings, 0 replies; 200+ results
From: Louis DeLosSantos @ 2023-03-28 4:08 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
> Thats a lot of tail processes....
Ugh, sorry to waste your time.
All the tails were from a run away UI program I'm working on.
Once I killed them, `let` runs just fine as non-root.
Thanks for the free tech support, hope I didn't steal your attention
from something valuable :-D.
On Mon, Mar 27, 2023 at 11:38 PM Eric Wong <e@80x24.org> wrote:
>
> Louis DeLosSantos <louis.delos@gmail.com> wrote:
> > > Definitely not; the lei-daemon is per-user.
> >
> > Okay, maybe this is the issue to begin with? I installed lei from dnf.
> > I'm not sure what launches the daemon, is it launched on first run?
> >
> > If that is the case, it was probably launched when I restored to `sudo
> > lei q ....` command.
> > But, if its running as systemd service, I could move it to user service.
>
> You shouldn't need to manage it as a service; it's auto-started
> and killing it is harmless in most cases. I'm considering it
> have it auto-exit if it stays idle for a long time and there's
> no active inotify watches.
>
> lei-daemon doesn't start until any other lei command is invoked;
> so it shouldn't be started on installation.
>
> > # show system-wide limits
>
> > ==> /proc/sys/fs/inotify/max_user_instances <==
> > 128
>
> <snip>
>
> > tail 367093 louis 4r a_inode 0,14 0
>
> Thats a lot of tail processes....
> I wonder if they were spawned by `lei q -v' for emitting curl stderr?
> They should be auto-killed.
> (or if you have some other reason for running tail on your system).
^ permalink raw reply [relevance 71%]
* [PATCH] t/lei-refresh-mail-sync: improve test reliability
@ 2023-03-28 10:53 71% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2023-03-28 10:53 UTC (permalink / raw)
To: meta
Lack of signalfd/EVFILT_SIGNAL means we need to kill a
process repeatedly to ensure it wakes up.
---
t/lei-refresh-mail-sync.t | 8 ++++++--
1 file changed, 6 insertions(+), 2 deletions(-)
diff --git a/t/lei-refresh-mail-sync.t b/t/lei-refresh-mail-sync.t
index 0498a0c4..8ccc68c6 100644
--- a/t/lei-refresh-mail-sync.t
+++ b/t/lei-refresh-mail-sync.t
@@ -137,8 +137,12 @@ SKIP: {
my $ar = PublicInbox::AutoReap->new($pid);
ok(!(lei 'refresh-mail-sync', $url), 'URL fails on dead -imapd');
ok(!(lei 'refresh-mail-sync', '--all'), '--all fails on dead -imapd');
- $ar->kill for qw(avoid sig wake miss-no signalfd or EVFILT_SIG);
- $ar->join('TERM');
+ {
+ local $SIG{CHLD} = sub { $ar->join('TERM'); undef $ar };
+ do {
+ eval { $ar->kill and tick(0.01) }
+ } while (defined($ar));
+ }
my $cmd = $srv->{imapd}->{cmd};
my $s = $srv->{imapd}->{s};
^ permalink raw reply related [relevance 71%]
* [PATCH] t/lei-import-nntp: dump $lei_err on failure
@ 2023-04-29 7:18 71% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2023-04-29 7:18 UTC (permalink / raw)
To: meta
I hit an error on the backwards range import test and can't
reproduce it, perhaps dumping $lei_err can help diagnose it
in the future.
---
t/lei-import-nntp.t | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/t/lei-import-nntp.t b/t/lei-import-nntp.t
index eb1ae312..2c48d973 100644
--- a/t/lei-import-nntp.t
+++ b/t/lei-import-nntp.t
@@ -43,7 +43,8 @@ test_lei({ tmpdir => $tmpdir }, sub {
lei_ok 'ls-mail-sync';
like($lei_out, qr!\A\Q$url\E\n\z!, 'ls-mail-sync output as-expected');
- ok(!lei(qw(import), "$url/12-1"), 'backwards range rejected');
+ ok(!lei(qw(import), "$url/12-1"), 'backwards range rejected') or
+ diag $lei_err;
# new home
local $ENV{HOME} = "$tmpdir/h2";
^ permalink raw reply related [relevance 71%]
* [PATCH] t/lei.t: quiet newline warning on older Perls
@ 2023-06-08 18:26 71% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2023-06-08 18:26 UTC (permalink / raw)
To: meta
Perl < 5.22 warned on newlines in the middle of a string instead
of just the end. Workaround it by disabling all warnings on older
Perls while running File::Path::mkpath.
---
t/lei.t | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/t/lei.t b/t/lei.t
index a80143ef..5d0fa622 100644
--- a/t/lei.t
+++ b/t/lei.t
@@ -149,7 +149,10 @@ my $test_fail = sub {
for my $lk (qw(ei inbox)) {
my $d = "$home/newline\n$lk";
my $all = $lk eq 'ei' ? 'ALL' : 'all';
- File::Path::mkpath("$d/$all.git/objects");
+ { # quiet newline warning on older Perls
+ local $^W = undef if $^V lt v5.22.0;
+ File::Path::mkpath("$d/$all.git/objects");
+ }
open my $fh, '>', "$d/$lk.lock" or BAIL_OUT "open $d/$lk.lock";
for my $fl (qw(-I --only)) {
ok(!lei('q', $fl, $d, 'whatever'),
^ permalink raw reply related [relevance 71%]
* [PATCH] doc: lei q: document v2:$INBOX_DIR output format
@ 2023-06-15 0:08 71% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2023-06-15 0:08 UTC (permalink / raw)
To: meta
This has been supported in every lei release, actually.
---
Documentation/lei-q.pod | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/Documentation/lei-q.pod b/Documentation/lei-q.pod
index 5e9a5658..c0254ba0 100644
--- a/Documentation/lei-q.pod
+++ b/Documentation/lei-q.pod
@@ -50,6 +50,10 @@ A prefix can specify the format of the output: C<maildir>,
C<mboxrd>, C<mboxcl2>, C<mboxcl>, C<mboxo>. For a description of
mail formats, see L<lei-mail-formats(5)>.
+C<v2:/path/to/inbox> may be used to create a new inbox of
+L<public-inbox-v2-format(5)>. The new inbox will not be configured
+in the L<public-inbox-config(5)> file.
+
C<maildir> is the default for an existing directory or non-existing path.
Default: C<-> (stdout)
^ permalink raw reply related [relevance 71%]
* [PATCH] lei import: set +(L|kw) on already-imported blobs
@ 2023-06-15 8:46 60% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2023-06-15 8:46 UTC (permalink / raw)
To: meta
When import hits blobs it's already seen, we'll add labels
regardless in order to match the behavior of other inexact
matches. This is useful when importing exact copies of
messages which exist in multiple mailboxes.
I noticed this when I had a message imported from my normal IMAP
`INBOX', but also copied it to a different folder for future
reference.
---
Documentation/RelNotes/v2.0.0.wip | 3 +++
lib/PublicInbox/LeiStore.pm | 8 +++++++-
t/lei-import.t | 17 ++++++++++++++++-
3 files changed, 26 insertions(+), 2 deletions(-)
diff --git a/Documentation/RelNotes/v2.0.0.wip b/Documentation/RelNotes/v2.0.0.wip
index cd90bdae..cccf11ae 100644
--- a/Documentation/RelNotes/v2.0.0.wip
+++ b/Documentation/RelNotes/v2.0.0.wip
@@ -60,6 +60,9 @@ lei
* fix `lei q -tt' on locally-indexed messages (still broken for remotes:
https://public-inbox.org/meta/20230226170931.M947721@dcvr/ )
+ * `lei import' now set labels+keywords consistently on all
+ already-imported messages
+
solver (used by lei (rediff|blob), and PublicInbox::WWW)
* handle copies in patches properly
diff --git a/lib/PublicInbox/LeiStore.pm b/lib/PublicInbox/LeiStore.pm
index cf5a03a0..727de066 100644
--- a/lib/PublicInbox/LeiStore.pm
+++ b/lib/PublicInbox/LeiStore.pm
@@ -387,8 +387,14 @@ sub add_eml {
_lms_rw($self)->set_src($smsg->oidbin, @{$vmd->{sync_info}});
}
unless ($im_mark) { # duplicate blob returns undef
- return unless wantarray;
+ return unless wantarray || $vmd;
my @docids = $oidx->blob_exists($smsg->{blob});
+ if ($vmd) {
+ for my $docid (@docids) {
+ my $idx = $eidx->idx_shard($docid);
+ _add_vmd($self, $idx, $docid, $vmd);
+ }
+ }
return _docids_and_maybe_kw $self, \@docids;
}
diff --git a/t/lei-import.t b/t/lei-import.t
index 6e9a853c..c9e668a3 100644
--- a/t/lei-import.t
+++ b/t/lei-import.t
@@ -1,5 +1,5 @@
#!perl -w
-# Copyright (C) 2020-2021 all contributors <meta@public-inbox.org>
+# Copyright (C) all contributors <meta@public-inbox.org>
# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
use strict; use v5.10.1; use PublicInbox::TestCommon;
test_lei(sub {
@@ -110,6 +110,21 @@ $res = json_utf8->decode($lei_out);
is_deeply($res->[0]->{kw}, ['seen'], 'keyword set');
is_deeply($res->[0]->{L}, ['inbox'], 'label set');
+# idempotent import can add label
+lei_ok([qw(import -F eml - +L:boombox)],
+ undef, { %$lei_opt, 0 => \$eml_str });
+lei_ok(qw(q m:inbox@example.com));
+$res = json_utf8->decode($lei_out);
+is_deeply($res->[0]->{kw}, ['seen'], 'keyword remains set');
+is_deeply($res->[0]->{L}, [qw(boombox inbox)], 'new label added');
+
+# idempotent import can add keyword
+lei_ok([qw(import -F eml - +kw:answered)],
+ undef, { %$lei_opt, 0 => \$eml_str });
+lei_ok(qw(q m:inbox@example.com));
+$res = json_utf8->decode($lei_out);
+is_deeply($res->[0]->{kw}, [qw(answered seen)], 'keyword added');
+is_deeply($res->[0]->{L}, [qw(boombox inbox)], 'labels preserved');
# see t/lei_to_mail.t for "import -F mbox*"
});
^ permalink raw reply related [relevance 60%]
* [PATCH] lei: make --dedupe=content always account for Message-IDs
@ 2023-06-15 9:50 51% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2023-06-15 9:50 UTC (permalink / raw)
To: meta
The content dedupe logic was originally designed for v2 public
inboxes as a fallback for when the importer sees identical
Message-IDs. Thus it did not account for Message-ID(s) in
the message itself.
This change doesn't affect saved searches (the default when
writing to a pathname or IMAP). It affects --no-save, and
outputs to stdout (even if stdout is redirected to a file).
Prior to this change, lei reused the v2 logic as-is without
accounting for Message-IDs anywhere with `--dedupe=content'
(the default). This could cause messages to be skipped when
the content matches despite Message-IDs being different.
So with this change, `lei q --dedupe=content' will hash the
Message-ID(s) in the message to ensure messages with different
Message-IDs are NOT deduplicated.
Whether or not this change is a bug fix or introduces regression
is actually debatable. In my mind, it is better to err on the
side of showing too many messages rather than too few, even if
the actual contents of the message are identical. Making saved
searches deduplicate without accounting for Message-IDs would be
more difficult, too.
---
lib/PublicInbox/ContentHash.pm | 15 +++++++++++----
lib/PublicInbox/LeiDedupe.pm | 9 +++++++--
t/lei_dedupe.t | 6 +++++-
3 files changed, 23 insertions(+), 7 deletions(-)
diff --git a/lib/PublicInbox/ContentHash.pm b/lib/PublicInbox/ContentHash.pm
index fc94257c..95ca2929 100644
--- a/lib/PublicInbox/ContentHash.pm
+++ b/lib/PublicInbox/ContentHash.pm
@@ -54,16 +54,23 @@ sub content_dig_i {
$dig->add($s);
}
-sub content_digest ($;$) {
- my ($eml, $dig) = @_;
+sub content_digest ($;$$) {
+ my ($eml, $dig, $hash_mids) = @_;
$dig //= Digest::SHA->new(256);
# References: and In-Reply-To: get used interchangeably
# in some "duplicates" in LKML. We treat them the same
# in SearchIdx, so treat them the same for this:
# do NOT consider the Message-ID as part of the content_hash
- # if we got here, we've already got Message-ID reuse
- my %seen = map { $_ => 1 } @{mids($eml)};
+ # if we got here, we've already got Message-ID reuse for v2.
+ #
+ # However, `lei q --dedupe=content' does use $hash_mids since
+ # it doesn't have any other dedupe
+ my $mids = mids($eml);
+ if ($hash_mids) {
+ $dig->add("mid\0$_\0") for @$mids;
+ }
+ my %seen = map { $_ => 1 } @$mids;
for (grep { !$seen{$_}++ } @{references($eml)}) {
utf8::encode($_);
$dig->add("ref\0$_\0");
diff --git a/lib/PublicInbox/LeiDedupe.pm b/lib/PublicInbox/LeiDedupe.pm
index 86cd8490..eda54d79 100644
--- a/lib/PublicInbox/LeiDedupe.pm
+++ b/lib/PublicInbox/LeiDedupe.pm
@@ -2,7 +2,7 @@
# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
package PublicInbox::LeiDedupe;
use v5.12;
-use PublicInbox::ContentHash qw(content_hash git_sha);
+use PublicInbox::ContentHash qw(content_hash content_digest git_sha);
use PublicInbox::SHA qw(sha256);
# n.b. mutt sets most of these headers not sure about Bytes
@@ -69,7 +69,12 @@ sub dedupe_content ($) {
my ($skv) = @_;
(sub { # may be called in a child process
my ($eml) = @_; # $oidhex = $_[1], ignored
- $skv->set_maybe(content_hash($eml), '');
+
+ # we must account for Message-ID via hash_mids, since
+ # (unlike v2 dedupe) Message-ID is not accounted for elsewhere:
+ $skv->set_maybe(content_digest($eml, PublicInbox::SHA->new(256),
+ 1 # hash_mids
+ )->digest, '');
}, sub {
my ($smsg) = @_;
$skv->set_maybe(smsg_hash($smsg), '');
diff --git a/t/lei_dedupe.t b/t/lei_dedupe.t
index e1944d02..13fc1f3b 100644
--- a/t/lei_dedupe.t
+++ b/t/lei_dedupe.t
@@ -1,5 +1,5 @@
#!perl -w
-# Copyright (C) 2020-2021 all contributors <meta@public-inbox.org>
+# Copyright (C) all contributors <meta@public-inbox.org>
# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
use strict;
use v5.10.1;
@@ -10,6 +10,8 @@ use PublicInbox::Smsg;
require_mods(qw(DBD::SQLite));
use_ok 'PublicInbox::LeiDedupe';
my $eml = eml_load('t/plack-qp.eml');
+my $sameish = eml_load('t/plack-qp.eml');
+$sameish->header_set('Message-ID', '<cuepee@example.com>');
my $mid = $eml->header_raw('Message-ID');
my $different = eml_load('t/msg_iter-order.eml');
$different->header_set('Message-ID', $mid);
@@ -47,6 +49,8 @@ for my $strat (undef, 'content') {
ok(!$dd->is_dup($different), "different is_dup with $desc dedupe");
ok(!$dd->is_smsg_dup($smsg), "is_smsg_dup pass w/ $desc dedupe");
ok($dd->is_smsg_dup($smsg), "is_smsg_dup reject w/ $desc dedupe");
+ ok(!$dd->is_dup($sameish),
+ "Message-ID accounted for w/ same content otherwise");
}
$lei->{opt}->{dedupe} = 'bogus';
eval { PublicInbox::LeiDedupe->new($lei) };
^ permalink raw reply related [relevance 51%]
* [PATCH] t/lei-mirror: do not bail out on `make help' failure
@ 2023-09-14 12:12 71% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2023-09-14 12:12 UTC (permalink / raw)
To: meta
I'm not sure why, but this test occasionally fails on OpenBSD
and I can't reproduce it on a repeatable basis. In any case,
there's no reason we can't continue the rest of the test if
`make help' fails.
---
t/lei-mirror.t | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/t/lei-mirror.t b/t/lei-mirror.t
index 2400578a..9b5d73ec 100644
--- a/t/lei-mirror.t
+++ b/t/lei-mirror.t
@@ -27,7 +27,8 @@ test_lei({ tmpdir => $tmpdir }, sub {
"mirror of $http/t1/\n", 'description set');
ok(-f "$t1/Makefile", 'convenience Makefile added (v1)');
my $make = which('make');
- xsys_e([$make, 'help'], undef, { -C => $t1, 1 => \(my $help) });
+ is(xsys([$make, 'help'], undef, { -C => $t1, 1 => \(my $help) }), 0,
+ 'make help');
ok(-f "$t1/inbox.config.example", 'inbox.config.example downloaded');
isnt($help, '', 'make help worked');
is((stat(_))[9], $created{v1},
^ permalink raw reply related [relevance 71%]
* [PATCH] lei: ensure --stdin sets %ENV and $current_lei
@ 2023-09-14 23:10 64% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2023-09-14 23:10 UTC (permalink / raw)
To: meta
--stdin usage means the current request can be delayed
indefinitely while other requests with different %ENV
come in. So make sure our warnings and %ENV can match
non-stdin behavior.
This probably fix segfaults during process cleanup on OpenBSD
since _lei_atfork_child use non-localized assignment of
$current_lei. But it could be another red herring. Either way,
it's the right thing to do from an environment replication
perspective.
---
lib/PublicInbox/LeiInspect.pm | 3 +++
lib/PublicInbox/LeiLcat.pm | 2 ++
lib/PublicInbox/LeiQuery.pm | 2 ++
3 files changed, 7 insertions(+)
diff --git a/lib/PublicInbox/LeiInspect.pm b/lib/PublicInbox/LeiInspect.pm
index d1dca4ef..0455e739 100644
--- a/lib/PublicInbox/LeiInspect.pm
+++ b/lib/PublicInbox/LeiInspect.pm
@@ -255,6 +255,9 @@ sub ins_add { # InputPipe->consume callback
my ($lei) = @_; # $_[1] = $rbuf
if (defined $_[1]) {
$_[1] eq '' and return eval {
+ $lei->fchdir;
+ local %ENV = %{$lei->{env}};
+ local $PublicInbox::LEI::current_lei = $lei;
my $str = delete $lei->{istr};
$str =~ s/\A[\r\n]*From [^\r\n]*\r?\n//s;
my $eml = PublicInbox::Eml->new(\$str);
diff --git a/lib/PublicInbox/LeiLcat.pm b/lib/PublicInbox/LeiLcat.pm
index 8d89cb73..7ed191c3 100644
--- a/lib/PublicInbox/LeiLcat.pm
+++ b/lib/PublicInbox/LeiLcat.pm
@@ -128,6 +128,8 @@ sub _stdin { # PublicInbox::InputPipe::consume callback for --stdin
return $lei->{mset_opt}->{qstr} .= $_[1] if $_[1] ne '';
eval {
$lei->fchdir;
+ local %ENV = %{$lei->{env}};
+ local $PublicInbox::LEI::current_lei = $lei;
my @argv = split(/\s+/, $lei->{mset_opt}->{qstr});
$lei->{mset_opt}->{qstr} = extract_all($lei, @argv) or return;
$lei->_start_query;
diff --git a/lib/PublicInbox/LeiQuery.pm b/lib/PublicInbox/LeiQuery.pm
index 26cfb3fd..a23354f0 100644
--- a/lib/PublicInbox/LeiQuery.pm
+++ b/lib/PublicInbox/LeiQuery.pm
@@ -65,6 +65,8 @@ sub qstr_add { # PublicInbox::InputPipe::consume callback for --stdin
return $lei->{mset_opt}->{qstr} .= $_[1] if $_[1] ne '';
eval {
$lei->fchdir;
+ local %ENV = %{$lei->{env}};
+ local $PublicInbox::LEI::current_lei = $lei;
$lei->{mset_opt}->{q_raw} = $lei->{mset_opt}->{qstr};
$lei->{lse}->query_approxidate($lei->{lse}->git,
$lei->{mset_opt}->{qstr});
^ permalink raw reply related [relevance 64%]
* [PATCH] lei: ensure we run DESTROY|END at daemon exit w/ kqueue
@ 2023-09-15 10:11 63% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2023-09-15 10:11 UTC (permalink / raw)
To: meta
The fundamental difference which I originally missed when
implementing kqueue EVFILT_SIGNAL support is that it does not
consume signals like signalfd(2) does. In other words, with
EVFILT_SIGNAL, it's possible for a single signal to be delivered
twice if we unblock signals upon leaving the event loop as we do
in lei.
Note: Our DS->event_loop and Sigfd APIs can/should probably be
changed to better accomodate EVFILT_SIGNAL differences from
signalfd without sacrificing usability of either.
This fixes the problem of leftover lei-ovv.dst*, lei_cfg-* and
skv.* files in $TMPDIR at the end of test suite runs on *BSD
when IO::KQueue is installed.
---
lib/PublicInbox/LEI.pm | 14 +++++++++++++-
1 file changed, 13 insertions(+), 1 deletion(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 5fbb1211..c61ce76d 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -1318,6 +1318,9 @@ sub lazy_start {
USR1 => \&noop,
USR2 => \&noop,
};
+ # for EVFILT_SIGNAL and signalfd behavioral difference:
+ my @kq_ign = eval { require PublicInbox::DSKQXS } ? keys(%$sig) : ();
+
require PublicInbox::DirIdle;
local $dir_idle = PublicInbox::DirIdle->new(sub {
# just rely on wakeup to hit post_loop_do
@@ -1356,13 +1359,22 @@ sub lazy_start {
$current_lei ? err($current_lei, @_) : warn(
strftime('%Y-%m-%dT%H:%M:%SZ', gmtime(time))," $$ ", @_);
};
+ local $SIG{PIPE} = 'IGNORE';
open STDERR, '>&STDIN' or die "redirect stderr failed: $!";
open STDOUT, '>&STDIN' or die "redirect stdout failed: $!";
# $daemon pipe to `lei' closed, main loop begins:
eval { PublicInbox::DS::event_loop($sig, $oldset) };
warn "event loop error: $@\n" if $@;
+
+ # EVFILT_SIGNAL will get a duplicate of all the signals it was sent
+ local @SIG{@kq_ign} = map 'IGNORE', @kq_ign;
+ PublicInbox::DS::sig_setmask($oldset) if @kq_ign;
+
# exit() may trigger waitpid via various DESTROY, ensure interruptible
- PublicInbox::DS::sig_setmask($oldset);
+ local @SIG{TERM} = sub { exit(POSIX::SIGTERM + 128) };
+ local @SIG{INT} = sub { exit(POSIX::SIGINT + 128) };
+ local @SIG{QUIT} = sub { exit(POSIX::SIGQUIT + 128) };
+ PublicInbox::DS::sig_setmask($oldset) if !@kq_ign;
dump_and_clear_log();
exit($exit_code // 0);
}
^ permalink raw reply related [relevance 63%]
* RFC: lei searches managed by users in git
@ 2023-09-15 21:08 67% Konstantin Ryabitsev
2023-09-15 22:47 55% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Konstantin Ryabitsev @ 2023-09-15 21:08 UTC (permalink / raw)
To: meta
Hello:
I am curious what is the best approach to have a centrally managed set of lei
searches, for example via config files tracked in git. For example, the file
could look like this:
mricon.toml:
[search.torvalds]
# All mail sent by torvalds
q = 'f:torvalds@linux-foundation.org'
[search.floppy]
# Any messages talking about floppies or touching floppy code
q = 'dfhh:floppy_* OR dfn:drivers/block/floppy.c OR s:floppy OR ((nq:bug OR nq:regression) AND nq:floppy)'
I could then have a small wrapper maintaining saved searches and making the
mailboxes available via special newsgroups like:
org.kernel.lei.mricon.torvalds
org.kernel.lei.mricon.floppy
The goal is to make it possible for maintainers to define their own set of
saved searches and have access to them at kernel.org via imap/pop3/nntp.
It's easy to write a simple wrapper that would invoke lei-edit-search and
replace the search string when there are updates to the config files, but I'm
curious if you already have thoughts on how to best implement something like
this.
My biggest concern is someone committing an invalid query and not receiving
any more email as a result -- so having a sane way to validate the query
before sticking it into the saved search would be handy.
-K
^ permalink raw reply [relevance 67%]
* Re: RFC: lei searches managed by users in git
2023-09-15 21:08 67% RFC: lei searches managed by users in git Konstantin Ryabitsev
@ 2023-09-15 22:47 55% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2023-09-15 22:47 UTC (permalink / raw)
To: Konstantin Ryabitsev; +Cc: meta
Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> Hello:
>
> I am curious what is the best approach to have a centrally managed set of lei
> searches, for example via config files tracked in git. For example, the file
> could look like this:
I don't have one nor have I thought about it until now...
I wonder if `lei edit-search --all' could be a thing...
As would adding gitconfig output to ls-search
(`lei ls-search -l -f gitconfig').
Maybe `lei edit-search --all=lei.q' could cut down
on non-interesting fields when editing all searches.
I also wouldn't be opposed to supporting non-interactive use
for automation:
lei edit-search --set lei.q=foo OUTPUT
lei edit-search --add lei.q=foo OUTPUT
...but maybe it's not needed (see below on `lei q'):
> mricon.toml:
Any particular reason for toml? I don't have a toml parser
installed and I suspect many don't, either. git-config is
widely available and installed, of course. I've been trying
hard to avoid data format and dependency proliferation.
> [search.torvalds]
> # All mail sent by torvalds
> q = 'f:torvalds@linux-foundation.org'
> [search.floppy]
> # Any messages talking about floppies or touching floppy code
> q = 'dfhh:floppy_* OR dfn:drivers/block/floppy.c OR s:floppy OR ((nq:bug OR nq:regression) AND nq:floppy)'
>
> I could then have a small wrapper maintaining saved searches and making the
> mailboxes available via special newsgroups like:
>
> org.kernel.lei.mricon.torvalds
> org.kernel.lei.mricon.floppy
Sidenote: I think `query' or similar is more appropriate than
using `lei' in a public newsgroup name since it's not `local' :>
> The goal is to make it possible for maintainers to define their own set of
> saved searches and have access to them at kernel.org via imap/pop3/nntp.
>
> It's easy to write a simple wrapper that would invoke lei-edit-search and
> replace the search string when there are updates to the config files, but I'm
> curious if you already have thoughts on how to best implement something like
> this.
Rerunning `lei q' on existing destinations is fine for updating
existing queries. This is especially nice with `-f v2', since
dedupe is faster on v2 than any other output format.
Thus there's no need to even bother with `lei edit-search' if
you're doing this non-interactively with your own files/formats
(e.g. via web UI or something).
> My biggest concern is someone committing an invalid query and not receiving
> any more email as a result -- so having a sane way to validate the query
> before sticking it into the saved search would be handy.
Perhaps `lei q --limit=1' could be appropriate for validating queries
with the below fix. But validating searches can ahead of time is
a TOCTOU problem. There can be added support for per-inbox prefixes
(e.g. altid with gmane: on public-inbox.org/git), and also when mixing
old/new versions via different HTTP(S) externals.
-------8<-------
Subject: [PATCH] lei q: set exit code for invalid Xapian queries
Xapian can't parse every query, so ensure we set the
exit code for the client.
---
lib/PublicInbox/LeiXSearch.pm | 6 ++++--
t/lei.t | 5 +++++
2 files changed, 9 insertions(+), 2 deletions(-)
diff --git a/lib/PublicInbox/LeiXSearch.pm b/lib/PublicInbox/LeiXSearch.pm
index 5965274c..7f4911b3 100644
--- a/lib/PublicInbox/LeiXSearch.pm
+++ b/lib/PublicInbox/LeiXSearch.pm
@@ -186,7 +186,8 @@ sub query_one_mset { # for --threads and l2m w/o sort
}
my $first_ids;
do {
- $mset = $srch->mset($mo->{qstr}, $mo);
+ $mset = eval { $srch->mset($mo->{qstr}, $mo) };
+ return $lei->child_error(22 << 8, "E: $@") if $@; # 22 from curl
mset_progress($lei, $dir, $mo->{offset} + $mset->size,
$mset->get_matches_estimated);
wait_startq($lei); # wait for keyword updates
@@ -249,7 +250,8 @@ sub query_combined_mset { # non-parallel for non-"--threads" users
}
my $each_smsg = $lei->{ovv}->ovv_each_smsg_cb($lei);
do {
- $mset = $self->mset($mo->{qstr}, $mo);
+ $mset = eval { $self->mset($mo->{qstr}, $mo) };
+ return $lei->child_error(22 << 8, "E: $@") if $@; # 22 from curl
mset_progress($lei, 'xsearch', $mo->{offset} + $mset->size,
$mset->get_matches_estimated);
wait_startq($lei); # wait for keyword updates
diff --git a/t/lei.t b/t/lei.t
index d83bde69..1199ca75 100644
--- a/t/lei.t
+++ b/t/lei.t
@@ -147,6 +147,11 @@ my $test_fail = sub {
lei_ok('q', "foo\n");
like($lei_err, qr/trailing `\\n' removed/s, "noted `\\n' removal");
+ lei(qw(q from:infinity..));
+ is($? >> 8, 22, 'combined query fails on invalid range op');
+ lei(qw(q -t from:infinity..));
+ is($? >> 8, 22, 'single query fails on invalid range op');
+
for my $lk (qw(ei inbox)) {
my $d = "$home/newline\n$lk";
my $all = $lk eq 'ei' ? 'ALL' : 'all';
^ permalink raw reply related [relevance 55%]
* Re: [PATCH 03/15] t/lei-p2q: extra diagnostics
@ 2023-09-21 10:23 71% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2023-09-21 10:23 UTC (permalink / raw)
To: meta
Eric Wong <e@80x24.org> wrote:
> I got one mysterious test failure here, once, and can't seem
> to reproduce it...
nearly 2 years later: I've hit it again, but once again cannot
reproduce it...
> +++ b/t/lei-p2q.t
> @@ -7,7 +7,7 @@ require_mods(qw(json DBD::SQLite Search::Xapian));
>
> test_lei(sub {
> ok(!lei(qw(p2q this-better-cause-format-patch-to-fail)),
> - 'p2q fails on bogus arg');
> + 'p2q fails on bogus arg') or diag $lei_err;
> like($lei_err, qr/format-patch.*failed/, 'notes format-patch failure');
$? does get printed (32768, so (>> 8) means 128), as it should be.
So I wonder if there's a place we drop the socket prematurely or
something else is amiss...
On a side note, lei's internal IPC could probably be simplified
a bit w/o sacrificing parallelism.
^ permalink raw reply [relevance 71%]
* [PATCH] t/lei-mirror: avoid make(1) jobserver warning
@ 2023-09-22 18:37 66% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2023-09-22 18:37 UTC (permalink / raw)
To: meta
We can't control `make test' nor user-defined targets in
config.mak. There's no need for a jobserver to run `make help',
anyways, so just let things be.
This also fixes the use of `gmake check' et al. on *BSDs where
various make flags confuse BSD make(1)
While we're at it, allow the test to run in the odd case make(1)
isn't available at all...
---
t/lei-mirror.t | 13 ++++++++-----
1 file changed, 8 insertions(+), 5 deletions(-)
diff --git a/t/lei-mirror.t b/t/lei-mirror.t
index 9b5d73ec..08961491 100644
--- a/t/lei-mirror.t
+++ b/t/lei-mirror.t
@@ -1,7 +1,7 @@
#!perl -w
# Copyright (C) all contributors <meta@public-inbox.org>
# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
-use strict; use v5.10.1; use PublicInbox::TestCommon;
+use v5.12; use PublicInbox::TestCommon;
use PublicInbox::Inbox;
require_mods(qw(-httpd lei DBD::SQLite));
require_cmd('curl');
@@ -26,11 +26,14 @@ test_lei({ tmpdir => $tmpdir }, sub {
is(PublicInbox::Git::try_cat("$t1/description"),
"mirror of $http/t1/\n", 'description set');
ok(-f "$t1/Makefile", 'convenience Makefile added (v1)');
- my $make = which('make');
- is(xsys([$make, 'help'], undef, { -C => $t1, 1 => \(my $help) }), 0,
- 'make help');
+ SKIP: {
+ my $make = require_cmd('make', 1);
+ delete local @ENV{qw(MFLAGS MAKEFLAGS MAKELEVEL)};
+ is(xsys([$make, 'help'], undef, { -C => $t1, 1 => \(my $help) }),
+ 0, "$make handled Makefile without errors");
+ isnt($help, '', 'make help worked');
+ }
ok(-f "$t1/inbox.config.example", 'inbox.config.example downloaded');
- isnt($help, '', 'make help worked');
is((stat(_))[9], $created{v1},
'inbox.config.example mtime is ->created_at');
is((stat(_))[2] & 0222, 0, 'inbox.config.example not writable');
^ permalink raw reply related [relevance 66%]
* lei interactive TUIs (ncurses/vim/emacs)
@ 2023-09-22 20:33 69% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2023-09-22 20:33 UTC (permalink / raw)
To: meta
I hardly have any experience in this area; but automatic
end-to-end tests for ncurses and other TUI stuff seems
like a huge pain...
I've also noticed vim has scripting abilities (like Emacs?) and
notmuch bundles a vim extension we can take inspiration from.
Perhaps we could bundle vim and Emacs extensions for lei, too...
While I use vim[1], I've always kept my vim decoupled from Perl
(or Lua, Python, Ruby, TCL, etc) and rather tie stuff together
with pipes. IOW, I'm happy my editor can run arbitrary shell
commands; but don't want stuff linked into my editor (since more
code is usually more fragile).
So, any thoughts on this matter?
Anybody willing to maintain an lei TUI for emacs?
I might give a vim TUI a shot...
But writing and testing a FUSE FS is much more natural...
[1] My choice to use vi/vim is merely because it's the most
widely-installed editor on random systems I ssh into.
It also mostly works the same after a few decades w/o
needing UI changes; same goes for Perl.
^ permalink raw reply [relevance 69%]
* [PATCH 1/4] lei blob|rediff: fix usage of lei->fail
2023-09-22 21:13 71% [PATCH 0/4] small lei fixes Eric Wong
@ 2023-09-22 21:13 90% ` Eric Wong
2023-09-22 21:13 66% ` [PATCH 2/4] lei: improve ->fail internal API Eric Wong
` (2 subsequent siblings)
3 siblings, 0 replies; 200+ results
From: Eric Wong @ 2023-09-22 21:13 UTC (permalink / raw)
To: meta
lei->fail only takes one message argument, presently;
but it's probably a good idea to change the API...
---
lib/PublicInbox/LeiBlob.pm | 2 +-
lib/PublicInbox/LeiRediff.pm | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/lib/PublicInbox/LeiBlob.pm b/lib/PublicInbox/LeiBlob.pm
index 1692289c..5fc6d902 100644
--- a/lib/PublicInbox/LeiBlob.pm
+++ b/lib/PublicInbox/LeiBlob.pm
@@ -158,7 +158,7 @@ sub lei_blob {
if ($lxs->remotes) {
require PublicInbox::LeiRemote;
$lei->{curl} //= which('curl') or return
- $lei->fail('curl needed for', $lxs->remotes);
+ $lei->fail('curl needed for '.join(', ',$lxs->remotes));
$lei->_lei_store(1)->write_prepare($lei);
}
require PublicInbox::SolverGit;
diff --git a/lib/PublicInbox/LeiRediff.pm b/lib/PublicInbox/LeiRediff.pm
index c312d90f..9cf95c08 100644
--- a/lib/PublicInbox/LeiRediff.pm
+++ b/lib/PublicInbox/LeiRediff.pm
@@ -268,7 +268,7 @@ sub lei_rediff {
if ($lxs->remotes) {
require PublicInbox::LeiRemote;
$lei->{curl} //= which('curl') or return
- $lei->fail('curl needed for', $lxs->remotes);
+ $lei->fail('curl needed for '.join(', ',$lxs->remotes));
}
$lei->ale->refresh_externals($lxs, $lei);
my $self = bless {
^ permalink raw reply related [relevance 90%]
* [PATCH 0/4] small lei fixes
@ 2023-09-22 21:13 71% Eric Wong
2023-09-22 21:13 90% ` [PATCH 1/4] lei blob|rediff: fix usage of lei->fail Eric Wong
` (3 more replies)
0 siblings, 4 replies; 200+ results
From: Eric Wong @ 2023-09-22 21:13 UTC (permalink / raw)
To: meta
Only noticed while working on bigger fixes...
Eric Wong (4):
lei blob|rediff: fix usage of lei->fail
lei: improve ->fail internal API
lei_to_mail: drop awkward duplication of $lei object
lei: use File::Temp for listing saved searches
lib/PublicInbox/LEI.pm | 19 ++++++++++++-------
lib/PublicInbox/LeiBlob.pm | 2 +-
lib/PublicInbox/LeiRediff.pm | 2 +-
lib/PublicInbox/LeiSavedSearch.pm | 20 +++++++++-----------
lib/PublicInbox/LeiToMail.pm | 12 +++++-------
5 files changed, 28 insertions(+), 27 deletions(-)
^ permalink raw reply [relevance 71%]
* [PATCH 3/4] lei_to_mail: drop awkward duplication of $lei object
2023-09-22 21:13 71% [PATCH 0/4] small lei fixes Eric Wong
2023-09-22 21:13 90% ` [PATCH 1/4] lei blob|rediff: fix usage of lei->fail Eric Wong
2023-09-22 21:13 66% ` [PATCH 2/4] lei: improve ->fail internal API Eric Wong
@ 2023-09-22 21:13 70% ` Eric Wong
2023-09-22 21:13 65% ` [PATCH 4/4] lei: use File::Temp for listing saved searches Eric Wong
3 siblings, 0 replies; 200+ results
From: Eric Wong @ 2023-09-22 21:13 UTC (permalink / raw)
To: meta
Our awaitpid API now exists and ProcessPipe uses it, so it's
immune to cyclic references. Thus there's no need to create
a duplicate of the lei object to prevent leaks.
---
lib/PublicInbox/LeiToMail.pm | 12 +++++-------
1 file changed, 5 insertions(+), 7 deletions(-)
diff --git a/lib/PublicInbox/LeiToMail.pm b/lib/PublicInbox/LeiToMail.pm
index 7c7967c8..4adcc33e 100644
--- a/lib/PublicInbox/LeiToMail.pm
+++ b/lib/PublicInbox/LeiToMail.pm
@@ -151,10 +151,9 @@ sub git_to_mail { # git->cat_async callback
}
sub reap_compress { # awaitpid callback
- my ($pid, $lei) = @_;
- my $cmd = delete $lei->{"pid.$pid"};
- return if $? == 0;
- $lei->fail($?, "@$cmd failed");
+ my ($pid, $lei, $cmd, $old_out) = @_;
+ $lei->{1} = $old_out;
+ $lei->fail($?, "@$cmd failed") if $?;
}
sub _post_augment_mbox { # open a compressor process from top-level process
@@ -165,9 +164,8 @@ sub _post_augment_mbox { # open a compressor process from top-level process
my $rdr = { 0 => $r, 1 => $lei->{1}, 2 => $lei->{2}, pgid => 0 };
my $pid = spawn($cmd, undef, $rdr);
my $pp = gensym;
- my $dup = bless { "pid.$pid" => $cmd }, ref($lei);
- $dup->{$_} = $lei->{$_} for qw(2 sock);
- tie *$pp, 'PublicInbox::ProcessPipe', $pid, $w, \&reap_compress, $dup;
+ tie *$pp, 'PublicInbox::ProcessPipe', $pid, $w,
+ \&reap_compress, $lei, $cmd, $lei->{1};
$lei->{1} = $pp;
}
^ permalink raw reply related [relevance 70%]
* [PATCH 2/4] lei: improve ->fail internal API
2023-09-22 21:13 71% [PATCH 0/4] small lei fixes Eric Wong
2023-09-22 21:13 90% ` [PATCH 1/4] lei blob|rediff: fix usage of lei->fail Eric Wong
@ 2023-09-22 21:13 66% ` Eric Wong
2023-09-22 21:13 70% ` [PATCH 3/4] lei_to_mail: drop awkward duplication of $lei object Eric Wong
2023-09-22 21:13 65% ` [PATCH 4/4] lei: use File::Temp for listing saved searches Eric Wong
3 siblings, 0 replies; 200+ results
From: Eric Wong @ 2023-09-22 21:13 UTC (permalink / raw)
To: meta
Allow the exit code to be the first argument intead of the last
to match our ->child_error, as well as the BSD err(3) API.
We'll also avoid shifting user-passed exit codes so $? can be
passed as-is without losing signal information.
---
lib/PublicInbox/LEI.pm | 19 ++++++++++++-------
lib/PublicInbox/LeiToMail.pm | 2 +-
2 files changed, 13 insertions(+), 8 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index c61ce76d..368f9357 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -29,6 +29,7 @@ use File::Path ();
use File::Spec;
use Carp ();
use Sys::Syslog qw(openlog syslog closelog);
+use Scalar::Util qw(looks_like_number);
our $quit = \&CORE::exit;
our ($current_lei, $errors_log, $listener, $oldset, $dir_idle);
my $GLP = Getopt::Long::Parser->new;
@@ -518,13 +519,17 @@ sub sigpipe_handler { # handles SIGPIPE from @WQ_KEYS workers
fail_handler($_[0], 13, delete $_[0]->{1});
}
-sub fail ($$;$) {
- my ($self, $msg, $exit_code) = @_;
- local $current_lei = $self;
- $self->{failed}++;
- warn(substr($msg, -1, 1) eq "\n" ? $msg : "$msg\n") if defined $msg;
- $self->{pkt_op_p}->pkt_do('fail_handler') if $self->{pkt_op_p};
- x_it($self, ($exit_code // 1) << 8);
+sub fail ($;@) {
+ my ($lei, @msg) = @_;
+ my $exit_code = looks_like_number($msg[0]) ? shift(@msg) : undef;
+ local $current_lei = $lei;
+ $lei->{failed}++;
+ if (@msg) {
+ push @msg, "\n" if substr($msg[-1], -1, 1);
+ warn @msg;
+ }
+ $lei->{pkt_op_p}->pkt_do('fail_handler') if $lei->{pkt_op_p};
+ x_it($lei, $exit_code // (1 << 8));
undef;
}
diff --git a/lib/PublicInbox/LeiToMail.pm b/lib/PublicInbox/LeiToMail.pm
index e357ee00..7c7967c8 100644
--- a/lib/PublicInbox/LeiToMail.pm
+++ b/lib/PublicInbox/LeiToMail.pm
@@ -154,7 +154,7 @@ sub reap_compress { # awaitpid callback
my ($pid, $lei) = @_;
my $cmd = delete $lei->{"pid.$pid"};
return if $? == 0;
- $lei->fail("@$cmd failed", $? >> 8);
+ $lei->fail($?, "@$cmd failed");
}
sub _post_augment_mbox { # open a compressor process from top-level process
^ permalink raw reply related [relevance 66%]
* [PATCH 4/4] lei: use File::Temp for listing saved searches
2023-09-22 21:13 71% [PATCH 0/4] small lei fixes Eric Wong
` (2 preceding siblings ...)
2023-09-22 21:13 70% ` [PATCH 3/4] lei_to_mail: drop awkward duplication of $lei object Eric Wong
@ 2023-09-22 21:13 65% ` Eric Wong
3 siblings, 0 replies; 200+ results
From: Eric Wong @ 2023-09-22 21:13 UTC (permalink / raw)
To: meta
I have no idea how badly my brain malfunctioned here when I
wrote the code to create a temporary file without O_EXCL :x
I'm still not sure if users have enough saved searches for
justifying a cache, here.
---
lib/PublicInbox/LeiSavedSearch.pm | 20 +++++++++-----------
1 file changed, 9 insertions(+), 11 deletions(-)
diff --git a/lib/PublicInbox/LeiSavedSearch.pm b/lib/PublicInbox/LeiSavedSearch.pm
index e5396342..2811c46d 100644
--- a/lib/PublicInbox/LeiSavedSearch.pm
+++ b/lib/PublicInbox/LeiSavedSearch.pm
@@ -3,8 +3,7 @@
# pretends to be like LeiDedupe and also PublicInbox::Inbox
package PublicInbox::LeiSavedSearch;
-use strict;
-use v5.10.1;
+use v5.12;
use parent qw(PublicInbox::Lock);
use PublicInbox::Git;
use PublicInbox::OverIdx;
@@ -14,6 +13,8 @@ use PublicInbox::Spawn qw(run_die);
use PublicInbox::ContentHash qw(git_sha);
use PublicInbox::MID qw(mids_for_index);
use PublicInbox::SHA qw(sha256_hex);
+use File::Temp ();
+use IO::Handle ();
our $LOCAL_PFX = qr!\A(?:maildir|mh|mbox.+|mmdf|v2):!i; # TODO: put in LeiToMail?
# move this to PublicInbox::Config if other things use it:
@@ -76,20 +77,17 @@ sub list {
my $lss_dir = $lei->share_path.'/saved-searches';
return () unless -d $lss_dir;
# TODO: persist the cache? Use another format?
- my $f = $lei->cache_dir."/saved-tmp.$$.".time.'.config';
- open my $fh, '>', $f or die "open $f: $!";
+ my $fh = File::Temp->new(TEMPLATE => 'lss_list-XXXX', TMPDIR => 1) or
+ die "File::Temp->new: $!";
print $fh "[include]\n";
for my $p (glob("$lss_dir/*/lei.saved-search")) {
print $fh "\tpath = ", cquote_val($p), "\n";
}
- close $fh or die "close $f: $!";
- my $cfg = $lei->cfg_dump($f);
- unlink($f);
+ $fh->flush or die "flush: $fh";
+ my $cfg = $lei->cfg_dump($fh->filename);
my $out = $cfg ? $cfg->get_all('lei.q.output') : [];
- map {;
- s!$LOCAL_PFX!!;
- $_;
- } @$out
+ s!$LOCAL_PFX!! for @$out;;
+ @$out;
}
sub translate_dedupe ($$) {
^ permalink raw reply related [relevance 65%]
* [PATCH 2/6] lei view_text: used tied ProcessPipe for `git config'
2023-09-24 5:42 69% [PATCH 0/6] lei config fixes and improvements Eric Wong
2023-09-24 5:42 56% ` [PATCH 1/6] lei: check git-config(1) failures Eric Wong
@ 2023-09-24 5:42 71% ` Eric Wong
2023-09-24 5:42 66% ` [PATCH 4/6] lei config: send `git config' errors to pager Eric Wong
2023-09-24 5:42 30% ` [PATCH 5/6] lei: fix `-c NAME=VALUE' config support Eric Wong
3 siblings, 0 replies; 200+ results
From: Eric Wong @ 2023-09-24 5:42 UTC (permalink / raw)
To: meta
The code exists and is loaded anyways, so we might as well
save an explicit call to waitpid. Noticed while checking
over our uses of `git config'
---
lib/PublicInbox/LeiViewText.pm | 9 ++++-----
1 file changed, 4 insertions(+), 5 deletions(-)
diff --git a/lib/PublicInbox/LeiViewText.pm b/lib/PublicInbox/LeiViewText.pm
index 53555467..70441867 100644
--- a/lib/PublicInbox/LeiViewText.pm
+++ b/lib/PublicInbox/LeiViewText.pm
@@ -72,12 +72,11 @@ sub new {
my $self = bless { %{$lei->{opt}}, -colored => \&uncolored }, $cls;
$self->{-quote_reply} = 1 if $fmt eq 'reply';
return $self unless $self->{color} //= -t $lei->{1};
- my $cmd = [ qw(git config -z --includes -l) ];
- my ($r, $pid) = popen_rd($cmd, undef, { 2 => $lei->{2} });
+ my @cmd = qw(git config -z --includes -l); # reuse normal git config
+ my $r = popen_rd(\@cmd, undef, { 2 => $lei->{2} });
my $cfg = PublicInbox::Config::config_fh_parse($r, "\0", "\n");
- waitpid($pid, 0);
- if ($?) {
- warn "# git-config failed, no color (non-fatal)\n";
+ if (!close($r)) {
+ warn "# @cmd failed, no color (non-fatal \$?=$?)\n";
return $self;
}
$self->{-colored} = \&my_colored;
^ permalink raw reply related [relevance 71%]
* [PATCH 0/6] lei config fixes and improvements
@ 2023-09-24 5:42 69% Eric Wong
2023-09-24 5:42 56% ` [PATCH 1/6] lei: check git-config(1) failures Eric Wong
` (3 more replies)
0 siblings, 4 replies; 200+ results
From: Eric Wong @ 2023-09-24 5:42 UTC (permalink / raw)
To: meta
Fixing `-c NAME=VALUE' was something I noticed working on
improving *BSD support the other week. Everything else
here was noticed while fixing -c. And 6/6 is because I'm
annoyed at seeing test-only code in Config.pm
Eric Wong (6):
lei: check git-config(1) failures
lei view_text: used tied ProcessPipe for `git config'
config: handle key-only entries as booleans
lei config: send `git config' errors to pager
lei: fix `-c NAME=VALUE' config support
config: drop scalar ref support from internal API
lib/PublicInbox/Config.pm | 78 ++++++++++------
lib/PublicInbox/LEI.pm | 102 +++++++++++----------
lib/PublicInbox/LeiAddWatch.pm | 7 +-
lib/PublicInbox/LeiConfig.pm | 35 +++++---
lib/PublicInbox/LeiForgetExternal.pm | 3 +-
lib/PublicInbox/LeiInit.pm | 4 +-
lib/PublicInbox/LeiMirror.pm | 5 +-
lib/PublicInbox/LeiRmWatch.pm | 2 +-
lib/PublicInbox/LeiViewText.pm | 9 +-
lib/PublicInbox/TestCommon.pm | 20 +++--
t/config.t | 128 +++++++++++++++------------
t/config_limiter.t | 31 +++----
t/inbox_idle.t | 15 ++--
t/lei.t | 17 +++-
t/psgi_bad_mids.t | 18 ++--
t/psgi_mount.t | 14 ++-
t/psgi_multipart_not.t | 16 ++--
t/psgi_scan_all.t | 18 ++--
t/psgi_search.t | 12 ++-
t/psgi_text.t | 21 ++---
t/watch_filter_rubylang.t | 30 +++----
t/watch_imap.t | 20 +++--
t/watch_maildir.t | 24 ++---
t/watch_maildir_v2.t | 44 ++++-----
t/watch_multiple_headers.t | 21 +++--
25 files changed, 381 insertions(+), 313 deletions(-)
^ permalink raw reply [relevance 69%]
* [PATCH 4/6] lei config: send `git config' errors to pager
2023-09-24 5:42 69% [PATCH 0/6] lei config fixes and improvements Eric Wong
2023-09-24 5:42 56% ` [PATCH 1/6] lei: check git-config(1) failures Eric Wong
2023-09-24 5:42 71% ` [PATCH 2/6] lei view_text: used tied ProcessPipe for `git config' Eric Wong
@ 2023-09-24 5:42 66% ` Eric Wong
2023-09-24 5:42 30% ` [PATCH 5/6] lei: fix `-c NAME=VALUE' config support Eric Wong
3 siblings, 0 replies; 200+ results
From: Eric Wong @ 2023-09-24 5:42 UTC (permalink / raw)
To: meta
Our previous use of lei->cfg_dump was wrong as the extra arg was
never supported. Instead, we need to capture the output of
`git config' and send it to the pager if ->cfg_dump fails. We'll
also add a note to the user to quit the pager to continue.
---
lib/PublicInbox/LEI.pm | 2 +-
lib/PublicInbox/LeiConfig.pm | 12 ++++++++++--
2 files changed, 11 insertions(+), 3 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index a6d92eec..488006e0 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -1098,7 +1098,7 @@ sub pgr_err {
my ($self, @msg) = @_;
return warn(@msg) unless $self->{sock} && -t $self->{2};
start_pager($self, { LESS => 'RX' }); # no 'F' so we prompt
- print { $self->{2} } @msg;
+ say { $self->{2} } @msg, '# -quit pager to continue-';
$self->{2}->autoflush(1);
stop_pager($self);
send($self->{sock}, 'wait', 0); # wait for user to quit pager
diff --git a/lib/PublicInbox/LeiConfig.pm b/lib/PublicInbox/LeiConfig.pm
index 23be9aaf..fd4b0eca 100644
--- a/lib/PublicInbox/LeiConfig.pm
+++ b/lib/PublicInbox/LeiConfig.pm
@@ -4,6 +4,8 @@ package PublicInbox::LeiConfig;
use strict;
use v5.10.1;
use PublicInbox::PktOp;
+use Fcntl qw(SEEK_SET);
+use autodie qw(open seek);
sub cfg_do_edit ($;$) {
my ($self, $reason) = @_;
@@ -22,8 +24,14 @@ sub cfg_do_edit ($;$) {
sub cfg_edit_done { # PktOp
my ($self) = @_;
eval {
- my $cfg = $self->{lei}->cfg_dump($self->{-f}, $self->{lei}->{2})
- // return cfg_do_edit($self, "\n");
+ open my $fh, '+>', undef or die "open($!)";
+ my $cfg = do {
+ local $self->{lei}->{2} = $fh;
+ $self->{lei}->cfg_dump($self->{-f});
+ } or do {
+ seek($fh, 0, SEEK_SET);
+ return cfg_do_edit($self, do { local $/; <$fh> });
+ };
$self->cfg_verify($cfg) if $self->can('cfg_verify');
};
$self->{lei}->fail($@) if $@;
^ permalink raw reply related [relevance 66%]
* [PATCH 1/6] lei: check git-config(1) failures
2023-09-24 5:42 69% [PATCH 0/6] lei config fixes and improvements Eric Wong
@ 2023-09-24 5:42 56% ` Eric Wong
2023-09-24 5:42 71% ` [PATCH 2/6] lei view_text: used tied ProcessPipe for `git config' Eric Wong
` (2 subsequent siblings)
3 siblings, 0 replies; 200+ results
From: Eric Wong @ 2023-09-24 5:42 UTC (permalink / raw)
To: meta
2020-2021 were bad times and I somehow got deluded into
believing git-config(1) would always succeed :x
---
lib/PublicInbox/LEI.pm | 9 ++++++---
lib/PublicInbox/LeiAddWatch.pm | 7 ++++---
lib/PublicInbox/LeiForgetExternal.pm | 3 +--
lib/PublicInbox/LeiInit.pm | 4 ++--
lib/PublicInbox/LeiRmWatch.pm | 2 +-
5 files changed, 14 insertions(+), 11 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 368f9357..a6d92eec 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -776,9 +776,8 @@ EOM
/\A([^=\.]+\.[^=]+)(?:=(.*))?\z/ or return fail($self, <<EOM);
`-c $_' is not of the form -c <name>=<value>'
EOM
- my $name = $1;
- my $value = $2 // 1;
- _config($self, '--add', $name, $value);
+ my ($name, $value) = ($1, $2 // 1);
+ _config($self, '--add', $name, $value) or return;
if (defined(my $v = $tmp->{$name})) {
if (ref($v) eq 'ARRAY') {
push @$v, $value;
@@ -894,13 +893,17 @@ sub _lei_store ($;$) {
};
}
+# returns true on success, undef
+# argv[0] eq `+e' means errors do not ->fail # (like `sh +e')
sub _config {
my ($self, @argv) = @_;
+ my $err_ok = ($argv[0] // '') eq '+e' ? shift(@argv) : undef;
my %env = (%{$self->{env}}, GIT_CONFIG => undef);
my $cfg = _lei_cfg($self, 1);
my $cmd = [ qw(git config -f), $cfg->{'-f'}, @argv ];
my %rdr = map { $_ => $self->{$_} } (0..2);
waitpid(spawn($cmd, \%env, \%rdr), 0);
+ $? == 0 ? 1 : ($err_ok ? undef : fail($self, $?));
}
sub lei_daemon_pid { puts shift, $$ }
diff --git a/lib/PublicInbox/LeiAddWatch.pm b/lib/PublicInbox/LeiAddWatch.pm
index 97e7a342..f61e2de4 100644
--- a/lib/PublicInbox/LeiAddWatch.pm
+++ b/lib/PublicInbox/LeiAddWatch.pm
@@ -26,13 +26,14 @@ sub lei_add_watch {
for my $w (@{$self->{inputs}}) {
# clobber existing, allow multiple
if (defined($vmd0)) {
- $lei->_config("watch.$w.vmd", '--replace-all', $vmd0);
+ $lei->_config("watch.$w.vmd", '--replace-all', $vmd0)
+ or return;
for my $v (@vmd) {
- $lei->_config("watch.$w.vmd", $v);
+ $lei->_config("watch.$w.vmd", $v) or return;
}
}
next if defined $cfg->{"watch.$w.state"};
- $lei->_config("watch.$w.state", $state);
+ $lei->_config("watch.$w.state", $state) or return;
}
$lei->_lei_store(1); # create
$lei->lms(1)->lms_write_prepare->add_folders(@{$self->{inputs}});
diff --git a/lib/PublicInbox/LeiForgetExternal.pm b/lib/PublicInbox/LeiForgetExternal.pm
index 39bfc60b..c8d1df38 100644
--- a/lib/PublicInbox/LeiForgetExternal.pm
+++ b/lib/PublicInbox/LeiForgetExternal.pm
@@ -16,8 +16,7 @@ sub lei_forget_external {
next if $seen{$l}++;
my $key = "external.$l.boost";
delete($cfg->{$key});
- $lei->_config('--unset', $key);
- if ($? == 0) {
+ if ($lei->_config('+e', '--unset', $key)) {
$lei->qerr("# $l forgotten ");
} elsif (($? >> 8) == 5) {
warn("# $l not found\n");
diff --git a/lib/PublicInbox/LeiInit.pm b/lib/PublicInbox/LeiInit.pm
index 27ce8169..94897e61 100644
--- a/lib/PublicInbox/LeiInit.pm
+++ b/lib/PublicInbox/LeiInit.pm
@@ -23,7 +23,7 @@ sub lei_init {
# some folks like symlinks and bind mounts :P
if (@dir && "@cur[1,0]" eq "@dir[1,0]") {
- $self->_config('leistore.dir', $dir);
+ $self->_config('leistore.dir', $dir) or return;
$self->_lei_store(1)->done;
return $self->qerr("$exists (as $cur)");
}
@@ -31,7 +31,7 @@ sub lei_init {
E: leistore.dir=$cur already initialized and it is not $dir
}
- $self->_config('leistore.dir', $dir);
+ $self->_config('leistore.dir', $dir) or return;
$self->_lei_store(1)->done;
$exists //= "# leistore.dir=$dir newly initialized";
$self->qerr($exists);
diff --git a/lib/PublicInbox/LeiRmWatch.pm b/lib/PublicInbox/LeiRmWatch.pm
index c0f336f0..19bee3ab 100644
--- a/lib/PublicInbox/LeiRmWatch.pm
+++ b/lib/PublicInbox/LeiRmWatch.pm
@@ -14,7 +14,7 @@ sub lei_rm_watch {
my $self = bless { missing_ok => 1 }, __PACKAGE__;
$self->prepare_inputs($lei, \@argv) or return;
for my $w (@{$self->{inputs}}) {
- $lei->_config('--remove-section', "watch.$w");
+ $lei->_config('--remove-section', "watch.$w") or return;
}
delete $lei->{cfg}; # force reload
$lei->refresh_watches;
^ permalink raw reply related [relevance 56%]
* [PATCH 5/6] lei: fix `-c NAME=VALUE' config support
2023-09-24 5:42 69% [PATCH 0/6] lei config fixes and improvements Eric Wong
` (2 preceding siblings ...)
2023-09-24 5:42 66% ` [PATCH 4/6] lei config: send `git config' errors to pager Eric Wong
@ 2023-09-24 5:42 30% ` Eric Wong
3 siblings, 0 replies; 200+ results
From: Eric Wong @ 2023-09-24 5:42 UTC (permalink / raw)
To: meta
We can pass `-c NAME=VALUE' args directly to git-config without
needing a temporary directory nor file. Furthermore, this opens
the door to us being able to correctly handle `-c NAME=VALUE'
after `delete $lei->{cfg}' if we need to reload the config
during a command.
This tightens up error-checking for `lei config' and ensures we
can make config settings changes while using `-c NAME=VALUE'
instead of editing the temporary file.
The non-obvious part was avoiding the use of the -f/--file arg for
`git config' for read-only operations and include relying on
`-c include.path=$ABS_PATH'. This is done by parsing the
switches to be passed to `git config' to determine if it's a
read-only operation or not.
---
lib/PublicInbox/Config.pm | 52 +++++++++++++++-----
lib/PublicInbox/LEI.pm | 95 +++++++++++++++++++-----------------
lib/PublicInbox/LeiConfig.pm | 23 ++++++---
lib/PublicInbox/LeiMirror.pm | 5 +-
t/lei.t | 17 ++++++-
5 files changed, 123 insertions(+), 69 deletions(-)
diff --git a/lib/PublicInbox/Config.pm b/lib/PublicInbox/Config.pm
index f6236d84..533f4a52 100644
--- a/lib/PublicInbox/Config.pm
+++ b/lib/PublicInbox/Config.pm
@@ -22,7 +22,7 @@ sub _array ($) { ref($_[0]) eq 'ARRAY' ? $_[0] : [ $_[0] ] }
# returns key-value pairs of config directives in a hash
# if keys may be multi-value, the value is an array ref containing all values
sub new {
- my ($class, $file, $errfh) = @_;
+ my ($class, $file, $lei) = @_;
$file //= default_file();
my $self;
my $set_dedupe;
@@ -36,7 +36,7 @@ sub new {
$self = $DEDUPE->{$file} and return $self;
$set_dedupe = 1;
}
- $self = git_config_dump($class, $file, $errfh);
+ $self = git_config_dump($class, $file, $lei);
$self->{'-f'} = $file;
}
# caches
@@ -174,13 +174,34 @@ sub config_fh_parse ($$$) {
\%rv;
}
+sub tmp_cmd_opt ($$) {
+ my ($env, $opt) = @_;
+ # quiet global and system gitconfig if supported by installed git,
+ # but normally harmless if too noisy (NOGLOBAL no longer exists)
+ $env->{GIT_CONFIG_NOSYSTEM} = 1;
+ $env->{GIT_CONFIG_GLOBAL} = '/dev/null'; # git v2.32+
+ $opt->{-C} = '/'; # avoid $worktree/.git/config on MOST systems :P
+}
+
sub git_config_dump {
- my ($class, $file, $errfh) = @_;
- return bless {}, $class unless -e $file;
- my $cmd = [ qw(git config -z -l --includes), "--file=$file" ];
- my $fh = popen_rd($cmd, undef, { 2 => $errfh // 2 });
+ my ($class, $file, $lei) = @_;
+ my @opt_c = map { ('-c', $_) } @{$lei->{opt}->{c} // []};
+ $file = undef if !-e $file;
+ # XXX should we set {-f} if !-e $file?
+ return bless {}, $class if (!@opt_c && !defined($file));
+ my %env;
+ my $opt = { 2 => $lei->{2} // 2 };
+ if (@opt_c) {
+ unshift(@opt_c, '-c', "include.path=$file") if defined($file);
+ tmp_cmd_opt(\%env, $opt);
+ }
+ my @cmd = ('git', @opt_c, qw(config -z -l --includes));
+ push(@cmd, '-f', $file) if !@opt_c && defined($file);
+ my $fh = popen_rd(\@cmd, \%env, $opt);
my $rv = config_fh_parse($fh, "\0", "\n");
- close $fh or die "@$cmd failed: \$?=$?\n";
+ close $fh or die "@cmd failed: \$?=$?\n";
+ $rv->{-opt_c} = \@opt_c if @opt_c; # for ->urlmatch
+ $rv->{-f} = $file;
bless $rv, $class;
}
@@ -544,14 +565,23 @@ sub _fill_ei ($$) {
$es;
}
+sub config_cmd {
+ my ($self, $env, $opt) = @_;
+ my $f = $self->{-f} // default_file();
+ my @opt_c = @{$self->{-opt_c} // []};
+ my @cmd = ('git', @opt_c, 'config');
+ @opt_c ? tmp_cmd_opt($env, $opt) : push(@cmd, '-f', $f);
+ \@cmd;
+}
+
sub urlmatch {
my ($self, $key, $url, $try_git) = @_;
state $urlmatch_broken; # requires git 1.8.5
return if $urlmatch_broken;
- my $file = $self->{'-f'} // default_file();
- my $cmd = [qw/git config -z --includes --get-urlmatch/,
- "--file=$file", $key, $url ];
- my $fh = popen_rd($cmd);
+ my (%env, %opt);
+ my $cmd = $self->config_cmd(\%env, \%opt);
+ push @$cmd, qw(-z --includes --get-urlmatch), $key, $url;
+ my $fh = popen_rd($cmd, \%env, \%opt);
local $/ = "\0";
my $val = <$fh>;
if (!close($fh)) {
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 488006e0..8b62def2 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -281,7 +281,8 @@ our %CMD = ( # sorted in order of importance/use:
"use a patch to generate a query for `lei q --stdin'",
qw(stdin| in-format|F=s want|w=s@ uri debug), @net_opt, @c_opt ],
'config' => [ '[...]', sub {
- 'git-config(1) wrapper for '._config_path($_[0]);
+ 'git-config(1) wrapper for '._config_path($_[0]). "\n" .
+ '-l/--list and other common git-config uses are supported'
}, qw(config-file|system|global|file|f=s), # for conflict detection
qw(edit|e c=s@ C=s@), pass_through('git config') ],
'inspect' => [ 'ITEMS...|--stdin', 'inspect lei/store and/or local external',
@@ -456,6 +457,7 @@ my %OPTDESC = (
'z|0' => 'use NUL \\0 instead of newline (CR) to delimit lines',
'signal|s=s' => [ 'SIG', 'signal to send lei-daemon (default: TERM)' ],
+'edit|e config' => 'open an editor to modify the lei config file',
); # %OPTDESC
my %CONFIG_KEYS = (
@@ -760,36 +762,6 @@ sub optparse ($$$) {
$err ? fail($self, "usage: lei $cmd $proto\nE: $err") : 1;
}
-sub _tmp_cfg { # for lei -c <name>=<value> ...
- my ($self) = @_;
- my $cfg = _lei_cfg($self, 1);
- require File::Temp;
- my $ft = File::Temp->new(TEMPLATE => 'lei_cfg-XXXX', TMPDIR => 1);
- my $tmp = { '-f' => $ft->filename, -tmp => $ft };
- $ft->autoflush(1);
- print $ft <<EOM or return fail($self, "$tmp->{-f}: $!");
-[include]
- path = $cfg->{-f}
-EOM
- $tmp = $self->{cfg} = bless { %$cfg, %$tmp }, ref($cfg);
- for (@{$self->{opt}->{c}}) {
- /\A([^=\.]+\.[^=]+)(?:=(.*))?\z/ or return fail($self, <<EOM);
-`-c $_' is not of the form -c <name>=<value>'
-EOM
- my ($name, $value) = ($1, $2 // 1);
- _config($self, '--add', $name, $value) or return;
- if (defined(my $v = $tmp->{$name})) {
- if (ref($v) eq 'ARRAY') {
- push @$v, $value;
- } else {
- $tmp->{$name} = [ $v, $value ];
- }
- } else {
- $tmp->{$name} = $value;
- }
- }
-}
-
sub lazy_cb ($$$) { # $pfx is _complete_ or lei_
my ($self, $cmd, $pfx) = @_;
my $ucmd = $cmd;
@@ -819,7 +791,6 @@ sub dispatch {
}
if (my $cb = lazy_cb(__PACKAGE__, $cmd, 'lei_')) {
optparse($self, $cmd, \@argv) or return;
- $self->{opt}->{c} and (_tmp_cfg($self) // return);
if (my $chdir = $self->{opt}->{C}) {
for my $d (@$chdir) {
next if $d eq ''; # same as git(1)
@@ -844,17 +815,20 @@ sub _lei_cfg ($;$) {
my $f = _config_path($self);
my @st = stat($f);
my $cur_st = @st ? pack('dd', $st[10], $st[7]) : ''; # 10:ctime, 7:size
- my ($sto, $sto_dir, $watches, $lne);
- if (my $cfg = $PATH2CFG{$f}) { # reuse existing object in common case
- return ($self->{cfg} = $cfg) if $cur_st eq $cfg->{-st};
+ my ($sto, $sto_dir, $watches, $lne, $cfg);
+ if ($cfg = $PATH2CFG{$f}) { # reuse existing object in common case
+ ($cur_st eq $cfg->{-st} && !$self->{opt}->{c}) and
+ return ($self->{cfg} = $cfg);
+ # reuse some fields below if they match:
($sto, $sto_dir, $watches, $lne) =
@$cfg{qw(-lei_store leistore.dir -watches
-lei_note_event)};
}
if (!@st) {
- unless ($creat) {
- delete $self->{cfg};
- return bless {}, 'PublicInbox::Config';
+ unless ($creat) { # any commands which write to cfg must creat
+ $cfg = PublicInbox::Config->git_config_dump(
+ '/dev/null', $self);
+ return ($self->{cfg} = $cfg);
}
my ($cfg_dir) = ($f =~ m!(.*?/)[^/]+\z!);
File::Path::mkpath($cfg_dir);
@@ -863,9 +837,8 @@ sub _lei_cfg ($;$) {
$cur_st = pack('dd', $st[10], $st[7]);
qerr($self, "# $f created") if $self->{cmd} ne 'config';
}
- my $cfg = PublicInbox::Config->git_config_dump($f, $self->{2});
+ $cfg = PublicInbox::Config->git_config_dump($f, $self);
$cfg->{-st} = $cur_st;
- $cfg->{'-f'} = $f;
if ($sto && canonpath_harder($sto_dir // store_path($self))
eq canonpath_harder($cfg->{'leistore.dir'} //
store_path($self))) {
@@ -877,7 +850,7 @@ sub _lei_cfg ($;$) {
# FIXME: use inotify/EVFILT_VNODE to detect unlinked configs
delete(@PATH2CFG{grep(!-f, keys %PATH2CFG)});
}
- $self->{cfg} = $PATH2CFG{$f} = $cfg;
+ $self->{cfg} = $self->{opt}->{c} ? $cfg : ($PATH2CFG{$f} = $cfg);
refresh_watches($self);
$cfg;
}
@@ -898,11 +871,41 @@ sub _lei_store ($;$) {
sub _config {
my ($self, @argv) = @_;
my $err_ok = ($argv[0] // '') eq '+e' ? shift(@argv) : undef;
- my %env = (%{$self->{env}}, GIT_CONFIG => undef);
+ my %env;
+ my %opt = map { $_ => $self->{$_} } (0..2);
my $cfg = _lei_cfg($self, 1);
- my $cmd = [ qw(git config -f), $cfg->{'-f'}, @argv ];
- my %rdr = map { $_ => $self->{$_} } (0..2);
- waitpid(spawn($cmd, \%env, \%rdr), 0);
+ my $opt_c = delete local $cfg->{-opt_c};
+ my @file_arg;
+ if ($opt_c) {
+ my ($set, $get, $nondash);
+ for (@argv) { # order matters for git-config
+ if (!$nondash) {
+ if (/\A--(?:add|rename-section|remove-section|
+ replace-all|
+ unset-all|unset)\z/x) {
+ ++$set;
+ } elsif ($_ eq '-l' || $_ eq '--list' ||
+ /\A--get/) {
+ ++$get;
+ } elsif (/\A-/) { # -z and such
+ } else {
+ ++$nondash;
+ }
+ } else {
+ ++$nondash;
+ }
+ }
+ if ($set || ($nondash//0) > 1 && !$get) {
+ @file_arg = ('-f', $cfg->{-f});
+ $env{GIT_CONFIG} = $file_arg[1];
+ } else { # OK, we can use `-c n=v' for read-only
+ $cfg->{-opt_c} = $opt_c;
+ $env{GIT_CONFIG} = undef;
+ }
+ }
+ my $cmd = $cfg->config_cmd(\%env, \%opt);
+ push @$cmd, @file_arg, @argv;
+ waitpid(spawn($cmd, \%env, \%opt), 0);
$? == 0 ? 1 : ($err_ok ? undef : fail($self, $?));
}
@@ -1545,7 +1548,7 @@ sub sto_done_request {
sub cfg_dump ($$) {
my ($lei, $f) = @_;
- my $ret = eval { PublicInbox::Config->git_config_dump($f, $lei->{2}) };
+ my $ret = eval { PublicInbox::Config->git_config_dump($f, $lei) };
return $ret if !$@;
warn($@);
undef;
diff --git a/lib/PublicInbox/LeiConfig.pm b/lib/PublicInbox/LeiConfig.pm
index fd4b0eca..76fc43e7 100644
--- a/lib/PublicInbox/LeiConfig.pm
+++ b/lib/PublicInbox/LeiConfig.pm
@@ -1,8 +1,7 @@
-# Copyright (C) 2021 all contributors <meta@public-inbox.org>
+# Copyright (C) all contributors <meta@public-inbox.org>
# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
-package PublicInbox::LeiConfig;
-use strict;
-use v5.10.1;
+package PublicInbox::LeiConfig; # subclassed by LeiEditSearch
+use v5.12;
use PublicInbox::PktOp;
use Fcntl qw(SEEK_SET);
use autodie qw(open seek);
@@ -41,10 +40,18 @@ sub lei_config {
my ($lei, @argv) = @_;
$lei->{opt}->{'config-file'} and return $lei->fail(
"config file switches not supported by `lei config'");
- return $lei->_config(@argv) unless $lei->{opt}->{edit};
- my $f = $lei->_lei_cfg(1)->{-f};
- my $self = bless { lei => $lei, -f => $f }, __PACKAGE__;
- cfg_do_edit($self);
+ if ($lei->{opt}->{edit}) {
+ @argv and return $lei->fail(
+'--edit must be used without other arguments');
+ $lei->{opt}->{c} and return $lei->fail(
+"`-c $lei->{opt}->{c}->[0]' not allowed with --edit");
+ my $f = $lei->_lei_cfg(1)->{-f};
+ cfg_do_edit(bless { lei => $lei, -f => $f }, __PACKAGE__);
+ } elsif (@argv) { # let git-config do error-checking
+ $lei->_config(@argv);
+ } else {
+ $lei->_help('no options given');
+ }
}
1;
diff --git a/lib/PublicInbox/LeiMirror.pm b/lib/PublicInbox/LeiMirror.pm
index bed034f1..fed6b668 100644
--- a/lib/PublicInbox/LeiMirror.pm
+++ b/lib/PublicInbox/LeiMirror.pm
@@ -193,7 +193,8 @@ sub _write_inbox_config {
} elsif (!$!{EEXIST}) {
die "open($f): $!";
}
- my $cfg = PublicInbox::Config->git_config_dump($f, $self->{lei}->{2});
+ my $cfg = PublicInbox::Config->git_config_dump($f,
+ { 2 => $self->{lei}->{2} });
my $ibx = $self->{ibx} = {}; # for indexing
for my $sec (grep(/\Apublicinbox\./, @{$cfg->{-section_order}})) {
for (qw(address newsgroup nntpmirror)) {
@@ -238,7 +239,7 @@ sub index_cloned_inbox {
}
# force synchronous awaitpid for v2:
local $PublicInbox::DS::in_loop = 0;
- my $cfg = PublicInbox::Config->new(undef, $lei->{2});
+ my $cfg = PublicInbox::Config->new(undef, { 2 => $lei->{2} });
my $env = PublicInbox::Admin::index_prepare($opt, $cfg);
local %ENV = (%ENV, %$env) if $env;
PublicInbox::Admin::progress_prepare($opt, $lei->{2});
diff --git a/t/lei.t b/t/lei.t
index 1199ca75..3ac804a8 100644
--- a/t/lei.t
+++ b/t/lei.t
@@ -40,10 +40,21 @@ my $test_help = sub {
lei_ok(qw(config -h));
like($lei_out, qr! \Q$home\E/\.config/lei/config\b!,
'actual path shown in config -h');
+ my $exp_help = qr/\Q$lei_out\E/s;
+ ok(!lei('config'), 'config w/o args fails');
+ like($lei_err, $exp_help, 'config w/o args shows our help in stderr');
lei_ok(qw(config -h), { XDG_CONFIG_HOME => '/XDC' },
\'config with XDG_CONFIG_HOME');
like($lei_out, qr! /XDC/lei/config\b!, 'XDG_CONFIG_HOME in config -h');
is($lei_err, '', 'no errors from config -h');
+
+ lei_ok(qw(-c foo.bar config dash.c works));
+ lei_ok(qw(config dash.c));
+ is($lei_out, "works\n", 'config set w/ -c');
+
+ lei_ok(qw(-c foo.bar config --add dash.c add-works));
+ lei_ok(qw(config --get-all dash.c));
+ is($lei_out, "works\nadd-works\n", 'config --add w/ -c');
};
my $ok_err_info = sub {
@@ -101,9 +112,11 @@ my $test_config = sub {
is($lei_out, "tr00\n", "-c string value passed as-is");
lei_ok(qw(-c imap.debug=a -c imap.debug=b config --get-all imap.debug));
is($lei_out, "a\nb\n", '-c and --get-all work together');
-
- lei_ok([qw(config -e)], { VISUAL => 'cat', EDITOR => 'cat' });
+ my $env = { VISUAL => 'cat', EDITOR => 'cat' };
+ lei_ok([qw(config -e)], $env);
is($lei_out, "[a]\n\tb = c\n", '--edit works');
+ ok(!lei([qw(-c a.b=c config -e)], $env), '-c conflicts with -e');
+ like($lei_err, qr/not allowed/, 'error message shown');
};
my $test_completion = sub {
^ permalink raw reply related [relevance 30%]
* [PATCH 3/4] lei: use scalar %SIG assignment
@ 2023-09-24 21:08 71% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2023-09-24 21:08 UTC (permalink / raw)
To: meta
Perl v5.16.3 (and possibly some later versions) complain about
this, but newer (v5.32.1) are fine with it.
Fixes: e281363ba937 ("lei: ensure we run DESTROY|END at daemon exit w/ kqueue")
---
lib/PublicInbox/LEI.pm | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 1ead9bf6..be77fa90 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -1381,9 +1381,9 @@ sub lazy_start {
PublicInbox::DS::sig_setmask($oldset) if @kq_ign;
# exit() may trigger waitpid via various DESTROY, ensure interruptible
- local @SIG{TERM} = sub { exit(POSIX::SIGTERM + 128) };
- local @SIG{INT} = sub { exit(POSIX::SIGINT + 128) };
- local @SIG{QUIT} = sub { exit(POSIX::SIGQUIT + 128) };
+ local $SIG{TERM} = sub { exit(POSIX::SIGTERM + 128) };
+ local $SIG{INT} = sub { exit(POSIX::SIGINT + 128) };
+ local $SIG{QUIT} = sub { exit(POSIX::SIGQUIT + 128) };
PublicInbox::DS::sig_setmask($oldset) if !@kq_ign;
dump_and_clear_log();
exit($exit_code // 0);
^ permalink raw reply related [relevance 71%]
* [PATCH 3/3] lei: don't gzip --rsyncable by default for mbox*
@ 2023-09-27 6:02 46% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2023-09-27 6:02 UTC (permalink / raw)
To: meta
Using and memoizing the usability of `--rsyncable' is unsafe
since pigz (or GNU gzip) can be uninstalled and leave a user
with a non-rsync-aware gzip implementation in the long-running
daemon. So we stop passing --rsyncable by default to pigz/gzip
and no longer attempt to check for it (since it was a TOCTTOU
error, anyways).
Specifying --rsyncable explicitly didn't work, either, and
ended up passing `1' to the gzip/pigz argv :x
Finally, we now test --rsyncable on the CLI by adding support
for it in `lei convert' and testing it in t/lei-convert.t
---
lib/PublicInbox/LEI.pm | 3 ++-
lib/PublicInbox/LeiToMail.pm | 2 +-
lib/PublicInbox/MboxReader.pm | 36 ++++++++---------------------------
t/lei-convert.t | 27 ++++++++++++++++++++++++--
4 files changed, 36 insertions(+), 32 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 2f8d7a96..beb0f897 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -275,7 +275,8 @@ our %CMD = ( # sorted in order of importance/use:
qw(all:s mode=s), @net_opt, @c_opt ],
'convert' => [ 'LOCATION...|--stdin',
'one-time conversion from URL or filesystem to another format',
- qw(stdin| in-format|F=s out-format|f=s output|mfolder|o=s lock=s@ kw!),
+ qw(stdin| in-format|F=s out-format|f=s output|mfolder|o=s lock=s@ kw!
+ rsyncable),
@net_opt, @c_opt ],
'p2q' => [ 'LOCATION_OR_COMMIT...|--stdin',
"use a patch to generate a query for `lei q --stdin'",
diff --git a/lib/PublicInbox/LeiToMail.pm b/lib/PublicInbox/LeiToMail.pm
index a2cd8650..2dddf00b 100644
--- a/lib/PublicInbox/LeiToMail.pm
+++ b/lib/PublicInbox/LeiToMail.pm
@@ -155,7 +155,7 @@ sub reap_compress { # awaitpid callback
$lei->fail($?, "@$cmd failed") if $?;
}
-sub _post_augment_mbox { # open a compressor process from top-level process
+sub _post_augment_mbox { # open a compressor process from top-level lei-daemon
my ($self, $lei) = @_;
my $zsfx = $self->{zsfx} or return;
my $cmd = PublicInbox::MboxReader::zsfx2cmd($zsfx, undef, $lei);
diff --git a/lib/PublicInbox/MboxReader.pm b/lib/PublicInbox/MboxReader.pm
index beffabe8..e4209022 100644
--- a/lib/PublicInbox/MboxReader.pm
+++ b/lib/PublicInbox/MboxReader.pm
@@ -1,10 +1,10 @@
-# Copyright (C) 2020-2021 all contributors <meta@public-inbox.org>
+# Copyright (C) all contributors <meta@public-inbox.org>
# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
-# reader for mbox variants we support
+# reader for mbox variants we support (and also sets up commands for writing)
package PublicInbox::MboxReader;
use strict;
-use v5.10.1;
+use v5.10.1; # check regexps before v5.12
use Data::Dumper;
$Data::Dumper::Useqq = 1; # should've been the default, for bad data
@@ -141,10 +141,9 @@ sub reads {
# all of these support -c for stdout and -d for decompression,
# mutt is commonly distributed with hooks for gz, bz2 and xz, at least
-# { foo => '' } means "--foo" is passed to the command-line,
-# otherwise { foo => '--bar' } passes "--bar"
+# { foo => '' } means "--foo" is passed to the command-line
my %zsfx2cmd = (
- gz => [ qw(GZIP pigz gzip) ],
+ gz => [ qw(GZIP pigz gzip), { rsyncable => '' } ],
bz2 => [ 'bzip2', {} ],
xz => [ 'xz', {} ],
# don't add new entries here unless MUA support is widely available
@@ -173,28 +172,9 @@ sub zsfx2cmd ($$$) {
}
$cmd[0] // die join(' or ', @info)." missing for .$zsfx";
- # not all gzip support --rsyncable, FreeBSD gzip doesn't even exit
- # with an error code
- if (!$decompress && $cmd[0] =~ m!/gzip\z! && !defined($cmd_opt)) {
- pipe(my ($r, $w)) or die "pipe: $!";
- open my $null, '+>', '/dev/null' or die "open: $!";
- my $rdr = { 0 => $null, 1 => $null, 2 => $w };
- my $tst = [ $cmd[0], '--rsyncable' ];
- my $pid = PublicInbox::Spawn::spawn($tst, undef, $rdr);
- close $w;
- my $err = do { local $/; <$r> };
- waitpid($pid, 0) == $pid or die "BUG: waitpid: $!";
- $cmd_opt = $err ? {} : { rsyncable => '' };
- push(@$x, $cmd_opt);
- }
- for my $bool (keys %$cmd_opt) {
- my $switch = $cmd_opt->{$bool} // next;
- push @cmd, '--'.($switch || $bool);
- }
- for my $key (qw(rsyncable)) { # support compression level?
- my $switch = $cmd_opt->{$key} // next;
- my $val = $lei->{opt}->{$key} // next;
- push @cmd, $switch, $val;
+ # only for --rsyncable. TODO: support compression level?
+ for my $key (keys %$cmd_opt) {
+ push @cmd, '--'.$key if $lei->{opt}->{$key};
}
\@cmd;
}
diff --git a/t/lei-convert.t b/t/lei-convert.t
index e1849ff7..115e7ed0 100644
--- a/t/lei-convert.t
+++ b/t/lei-convert.t
@@ -1,12 +1,13 @@
#!perl -w
-# Copyright (C) 2021 all contributors <meta@public-inbox.org>
+# Copyright (C) all contributors <meta@public-inbox.org>
# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
-use strict; use v5.10.1; use PublicInbox::TestCommon;
+use v5.12; use PublicInbox::TestCommon;
use PublicInbox::MboxReader;
use PublicInbox::MdirReader;
use PublicInbox::NetReader;
use PublicInbox::Eml;
use IO::Uncompress::Gunzip;
+use autodie qw(open);
require_mods(qw(lei -imapd -nntpd Mail::IMAPClient Net::NNTP));
my ($tmpdir, $for_destroy) = tmpdir;
my $sock = tcp_server;
@@ -125,5 +126,27 @@ test_lei({ tmpdir => $tmpdir }, sub {
like($md[0], qr/:2,S\z/, "`seen' flag set in Maildir");
lei_ok(qw(convert -o mboxrd:/dev/stdout), "$d/md2");
like($lei_out, qr/^Status: RO/sm, "`seen' flag preserved");
+
+ SKIP: {
+ my $ok;
+ for my $x (($ENV{GZIP}//''), qw(pigz gzip)) {
+ $x && `$x -h 2>&1` =~ /--rsyncable\b/s or next;
+ $ok = $x;
+ last;
+ }
+ skip 'pigz || gzip do not support --rsyncable' if !$ok;
+ lei_ok qw(convert --rsyncable), "mboxrd:$d/qp.gz",
+ '-o', "mboxcl2:$d/qp2.gz";
+ undef $fh; # necessary to make IO::Uncompress::Gunzip happy
+ open $fh, '<', "$d/qp2.gz";
+ $fh = IO::Uncompress::Gunzip->new($fh, MultiStream => 1);
+ my @tmp;
+ PublicInbox::MboxReader->mboxcl2($fh, sub {
+ my ($eml) = @_;
+ $eml->header_set($_) for qw(Content-Length Lines);
+ push @tmp, $eml;
+ });
+ is_deeply(\@tmp, \@bar, 'read rsyncable-gzipped mboxcl2');
+ }
});
done_testing;
^ permalink raw reply related [relevance 46%]
* [PATCH 0/2] lei: support reading inboxes & extindex w/o search
@ 2023-09-30 0:36 71% Eric Wong
2023-09-30 0:36 41% ` [PATCH 2/2] lei convert: support reading from v1, v2, and extindex Eric Wong
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2023-09-30 0:36 UTC (permalink / raw)
To: meta
This works on completely unindexed inboxes, even, as long as the
inbox.lock (or ssoma.lock) file exists.
Eric Wong (2):
lei_input: always prefix `maildir:' internally
lei convert: support reading from v1, v2, and extindex
lib/PublicInbox/ExtSearch.pm | 6 +-
lib/PublicInbox/LeiInput.pm | 113 ++++++++++++++++++++++++++---------
t/extsearch.t | 24 ++++++++
t/lei-convert.t | 40 +++++++++++++
4 files changed, 153 insertions(+), 30 deletions(-)
^ permalink raw reply [relevance 71%]
* [PATCH 2/2] lei convert: support reading from v1, v2, and extindex
2023-09-30 0:36 71% [PATCH 0/2] lei: support reading inboxes & extindex w/o search Eric Wong
@ 2023-09-30 0:36 41% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2023-09-30 0:36 UTC (permalink / raw)
To: meta
We should be able to dump all public-inbox and extindex directories
to Maildir/mbox* or IMAP folders. Even unindexed inboxes can be
dumped as long as inbox.lock (or ssoma.lock) exists.
This change likely works for `lei tag' and other lei_input-using
things, as well, but that's untested at the moment. I mainly
want to be able to use `lei convert' to benchmark some upcoming
changes...
---
lib/PublicInbox/ExtSearch.pm | 6 ++--
lib/PublicInbox/LeiInput.pm | 70 +++++++++++++++++++++++++++++++-----
t/extsearch.t | 24 +++++++++++++
t/lei-convert.t | 40 +++++++++++++++++++++
4 files changed, 129 insertions(+), 11 deletions(-)
diff --git a/lib/PublicInbox/ExtSearch.pm b/lib/PublicInbox/ExtSearch.pm
index fa49a1d0..d43c23e6 100644
--- a/lib/PublicInbox/ExtSearch.pm
+++ b/lib/PublicInbox/ExtSearch.pm
@@ -33,9 +33,11 @@ sub misc {
# same as per-inbox ->over, for now...
sub over {
my ($self) = @_;
- $self->{over} //= do {
+ $self->{over} // eval {
PublicInbox::Inbox::_cleanup_later($self);
- PublicInbox::Over->new("$self->{xpfx}/over.sqlite3");
+ my $over = PublicInbox::Over->new("$self->{xpfx}/over.sqlite3");
+ $over->dbh; # may die
+ $self->{over} = $over;
};
}
diff --git a/lib/PublicInbox/LeiInput.pm b/lib/PublicInbox/LeiInput.pm
index f88c5374..58069b0a 100644
--- a/lib/PublicInbox/LeiInput.pm
+++ b/lib/PublicInbox/LeiInput.pm
@@ -125,6 +125,51 @@ sub handle_http_input ($$@) {
$lei->child_error($?, "@$cmd failed: @err") if @err;
}
+sub oid2eml { # git->cat_async cb
+ my ($bref, $oid, $type, $size, $self) = @_;
+ if ($type eq 'blob') {
+ $self->input_eml_cb(PublicInbox::Eml->new($bref));
+ } else {
+ warn "W: $oid is type=$type\n";
+ }
+}
+
+sub each_ibx_eml_unindexed {
+ my ($self, $ibx, @args) = @_;
+ $ibx->isa('PublicInbox::Inbox') or return $self->{lei}->fail(<<EOM);
+unindexed extindex $ibx->{topdir} not supported
+EOM
+ require PublicInbox::SearchIdx;
+ my $n = $ibx->max_git_epoch;
+ my @g = defined($n) ? map { $ibx->git_epoch($_) } (0..$n) : ($ibx->git);
+ my $sync = { D => {}, ibx => $ibx }; # D => {} filters out deletes
+ my ($f, $at, $ct, $oid, $cmt);
+ for my $git (grep defined, @g) {
+ my $s = PublicInbox::SearchIdx::log2stack($sync, $git, 'HEAD');
+ while (($f, $at, $ct, $oid, $cmt) = $s->pop_rec) {
+ $git->cat_async($oid, \&oid2eml, $self) if $f eq 'm';
+ }
+ $git->cleanup; # wait all
+ }
+}
+
+sub each_ibx_eml {
+ my ($self, $ibx, @args) = @_; # TODO: is @args used at all?
+ my $over = $ibx->over or return each_ibx_eml_unindexed(@_);
+ my $git = $ibx->git;
+ my $prev = 0;
+ my $smsg;
+ my $ids = $over->ids_after(\$prev);
+ while (@$ids) {
+ for (@$ids) {
+ $smsg = $over->get_art($_) // next;
+ $git->cat_async($smsg->{blob}, \&oid2eml, $self);
+ }
+ $ids = $over->ids_after(\$prev);
+ }
+ $git->cat_async_wait;
+}
+
sub input_path_url {
my ($self, $input, @args) = @_;
my $lei = $self->{lei};
@@ -191,6 +236,12 @@ sub input_path_url {
$self->can('input_maildir_cb'),
$self, @args);
}
+ } elsif (-d _ && $ifmt =~ /\A(?:v1|v2)\z/) {
+ my $ibx = PublicInbox::Inbox->new({inboxdir => $input});
+ each_ibx_eml($self, $ibx, @args);
+ } elsif (-d _ && $ifmt eq 'extindex') {
+ my $esrch = PublicInbox::ExtSearch->new($input);
+ each_ibx_eml($self, $esrch, @args);
} elsif ($self->{missing_ok} && !-e $input) { # don't ->fail
if ($lei->{cmd} eq 'p2q') {
my $fp = [ qw(git format-patch --stdout -1), $input ];
@@ -308,9 +359,9 @@ sub prepare_inputs { # returns undef on error
require PublicInbox::MboxReader;
PublicInbox::MboxReader->reads($ifmt) or return
$lei->fail("$ifmt not supported");
- } elsif (-d $input_path) {
- $ifmt eq 'maildir' or return # TODO v1/v2/ei
- $lei->fail("$ifmt not supported");
+ } elsif (-d $input_path) { # TODO extindex
+ $ifmt =~ /\A(?:maildir|v1|v2|extindex)\z/ or
+ return$lei->fail("$ifmt not supported");
$input = $input_path;
add_dir $lei, $istate, $ifmt, \$input;
} elsif ($self->{missing_ok} && !-e _) {
@@ -350,12 +401,12 @@ $input is `eml', not --in-format=$in_fmt
push @f, $input;
} elsif (-d "$input/new" && -d "$input/cur") {
add_dir $lei, $istate, 'maildir', \$input;
- } elsif (-e "$input/inbox.lock") { # TODO
- $lei->fail('v2 inputs not yet supported (TODO)');
- #add_dir $lei, $istate, 'v2', \$input;
- } elsif (-e "$input/ssoma.lock") { # TODO
- $lei->fail('v1 inputs not yet supported (TODO)');
- #add_dir $lei, $istate, 'v1', \$input;
+ } elsif (-e "$input/inbox.lock") {
+ add_dir $lei, $istate, 'v2', \$input;
+ } elsif (-e "$input/ssoma.lock") {
+ add_dir $lei, $istate, 'v1', \$input;
+ } elsif (-e "$input/ei.lock") {
+ add_dir $lei, $istate, 'extindex', \$input;
} elsif ($self->{missing_ok} && !-e $input) {
if ($lei->{cmd} eq 'p2q') {
# will run "git format-patch"
@@ -401,6 +452,7 @@ $input is `eml', not --in-format=$in_fmt
$lei->refresh_watches;
}
}
+ require PublicInbox::ExtSearch if $istate->{extindex};
$self->{inputs} = $inputs;
}
diff --git a/t/extsearch.t b/t/extsearch.t
index 8ded3382..19eaf3b5 100644
--- a/t/extsearch.t
+++ b/t/extsearch.t
@@ -581,4 +581,28 @@ EOM
}
}
+test_lei(sub {
+ my $d = "$home/extindex";
+ lei_ok('convert', '-o', "$home/md1", $d);
+ lei_ok('convert', '-o', "$home/md2", "extindex:$d");
+ my $dst = [];
+ my $cb = sub { push @$dst, $_[2]->as_string };
+ require PublicInbox::MdirReader;
+ PublicInbox::MdirReader->new->maildir_each_eml("$home/md1", $cb);
+ my @md1 = sort { $a cmp $b } @$dst;
+ ok(scalar(@md1), 'dumped messages to md1');
+ $dst = [];
+ PublicInbox::MdirReader->new->maildir_each_eml("$home/md2", $cb);
+ @$dst = sort { $a cmp $b } @$dst;
+ is_deeply($dst, \@md1,
+ "convert from extindex w/ or w/o `extindex' prefix");
+
+ use autodie qw(unlink);
+ my @o = glob "$home/extindex/ei*/over.sqlite*";
+ unlink(@o);
+ ok(!lei('convert', '-o', "$home/fail", "extindex:$d"));
+ like($lei_err, qr/unindexed .*?not supported/,
+ 'noted unindexed extindex is unsupported');
+});
+
done_testing;
diff --git a/t/lei-convert.t b/t/lei-convert.t
index 115e7ed0..d75110cb 100644
--- a/t/lei-convert.t
+++ b/t/lei-convert.t
@@ -7,6 +7,8 @@ use PublicInbox::MdirReader;
use PublicInbox::NetReader;
use PublicInbox::Eml;
use IO::Uncompress::Gunzip;
+use File::Path qw(remove_tree);
+use PublicInbox::Spawn qw(which);
use autodie qw(open);
require_mods(qw(lei -imapd -nntpd Mail::IMAPClient Net::NNTP));
my ($tmpdir, $for_destroy) = tmpdir;
@@ -148,5 +150,43 @@ test_lei({ tmpdir => $tmpdir }, sub {
});
is_deeply(\@tmp, \@bar, 'read rsyncable-gzipped mboxcl2');
}
+ my $cp = which('cp') or xbail 'cp(1) not available (WTF?)';
+ for my $v (1, 2) {
+ my $ibx_dir = "$ro_home/t$v";
+ lei_ok qw(convert -f mboxrd), $ibx_dir,
+ \"dump v$v inbox to mboxrd";
+ my $out = $lei_out;
+ lei_ok qw(convert -f mboxrd), "v$v:$ibx_dir",
+ \"dump v$v inbox to mboxrd w/ v$v:// prefix";
+ is $out, $lei_out, "v$v:// prefix accepted";
+ open my $fh, '<', \$out;
+ my (@mb, @md, @md2);
+ PublicInbox::MboxReader->mboxrd($fh, sub {
+ $_[0]->header_set('Status');
+ push @mb, $_[0]->as_string;
+ });
+ undef $out;
+ ok(scalar(@mb), 'got messages output');
+ my $mdir = "$d/v$v-mdir";
+ lei_ok qw(convert -o), $mdir, $ibx_dir,
+ \"dump v$v inbox to Maildir";
+ PublicInbox::MdirReader->new->maildir_each_eml($mdir, sub {
+ push @md, $_[2]->as_string;
+ });
+ @md = sort { $a cmp $b } @md;
+ @mb = sort { $a cmp $b } @mb;
+ is_deeply(\@mb, \@md, 'got matching inboxes');
+ xsys_e([$cp, '-Rp', $ibx_dir, "$d/tv$v" ]);
+ remove_tree($mdir, "$d/tv$v/public-inbox",
+ glob("$d/tv$v/xap*"));
+
+ lei_ok qw(convert -o), $mdir, "$d/tv$v",
+ \"dump u indexed v$v inbox to Maildir";
+ PublicInbox::MdirReader->new->maildir_each_eml($mdir, sub {
+ push @md2, $_[2]->as_string;
+ });
+ @md2 = sort { $a cmp $b } @md2;
+ is_deeply(\@md, \@md2, 'got matching inboxes even unindexed');
+ }
});
done_testing;
^ permalink raw reply related [relevance 41%]
* [PATCH] t/lei-convert: fix uninitialized variable w/o pigz
@ 2023-09-30 16:17 71% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2023-09-30 16:17 UTC (permalink / raw)
To: meta
`backtick` captures return `undef' when a command is missing
Fixes: 5df0446abcca (lei: don't gzip --rsyncable by default for mbox*)
---
t/lei-convert.t | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/t/lei-convert.t b/t/lei-convert.t
index d75110cb..84b57f81 100644
--- a/t/lei-convert.t
+++ b/t/lei-convert.t
@@ -132,7 +132,7 @@ test_lei({ tmpdir => $tmpdir }, sub {
SKIP: {
my $ok;
for my $x (($ENV{GZIP}//''), qw(pigz gzip)) {
- $x && `$x -h 2>&1` =~ /--rsyncable\b/s or next;
+ $x && (`$x -h 2>&1`//'') =~ /--rsyncable\b/s or next;
$ok = $x;
last;
}
^ permalink raw reply related [relevance 71%]
Results 1001-1200 of ~1329 next (newer) | prev (older) | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2021-10-21 21:10 [PATCH 00/15] use RENAME_NOREPLACE on Linux 3.15+ Eric Wong
2021-10-21 21:10 ` [PATCH 03/15] t/lei-p2q: extra diagnostics Eric Wong
2023-09-21 10:23 71% ` Eric Wong
2021-10-25 22:24 [PATCH] t/index-git-times: support non-master default branch Thomas Weißschuh
2021-10-25 22:58 ` Eric Wong
2021-10-26 5:05 ` Thomas Weißschuh
2021-10-26 5:28 ` Eric Wong
2021-10-26 18:43 ` Thomas Weißschuh
2021-10-27 21:15 ` lei-q-remote-import failures [was: [PATCH] t/index-git-times: support non-master default branch] Eric Wong
2021-10-27 21:24 ` Thomas Weißschuh
2021-10-27 23:48 63% ` Eric Wong
2021-10-28 14:03 71% ` Thomas Weißschuh
2021-10-28 19:16 68% ` [PATCH] test_common: clear XDG_CACHE_HOME before lei tests Eric Wong
2021-10-28 19:22 71% ` Thomas Weißschuh
2021-10-27 21:09 [PATCH] lei q: fix remote import accounting Eric Wong
2021-10-27 21:28 71% ` Thomas Weißschuh
2021-10-28 6:17 71% [PATCH] lei add-watch: ensure folders are known to mail_sync.sqlite3 Eric Wong
2021-10-28 11:14 71% [PATCH 0/8] lei: docs and cleanups Eric Wong
2021-10-28 11:14 68% ` [PATCH 1/8] xt/net_writer_imap: test "lei convert" w/ IMAP source Eric Wong
2021-10-28 11:14 71% ` [PATCH 2/8] lei convert: use "--output" in failure message Eric Wong
2021-10-28 11:14 62% ` [PATCH 3/8] doc: lei-convert: various updates and cleanups Eric Wong
2021-10-28 11:14 70% ` [PATCH 4/8] doc: lei blob: wording fixups, describe --remote Eric Wong
2021-10-28 11:14 71% ` [PATCH 5/8] lei convert: remove redundant input_net_cb Eric Wong
2021-10-28 11:14 70% ` [PATCH 6/8] doc: lei-add-watch: add warning about unreliability Eric Wong
2021-10-28 11:15 71% ` [PATCH 7/8] lei sucks: show nproc in CPU info Eric Wong
2021-10-28 11:15 63% ` [PATCH 8/8] lei rm: move generic input_maildir_cb to LeiInput parent class Eric Wong
2021-10-30 8:11 71% [PATCH 0/5] lei: fix various SIGPIPE problems Eric Wong
2021-10-30 8:11 71% ` [PATCH 1/5] lei: do not access {sock} after SIGPIPE Eric Wong
2021-10-30 8:11 71% ` [PATCH 5/5] doc: lei-security: add a note about core dumps Eric Wong
2021-10-31 9:10 71% [PATCH 0/2] lei: mail-sync docs + compat fix Eric Wong
2021-10-31 9:10 29% ` [PATCH 1/2] doc: add lei-mail-sync-overview manpage Eric Wong
2021-10-31 16:32 71% ` Kyle Meyer
2021-10-31 17:30 64% ` Eric Wong
2021-11-01 19:00 71% [PATCH] doc: lei-config: fix missing =back Eric Wong
2021-11-02 9:24 60% [PATCH] t/lei-refresh-mail-sync: speed up test on FreeBSD 12 Eric Wong
2021-11-02 18:14 71% [PATCH 0/3] lei: more coherent input handling Eric Wong
2021-11-02 18:14 71% ` [PATCH 1/3] lei mail-diff: do not default to 'eml' Eric Wong
2021-11-02 18:14 41% ` [PATCH 2/3] lei: simplify common LeiInput users with ->wq1_start Eric Wong
2021-11-02 18:47 71% ` Eric Wong
2021-11-02 18:14 65% ` [PATCH 3/3] lei <rediff|rm|tag>: stdin implies `-F eml' Eric Wong
2021-11-02 23:55 43% [PATCH] doc: lei-q: document SEARCH TERMS prefixes Eric Wong
2021-11-03 0:28 71% ` Kyle Meyer
2021-11-03 1:28 71% Initial Fedora packaging for lei Konstantin Ryabitsev
2021-11-03 8:34 [PATCH 0/3] doc: extindex-related updates Eric Wong
2021-11-03 8:34 68% ` [PATCH 2/3] doc: -clone|lei add-external: add bit about the Makefile Eric Wong
[not found] <lorelei.part1.202111051304.mdtebsxahljcrxak@meerkat.local>
[not found] ` <CAL_JsqJBh1O3H2-P07AHzVq0x89BoP_N6P=rT5up6=3QyF_B0Q@mail.gmail.com>
2021-11-08 20:22 86% ` lei: incorrect quoting on saved searches (was Re: lore+lei: getting started) Konstantin Ryabitsev
2021-11-08 20:49 90% ` Eric Wong
2021-11-08 21:36 90% ` Konstantin Ryabitsev
2021-11-08 21:48 90% ` Eric Wong
2021-11-08 22:36 88% ` Konstantin Ryabitsev
2021-11-08 22:57 90% ` Eric Wong
2021-11-08 20:53 90% ` Rob Herring
2021-11-08 23:39 53% [PATCH] lei q|up: fix saved searches for single-phrase search Eric Wong
2021-11-10 10:25 71% [PATCH 0/4] "lei q" rawstr fallout + fixes Eric Wong
2021-11-10 10:28 68% ` [PATCH 2/4] lei up: infer rawstr from old searches via trailing "\n" Eric Wong
2021-11-10 10:28 70% ` [PATCH 3/4] lei q: disallow "\n" in argv[] elements Eric Wong
2021-11-10 10:28 71% ` [PATCH 4/4] lei q: make HTTP(S) query strings even less ugly Eric Wong
2021-11-10 10:33 71% [PATCH] t/lei-watch: test with with higher sleep Eric Wong
2021-11-12 11:08 71% [PATCH] lei forget-search: add help for --prune Eric Wong
2021-11-14 20:41 71% lei spawns mua before results are written Leah Neukirchen
2021-11-15 0:31 71% ` Eric Wong
2021-11-19 12:42 'make test' failures while packaging for openSUSE Jörg Rödel
2021-11-19 18:54 ` Eric Wong
2021-11-19 21:46 ` Jörg Rödel
2021-11-22 6:55 ` [PATCH] searchidx: add some extra diagnostics for odd message Eric Wong
2021-11-22 7:42 96% ` [PATCH] t/lei-mirror: skip lei comparisons if lei missing Eric Wong
2021-11-22 13:47 99% ` Jörg Rödel
2021-11-22 17:24 99% ` Eric Wong
2021-11-19 20:49 71% RFC: should lei inject its own "Received:" header? Konstantin Ryabitsev
2021-11-21 10:13 71% ` Eric Wong
2021-11-20 15:30 How to delete caches Johannes Altmanninger
2021-11-20 16:36 71% ` lei q error after deleting cache (can't find mail_sync.sqlite3) Johannes Altmanninger
2021-11-22 18:38 71% [PATCH] lei: always use 3-arg open perlop Eric Wong
2021-12-08 1:07 Test failures with 1.7.0 Julien Moutinho
2021-12-08 4:08 ` Eric Wong
2021-12-08 10:56 ` Dominique Martinet
2021-12-09 1:37 ` Julien Moutinho
2022-02-17 21:02 63% ` [PATCH] t/lei-sigpipe: attempt to improve diagnostics for stuck test Eric Wong
2022-02-20 1:38 51% ` Julien Moutinho
2022-02-22 6:44 71% ` Eric Wong
2022-02-27 4:15 71% ` Julien Moutinho
2022-02-27 6:41 71% ` Julien Moutinho
2022-02-27 7:23 71% ` Dominique Martinet
2022-02-27 8:04 71% ` Julien Moutinho
2022-02-27 11:17 62% ` [PATCH] t/lei-sigpipe: ensure SIGPIPE is unblocked for this test Eric Wong
2022-03-11 10:42 69% ` [PATCH] t/lei-sigpipe.t: ensure SIGPIPE is not ignored instead of not blocked Julien Moutinho
2022-03-14 22:14 71% ` Eric Wong
2022-03-15 2:56 71% ` Julien Moutinho
2021-12-30 23:04 60% lei q: importing messages when specifying '-f *json*'? Kyle Meyer
[not found] <CABPp-BF_xsOpQ6GSaWs9u9JcnPQT_OXP-gCsAuxPtMj-X1tgOg@mail.gmail.com>
[not found] ` <211203.86sfv9qwdm.gmgdl@evledraar.gmail.com>
[not found] ` <20211203202427.o575sgrx4auqkmjp@meerkat.local>
2021-12-06 16:12 ` Large delays in mailing list delivery? Ævar Arnfjörð Bjarmason
2021-12-06 16:36 ` Eric Wong
2022-02-02 9:34 48% ` Using public-inbox+lei+Emacs+mu+mu4e (was: Large delays in mailing list delivery?) Ævar Arnfjörð Bjarmason
2022-02-07 21:27 69% ` Eric Wong
2022-03-12 21:04 87% Failed 'lei q' blocks 'lei init' from working Nícolas F. R. A. Prado
2022-03-13 0:06 90% ` Kyle Meyer
2022-04-05 8:18 37% [PATCH] lei: always open mail_sync.sqlite3 R/W Eric Wong
2022-04-18 9:50 71% [PATCH 0/4] lei: finish wiring up pure-Perl stuff for Linux Eric Wong
2022-04-18 9:50 71% ` [PATCH 1/4] lei: clobber recvmsg buffer on errors Eric Wong
2022-04-18 9:50 53% ` [PATCH 3/4] lei: wire up pure Perl sendmsg/recvmsg for Linux users Eric Wong
2022-04-21 11:59 58% [PATCH] lei: commit store on interrupted partial imports Eric Wong
2022-04-23 22:03 [PATCH 0/2] version bumps, Perl 5.12 in _some_ places Eric Wong
2022-04-23 22:03 61% ` [PATCH 2/2] lei: move to v5.12 to avoid "use strict" Eric Wong
2022-04-30 21:04 71% [PATCH] lei: improve diagnosis of errors from children Eric Wong
2022-04-30 21:29 71% [PATCH] lei refresh-mail-sync: filter NNTP(S) from --all Eric Wong
2022-05-02 18:10 71% [PATCH] lei import: add label completions (+L:$LABEL) Eric Wong
2022-05-03 11:15 70% Trouble running lei Filipe Manana
2022-05-03 11:37 71% ` Eric Wong
2022-05-03 12:50 71% ` Filipe Manana
2022-05-03 15:24 71% ` Konstantin Ryabitsev
2022-05-03 20:32 71% ` Eric Wong
2022-06-20 19:27 [PATCH 0/3] search indexing improvements Eric Wong
2022-06-20 19:27 ` [PATCH 2/3] search: support "patchid:" prefix (git patch-id --stable) Eric Wong
2022-06-20 20:01 ` Kyle Meyer
2022-06-21 10:37 ` [PATCH 4/3] search: add help for patchid: prefix Eric Wong
2022-06-22 7:50 58% ` [PATCH 5/3] doc: lei-q: regenerate for patchid: help Eric Wong
2022-06-29 16:15 71% lei missing mails Rob Herring
2022-06-29 16:30 71% ` Eric Wong
2022-06-29 16:53 69% ` Rob Herring
2022-06-29 17:27 71% ` Eric Wong
2022-06-29 22:01 59% ` Rob Herring
2022-06-30 8:55 71% ` Eric Wong
2022-07-07 9:48 71% ` Eric Wong
2022-07-11 21:17 71% ` Rob Herring
2022-07-11 21:59 68% ` Rob Herring
2022-07-18 23:41 71% ` Eric Wong
2022-07-07 9:40 71% [PATCH 0/2] lei: minor diagnostic improvement Eric Wong
2022-07-07 9:40 71% ` [PATCH 1/2] lei_xsearch: simplify lei/store import check Eric Wong
2022-07-07 9:40 53% ` [PATCH 2/2] lei: track seen messages to note duplicates Eric Wong
2022-07-19 22:42 71% [PATCH 0/2] lei inotify/EVFILT_VNODE deadlock fix Eric Wong
2022-07-19 22:42 51% ` [PATCH 1/2] lei: avoid deadlock on inotify/EVFILT_VNODE wakeups Eric Wong
2022-07-19 22:42 71% ` [PATCH 2/2] lei note-event: inline note_event_arm_done Eric Wong
2022-08-04 7:23 [PATCH] TODO: remove done items, adjust/add/abandon some Eric Wong
2022-12-09 1:41 65% ` FUSE3 vs read-write IMAP for lei Eric Wong
2023-02-20 19:27 71% ` Eric Wong
2022-08-16 3:44 71% [PATCH] lei: do not wait for sto->done on disconnected EOF Eric Wong
2022-08-17 9:33 71% [PATCH 0/4] lei reindex, minor tweaks Eric Wong
2022-08-17 9:33 71% ` [PATCH 2/4] lei inspect: less scary exception for invalid "docid:" inspect Eric Wong
2022-08-17 9:33 71% ` [PATCH 3/4] lei/store: reduce work when accessing mail_sync.sqlite3 Eric Wong
2022-08-17 9:33 69% ` [PATCH 4/4] lei reindex: new command to reindex lei/store Eric Wong
2022-08-18 7:22 90% ` Eric Wong
2022-08-19 9:07 71% [PATCH 0/4] lei reindex-related stuff Eric Wong
2022-08-19 9:07 90% ` [PATCH 1/4] lei reindex: account for parallel lei/store users Eric Wong
2022-08-19 9:07 62% ` [PATCH 2/4] tests: add some basic "lei reindex" tests Eric Wong
2022-08-19 9:07 71% ` [PATCH 4/4] lei/store: reindex culls over-indexed messages Eric Wong
2022-08-30 9:10 [PATCH 0/3] misc doc updates I missed for 1.9 :x Eric Wong
2022-08-30 9:10 71% ` [PATCH 1/3] Makefile.PL: add lei-reindex manpage Eric Wong
2022-09-02 18:26 71% [PATCH 0/2] lei-related import tweaks Eric Wong
2022-09-02 18:26 69% ` [PATCH 1/2] lei/store: do not write info/refs file Eric Wong
2022-09-09 10:09 imap: "Can't use an undefined value as a subroutine reference" Ricardo Ribalda
2022-09-09 17:44 64% ` [PATCH] lei: add diagnostics for IMAP writer failures Eric Wong
2022-09-09 18:00 86% ` [PATCH] doc: document --jobs for `lei q' and `lei up' Eric Wong
2022-09-09 20:35 71% ` [PATCH] lei: add diagnostics for IMAP writer failures Ricardo Ribalda
2022-09-10 1:18 59% ` [PATCH v2] lei: bail out earlier on " Eric Wong
2022-09-10 19:34 37% ` Ricardo Ribalda
2022-09-10 19:50 71% ` Eric Wong
2022-09-10 19:53 70% ` Ricardo Ribalda
2022-09-10 20:19 71% ` Eric Wong
2022-11-14 8:07 64% ` [PATCH] lei q|up: limit default write --jobs for IMAP(S) Eric Wong
2022-09-10 1:35 71% [PATCH] lei: fix --help for --jobs with `up' and `q' Eric Wong
2022-09-30 9:21 71% [PATCH 0/4] fixes noticed while diagnosing t/lei-up.t Eric Wong
2022-09-30 9:21 56% ` [PATCH 2/4] t/lei-up: improve diagnostics for this test Eric Wong
2022-09-30 9:21 61% ` [PATCH 3/4] lei_to_mail: propagate errors to script/lei Eric Wong
2022-09-30 17:20 65% ` SQLite <3.8.3 was broken on fork (was: fixes noticed while diagnosing t/lei-up.t) Eric Wong
2022-10-01 0:33 60% [PATCH] lei: force --jobs=1,1 for SQLite < 3.8.3 Eric Wong
2022-10-30 4:03 70% [Need Help] lei add quotes at the search Hangbin Liu
2022-10-30 5:13 71% ` Eric Wong
2022-10-30 7:08 59% ` Hangbin Liu
2022-10-30 23:06 63% ` Eric Wong
2022-10-31 7:36 71% ` Hangbin Liu
2022-10-31 7:47 71% ` Hangbin Liu
2022-10-31 21:52 90% [PATCH] lei up: improve error for multiple lei.q values Eric Wong
2022-11-01 9:36 60% [PATCH] lei: fix globbing semantics to match end-of-filename Eric Wong
2022-11-03 0:48 [PATCH 0/6] doc: linkify HTML harder Eric Wong
2022-11-03 0:48 62% ` [PATCH 2/6] doc: lei: improve description of *-search commands Eric Wong
2022-11-03 0:48 71% ` [PATCH 3/6] doc: txt2pre: linkify "lei COMMAND" form Eric Wong
2022-11-03 0:48 90% ` [PATCH 5/6] doc: lei-import: link to lei-store-format(5) Eric Wong
2022-11-03 2:03 90% ` Eric Wong
2022-11-03 0:48 90% ` [PATCH 6/6] txt2pre: linkify lei/store => lei-store-format.html Eric Wong
2022-12-01 11:21 90% [PATCH 0/2] lei - expanding relative paths for `lei up' Eric Wong
2022-12-01 11:21 67% ` [PATCH 1/2] lei: stricter external checks for valid $GIT_DIR/objects Eric Wong
2023-01-11 10:55 [PATCH] www: /$INBOX/$MSGID/d/ to diff reused Message-IDs Eric Wong
2023-01-11 11:00 42% ` [1/2 PATCH] hoist MailDiff and ContentDigestDbg out of lei Eric Wong
2023-01-17 7:18 [PATCH 00/12] improve process reaping Eric Wong
2023-01-17 7:19 39% ` [PATCH 11/12] ipc+lei: switch to awaitpid Eric Wong
2023-01-29 10:30 [PATCH 0/2] allow OpenSSL SHA-(1|256) use if installed Eric Wong
2023-01-29 10:30 63% ` [PATCH 2/2] content_digest_dbg: convert to arrayref and limit to lei Eric Wong
2023-01-29 22:58 71% [PATCH 0/2] fix xt/lei-auth-fail.t Eric Wong
2023-01-29 22:58 69% ` [PATCH 2/2] xt/lei-auth-fail: use valid label name Eric Wong
2023-01-31 0:05 70% [PATCH] lei: drop -watches and -lei_note_event from workers Eric Wong
2023-02-12 3:12 61% [PATCH] t/lei-refresh-mail-sync: avoid kill+sleep loop Eric Wong
2023-02-13 16:06 63% lei q -tt doesn't work properly? Maxim Mikityanskiy
2023-02-14 2:42 66% ` [PATCH] lei q: do not collapse threads with `-tt' Eric Wong
2023-02-26 12:17 71% ` Maxim Mikityanskiy
2023-02-26 17:09 69% ` Eric Wong
2023-02-26 17:15 71% ` [PATCH] doc: note "lei q -tt" is broken with HTTP(S) remotes Eric Wong
2023-03-09 19:28 [PATCH 0/6] various doc updates Eric Wong
2023-03-09 19:28 71% ` [PATCH 4/6] doc: lei import: add hints about nntp.* and imap.* config options Eric Wong
2023-03-09 19:28 68% ` [PATCH 5/6] doc: lei config: update with --edit and --list examples Eric Wong
2023-03-23 21:45 46% [PATCH] lei: improve bash completion involving colons Eric Wong
2023-03-23 22:05 71% repeat `lei import' users? Eric Wong
2023-03-28 1:00 71% Issues with `lei` as non-root Louis DeLosSantos
2023-03-28 1:32 71% ` Eric Wong
2023-03-28 2:30 45% ` Louis DeLosSantos
2023-03-28 2:52 71% ` Eric Wong
2023-03-28 3:05 42% ` Louis DeLosSantos
2023-03-28 3:38 71% ` Eric Wong
2023-03-28 4:08 71% ` Louis DeLosSantos
2023-03-28 10:53 71% [PATCH] t/lei-refresh-mail-sync: improve test reliability Eric Wong
2023-04-29 7:18 71% [PATCH] t/lei-import-nntp: dump $lei_err on failure Eric Wong
2023-06-08 18:26 71% [PATCH] t/lei.t: quiet newline warning on older Perls Eric Wong
2023-06-15 0:08 71% [PATCH] doc: lei q: document v2:$INBOX_DIR output format Eric Wong
2023-06-15 8:46 60% [PATCH] lei import: set +(L|kw) on already-imported blobs Eric Wong
2023-06-15 9:50 51% [PATCH] lei: make --dedupe=content always account for Message-IDs Eric Wong
2023-09-14 12:12 71% [PATCH] t/lei-mirror: do not bail out on `make help' failure Eric Wong
2023-09-14 23:10 64% [PATCH] lei: ensure --stdin sets %ENV and $current_lei Eric Wong
2023-09-15 10:11 63% [PATCH] lei: ensure we run DESTROY|END at daemon exit w/ kqueue Eric Wong
2023-09-15 21:08 67% RFC: lei searches managed by users in git Konstantin Ryabitsev
2023-09-15 22:47 55% ` Eric Wong
2023-09-22 18:37 66% [PATCH] t/lei-mirror: avoid make(1) jobserver warning Eric Wong
2023-09-22 20:33 69% lei interactive TUIs (ncurses/vim/emacs) Eric Wong
2023-09-22 21:13 71% [PATCH 0/4] small lei fixes Eric Wong
2023-09-22 21:13 90% ` [PATCH 1/4] lei blob|rediff: fix usage of lei->fail Eric Wong
2023-09-22 21:13 66% ` [PATCH 2/4] lei: improve ->fail internal API Eric Wong
2023-09-22 21:13 70% ` [PATCH 3/4] lei_to_mail: drop awkward duplication of $lei object Eric Wong
2023-09-22 21:13 65% ` [PATCH 4/4] lei: use File::Temp for listing saved searches Eric Wong
2023-09-24 5:42 69% [PATCH 0/6] lei config fixes and improvements Eric Wong
2023-09-24 5:42 56% ` [PATCH 1/6] lei: check git-config(1) failures Eric Wong
2023-09-24 5:42 71% ` [PATCH 2/6] lei view_text: used tied ProcessPipe for `git config' Eric Wong
2023-09-24 5:42 66% ` [PATCH 4/6] lei config: send `git config' errors to pager Eric Wong
2023-09-24 5:42 30% ` [PATCH 5/6] lei: fix `-c NAME=VALUE' config support Eric Wong
2023-09-24 21:08 [PATCH 0/4] various CentOS 7.x related fixes Eric Wong
2023-09-24 21:08 71% ` [PATCH 3/4] lei: use scalar %SIG assignment Eric Wong
2023-09-27 6:02 [PATCH 0/3] more process management cleanups + bugfix Eric Wong
2023-09-27 6:02 46% ` [PATCH 3/3] lei: don't gzip --rsyncable by default for mbox* Eric Wong
2023-09-30 0:36 71% [PATCH 0/2] lei: support reading inboxes & extindex w/o search Eric Wong
2023-09-30 0:36 41% ` [PATCH 2/2] lei convert: support reading from v1, v2, and extindex Eric Wong
2023-09-30 16:17 71% [PATCH] t/lei-convert: fix uninitialized variable w/o pigz Eric Wong
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).