* [PATCH 1/3] imap_tracker: prepare for use with lei
2021-04-22 9:08 71% [PATCH 0/3] lei import: network sync things Eric Wong
@ 2021-04-22 9:08 70% ` Eric Wong
2021-04-22 9:08 51% ` [PATCH 2/3] lei import: --incremental default for NNTP and IMAP Eric Wong
2021-04-22 9:08 59% ` [PATCH 3/3] lei import|convert: drop --no-kw aliases Eric Wong
2 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-22 9:08 UTC (permalink / raw)
To: meta
We'll support nodatacow as we do in other SQLite DBs
---
lib/PublicInbox/IMAPTracker.pm | 22 ++++++++++++++--------
1 file changed, 14 insertions(+), 8 deletions(-)
diff --git a/lib/PublicInbox/IMAPTracker.pm b/lib/PublicInbox/IMAPTracker.pm
index 6d4fb227..bcf7af2e 100644
--- a/lib/PublicInbox/IMAPTracker.pm
+++ b/lib/PublicInbox/IMAPTracker.pm
@@ -62,21 +62,27 @@ VALUES (?, ?, ?)
}
sub new {
- my ($class, $url) = @_;
+ my ($class, $url, $dbname) = @_;
- # original name for compatibility with old setups:
- my $dbname = PublicInbox::Config->config_dir() . "/imap.sqlite3";
+ unless (defined($dbname)) {
+ # original name for compatibility with old setups:
+ $dbname = PublicInbox::Config->config_dir() . '/imap.sqlite3';
- # use the new XDG-compliant name for new setups:
- if (!-f $dbname) {
- $dbname = ($ENV{XDG_DATA_HOME} //
- (($ENV{HOME} // '/nonexistent').'/.local/share')) .
- '/public-inbox/imap.sqlite3';
+ # use the new XDG-compliant name for new setups:
+ if (!-f $dbname) {
+ $dbname = ($ENV{XDG_DATA_HOME} //
+ (($ENV{HOME} // '/nonexistent').
+ '/.local/share')) .
+ '/public-inbox/imap.sqlite3';
+ }
}
if (!-f $dbname) {
require File::Path;
require File::Basename;
+ require PublicInbox::Spawn;
File::Path::mkpath(File::Basename::dirname($dbname));
+ open my $fh, '+>>', $dbname or die "failed to open $dbname: $!";
+ PublicInbox::Spawn::nodatacow_fd(fileno($fh));
}
my $self = bless { lock_path => "$dbname.lock", url => $url }, $class;
$self->lock_acquire;
^ permalink raw reply related [relevance 70%]
* [PATCH 3/3] lei import|convert: drop --no-kw aliases
2021-04-22 9:08 71% [PATCH 0/3] lei import: network sync things Eric Wong
2021-04-22 9:08 70% ` [PATCH 1/3] imap_tracker: prepare for use with lei Eric Wong
2021-04-22 9:08 51% ` [PATCH 2/3] lei import: --incremental default for NNTP and IMAP Eric Wong
@ 2021-04-22 9:08 59% ` Eric Wong
2 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-22 9:08 UTC (permalink / raw)
To: meta
Supporting --no-keywords and --no-flags aliases is harmful
if users end up assuming "keywords:" and "flags:" are valid
search prefixes (they're not).
---
Documentation/lei-import.pod | 2 +-
lib/PublicInbox/LEI.pm | 9 ++++-----
t/lei.t | 3 +--
3 files changed, 6 insertions(+), 8 deletions(-)
diff --git a/Documentation/lei-import.pod b/Documentation/lei-import.pod
index acc4f776..7d70191d 100644
--- a/Documentation/lei-import.pod
+++ b/Documentation/lei-import.pod
@@ -40,7 +40,7 @@ C<none>.
Default: fcntl,dotlock
-=item --no-kw, --no-keywords, --no-flags
+=item --no-kw
Don't import message keywords (or "flags" in IMAP terminology).
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index d9e644eb..9f49fc03 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -181,7 +181,7 @@ our %CMD = ( # sorted in order of importance/use:
qw(exact! all jobs:i indexed), @c_opt ],
'add-watch' => [ 'LOCATION', 'watch for new messages and flag changes',
- qw(import! kw|keywords|flags! interval=s recursive|r
+ qw(import! kw! interval=s recursive|r
exclude=s include=s), @c_opt ],
'ls-watch' => [ '[FILTER...]', 'list active watches with numbers and status',
qw(format|f=s z), @c_opt ],
@@ -193,12 +193,11 @@ our %CMD = ( # sorted in order of importance/use:
'import' => [ 'LOCATION...|--stdin',
'one-time import/update from URL or filesystem',
qw(stdin| offset=i recursive|r exclude=s include|I=s
- lock=s@ in-format|F=s kw|keywords|flags! verbose|v+
- incremental!), @c_opt ],
+ lock=s@ in-format|F=s kw! verbose|v+ incremental!), @c_opt ],
'convert' => [ 'LOCATION...|--stdin',
'one-time conversion from URL or filesystem to another format',
qw(stdin| in-format|F=s out-format|f=s output|mfolder|o=s
- lock=s@ kw|keywords|flags!), @c_opt ],
+ lock=s@ kw!), @c_opt ],
'p2q' => [ 'FILE|COMMIT_OID|--stdin',
"use a patch to generate a query for `lei q --stdin'",
qw(stdin| want|w=s@ uri debug), @c_opt ],
@@ -350,7 +349,7 @@ my %OPTDESC = (
'by-mid|mid:s' => [ 'MID', 'match only by Message-ID, ignoring contents' ],
-'kw|keywords|flags!' => 'disable/enable importing flags',
+'kw!' => 'disable/enable importing keywords (aka "flags")',
# xargs, env, use "-0", git(1) uses "-z". We support z|0 everywhere
'z|0' => 'use NUL \\0 instead of newline (CR) to delimit lines',
diff --git a/t/lei.t b/t/lei.t
index 6ade2f18..6d276050 100644
--- a/t/lei.t
+++ b/t/lei.t
@@ -131,8 +131,7 @@ my $test_completion = sub {
}
lei_ok(qw(_complete lei import));
%out = map { $_ => 1 } split(/\s+/s, $lei_out);
- for my $sw (qw(--flags --no-flags --no-kw --kw --no-keywords
- --keywords)) {
+ for my $sw (qw(--no-kw --kw)) {
ok($out{$sw}, "$sw offered as `lei import' completion");
}
};
^ permalink raw reply related [relevance 59%]
* [PATCH 2/3] lei import: --incremental default for NNTP and IMAP
2021-04-22 9:08 71% [PATCH 0/3] lei import: network sync things Eric Wong
2021-04-22 9:08 70% ` [PATCH 1/3] imap_tracker: prepare for use with lei Eric Wong
@ 2021-04-22 9:08 51% ` Eric Wong
2021-04-22 9:08 59% ` [PATCH 3/3] lei import|convert: drop --no-kw aliases Eric Wong
2 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-22 9:08 UTC (permalink / raw)
To: meta
No point in burning through bandwidth to import stuff we already
saw. All this logic is shared with -watch but uses a different
pathname for lei since it's tied to lei/store (and not a
public-inbox).
---
Documentation/lei-store-format.pod | 1 +
lib/PublicInbox/LEI.pm | 4 +++-
lib/PublicInbox/LeiImport.pm | 5 +++++
lib/PublicInbox/NetReader.pm | 13 +++++++++----
t/lei-import-imap.t | 3 +++
t/lei-import-nntp.t | 3 +++
6 files changed, 24 insertions(+), 5 deletions(-)
diff --git a/Documentation/lei-store-format.pod b/Documentation/lei-store-format.pod
index a42c770e..3e1ddc65 100644
--- a/Documentation/lei-store-format.pod
+++ b/Documentation/lei-store-format.pod
@@ -32,6 +32,7 @@ prevent them from being accidentally treated as a v2 inbox.
~/.local/share/lei/store
- ipc.lock # lock file for internal lei IPC
- local/$EPOCH.git # normal bare git repositories
+ - net_last.sqlite3 # import state for IMAP & NNTP
Additionally, the following share the same roles they do in extindex:
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 2e1aa246..d9e644eb 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -193,7 +193,8 @@ our %CMD = ( # sorted in order of importance/use:
'import' => [ 'LOCATION...|--stdin',
'one-time import/update from URL or filesystem',
qw(stdin| offset=i recursive|r exclude=s include|I=s
- lock=s@ in-format|F=s kw|keywords|flags! verbose|v+), @c_opt ],
+ lock=s@ in-format|F=s kw|keywords|flags! verbose|v+
+ incremental!), @c_opt ],
'convert' => [ 'LOCATION...|--stdin',
'one-time conversion from URL or filesystem to another format',
qw(stdin| in-format|F=s out-format|f=s output|mfolder|o=s
@@ -244,6 +245,7 @@ my %OPTDESC = (
'lock=s@' => [ 'METHOD|dotlock|fcntl|flock|none',
'mbox(5) locking method(s) to use (default: fcntl,dotlock)' ],
+'incremental! import' => 'import already seen IMAP and NNTP articles',
'globoff|g' => "do not match locations using '*?' wildcards ".
"and\xa0'[]'\x{a0}ranges",
'verbose|v+' => 'be more verbose',
diff --git a/lib/PublicInbox/LeiImport.pm b/lib/PublicInbox/LeiImport.pm
index 16271603..accf08f5 100644
--- a/lib/PublicInbox/LeiImport.pm
+++ b/lib/PublicInbox/LeiImport.pm
@@ -58,6 +58,11 @@ sub lei_import { # the main "lei import" method
my $j = $lei->{opt}->{jobs} // scalar(@{$self->{inputs}}) || 1;
if (my $net = $lei->{net}) {
# $j = $net->net_concurrency($j); TODO
+ if ($lei->{opt}->{incremental} // 1) {
+ $net->{incremental} = 1;
+ $net->{itrk_fn} = $lei->store_path .
+ '/net_last.sqlite3';
+ }
} else {
my $nproc = $self->detect_nproc;
$j = $nproc if $j > $nproc;
diff --git a/lib/PublicInbox/NetReader.pm b/lib/PublicInbox/NetReader.pm
index 0ef66fd8..c7b43f01 100644
--- a/lib/PublicInbox/NetReader.pm
+++ b/lib/PublicInbox/NetReader.pm
@@ -373,6 +373,13 @@ sub run_commit_cb ($) {
$cb->(@args);
}
+sub _itrk ($$) {
+ my ($self, $uri) = @_;
+ return unless $self->{incremental};
+ # itrk_fn is set by lei
+ PublicInbox::IMAPTracker->new($$uri, $self->{itrk_fn});
+}
+
sub _imap_fetch_all ($$$) {
my ($self, $mic, $uri) = @_;
my $sec = uri_section($uri);
@@ -389,8 +396,7 @@ sub _imap_fetch_all ($$$) {
return "E: $uri cannot get UIDVALIDITY";
$r_uidnext //= $mic->uidnext($mbx) //
return "E: $uri cannot get UIDNEXT";
- my $itrk = $self->{incremental} ?
- PublicInbox::IMAPTracker->new($$uri) : 0;
+ my $itrk = _itrk($self, $uri);
my ($l_uidval, $l_uid) = $itrk ? $itrk->get_last : ();
$l_uidval //= $r_uidval; # first time
$l_uid //= 0;
@@ -543,8 +549,7 @@ sub _nntp_fetch_all ($$$) {
# IMAPTracker is also used for tracking NNTP, UID == article number
# LIST.ACTIVE can get the equivalent of UIDVALIDITY, but that's
# expensive. So we assume newsgroups don't change:
- my $itrk = $self->{incremental} ?
- PublicInbox::IMAPTracker->new($$uri) : 0;
+ my $itrk = _itrk($self, $uri);
my (undef, $l_art) = $itrk ? $itrk->get_last : ();
# allow users to specify articles to refetch
diff --git a/t/lei-import-imap.t b/t/lei-import-imap.t
index 7e4d44b9..490ea9be 100644
--- a/t/lei-import-imap.t
+++ b/t/lei-import-imap.t
@@ -24,5 +24,8 @@ test_lei({ tmpdir => $tmpdir }, sub {
for (@$out) { $r{ref($_)}++ }
is_deeply(\%r, { 'HASH' => scalar(@$out) }, 'all hashes');
lei_ok([qw(tag +kw:seen), "imap://$host_port/t.v2.0"], undef, undef);
+
+ my $f = "$ENV{HOME}/.local/share/lei/store/net_last.sqlite3";
+ ok(-s $f, 'net tracked for redundant imports');
});
done_testing;
diff --git a/t/lei-import-nntp.t b/t/lei-import-nntp.t
index 1fc6dbad..d795a86a 100644
--- a/t/lei-import-nntp.t
+++ b/t/lei-import-nntp.t
@@ -26,5 +26,8 @@ test_lei({ tmpdir => $tmpdir }, sub {
my %r;
for (@$out) { $r{ref($_)}++ }
is_deeply(\%r, { 'HASH' => scalar(@$out) }, 'all hashes');
+
+ my $f = "$ENV{HOME}/.local/share/lei/store/net_last.sqlite3";
+ ok(-s $f, 'net tracked for redundant imports');
});
done_testing;
^ permalink raw reply related [relevance 51%]
* [PATCH 0/3] lei import: network sync things
@ 2021-04-22 9:08 71% Eric Wong
2021-04-22 9:08 70% ` [PATCH 1/3] imap_tracker: prepare for use with lei Eric Wong
` (2 more replies)
0 siblings, 3 replies; 200+ results
From: Eric Wong @ 2021-04-22 9:08 UTC (permalink / raw)
To: meta
Once keyword synchronization exists; I'll be able to
retire my personal uses of offlineimap and mbsync...
Eric Wong (3):
imap_tracker: prepare for use with lei
lei import: --incremental default for NNTP and IMAP
lei import|convert: drop --no-kw aliases
Documentation/lei-import.pod | 2 +-
Documentation/lei-store-format.pod | 1 +
lib/PublicInbox/IMAPTracker.pm | 22 ++++++++++++++--------
lib/PublicInbox/LEI.pm | 9 +++++----
lib/PublicInbox/LeiImport.pm | 5 +++++
lib/PublicInbox/NetReader.pm | 13 +++++++++----
t/lei-import-imap.t | 3 +++
t/lei-import-nntp.t | 3 +++
t/lei.t | 3 +--
9 files changed, 42 insertions(+), 19 deletions(-)
^ permalink raw reply [relevance 71%]
* [PATCH] lei: flesh out `forwarded' kw support for Maildir and IMAP
@ 2021-04-21 23:50 50% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-21 23:50 UTC (permalink / raw)
To: meta
Maildir and IMAP can both handle `forwarded'. Ensure we don't
lose `forwarded' when reading from stores which do not support
it, but ensure we can set it when reading from IMAP and Maildir
stores.
---
lib/PublicInbox/LeiSearch.pm | 12 ++++++++++--
lib/PublicInbox/LeiToMail.pm | 2 +-
lib/PublicInbox/NetReader.pm | 1 +
t/lei-q-kw.t | 31 +++++++++++++++++++++++++++++++
xt/net_writer-imap.t | 14 +++++++++++---
5 files changed, 54 insertions(+), 6 deletions(-)
diff --git a/lib/PublicInbox/LeiSearch.pm b/lib/PublicInbox/LeiSearch.pm
index 082176e7..ff615d89 100644
--- a/lib/PublicInbox/LeiSearch.pm
+++ b/lib/PublicInbox/LeiSearch.pm
@@ -103,8 +103,16 @@ sub kw_changed {
my $xoids = xoids_for($self, $eml) // return;
$docids //= [];
@$docids = sort { $a <=> $b } values %$xoids;
- my @cur_kw = msg_keywords($self, $docids->[0]);
- join("\0", @$new_kw_sorted) eq join("\0", @cur_kw) ? 0 : 1;
+ my $cur_kw = msg_keywords($self, $docids->[0]);
+
+ # RFC 5550 sec 5.9 on the $Forwarded keyword states:
+ # "Once set, the flag SHOULD NOT be cleared"
+ if (exists($cur_kw->{forwarded}) &&
+ !grep(/\Aforwarded\z/, @$new_kw_sorted)) {
+ delete $cur_kw->{forwarded};
+ }
+ $cur_kw = join("\0", sort keys %$cur_kw);
+ join("\0", @$new_kw_sorted) eq $cur_kw ? 0 : 1;
}
sub all_terms {
diff --git a/lib/PublicInbox/LeiToMail.pm b/lib/PublicInbox/LeiToMail.pm
index 46a82a4b..0fa0bd9a 100644
--- a/lib/PublicInbox/LeiToMail.pm
+++ b/lib/PublicInbox/LeiToMail.pm
@@ -38,7 +38,7 @@ sub _mbox_hdr_buf ($$$) {
if (my $ent = $kw2status{$k}) {
push @{$hdr{$ent->[0]}}, $ent->[1];
} else { # X-Label?
- warn "TODO: keyword `$k' not supported for mbox\n";
+ warn "# keyword `$k' not supported for mbox\n";
}
}
# Messages are always 'O' (non-\Recent in IMAP), it saves
diff --git a/lib/PublicInbox/NetReader.pm b/lib/PublicInbox/NetReader.pm
index 821e5d7f..0ef66fd8 100644
--- a/lib/PublicInbox/NetReader.pm
+++ b/lib/PublicInbox/NetReader.pm
@@ -8,6 +8,7 @@ use v5.10.1;
use parent qw(Exporter PublicInbox::IPC);
use PublicInbox::Eml;
our %IMAPflags2kw = map {; "\\\u$_" => $_ } qw(seen answered flagged draft);
+$IMAPflags2kw{'$Forwarded'} = 'forwarded'; # RFC 5550
our @EXPORT = qw(uri_section imap_uri nntp_uri);
diff --git a/t/lei-q-kw.t b/t/lei-q-kw.t
index c17411fb..c00a0a43 100644
--- a/t/lei-q-kw.t
+++ b/t/lei-q-kw.t
@@ -205,5 +205,36 @@ open $fh, '<', \$lei_out or BAIL_OUT $!;
PublicInbox::MboxReader->mboxrd($fh, sub { push @another, shift });
is($another[0]->header('Status'), 'RO', 'seen kw set');
+# forwarded
+{
+ local $ENV{DBG} = 1;
+ $o = "$ENV{HOME}/forwarded";
+ lei_ok(qw(q -o), $o, "m:$m");
+ my @p = glob("$o/cur/*");
+ scalar(@p) == 1 or xbail('multiple when 1 expected', \@p);
+ my $passed = $p[0];
+ $passed =~ s/,S\z/,PS/ or xbail "failed to replace $passed";
+ rename($p[0], $passed) or xbail "rename $!";
+ lei_ok(qw(q -o), $o, 'm:bogus', \'clobber maildir');
+ is_deeply([glob("$o/cur/*")], [], 'old results clobbered');
+ lei_ok(qw(q -o), $o, "m:$m");
+ @p = glob("$o/cur/*");
+ scalar(@p) == 1 or xbail('multiple when 1 expected', \@p);
+ like($p[0], qr/,PS/, 'passed (Forwarded) flag kept');
+ lei_ok(qw(q -o), "mboxrd:$o.mboxrd", "m:$m");
+ open $fh, '<', "$o.mboxrd" or xbail $!;
+ my @res;
+ PublicInbox::MboxReader->mboxrd($fh, sub { push @res, shift });
+ scalar(@res) == 1 or xbail('multiple when 1 expected', \@res);
+ is($res[0]->header('Status'), 'RO', 'seen kw set');
+ is($res[0]->header('X-Status'), undef, 'no X-Status');
+
+ lei_ok(qw(q -o), "mboxrd:$o.mboxrd", 'bogus-for-import-before');
+ lei_ok(qw(q -o), $o, "m:$m");
+ @p = glob("$o/cur/*");
+ scalar(@p) == 1 or xbail('multiple when 1 expected', \@p);
+ like($p[0], qr/,PS/, 'passed (Forwarded) flag still kept');
+}
+
}); # test_lei
done_testing;
diff --git a/xt/net_writer-imap.t b/xt/net_writer-imap.t
index 11a10e74..007de35e 100644
--- a/xt/net_writer-imap.t
+++ b/xt/net_writer-imap.t
@@ -173,17 +173,18 @@ test_lei(sub {
is_deeply([@$res{qw(m kw)}], ['testmessage@example.com', ['seen']],
'kw set');
+ # prepare messages for watch
$mic = $nwr->mic_for_folder($folder_uri);
- for my $kw (qw(Deleted Seen Answered Draft)) {
+ for my $kw (qw(Deleted Seen Answered Draft forwarded)) {
my $buf = <<EOM;
From: x\@example.com
Message-ID: <$kw\@test.example.com>
EOM
- $mic->append_string($folder_uri->mailbox, $buf, "\\$kw")
+ my $f = $kw eq 'forwarded' ? '$Forwarded' : "\\$kw";
+ $mic->append_string($folder_uri->mailbox, $buf, $f)
or BAIL_OUT "append $kw $@";
}
- # $mic->expunge or BAIL_OUT "expunge: $@";
$mic->disconnect;
my $inboxdir = "$ENV{HOME}/wtest";
@@ -214,6 +215,13 @@ EOM
'-watch ignored \\Deleted');
ok(!defined($mm->num_for('Draft@test.example.com')),
'-watch ignored \\Draft');
+ ok(defined($mm->num_for('forwarded@test.example.com')),
+ '-watch takes forwarded message');
+ undef $w; # done with watch
+ lei_ok qw(import), $$folder_uri;
+ lei_ok qw(q m:forwarded@test.example.com);
+ is_deeply(json_utf8->decode($lei_out)->[0]->{kw}, ['forwarded'],
+ 'forwarded kw imported from IMAP');
});
undef $cleanup; # remove temporary folder
^ permalink raw reply related [relevance 50%]
* [PATCH] lei: share common *done_wait callbacks
@ 2021-04-21 18:36 61% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-21 18:36 UTC (permalink / raw)
To: meta
Code is the enemy, and there's no need to duplicate things, here.
There may be further opportunities along these lines to further
deduplicate things...
---
lib/PublicInbox/LEI.pm | 7 +++++++
lib/PublicInbox/LeiBlob.pm | 9 +--------
lib/PublicInbox/LeiImport.pm | 9 +--------
lib/PublicInbox/LeiTag.pm | 9 +--------
4 files changed, 10 insertions(+), 24 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 8fa89944..2e1aa246 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -1211,4 +1211,11 @@ sub DESTROY {
$? = $err if $err; # preserve ->fail or ->x_it code
}
+sub wq_done_wait { # dwaitpid callback
+ my ($arg, $pid) = @_;
+ my ($wq, $lei, $e) = @$arg;
+ $? and $lei->child_error($?, $e ? "$e errors during $lei->{cmd}" : ());
+ $lei->dclose;
+}
+
1;
diff --git a/lib/PublicInbox/LeiBlob.pm b/lib/PublicInbox/LeiBlob.pm
index ad885306..e4cd4cca 100644
--- a/lib/PublicInbox/LeiBlob.pm
+++ b/lib/PublicInbox/LeiBlob.pm
@@ -10,17 +10,10 @@ use parent qw(PublicInbox::IPC);
use PublicInbox::Spawn qw(spawn popen_rd which);
use PublicInbox::DS;
-sub sol_done_wait { # dwaitpid callback
- my ($arg, $pid) = @_;
- my (undef, $lei) = @$arg;
- $lei->child_error($?) if $?;
- $lei->dclose;
-}
-
sub sol_done { # EOF callback for main daemon
my ($lei) = @_;
my $sol = delete $lei->{sol} // return $lei->dclose; # already failed
- $sol->wq_wait_old(\&sol_done_wait, $lei);
+ $sol->wq_wait_old($lei->can('wq_done_wait'), $lei);
}
sub get_git_dir ($$) {
diff --git a/lib/PublicInbox/LeiImport.pm b/lib/PublicInbox/LeiImport.pm
index d33143ef..16271603 100644
--- a/lib/PublicInbox/LeiImport.pm
+++ b/lib/PublicInbox/LeiImport.pm
@@ -35,17 +35,10 @@ sub input_net_cb { # imap_each, nntp_each cb
input_eml_cb($self, $eml, $self->{-import_kw} ? { kw => $kw } : undef);
}
-sub import_done_wait { # dwaitpid callback
- my ($arg, $pid) = @_;
- my ($imp, $lei) = @$arg;
- $lei->child_error($?, 'non-fatal errors during import') if $?;
- $lei->dclose;
-}
-
sub import_done { # EOF callback for main daemon
my ($lei) = @_;
my $imp = delete $lei->{imp} // return $lei->fail('BUG: {imp} gone');
- $imp->wq_wait_old(\&import_done_wait, $lei);
+ $imp->wq_wait_old($lei->can('wq_done_wait'), $lei, 'non-fatal');
}
sub net_merge_complete { # callback used by LeiAuth
diff --git a/lib/PublicInbox/LeiTag.pm b/lib/PublicInbox/LeiTag.pm
index 1dfc841d..f019202f 100644
--- a/lib/PublicInbox/LeiTag.pm
+++ b/lib/PublicInbox/LeiTag.pm
@@ -71,17 +71,10 @@ sub input_eml_cb { # used by PublicInbox::LeiInput::input_fh
sub input_mbox_cb { input_eml_cb($_[1], $_[0]) }
-sub tag_done_wait { # dwaitpid callback
- my ($arg, $pid) = @_;
- my ($tag, $lei) = @$arg;
- $lei->child_error($?, 'non-fatal errors during tag') if $?;
- $lei->dclose;
-}
-
sub tag_done { # EOF callback for main daemon
my ($lei) = @_;
my $tag = delete $lei->{tag} or return;
- $tag->wq_wait_old(\&tag_done_wait, $lei);
+ $tag->wq_wait_old($lei->can('wq_done_wait'), $lei, 'non-fatal');
}
sub net_merge_complete { # callback used by LeiAuth
^ permalink raw reply related [relevance 61%]
* Re: [PATCH] t/lei-daemon: skip inaccessible socket test as root
2021-04-20 22:06 71% ` [PATCH] t/lei-daemon: skip inaccessible socket test as root Eric Wong
@ 2021-04-21 15:03 71% ` Konstantin Ryabitsev
0 siblings, 0 replies; 200+ results
From: Konstantin Ryabitsev @ 2021-04-21 15:03 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
On Tue, Apr 20, 2021 at 10:06:08PM +0000, Eric Wong wrote:
> Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> > While poking around, I've discovered that it only fails when "make test" runs
> > as root (don't judge -- this is in a throwaway lab VM):
>
> > Hope that helps.
>
> Yes :)
I can confirm that the error is gone now. Thanks!
-K
^ permalink raw reply [relevance 71%]
* [PATCH] doc: add lei_design_notes.txt and lei-store-format(5)
@ 2021-04-21 10:03 46% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-21 10:03 UTC (permalink / raw)
To: meta
lei itself is a somewhat weird design, but lei/store is
a fairly normal hybrid of extindex with v2-style storage.
---
Documentation/lei-store-format.pod | 91 ++++++++++++++++++++++++++++++
Documentation/lei_design_notes.txt | 20 +++++++
MANIFEST | 2 +
Makefile.PL | 2 +-
4 files changed, 114 insertions(+), 1 deletion(-)
create mode 100644 Documentation/lei-store-format.pod
create mode 100644 Documentation/lei_design_notes.txt
diff --git a/Documentation/lei-store-format.pod b/Documentation/lei-store-format.pod
new file mode 100644
index 00000000..a42c770e
--- /dev/null
+++ b/Documentation/lei-store-format.pod
@@ -0,0 +1,91 @@
+% public-inbox developer manual
+
+=head1 NAME
+
+lei-store-format - lei/store format description
+
+=head1 DESCRIPTION
+
+C<lei/store> is a hybrid store based on L<public-inbox-extindex-format(5)>
+("extindex") combined with L<public-inbox-v2-format(5)> ("v2") for blob
+storage. While v2 is ideal for archiving a single public mailing list;
+it was never intended for personal mail nor storing multiple
+blobs of the "same" message.
+
+As with extindex, it can index disparate C<List-Id> headers
+belonging to the "same" message with different git blob OIDs.
+Unlike v2 and extindex, C<Message-ID> headers are NOT required;
+allowing unsent draft messages to be stored and indexed.
+
+=head1 DIRECTORY LAYOUT
+
+Blob storage exists in the form of v2-style epochs. These epochs
+are under the C<local/> directory (instead of C<git/>) to
+prevent them from being accidentally treated as a v2 inbox.
+
+=head2 INDEX OVERVIEW AND DEFINITIONS
+
+ $EPOCH - Integer starting with 0 based on time
+ $SCHEMA_VERSION - DB schema version (for Xapian)
+ $SHARD - Integer starting with 0 based on parallelism
+
+ ~/.local/share/lei/store
+ - ipc.lock # lock file for internal lei IPC
+ - local/$EPOCH.git # normal bare git repositories
+
+Additionally, the following share the same roles they do in extindex:
+
+ - ei.lock # lock file to protect global state
+ - ALL.git # empty, alternates for local/*.git
+ - ei$SCHEMA_VERSION/$SHARD # per-shard Xapian DB
+ - ei$SCHEMA_VERSION/over.sqlite3 # overview DB for WWW, IMAP
+ - ei$SCHEMA_VERSION/misc # misc Xapian DB
+
+=head2 XREF3 DEDUPLICATION
+
+Index deduplication follows extindex, see
+L<public-inbox-extindex-format(5)/XREF3 DEDUPLICATION> for
+more information.
+
+=head2 BLOB DEDUPLICATION
+
+The contents of C<local/*.git> repos is deduplicated by git blob
+object IDs (currently SHA-1). This allows multiple copies of
+cross-posted and personally Cc-ed messages to be stored with
+different C<Received:>, C<X-Spam-Status:> and similar headers to
+allow troubleshooting.
+
+=head2 VOLATILE METADATA
+
+Keywords and label information (as described in RFC 8621 for JMAP)
+is stored in existing Xapian shards (C<ei$SCHEMA_VERSION/$SHARD>).
+It is possible to search for messages matching labels and
+keywords using C<L:> and C<kw:>, respectively. As with all data
+stored in Xapian indices, volatile metadata is associated with
+the Xapian document, thus it is shared across different blobs of
+the "same" message.
+
+=head1 IPC
+
+When L<lei(1)> is run in daemon mode, L<flock(2)> is used on
+C<ipc.lock> is used to serialize writes to C<lei/store> across
+multiple internal lei workers while minimizing commits.
+
+=head1 CAVEATS
+
+Reindexing and synchronization is not yet supported.
+
+=head1 THANKS
+
+Thanks to the Linux Foundation for sponsoring the development
+and testing.
+
+=head1 COPYRIGHT
+
+Copyright 2021 all contributors L<mailto:meta@public-inbox.org>
+
+License: AGPL-3.0+ L<http://www.gnu.org/licenses/agpl-3.0.txt>
+
+=head1 SEE ALSO
+
+L<public-inbox-v2-format(5)>, L<public-inbox-extindex(5)>
diff --git a/Documentation/lei_design_notes.txt b/Documentation/lei_design_notes.txt
new file mode 100644
index 00000000..a5606c05
--- /dev/null
+++ b/Documentation/lei_design_notes.txt
@@ -0,0 +1,20 @@
+lei design notes
+----------------
+
+Daemon architecture
+-------------------
+
+The use of a persistent daemon works around slow startup time of
+Perl. This is especially important for built-in support for
+shell completion. It will eventually support inotify and EVFILT_VNODE
+background monitoring of Maildir keyword changes.
+
+If lei were reimplemented in a language with faster startup
+time, the daemon architecture would likely remain since it also
+lets us easily decouple the local storage from slow IMAP/NNTP
+backends and allow us to serialize writes to git-fast-import,
+SQLite, and Xapian across multiple processes.
+
+The coupling of IMAP and NNTP network latency to local storage
+is a current weakness of public-inbox-watch. Therefore, -watch
+will likely adopt the daemon architecture of lei in the future.
diff --git a/MANIFEST b/MANIFEST
index 1a1d72a6..e0f9c35b 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -35,8 +35,10 @@ Documentation/lei-mail-formats.pod
Documentation/lei-overview.pod
Documentation/lei-p2q.pod
Documentation/lei-q.pod
+Documentation/lei-store-format.pod
Documentation/lei-tag.pod
Documentation/lei.pod
+Documentation/lei_design_notes.txt
Documentation/marketing.txt
Documentation/mknews.perl
Documentation/public-inbox-compact.pod
diff --git a/Makefile.PL b/Makefile.PL
index 129b082d..85b18e7d 100644
--- a/Makefile.PL
+++ b/Makefile.PL
@@ -50,7 +50,7 @@ $v->{-m1} = [ map {
lei-tag lei-p2q lei-q)];
$v->{-m5} = [ qw(public-inbox-config public-inbox-v1-format
public-inbox-v2-format public-inbox-extindex-format
- lei-mail-formats
+ lei-mail-formats lei-store-format
) ];
$v->{-m7} = [ qw(lei-overview public-inbox-overview public-inbox-tuning
public-inbox-glossary) ];
^ permalink raw reply related [relevance 46%]
* [PATCH] t/lei-daemon: skip inaccessible socket test as root
2021-04-20 21:37 68% ` Konstantin Ryabitsev
@ 2021-04-20 22:06 71% ` Eric Wong
2021-04-21 15:03 71% ` Konstantin Ryabitsev
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2021-04-20 22:06 UTC (permalink / raw)
To: meta
Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> While poking around, I've discovered that it only fails when "make test" runs
> as root (don't judge -- this is in a throwaway lab VM):
> Hope that helps.
Yes :)
----------8<--------
Subject: [PATCH] t/lei-daemon: skip inaccessible socket test as root
"chmod 0000" on a Unix socket can't stop root from connecting to it;
so just skip the test for rare cases when testing as root.
Reported-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
Link: https://public-inbox.org/meta/20210420213712.qfpftr2r543cqg7l@nitro.local/
---
t/lei-daemon.t | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/t/lei-daemon.t b/t/lei-daemon.t
index 35e059b9..84e2791d 100644
--- a/t/lei-daemon.t
+++ b/t/lei-daemon.t
@@ -74,7 +74,8 @@ test_lei({ daemon_only => 1 }, sub {
chomp $lei_out;
is($lei_out, $new_pid, 'PID unchanged after -0/-CHLD');
- if ('socket inaccessible') {
+ SKIP: { # socket inaccessible
+ skip "cannot test connect EPERM as root", 3 if $> == 0;
chmod 0000, $sock or BAIL_OUT "chmod 0000: $!";
lei_ok('help', \'connect fail, one-shot fallback works');
like($lei_err, qr/\bconnect\(/, 'connect error noted');
^ permalink raw reply related [relevance 71%]
* Re: t/lei-daemon.t failure when PERL_INLINE_DIRECTORY is set
2021-04-20 20:38 71% ` Eric Wong
@ 2021-04-20 21:37 68% ` Konstantin Ryabitsev
2021-04-20 22:06 71% ` [PATCH] t/lei-daemon: skip inaccessible socket test as root Eric Wong
0 siblings, 1 reply; 200+ results
From: Konstantin Ryabitsev @ 2021-04-20 21:37 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
On Tue, Apr 20, 2021 at 08:38:54PM +0000, Eric Wong wrote:
> > t/lei-daemon.t ............... 18/?
> > # Failed test 'connect error noted'
> > # at t/lei-daemon.t line 80.
> > # ''
> > # doesn't match '(?^:\bconnect\()'
> > # Looks like you failed 1 test of 34.
> > t/lei-daemon.t ............... Dubious, test returned 1 (wstat 256, 0x100)
> >
> > I'll be happy to help troubleshoot things, since I don't know how easy it
> > would be to reproduce this.
>
> I can't seem to reproduce it on my end with CentOS 7 and
> libgit2-devel 0.26.8. Try setting TEST_LEI_ERR_LOUD=1
While poking around, I've discovered that it only fails when "make test" runs
as root (don't judge -- this is in a throwaway lab VM):
t/lei-daemon.t ............... 1/? # lei_err=/tmp/pi-lei-daemon-6667-ojwj/lei-daemon/xdg_run/lei/errors.log from previous run
# phail
# Failed test 'connect error noted'
# at t/lei-daemon.t line 80.
# ''
# doesn't match '(?^:\bconnect\()'
# Looks like you failed 1 test of 34.
t/lei-daemon.t ............... Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/34 subtests
When I run as a non-root user, it passes:
t/lei-daemon.t ............... 1/? # lei_err=/tmp/pi-lei-daemon-11879-P3hC/lei-daemon/xdg_run/lei/errors.log from previous run
# phail
# lei_err=connect(/tmp/pi-lei-daemon-11879-P3hC/lei-daemon/xdg_run/lei/5.seq.sock): Permission denied at /home/mricon/public-inbox/blib/lib/PublicInbox/LEI.pm line 1073.
# lei-daemon could not start, exited with $?=3328
# connect(/tmp/pi-lei-daemon-11879-P3hC/lei-daemon/xdg_run/lei/5.seq.sock): Permission denied (after attempted daemon start)
# Falling back to (slow) one-shot mode
t/lei-daemon.t ............... ok
Hope that helps.
-K
^ permalink raw reply [relevance 68%]
* Re: t/lei-daemon.t failure when PERL_INLINE_DIRECTORY is set
2021-04-20 20:33 71% t/lei-daemon.t failure when PERL_INLINE_DIRECTORY is set Konstantin Ryabitsev
@ 2021-04-20 20:38 71% ` Eric Wong
2021-04-20 21:37 68% ` Konstantin Ryabitsev
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2021-04-20 20:38 UTC (permalink / raw)
To: meta
Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> While playing with libgit2/Gcf2, I've discovered that t/lei-daemon.t will fail
> when PERL_INLINE_DIRECTORY is set:
>
> t/lei-daemon.t ............... 18/?
> # Failed test 'connect error noted'
> # at t/lei-daemon.t line 80.
> # ''
> # doesn't match '(?^:\bconnect\()'
> # Looks like you failed 1 test of 34.
> t/lei-daemon.t ............... Dubious, test returned 1 (wstat 256, 0x100)
>
> I'll be happy to help troubleshoot things, since I don't know how easy it
> would be to reproduce this.
I can't seem to reproduce it on my end with CentOS 7 and
libgit2-devel 0.26.8. Try setting TEST_LEI_ERR_LOUD=1
^ permalink raw reply [relevance 71%]
* t/lei-daemon.t failure when PERL_INLINE_DIRECTORY is set
@ 2021-04-20 20:33 71% Konstantin Ryabitsev
2021-04-20 20:38 71% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Konstantin Ryabitsev @ 2021-04-20 20:33 UTC (permalink / raw)
To: meta
While playing with libgit2/Gcf2, I've discovered that t/lei-daemon.t will fail
when PERL_INLINE_DIRECTORY is set:
t/lei-daemon.t ............... 18/?
# Failed test 'connect error noted'
# at t/lei-daemon.t line 80.
# ''
# doesn't match '(?^:\bconnect\()'
# Looks like you failed 1 test of 34.
t/lei-daemon.t ............... Dubious, test returned 1 (wstat 256, 0x100)
I'll be happy to help troubleshoot things, since I don't know how easy it
would be to reproduce this.
-K
^ permalink raw reply [relevance 71%]
* [PATCH] lei-sigpipe: update and move test from xt => t
@ 2021-04-20 9:17 51% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-20 9:17 UTC (permalink / raw)
To: meta
We have "lei import" and better test infrastructure for lei,
now, so we can more easily test SIGPIPE without relying on
an already-configured instance.
---
MANIFEST | 2 +-
t/lei-sigpipe.t | 43 +++++++++++++++++++++++++++++++++++
xt/lei-sigpipe.t | 58 ------------------------------------------------
3 files changed, 44 insertions(+), 59 deletions(-)
create mode 100644 t/lei-sigpipe.t
delete mode 100644 xt/lei-sigpipe.t
diff --git a/MANIFEST b/MANIFEST
index f35c514c..f4a55687 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -398,6 +398,7 @@ t/lei-q-kw.t
t/lei-q-remote-import.t
t/lei-q-save.t
t/lei-q-thread.t
+t/lei-sigpipe.t
t/lei-tag.t
t/lei.t
t/lei_dedupe.t
@@ -501,7 +502,6 @@ xt/httpd-async-stream.t
xt/imapd-mbsync-oimap.t
xt/imapd-validate.t
xt/lei-auth-fail.t
-xt/lei-sigpipe.t
xt/mem-imapd-tls.t
xt/mem-msgview.t
xt/msgtime_cmp.t
diff --git a/t/lei-sigpipe.t b/t/lei-sigpipe.t
new file mode 100644
index 00000000..f84d6d22
--- /dev/null
+++ b/t/lei-sigpipe.t
@@ -0,0 +1,43 @@
+#!perl -w
+# Copyright (C) 2021 all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+use strict;
+use v5.10.1;
+use PublicInbox::TestCommon;
+use POSIX qw(WTERMSIG WIFSIGNALED SIGPIPE);
+test_lei(sub {
+ my $f = "$ENV{HOME}/big.eml";
+ my $imported;
+ for my $out ([], [qw(-f mboxcl2)]) {
+ pipe(my ($r, $w)) or BAIL_OUT $!;
+ my $size = 65536;
+ if ($^O eq 'linux' && fcntl($w, 1031, 4096)) {
+ $size = 4096;
+ }
+ unless (-f $f) {
+ open my $fh, '>', $f or xbail "open $f: $!";
+ print $fh <<'EOM' or xbail;
+From: big@example.com
+Message-ID: <big@example.com>
+EOM
+ print $fh 'Subject:';
+ print $fh (' '.('x' x 72)."\n") x (($size / 73) + 1);
+ print $fh "\nbody\n";
+ close $fh or xbail "close: $!";
+ }
+
+ lei_ok(qw(import), $f) if $imported++ == 0;
+ open my $errfh, '>>', "$ENV{HOME}/stderr.log" or xbail $!;
+ my $opt = { run_mode => 0, 2 => $errfh, 1 => $w };
+ my $cmd = [qw(lei q -q -t), @$out, 'z:1..'];
+ my $tp = start_script($cmd, undef, $opt);
+ close $w;
+ is(sysread($r, my $buf, 1), 1, 'read one byte');
+ close $r; # trigger SIGPIPE
+ $tp->join;
+ ok(WIFSIGNALED($?), "signaled @$out");
+ is(WTERMSIG($?), SIGPIPE, "got SIGPIPE @$out");
+ }
+});
+
+done_testing;
diff --git a/xt/lei-sigpipe.t b/xt/lei-sigpipe.t
deleted file mode 100644
index 44020bad..00000000
--- a/xt/lei-sigpipe.t
+++ /dev/null
@@ -1,58 +0,0 @@
-#!perl -w
-# Copyright (C) 2021 all contributors <meta@public-inbox.org>
-# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
-use strict;
-use v5.10.1;
-use Test::More;
-use PublicInbox::TestCommon;
-use POSIX qw(WTERMSIG WIFSIGNALED SIGPIPE);
-require_mods(qw(json DBD::SQLite Search::Xapian));
-# XXX this needs an already configured lei instance with many messages
-
-my $do_test = sub {
- my $env = shift // {};
- for my $out ([], [qw(-f mboxcl2)]) {
- pipe(my ($r, $w)) or BAIL_OUT $!;
- open my $err, '+>', undef or BAIL_OUT $!;
- my $opt = { run_mode => 0, 1 => $w, 2 => $err };
- my $cmd = [qw(lei q -q -t), @$out, 'z:1..'];
- my $tp = start_script($cmd, $env, $opt);
- close $w;
- sysread($r, my $buf, 1);
- close $r; # trigger SIGPIPE
- $tp->join;
- ok(WIFSIGNALED($?), "signaled @$out");
- is(WTERMSIG($?), SIGPIPE, "got SIGPIPE @$out");
- seek($err, 0, 0);
- my @err = grep(!m{mkdir /dev/null\b}, <$err>);
- is_deeply(\@err, [], "no errors @$out");
- }
-};
-
-my ($tmp, $for_destroy) = tmpdir();
-my $pid;
-my $opt = { run_mode => 0, 1 => \(my $out = '') };
-if (run_script([qw(lei daemon-pid)], undef, $opt)) {
- chomp($pid = $out);
- mkdir "$tmp/d" or BAIL_OUT $!;
- local $ENV{TMPDIR} = "$tmp/d";
- $do_test->();
- $out = '';
- ok(run_script([qw(lei daemon-pid)], undef, $opt), 'daemon-pid again');
- chomp($out);
- is($out, $pid, 'daemon-pid unchanged');
- ok(kill(0, $pid), 'daemon still running');
- $out = '';
-}
-{
- mkdir "$tmp/1" or BAIL_OUT $!;
- local $ENV{TMPDIR} = "$tmp/1";
- $do_test->({XDG_RUNTIME_DIR => '/dev/null'});
- is(unlink(glob("$tmp/1/*")), 0, 'nothing left over w/ oneshot');
-}
-
-# the one-shot test should be slow enough that the daemon has cleaned
-# up in the background:
-is_deeply([glob("$tmp/d/*")], [], 'nothing left over with daemon');
-
-done_testing;
^ permalink raw reply related [relevance 51%]
* [PATCH 0/2] lei {edit,forget}-search
@ 2021-04-20 7:16 71% Eric Wong
2021-04-20 7:16 56% ` [PATCH 1/2] lei forget-search: new command to forget saved searches Eric Wong
2021-04-20 7:16 64% ` [PATCH 2/2] lei edit-search: command to tweak search parameters Eric Wong
0 siblings, 2 replies; 200+ results
From: Eric Wong @ 2021-04-20 7:16 UTC (permalink / raw)
To: meta
Some convenience commands, since I expect users will want to
tweak and experiment with things.
I don't think "mv-search" (formerly "mv-query") is necessary
since it's easy enough to use "lei convert" (testing anything
requiring authentication is not fun nor easily automated, atm).
Eric Wong (2):
lei forget-search: new command to forget saved searches
lei edit-search: command to tweak search parameters
MANIFEST | 2 ++
lib/PublicInbox/LEI.pm | 6 ++++--
lib/PublicInbox/LeiEditSearch.pm | 25 +++++++++++++++++++++++
lib/PublicInbox/LeiForgetSearch.pm | 32 ++++++++++++++++++++++++++++++
t/lei-q-save.t | 11 ++++++++++
5 files changed, 74 insertions(+), 2 deletions(-)
create mode 100644 lib/PublicInbox/LeiEditSearch.pm
create mode 100644 lib/PublicInbox/LeiForgetSearch.pm
^ permalink raw reply [relevance 71%]
* [PATCH 2/2] lei edit-search: command to tweak search parameters
2021-04-20 7:16 71% [PATCH 0/2] lei {edit,forget}-search Eric Wong
2021-04-20 7:16 56% ` [PATCH 1/2] lei forget-search: new command to forget saved searches Eric Wong
@ 2021-04-20 7:16 64% ` Eric Wong
1 sibling, 0 replies; 200+ results
From: Eric Wong @ 2021-04-20 7:16 UTC (permalink / raw)
To: meta
This may be useful for users to tweak search parameters.
This command is also the reason lei.saved-search is a git-config
file rather than JSON.
---
MANIFEST | 1 +
lib/PublicInbox/LEI.pm | 2 ++
lib/PublicInbox/LeiEditSearch.pm | 25 +++++++++++++++++++++++++
3 files changed, 28 insertions(+)
create mode 100644 lib/PublicInbox/LeiEditSearch.pm
diff --git a/MANIFEST b/MANIFEST
index d4055af4..197da2c0 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -190,6 +190,7 @@ lib/PublicInbox/LeiBlob.pm
lib/PublicInbox/LeiConvert.pm
lib/PublicInbox/LeiCurl.pm
lib/PublicInbox/LeiDedupe.pm
+lib/PublicInbox/LeiEditSearch.pm
lib/PublicInbox/LeiExternal.pm
lib/PublicInbox/LeiForgetSearch.pm
lib/PublicInbox/LeiHelp.pm
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index c3c79631..8fa89944 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -162,6 +162,8 @@ our %CMD = ( # sorted in order of importance/use:
qw(format|f=s pretty l ascii z|0), @c_opt ],
'forget-search' => [ 'OUTPUT', 'forget a saved search',
qw(verbose|v+), @c_opt ],
+'edit-search' => [ 'OUTPUT', "edit saved search via `git config --edit'",
+ @c_opt ],
'plonk' => [ '--threads|--from=IDENT',
'exclude mail matching From: or threads from non-Message-ID searches',
diff --git a/lib/PublicInbox/LeiEditSearch.pm b/lib/PublicInbox/LeiEditSearch.pm
new file mode 100644
index 00000000..fb36fdcd
--- /dev/null
+++ b/lib/PublicInbox/LeiEditSearch.pm
@@ -0,0 +1,25 @@
+# Copyright (C) 2021 all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+
+# "lei edit-search" edit a saved search following "lei q --save"
+package PublicInbox::LeiEditSearch;
+use strict;
+use v5.10.1;
+use PublicInbox::LeiSavedSearch;
+use PublicInbox::LeiUp;
+
+sub lei_edit_search {
+ my ($lei, $out) = @_;
+ my $lss = PublicInbox::LeiSavedSearch->up($lei, $out) or return;
+ my @cmd = (qw(git config --edit -f), $lss->{'-f'});
+ $lei->qerr("# spawning @cmd");
+ if ($lei->{oneshot}) {
+ exec(@cmd) or die "exec @cmd: $!\n";
+ } else {
+ $lei->send_exec_cmd([], \@cmd, {});
+ }
+}
+
+*_complete_edit_search = \&PublicInbox::LeiUp::_complete_up;
+
+1;
^ permalink raw reply related [relevance 64%]
* [PATCH 1/2] lei forget-search: new command to forget saved searches
2021-04-20 7:16 71% [PATCH 0/2] lei {edit,forget}-search Eric Wong
@ 2021-04-20 7:16 56% ` Eric Wong
2021-04-20 7:16 64% ` [PATCH 2/2] lei edit-search: command to tweak search parameters Eric Wong
1 sibling, 0 replies; 200+ results
From: Eric Wong @ 2021-04-20 7:16 UTC (permalink / raw)
To: meta
Readers may lose interest in subscription topics. This lets
them avoid clutter by forgetting a saved search.
This does not and will not destroy the contents of an --output
mailbox. In other words, this is similar to unsubscribing
from an Atom/RSS feed or NNTP group.
I've also decided we won't support 'mv-search', since it'll
probably be rarely used and "lei convert" can be used, instead.
---
MANIFEST | 1 +
lib/PublicInbox/LEI.pm | 4 ++--
lib/PublicInbox/LeiForgetSearch.pm | 32 ++++++++++++++++++++++++++++++
t/lei-q-save.t | 11 ++++++++++
4 files changed, 46 insertions(+), 2 deletions(-)
create mode 100644 lib/PublicInbox/LeiForgetSearch.pm
diff --git a/MANIFEST b/MANIFEST
index f35c514c..d4055af4 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -191,6 +191,7 @@ lib/PublicInbox/LeiConvert.pm
lib/PublicInbox/LeiCurl.pm
lib/PublicInbox/LeiDedupe.pm
lib/PublicInbox/LeiExternal.pm
+lib/PublicInbox/LeiForgetSearch.pm
lib/PublicInbox/LeiHelp.pm
lib/PublicInbox/LeiImport.pm
lib/PublicInbox/LeiInit.pm
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index c0385eb5..c3c79631 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -160,8 +160,8 @@ our %CMD = ( # sorted in order of importance/use:
'ls-search' => [ '[PREFIX]', 'list saved search queries',
qw(format|f=s pretty l ascii z|0), @c_opt ],
-'rm-query' => [ 'QUERY_NAME', 'remove a saved search', @c_opt ],
-'mv-query' => [ qw(OLD_NAME NEW_NAME), 'rename a saved search', @c_opt ],
+'forget-search' => [ 'OUTPUT', 'forget a saved search',
+ qw(verbose|v+), @c_opt ],
'plonk' => [ '--threads|--from=IDENT',
'exclude mail matching From: or threads from non-Message-ID searches',
diff --git a/lib/PublicInbox/LeiForgetSearch.pm b/lib/PublicInbox/LeiForgetSearch.pm
new file mode 100644
index 00000000..b5fe5fb1
--- /dev/null
+++ b/lib/PublicInbox/LeiForgetSearch.pm
@@ -0,0 +1,32 @@
+# Copyright (C) 2021 all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+
+# "lei forget-search" forget/remove a saved search "lei q --save"
+package PublicInbox::LeiForgetSearch;
+use strict;
+use v5.10.1;
+use PublicInbox::LeiSavedSearch;
+use PublicInbox::LeiUp;
+use File::Path ();
+use SelectSaver;
+
+sub lei_forget_search {
+ my ($lei, $out) = @_;
+ my $d = PublicInbox::LeiSavedSearch::lss_dir_for($lei, \$out);
+ if (-e $d) {
+ my $save;
+ my $opt = { safe => 1 };
+ if ($lei->{opt}->{verbose}) {
+ $opt->{verbose} = 1;
+ $save = SelectSaver->new($lei->{2});
+ }
+ File::Path::remove_tree($d, $opt);
+ } else {
+ $lei->fail("--save was not used with $out cwd=".
+ $lei->rel2abs('.'));
+ }
+}
+
+*_complete_forget_search = \&PublicInbox::LeiUp::_complete_up;
+
+1;
diff --git a/t/lei-q-save.t b/t/lei-q-save.t
index 58342171..5a2f7fff 100644
--- a/t/lei-q-save.t
+++ b/t/lei-q-save.t
@@ -110,5 +110,16 @@ test_lei(sub {
like($mb, qr/<qp\@example\.com>/, 'new result written w/ -a');
lei_ok(qw(up --all=local));
+
+ ok(!lei(qw(forget-search), "$home/bogus"), 'bogus forget');
+ lei_ok qw(_complete lei forget-search);
+ like($lei_out, qr/mbrd-aug/, 'forget-search completion');
+ lei_ok(qw(forget-search -v), "$home/mbrd-aug");
+ is($lei_out, '', 'no output');
+ like($lei_err, qr/\bmbrd-aug\b/, '-v (verbose) reported unlinks');
+ lei_ok qw(_complete lei forget-search);
+ unlike($lei_out, qr/mbrd-aug/,
+ 'forget-search completion cleared after forget');
+ ok(!lei('up', "$home/mbrd-aug"), 'lei up fails after forget');
});
done_testing;
^ permalink raw reply related [relevance 56%]
* [PATCH 1/4] lei up: fix help output and ARGV handling
2021-04-19 23:48 71% [PATCH 0/4] "lei up --all=local" support Eric Wong
@ 2021-04-19 23:48 71% ` Eric Wong
2021-04-19 23:49 65% ` [PATCH 3/4] lei up: more error checking for config loading Eric Wong
2021-04-19 23:49 49% ` [PATCH 4/4] lei up: support --all=local Eric Wong
2 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-19 23:48 UTC (permalink / raw)
To: meta
We don't support changing search terms once "lei q --save" is
used.
---
lib/PublicInbox/LEI.pm | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index f641f0d9..5ee02f64 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -138,8 +138,8 @@ our %CMD = ( # sorted in order of importance/use:
sort|s=s reverse|r offset=i pretty jobs|j=s globoff|g augment|a
import-before! lock=s@ rsyncable alert=s@ mua=s verbose|v+), @c_opt,
opt_dash('limit|n=i', '[0-9]+') ],
-'up' => [ 'SEARCH_TERMS...', 'update saved search',
- qw(jobs|j=s lock=s@ alert=s@ mua=s verbose|v+), @c_opt ],
+'up' => [ 'OUTPUT', 'update saved search',
+ qw(jobs|j=s lock=s@ alert=s@ mua=s verbose|v+ all), @c_opt ],
'blob' => [ 'OID', 'show a git blob, reconstructing from mail if necessary',
qw(git-dir=s@ cwd! verbose|v+ mail! oid-a|A=s path-a|a=s path-b|b=s),
^ permalink raw reply related [relevance 71%]
* [PATCH 3/4] lei up: more error checking for config loading
2021-04-19 23:48 71% [PATCH 0/4] "lei up --all=local" support Eric Wong
2021-04-19 23:48 71% ` [PATCH 1/4] lei up: fix help output and ARGV handling Eric Wong
@ 2021-04-19 23:49 65% ` Eric Wong
2021-04-19 23:49 49% ` [PATCH 4/4] lei up: support --all=local Eric Wong
2 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-19 23:49 UTC (permalink / raw)
To: meta
We'll support editing the saved search config file, so user
errors may happen and we need to throw sensible errors in that
case.
---
lib/PublicInbox/LeiUp.pm | 13 ++++++++-----
1 file changed, 8 insertions(+), 5 deletions(-)
diff --git a/lib/PublicInbox/LeiUp.pm b/lib/PublicInbox/LeiUp.pm
index 63a7f996..e80ccf57 100644
--- a/lib/PublicInbox/LeiUp.pm
+++ b/lib/PublicInbox/LeiUp.pm
@@ -14,24 +14,27 @@ sub lei_up {
my $lss = PublicInbox::LeiSavedSearch->up($lei, $out) or return;
my $mset_opt = $lei->{mset_opt} = { relevance => -2 };
$mset_opt->{limit} = $lei->{opt}->{limit} // 10000;
+ my $f = $lss->{'-f'};
my $q = $mset_opt->{q_raw} = $lss->{-cfg}->{'lei.q'} //
- return $lei->fail("lei.q unset in $lss->{-f}");
+ return $lei->fail("lei.q unset in $f");
my $lse = $lei->{lse} // die 'BUG: {lse} missing';
if (ref($q)) {
$mset_opt->{qstr} = $lse->query_argv_to_string($lse->git, $q);
} else {
$lse->query_approxidate($lse->git, $mset_opt->{qstr} = $q);
}
- $lei->{opt}->{output} = $lss->{-cfg}->{'lei.q.output'} //
- return $lei->fail("lei.q.output unset in $lss->{-f}");
-
+ my $o = $lei->{opt}->{output} = $lss->{-cfg}->{'lei.q.output'} //
+ return $lei->fail("lei.q.output unset in $f");
+ ref($o) and return $lei->fail("multiple values of lei.q.output in $f");
for my $k (qw(only include exclude)) {
my $v = $lss->{-cfg}->get_all("lei.q.$k") // next;
$lei->{opt}->{$k} = $v;
}
for my $k (qw(external local remote
import-remote import-before threads)) {
- my $v = $lss->{-cfg}->{"lei.q.$k"} // next;
+ my $c = "lei.q.$k";
+ my $v = $lss->{-cfg}->{$c} // next;
+ ref($v) and return $lei->fail("multiple values of $c in $f");
$lei->{opt}->{$k} = $v;
}
$lei->{lss} = $lss; # for LeiOverview->new
^ permalink raw reply related [relevance 65%]
* [PATCH 4/4] lei up: support --all=local
2021-04-19 23:48 71% [PATCH 0/4] "lei up --all=local" support Eric Wong
2021-04-19 23:48 71% ` [PATCH 1/4] lei up: fix help output and ARGV handling Eric Wong
2021-04-19 23:49 65% ` [PATCH 3/4] lei up: more error checking for config loading Eric Wong
@ 2021-04-19 23:49 49% ` Eric Wong
2 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-19 23:49 UTC (permalink / raw)
To: meta
Users may wish to update several saved searches at once. We can
support parallel updates in lei-daemon so users won't have to do
it themselves via xargs or similar.
Supporting IMAP outputs would be significantly more involved
since we'd have to pre-authenticate for every single IMAP
output before entering the redispatch loop.
---
lib/PublicInbox/LEI.pm | 7 +++--
lib/PublicInbox/LeiQuery.pm | 2 +-
lib/PublicInbox/LeiUp.pm | 59 +++++++++++++++++++++++++++++++++----
t/lei-q-save.t | 2 ++
4 files changed, 62 insertions(+), 8 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 5ee02f64..c0385eb5 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -138,8 +138,9 @@ our %CMD = ( # sorted in order of importance/use:
sort|s=s reverse|r offset=i pretty jobs|j=s globoff|g augment|a
import-before! lock=s@ rsyncable alert=s@ mua=s verbose|v+), @c_opt,
opt_dash('limit|n=i', '[0-9]+') ],
-'up' => [ 'OUTPUT', 'update saved search',
- qw(jobs|j=s lock=s@ alert=s@ mua=s verbose|v+ all), @c_opt ],
+
+'up' => [ 'OUTPUT|--all', 'update saved search',
+ qw(jobs|j=s lock=s@ alert=s@ mua=s verbose|v+ all:s), @c_opt ],
'blob' => [ 'OID', 'show a git blob, reconstructing from mail if necessary',
qw(git-dir=s@ cwd! verbose|v+ mail! oid-a|A=s path-a|a=s path-b|b=s),
@@ -331,6 +332,8 @@ my %OPTDESC = (
'remote' => 'limit operations to those requiring network access',
'remote!' => 'prevent operations requiring network access',
+'all:s up' => ['local', 'update all (local) saved searches' ],
+
'mid=s' => 'specify the Message-ID of a message',
'oid=s' => 'specify the git object ID of a message',
diff --git a/lib/PublicInbox/LeiQuery.pm b/lib/PublicInbox/LeiQuery.pm
index 385ba0a9..4099b26c 100644
--- a/lib/PublicInbox/LeiQuery.pm
+++ b/lib/PublicInbox/LeiQuery.pm
@@ -14,6 +14,7 @@ sub prep_ext { # externals_each callback
sub _start_query { # used by "lei q" and "lei up"
my ($self) = @_;
+ require PublicInbox::LeiOverview;
PublicInbox::LeiOverview->new($self) or return;
my $opt = $self->{opt};
my ($xj, $mj) = split(/,/, $opt->{jobs} // '');
@@ -117,7 +118,6 @@ sub lxs_prepare {
# the main "lei q SEARCH_TERMS" method
sub lei_q {
my ($self, @argv) = @_;
- require PublicInbox::LeiOverview;
PublicInbox::Config->json; # preload before forking
my $lxs = lxs_prepare($self) or return;
$self->ale->refresh_externals($lxs);
diff --git a/lib/PublicInbox/LeiUp.pm b/lib/PublicInbox/LeiUp.pm
index e80ccf57..0fb9698b 100644
--- a/lib/PublicInbox/LeiUp.pm
+++ b/lib/PublicInbox/LeiUp.pm
@@ -6,15 +6,14 @@ package PublicInbox::LeiUp;
use strict;
use v5.10.1;
use PublicInbox::LeiSavedSearch;
-use PublicInbox::LeiOverview;
+use parent qw(PublicInbox::IPC);
-sub lei_up {
+sub up1 ($$) {
my ($lei, $out) = @_;
- $lei->{lse} = $lei->_lei_store(1)->search;
my $lss = PublicInbox::LeiSavedSearch->up($lei, $out) or return;
+ my $f = $lss->{'-f'};
my $mset_opt = $lei->{mset_opt} = { relevance => -2 };
$mset_opt->{limit} = $lei->{opt}->{limit} // 10000;
- my $f = $lss->{'-f'};
my $q = $mset_opt->{q_raw} = $lss->{-cfg}->{'lei.q'} //
return $lei->fail("lei.q unset in $f");
my $lse = $lei->{lse} // die 'BUG: {lse} missing';
@@ -40,10 +39,60 @@ sub lei_up {
$lei->{lss} = $lss; # for LeiOverview->new
my $lxs = $lei->lxs_prepare or return;
$lei->ale->refresh_externals($lxs);
- $lei->{opt}->{save} = -1;
$lei->_start_query;
}
+sub up1_redispatch {
+ my ($lei, $out, $op_p) = @_;
+ my $l = bless { %$lei }, ref($lei);
+ $l->{opt} = { %{$l->{opt}} };
+ delete $l->{sock};
+ $l->{''} = $op_p; # daemon only
+ eval {
+ $l->qerr("# updating $out");
+ up1($l, $out);
+ $l->qerr("# $out done");
+ };
+ $l->err($@) if $@;
+}
+
+sub lei_up {
+ my ($lei, $out) = @_;
+ $lei->{lse} = $lei->_lei_store(1)->search;
+ my $opt = $lei->{opt};
+ $opt->{save} = -1;
+ if (defined $opt->{all}) {
+ length($opt->{mua}//'') and return
+ $lei->fail('--all and --mua= are incompatible');
+
+ # supporting IMAP outputs is more involved due to
+ # git-credential prompts. TODO: add this in 1.8
+ $opt->{all} eq 'local' or return
+ $lei->fail('only --all=local works at the moment');
+ my @all = PublicInbox::LeiSavedSearch::list($lei);
+ my @local = grep(!m!\Aimaps?://!i, @all);
+ $lei->_lei_store->write_prepare($lei); # share early
+ if ($lei->{oneshot}) { # synchronous
+ up1_redispatch($lei, $_) for @local;
+ } else {
+ # daemon mode, re-dispatch into our event loop w/o
+ # creating an extra fork-level
+ require PublicInbox::DS;
+ require PublicInbox::PktOp;
+ my ($op_c, $op_p) = PublicInbox::PktOp->pair;
+ for my $o (@local) {
+ PublicInbox::DS::requeue(sub {
+ up1_redispatch($lei, $o, $op_p);
+ });
+ }
+ $lei->event_step_init;
+ $op_c->{ops} = { '' => [$lei->can('dclose'), $lei] };
+ }
+ } else {
+ up1($lei, $out);
+ }
+}
+
sub _complete_up {
my ($lei, @argv) = @_;
my ($cur, $re) = $lei->complete_url_common(\@argv);
diff --git a/t/lei-q-save.t b/t/lei-q-save.t
index c0c74581..58342171 100644
--- a/t/lei-q-save.t
+++ b/t/lei-q-save.t
@@ -108,5 +108,7 @@ test_lei(sub {
$mb = do { local $/; <$mb> };
like($mb, qr/pre-existing/, 'pre-existing message preserved w/ -a');
like($mb, qr/<qp\@example\.com>/, 'new result written w/ -a');
+
+ lei_ok(qw(up --all=local));
});
done_testing;
^ permalink raw reply related [relevance 49%]
* [PATCH 0/4] "lei up --all=local" support
@ 2021-04-19 23:48 71% Eric Wong
2021-04-19 23:48 71% ` [PATCH 1/4] lei up: fix help output and ARGV handling Eric Wong
` (2 more replies)
0 siblings, 3 replies; 200+ results
From: Eric Wong @ 2021-04-19 23:48 UTC (permalink / raw)
To: meta
Eventually, "--all" and "--all=remote" will be supported,
but "--all=local" was relatively easy and will probably be
the most popular for latency-sensitive users.
Eric Wong (4):
lei up: fix help output and ARGV handling
config: favor ->get_all when possible
lei up: more error checking for config loading
lei up: support --all=local
lib/PublicInbox/LEI.pm | 7 +++-
lib/PublicInbox/LeiQuery.pm | 2 +-
lib/PublicInbox/LeiUp.pm | 75 +++++++++++++++++++++++++++++++------
lib/PublicInbox/Watch.pm | 3 +-
t/lei-q-save.t | 2 +
5 files changed, 72 insertions(+), 17 deletions(-)
^ permalink raw reply [relevance 71%]
* [PATCH 5/6] lei_saved_search: split "lei q --save" and "lei up" init paths
2021-04-19 8:52 71% [PATCH 0/6] lei saved search improvements Eric Wong
2021-04-19 8:52 68% ` [PATCH 1/6] lei: support unlinked/missing saved searches Eric Wong
2021-04-19 8:52 59% ` [PATCH 2/6] lei q: implement import-before default for --save Eric Wong
@ 2021-04-19 8:52 73% ` Eric Wong
2021-04-19 8:52 65% ` [PATCH 6/6] lei q: --save and --augment may be combined Eric Wong
3 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-19 8:52 UTC (permalink / raw)
To: meta
They're more different than alike, and having two separate
methods seems less confusing to me.
---
lib/PublicInbox/LeiLsSearch.pm | 2 +-
lib/PublicInbox/LeiSavedSearch.pm | 79 ++++++++++++++++---------------
lib/PublicInbox/LeiUp.pm | 2 +-
3 files changed, 44 insertions(+), 39 deletions(-)
diff --git a/lib/PublicInbox/LeiLsSearch.pm b/lib/PublicInbox/LeiLsSearch.pm
index 2aa457c0..9ac4870f 100644
--- a/lib/PublicInbox/LeiLsSearch.pm
+++ b/lib/PublicInbox/LeiLsSearch.pm
@@ -29,7 +29,7 @@ sub do_ls_search_long {
my @x = sort(grep(/\A\Q$pfx/, PublicInbox::LeiSavedSearch::list($lei)));
while (my $x = shift @x) {
$ORS = '' if !scalar(@x);
- my $lss = PublicInbox::LeiSavedSearch->new($lei, $x) or next;
+ my $lss = PublicInbox::LeiSavedSearch->up($lei, $x) or next;
my $cfg = $lss->{-cfg};
my $ent = {
q => $cfg->get_all('lei.q'),
diff --git a/lib/PublicInbox/LeiSavedSearch.pm b/lib/PublicInbox/LeiSavedSearch.pm
index d3a32d36..948e4954 100644
--- a/lib/PublicInbox/LeiSavedSearch.pm
+++ b/lib/PublicInbox/LeiSavedSearch.pm
@@ -65,52 +65,57 @@ sub list {
} @$out
}
-sub new {
+sub up { # updating existing saved search via "lei up"
my ($cls, $lei, $dst) = @_;
+ my $f;
my $self = bless { ale => $lei->ale }, $cls;
- my $dir;
- if (defined $dst) { # updating existing saved search via "lei up"
- my $f;
- $dir = $dst;
- output2lssdir($self, $lei, \$dir, \$f) or
- return $lei->fail("--save was not used with $dst cwd=".
- $lei->rel2abs('.'));
- $self->{'-f'} = $f;
- } else { # new saved search "lei q --save"
- $dst = $lei->{ovv}->{dst};
- $dir = lss_dir_for($lei, \$dst);
- require File::Path;
- File::Path::make_path($dir); # raises on error
- $self->{-cfg} = {};
- my $f = $self->{'-f'} = "$dir/lei.saved-search";
- open my $fh, '>', $f or return $lei->fail("open $f: $!");
- my $sq_dst = PublicInbox::Config::squote_maybe($dst);
- my $q = $lei->{mset_opt}->{q_raw} // die 'BUG: {q_raw} missing';
- if (ref $q) {
- $q = join("\n", map { "\tq = ".cquote_val($_) } @$q);
- } else {
- $q = "\tq = ".cquote_val($q);
- }
- $dst = "$lei->{ovv}->{fmt}:$dst" if $dst !~ m!\Aimaps?://!i;
- print $fh <<EOM;
+ my $dir = $dst;
+ output2lssdir($self, $lei, \$dir, \$f) or
+ return $lei->fail("--save was not used with $dst cwd=".
+ $lei->rel2abs('.'));
+ $self->{-cfg} = PublicInbox::Config->git_config_dump($f);
+ $self->{-ovf} = "$dir/over.sqlite3";
+ $self->{'-f'} = $f;
+ $self->{lock_path} = "$self->{-f}.flock";
+ $self;
+}
+
+sub new { # new saved search "lei q --save"
+ my ($cls, $lei) = @_;
+ my $self = bless { ale => $lei->ale }, $cls;
+ my $dst = $lei->{ovv}->{dst};
+ my $dir = lss_dir_for($lei, \$dst);
+ require File::Path;
+ File::Path::make_path($dir); # raises on error
+ $self->{-cfg} = {};
+ my $f = $self->{'-f'} = "$dir/lei.saved-search";
+ open my $fh, '>', $f or return $lei->fail("open $f: $!");
+ my $sq_dst = PublicInbox::Config::squote_maybe($dst);
+ my $q = $lei->{mset_opt}->{q_raw} // die 'BUG: {q_raw} missing';
+ if (ref $q) {
+ $q = join("\n", map { "\tq = ".cquote_val($_) } @$q);
+ } else {
+ $q = "\tq = ".cquote_val($q);
+ }
+ $dst = "$lei->{ovv}->{fmt}:$dst" if $dst !~ m!\Aimaps?://!i;
+ print $fh <<EOM;
; to refresh with new results, run: lei up $sq_dst
[lei]
-$q
+ $q
[lei "q"]
output = $dst
EOM
- for my $k (ARRAY_FIELDS) {
- my $ary = $lei->{opt}->{$k} // next;
- for my $x (@$ary) {
- print $fh "\t$k = ".cquote_val($x)."\n";
- }
- }
- for my $k (BOOL_FIELDS) {
- my $val = $lei->{opt}->{$k} // next;
- print $fh "\t$k = ".($val ? 1 : 0)."\n";
+ for my $k (ARRAY_FIELDS) {
+ my $ary = $lei->{opt}->{$k} // next;
+ for my $x (@$ary) {
+ print $fh "\t$k = ".cquote_val($x)."\n";
}
- close($fh) or return $lei->fail("close $f: $!");
}
+ for my $k (BOOL_FIELDS) {
+ my $val = $lei->{opt}->{$k} // next;
+ print $fh "\t$k = ".($val ? 1 : 0)."\n";
+ }
+ close($fh) or return $lei->fail("close $f: $!");
$self->{lock_path} = "$self->{-f}.flock";
$self->{-ovf} = "$dir/over.sqlite3";
$self;
diff --git a/lib/PublicInbox/LeiUp.pm b/lib/PublicInbox/LeiUp.pm
index e4cbc825..23c5c606 100644
--- a/lib/PublicInbox/LeiUp.pm
+++ b/lib/PublicInbox/LeiUp.pm
@@ -11,7 +11,7 @@ use PublicInbox::LeiOverview;
sub lei_up {
my ($lei, $out) = @_;
$lei->{lse} = $lei->_lei_store(1)->search;
- my $lss = PublicInbox::LeiSavedSearch->new($lei, $out) or return;
+ my $lss = PublicInbox::LeiSavedSearch->up($lei, $out) or return;
my $mset_opt = $lei->{mset_opt} = { relevance => -2 };
$mset_opt->{limit} = $lei->{opt}->{limit} // 10000;
my $q = $mset_opt->{q_raw} = $lss->{-cfg}->{'lei.q'} //
^ permalink raw reply related [relevance 73%]
* [PATCH 0/6] lei saved search improvements
@ 2021-04-19 8:52 71% Eric Wong
2021-04-19 8:52 68% ` [PATCH 1/6] lei: support unlinked/missing saved searches Eric Wong
` (3 more replies)
0 siblings, 4 replies; 200+ results
From: Eric Wong @ 2021-04-19 8:52 UTC (permalink / raw)
To: meta
I think the "lei q --save" and "lei up" commands are in a good
state. There's no way to edit/forget saved searches, yet,
so I guess that's next...
Eric Wong (6):
lei: support unlinked/missing saved searches
lei q: implement import-before default for --save
lei_saved_search: avoid needless var shadowing
config: git_config_dump blesses
lei_saved_search: split "lei q --save" and "lei up" init paths
lei q: --save and --augment may be combined
lib/PublicInbox/Config.pm | 10 ++--
lib/PublicInbox/LEI.pm | 3 +-
lib/PublicInbox/LeiLsSearch.pm | 2 +-
lib/PublicInbox/LeiMirror.pm | 2 +-
lib/PublicInbox/LeiSavedSearch.pm | 94 +++++++++++++++----------------
lib/PublicInbox/LeiToMail.pm | 6 +-
lib/PublicInbox/LeiUp.pm | 4 +-
t/lei-q-save.t | 41 ++++++++++++++
8 files changed, 101 insertions(+), 61 deletions(-)
^ permalink raw reply [relevance 71%]
* [PATCH 1/6] lei: support unlinked/missing saved searches
2021-04-19 8:52 71% [PATCH 0/6] lei saved search improvements Eric Wong
@ 2021-04-19 8:52 68% ` Eric Wong
2021-04-19 8:52 59% ` [PATCH 2/6] lei q: implement import-before default for --save Eric Wong
` (2 subsequent siblings)
3 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-19 8:52 UTC (permalink / raw)
To: meta
It's conceivable a user will want to erase all previous
results but still rerun/refresh a search to get new results.
We probably won't support prune functionality, here, and
instead require explicit removal of saved searches.
---
lib/PublicInbox/LeiSavedSearch.pm | 7 ++-----
t/lei-q-save.t | 7 +++++++
2 files changed, 9 insertions(+), 5 deletions(-)
diff --git a/lib/PublicInbox/LeiSavedSearch.pm b/lib/PublicInbox/LeiSavedSearch.pm
index d67622c9..94920a4e 100644
--- a/lib/PublicInbox/LeiSavedSearch.pm
+++ b/lib/PublicInbox/LeiSavedSearch.pm
@@ -61,11 +61,8 @@ sub list {
bless $cfg, 'PublicInbox::Config';
my $out = $cfg->get_all('lei.q.output') or return ();
map {;
- if (s!\A(?:maildir|mh|mbox.+|mmdf):!!i) {
- -e $_ ? $_ : (); # TODO auto-prune somewhere?
- } else { # IMAP, maybe JMAP
- $_;
- }
+ s!\A(?:maildir|mh|mbox.+|mmdf):!!i;
+ $_;
} @$out
}
diff --git a/t/lei-q-save.t b/t/lei-q-save.t
index 761814b4..4e6ed642 100644
--- a/t/lei-q-save.t
+++ b/t/lei-q-save.t
@@ -67,5 +67,12 @@ test_lei(sub {
lei_ok qw(_complete lei up);
like($lei_out, qr!^\Q$home/mbcl2\E$!sm, 'complete got mbcl2 output');
like($lei_out, qr!^\Q$home/md/\E$!sm, 'complete got maildir output');
+
+ unlink("$home/mbcl2") or xbail "unlink $!";
+ lei_ok qw(_complete lei up);
+ like($lei_out, qr!^\Q$home/mbcl2\E$!sm,
+ 'mbcl2 output shown despite unlink');
+ lei_ok([qw(up mbcl2)], undef, { -C => $home, %$lei_opt });
+ ok(-f "$home/mbcl2" && -s _ == 0, 'up recreates on missing output');
});
done_testing;
^ permalink raw reply related [relevance 68%]
* [PATCH 6/6] lei q: --save and --augment may be combined
2021-04-19 8:52 71% [PATCH 0/6] lei saved search improvements Eric Wong
` (2 preceding siblings ...)
2021-04-19 8:52 73% ` [PATCH 5/6] lei_saved_search: split "lei q --save" and "lei up" init paths Eric Wong
@ 2021-04-19 8:52 65% ` Eric Wong
3 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-19 8:52 UTC (permalink / raw)
To: meta
This necessitated fixing pause_dedupe to release the handle
used by ->lock_for_scope_fast, but otherwise no changes to
the LeiToMail package.
---
lib/PublicInbox/LeiSavedSearch.pm | 1 +
t/lei-q-save.t | 13 +++++++++++++
2 files changed, 14 insertions(+)
diff --git a/lib/PublicInbox/LeiSavedSearch.pm b/lib/PublicInbox/LeiSavedSearch.pm
index 948e4954..cd9effce 100644
--- a/lib/PublicInbox/LeiSavedSearch.pm
+++ b/lib/PublicInbox/LeiSavedSearch.pm
@@ -182,6 +182,7 @@ sub git { $_[0]->{ale}->git }
sub pause_dedupe {
my ($self) = @_;
$self->{ale}->git->cleanup;
+ my $lockfh = delete $self->{lockfh}; # from lock_for_scope_fast;
my $oidx = delete($self->{oidx}) // return;
$oidx->commit_lazy;
}
diff --git a/t/lei-q-save.t b/t/lei-q-save.t
index 5bc8fb74..c0c74581 100644
--- a/t/lei-q-save.t
+++ b/t/lei-q-save.t
@@ -84,6 +84,7 @@ test_lei(sub {
lei_ok([qw(up mbcl2)], undef, { -C => $home, %$lei_opt });
ok(-f "$home/mbcl2" && -s _ == 0, 'up recreates on missing output');
+ # no --augment
open my $mb, '>', "$home/mbrd" or xbail "open $!";
print $mb $pre_existing;
close $mb or xbail "close: $!";
@@ -92,8 +93,20 @@ test_lei(sub {
open $mb, '<', "$home/mbrd" or xbail "open $!";
is_deeply([grep(/pre-existing/, <$mb>)], [],
'pre-existing messsage gone w/o augment');
+ close $mb;
lei_ok(qw(q m:import-before@example.com));
is(json_utf8->decode($lei_out)->[0]->{'s'},
'pre-existing', '--save imported before clobbering');
+
+ # --augment
+ open $mb, '>', "$home/mbrd-aug" or xbail "open $!";
+ print $mb $pre_existing;
+ close $mb or xbail "close: $!";
+ lei_ok(qw(q -a --save -o mboxrd:mbrd-aug m:qp@example.com -C), $home);
+ chdir($dh) or xbail "fchdir . $!";
+ open $mb, '<', "$home/mbrd-aug" or xbail "open $!";
+ $mb = do { local $/; <$mb> };
+ like($mb, qr/pre-existing/, 'pre-existing message preserved w/ -a');
+ like($mb, qr/<qp\@example\.com>/, 'new result written w/ -a');
});
done_testing;
^ permalink raw reply related [relevance 65%]
* [PATCH 2/6] lei q: implement import-before default for --save
2021-04-19 8:52 71% [PATCH 0/6] lei saved search improvements Eric Wong
2021-04-19 8:52 68% ` [PATCH 1/6] lei: support unlinked/missing saved searches Eric Wong
@ 2021-04-19 8:52 59% ` Eric Wong
2021-04-19 8:52 73% ` [PATCH 5/6] lei_saved_search: split "lei q --save" and "lei up" init paths Eric Wong
2021-04-19 8:52 65% ` [PATCH 6/6] lei q: --save and --augment may be combined Eric Wong
3 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-19 8:52 UTC (permalink / raw)
To: meta
This makes "lei q --save" as safe as "lei q" to prevent against
accidental data loss when clobbering an existing output,
---
lib/PublicInbox/LeiToMail.pm | 6 +++---
lib/PublicInbox/LeiUp.pm | 2 +-
t/lei-q-save.t | 21 +++++++++++++++++++++
3 files changed, 25 insertions(+), 4 deletions(-)
diff --git a/lib/PublicInbox/LeiToMail.pm b/lib/PublicInbox/LeiToMail.pm
index daa8084b..46a82a4b 100644
--- a/lib/PublicInbox/LeiToMail.pm
+++ b/lib/PublicInbox/LeiToMail.pm
@@ -375,7 +375,7 @@ sub _pre_augment_maildir {
sub _do_augment_maildir {
my ($self, $lei) = @_;
- return if defined($lei->{opt}->{save});
+ return if ($lei->{opt}->{save} // 0) < 0;
my $dst = $lei->{ovv}->{dst};
my $lse = $lei->{opt}->{'import-before'} ? $lei->{lse} : undef;
my $mdr = PublicInbox::MdirReader->new;
@@ -406,7 +406,7 @@ sub _imap_augment_or_delete { # PublicInbox::NetReader::imap_each cb
sub _do_augment_imap {
my ($self, $lei) = @_;
- return if defined($lei->{opt}->{save});
+ return if ($lei->{opt}->{save} // 0) < 0;
my $net = $lei->{net};
my $lse = $lei->{opt}->{'import-before'} ? $lei->{lse} : undef;
if ($lei->{opt}->{augment}) {
@@ -477,7 +477,7 @@ sub _do_augment_mbox {
my ($self, $lei) = @_;
return unless $self->{seekable};
my $opt = $lei->{opt};
- return if defined($opt->{save});
+ return if ($opt->{save} // 0) < 0;
my $out = $lei->{1};
my ($fmt, $dst) = @{$lei->{ovv}}{qw(fmt dst)};
return unless -s $out;
diff --git a/lib/PublicInbox/LeiUp.pm b/lib/PublicInbox/LeiUp.pm
index 73286ea2..e4cbc825 100644
--- a/lib/PublicInbox/LeiUp.pm
+++ b/lib/PublicInbox/LeiUp.pm
@@ -38,7 +38,7 @@ sub lei_up {
$lei->{lss} = $lss; # for LeiOverview->new
my $lxs = $lei->lxs_prepare or return;
$lei->ale->refresh_externals($lxs);
- $lei->{opt}->{save} = 1;
+ $lei->{opt}->{save} = -1;
$lei->_start_query;
}
diff --git a/t/lei-q-save.t b/t/lei-q-save.t
index 4e6ed642..5bc8fb74 100644
--- a/t/lei-q-save.t
+++ b/t/lei-q-save.t
@@ -10,6 +10,15 @@ $doc2->header_set('Date', PublicInbox::Smsg::date({ds => time - (86400 * 4)}));
my $doc3 = eml_load('t/msg_iter-order.eml');
$doc3->header_set('Date', PublicInbox::Smsg::date({ds => time - (86400 * 4)}));
+my $pre_existing = <<'EOF';
+From x Mon Sep 17 00:00:00 2001
+Message-ID: <import-before@example.com>
+Subject: pre-existing
+Date: Sat, 02 Oct 2010 00:00:00 +0000
+
+blah
+EOF
+
test_lei(sub {
my $home = $ENV{HOME};
my $in = $doc1->as_string;
@@ -74,5 +83,17 @@ test_lei(sub {
'mbcl2 output shown despite unlink');
lei_ok([qw(up mbcl2)], undef, { -C => $home, %$lei_opt });
ok(-f "$home/mbcl2" && -s _ == 0, 'up recreates on missing output');
+
+ open my $mb, '>', "$home/mbrd" or xbail "open $!";
+ print $mb $pre_existing;
+ close $mb or xbail "close: $!";
+ lei_ok(qw(q --save -o mboxrd:mbrd m:qp@example.com -C), $home);
+ chdir($dh) or xbail "fchdir . $!";
+ open $mb, '<', "$home/mbrd" or xbail "open $!";
+ is_deeply([grep(/pre-existing/, <$mb>)], [],
+ 'pre-existing messsage gone w/o augment');
+ lei_ok(qw(q m:import-before@example.com));
+ is(json_utf8->decode($lei_out)->[0]->{'s'},
+ 'pre-existing', '--save imported before clobbering');
});
done_testing;
^ permalink raw reply related [relevance 59%]
* [PATCH] lei ls-search: command to list saved searches
@ 2021-04-18 8:40 36% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-18 8:40 UTC (permalink / raw)
To: meta
Going forward, we'll probably support JSON for all the "ls-*"
subcommands. This also provides the basis for "lei up" shell
completion.
---
MANIFEST | 1 +
lib/PublicInbox/LEI.pm | 10 +--
lib/PublicInbox/LeiExternal.pm | 11 +--
lib/PublicInbox/LeiLsSearch.pm | 109 ++++++++++++++++++++++++++++++
lib/PublicInbox/LeiSavedSearch.pm | 37 ++++++++--
lib/PublicInbox/LeiUp.pm | 6 ++
t/lei-q-save.t | 12 ++++
7 files changed, 173 insertions(+), 13 deletions(-)
create mode 100644 lib/PublicInbox/LeiLsSearch.pm
diff --git a/MANIFEST b/MANIFEST
index 1b7d16ee..f35c514c 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -196,6 +196,7 @@ lib/PublicInbox/LeiImport.pm
lib/PublicInbox/LeiInit.pm
lib/PublicInbox/LeiInput.pm
lib/PublicInbox/LeiLsLabel.pm
+lib/PublicInbox/LeiLsSearch.pm
lib/PublicInbox/LeiMirror.pm
lib/PublicInbox/LeiOverview.pm
lib/PublicInbox/LeiP2q.pm
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index f223b3de..56640be1 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -157,8 +157,8 @@ our %CMD = ( # sorted in order of importance/use:
'exclude further results from a publicinbox|extindex',
qw(prune), @c_opt ],
-'ls-query' => [ '[FILTER...]', 'list saved search queries',
- qw(name-only format|f=s), @c_opt ],
+'ls-search' => [ '[PREFIX]', 'list saved search queries',
+ qw(format|f=s pretty l ascii z|0), @c_opt ],
'rm-query' => [ 'QUERY_NAME', 'remove a saved search', @c_opt ],
'mv-query' => [ qw(OLD_NAME NEW_NAME), 'rename a saved search', @c_opt ],
@@ -312,7 +312,9 @@ my %OPTDESC = (
'jobs|j=i add-external' => 'set parallelism when indexing after --mirror',
'in-format|F=s' => $stdin_formats,
-'format|f=s ls-query' => $ls_format,
+'format|f=s ls-search' => ['OUT|json|jsonl|concatjson',
+ 'listing output format' ],
+'l ls-search' => 'long listing format',
'format|f=s ls-external' => $ls_format,
'limit|n=i@' => ['NUM', 'limit on number of matches (default: 10000)' ],
@@ -353,7 +355,7 @@ my %CONFIG_KEYS = (
'leistore.dir' => 'top-level storage location',
);
-my @WQ_KEYS = qw(lxs l2m imp mrr cnv p2q tag sol); # internal workers
+my @WQ_KEYS = qw(lxs l2m imp mrr cnv p2q tag sol lsss); # internal workers
sub _drop_wq {
my ($self) = @_;
diff --git a/lib/PublicInbox/LeiExternal.pm b/lib/PublicInbox/LeiExternal.pm
index 5e8dc71a..b0ebe947 100644
--- a/lib/PublicInbox/LeiExternal.pm
+++ b/lib/PublicInbox/LeiExternal.pm
@@ -215,8 +215,8 @@ sub lei_forget_external {
}
}
-sub _complete_url_common ($) {
- my ($argv) = @_;
+sub complete_url_common {
+ my $argv = $_[-1];
# Workaround bash word-splitting URLs to ['https', ':', '//' ...]
# Maybe there's a better way to go about this in
# contrib/completion/lei-completion.bash
@@ -228,7 +228,8 @@ sub _complete_url_common ($) {
push @x, $cur;
$cur = '';
}
- while (@x > 2 && $x[0] !~ /\Ahttps?\z/ && $x[1] ne ':') {
+ while (@x > 2 && $x[0] !~ /\A(?:http|nntp|imap)s?\z/i &&
+ $x[1] ne ':') {
shift @x;
}
if (@x >= 2) { # qw(https : hostname : 443) or qw(http :)
@@ -245,7 +246,7 @@ sub _complete_url_common ($) {
sub _complete_forget_external {
my ($self, @argv) = @_;
my $cfg = $self->_lei_cfg;
- my ($cur, $re) = _complete_url_common(\@argv);
+ my ($cur, $re) = complete_url_common(\@argv);
# FIXME: bash completion off "http:" or "https:" when the last
# character is a colon doesn't work properly even if we're
# returning "//$HTTP_HOST/$PATH_INFO/", not sure why, could
@@ -261,7 +262,7 @@ sub _complete_forget_external {
sub _complete_add_external { # for bash, this relies on "compopt -o nospace"
my ($self, @argv) = @_;
my $cfg = $self->_lei_cfg;
- my ($cur, $re) = _complete_url_common(\@argv);
+ my ($cur, $re) = complete_url_common(\@argv);
require URI;
map {
my $u = URI->new(substr($_, length('external.')));
diff --git a/lib/PublicInbox/LeiLsSearch.pm b/lib/PublicInbox/LeiLsSearch.pm
new file mode 100644
index 00000000..2aa457c0
--- /dev/null
+++ b/lib/PublicInbox/LeiLsSearch.pm
@@ -0,0 +1,109 @@
+# Copyright (C) 2021 all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+
+# "lei ls-search" to display results saved via "lei q --save"
+package PublicInbox::LeiLsSearch;
+use strict;
+use v5.10.1;
+use PublicInbox::LeiSavedSearch;
+use parent qw(PublicInbox::IPC);
+
+sub do_ls_search_long {
+ my ($self, $pfx) = @_;
+ # TODO: share common JSON output code with LeiOverview
+ my $json = $self->{json}->new->utf8->canonical;
+ my $lei = $self->{lei};
+ $json->ascii(1) if $lei->{opt}->{ascii};
+ my $fmt = $lei->{opt}->{'format'};
+ $lei->{1}->autoflush(0);
+ my $ORS = "\n";
+ my $pretty = $lei->{opt}->{pretty};
+ my $EOR; # TODO: compact pretty like "lei q"
+ if ($fmt =~ /\A(concat)?json\z/ && $pretty) {
+ $EOR = ($1//'') eq 'concat' ? "\n}" : "\n},";
+ }
+ if ($fmt eq 'json') {
+ $lei->out('[');
+ $ORS = ",\n";
+ }
+ my @x = sort(grep(/\A\Q$pfx/, PublicInbox::LeiSavedSearch::list($lei)));
+ while (my $x = shift @x) {
+ $ORS = '' if !scalar(@x);
+ my $lss = PublicInbox::LeiSavedSearch->new($lei, $x) or next;
+ my $cfg = $lss->{-cfg};
+ my $ent = {
+ q => $cfg->get_all('lei.q'),
+ output => $cfg->{'lei.q.output'},
+ };
+ for my $k ($lss->ARRAY_FIELDS) {
+ my $ary = $cfg->get_all("lei.q.$k") // next;
+ $ent->{$k} = $ary;
+ }
+ for my $k ($lss->BOOL_FIELDS) {
+ my $val = $cfg->{"lei.q.$k"} // next;
+ $ent->{$k} = $val;
+ }
+ if (defined $EOR) { # pretty, but compact
+ $EOR = "\n}" if !scalar(@x);
+ my $buf = "{\n";
+ $buf .= join(",\n", map {;
+ my $f = $_;
+ if (my $v = $ent->{$f}) {
+ $v = $json->encode([$v]);
+ qq{ "$f": }.substr($v, 1, -1);
+ } else {
+ ();
+ }
+ # key order by importance
+ } (qw(output q), $lss->ARRAY_FIELDS,
+ $lss->BOOL_FIELDS) );
+ $lei->out($buf .= $EOR);
+ } else {
+ $lei->out($json->encode($ent), $ORS);
+ }
+ }
+ if ($fmt eq 'json') {
+ $lei->out("]\n");
+ } elsif ($fmt eq 'concatjson') {
+ $lei->out("\n");
+ }
+}
+
+sub bg_worker ($$$) {
+ my ($lei, $pfx, $json) = @_;
+ my $self = bless { -wq_nr_workers => 1, json => $json }, __PACKAGE__;
+ my ($op_c, $ops) = $lei->workers_start($self, 'ls-search', 1);
+ $lei->{lsss} = $self;
+ $self->wq_io_do('do_ls_search_long', [], $pfx);
+ $self->wq_close(1);
+ $op_c->op_wait_event($ops);
+}
+
+sub lei_ls_search {
+ my ($lei, $pfx) = @_;
+ my $fmt = $lei->{opt}->{'format'} // '';
+ if ($lei->{opt}->{l}) {
+ $lei->{opt}->{'format'} //= $fmt = 'json';
+ }
+ my $json;
+ my $tty = -t $lei->{1};
+ $lei->start_pager if $tty;
+ if ($fmt =~ /\A(ldjson|ndjson|jsonl|(?:concat)?json)\z/) {
+ $lei->{opt}->{pretty} //= $tty;
+ $json = ref(PublicInbox::Config->json);
+ } elsif ($fmt ne '') {
+ return $lei->fail("unknown format: $fmt");
+ }
+ my $ORS = "\n";
+ if ($lei->{opt}->{z}) {
+ return $lei->fail('-z and --format do not mix') if $json;
+ $ORS = "\0";
+ }
+ $pfx //= '';
+ return bg_worker($lei, $pfx, $json) if $json;
+ for (sort(grep(/\A\Q$pfx/, PublicInbox::LeiSavedSearch::list($lei)))) {
+ $lei->out($_, $ORS);
+ }
+}
+
+1;
diff --git a/lib/PublicInbox/LeiSavedSearch.pm b/lib/PublicInbox/LeiSavedSearch.pm
index 3076d14c..d67622c9 100644
--- a/lib/PublicInbox/LeiSavedSearch.pm
+++ b/lib/PublicInbox/LeiSavedSearch.pm
@@ -21,6 +21,11 @@ sub cquote_val ($) { # cf. git-config(1)
$val;
}
+sub ARRAY_FIELDS () { qw(only include exclude) }
+sub BOOL_FIELDS () {
+ qw(external local remote import-remote import-before threads)
+}
+
sub lss_dir_for ($$) {
my ($lei, $dstref) = @_;
my @n;
@@ -39,6 +44,31 @@ sub lss_dir_for ($$) {
$lei->share_path . '/saved-searches/' . join('-', @n);
}
+sub list {
+ my ($lei, $pfx) = @_;
+ my $lss_dir = $lei->share_path.'/saved-searches/';
+ return () unless -d $lss_dir;
+ # TODO: persist the cache? Use another format?
+ my $f = $lei->cache_dir."/saved-tmp.$$.".time.'.config';
+ open my $fh, '>', $f or die "open $f: $!";
+ print $fh "[include]\n";
+ for my $p (glob("$lss_dir/*/lei.saved-search")) {
+ print $fh "\tpath = ", cquote_val($p), "\n";
+ }
+ close $fh or die "close $f: $!";
+ my $cfg = PublicInbox::Config::git_config_dump($f);
+ unlink($f);
+ bless $cfg, 'PublicInbox::Config';
+ my $out = $cfg->get_all('lei.q.output') or return ();
+ map {;
+ if (s!\A(?:maildir|mh|mbox.+|mmdf):!!i) {
+ -e $_ ? $_ : (); # TODO auto-prune somewhere?
+ } else { # IMAP, maybe JMAP
+ $_;
+ }
+ } @$out
+}
+
sub new {
my ($cls, $lei, $dst) = @_;
my $self = bless { ale => $lei->ale }, $cls;
@@ -74,16 +104,15 @@ $q
[lei "q"]
output = $dst
EOM
- for my $k (qw(only include exclude)) {
+ for my $k (ARRAY_FIELDS) {
my $ary = $lei->{opt}->{$k} // next;
for my $x (@$ary) {
print $fh "\t$k = ".cquote_val($x)."\n";
}
}
- for my $k (qw(external local remote import-remote
- import-before threads)) {
+ for my $k (BOOL_FIELDS) {
my $val = $lei->{opt}->{$k} // next;
- print $fh "\t$k = ".cquote_val($val)."\n";
+ print $fh "\t$k = ".($val ? 1 : 0)."\n";
}
close($fh) or return $lei->fail("close $f: $!");
}
diff --git a/lib/PublicInbox/LeiUp.pm b/lib/PublicInbox/LeiUp.pm
index 9fe4901b..73286ea2 100644
--- a/lib/PublicInbox/LeiUp.pm
+++ b/lib/PublicInbox/LeiUp.pm
@@ -42,4 +42,10 @@ sub lei_up {
$lei->_start_query;
}
+sub _complete_up {
+ my ($lei, @argv) = @_;
+ my ($cur, $re) = $lei->complete_url_common(\@argv);
+ grep(/\A$re\Q$cur/, PublicInbox::LeiSavedSearch::list($lei));
+}
+
1;
diff --git a/t/lei-q-save.t b/t/lei-q-save.t
index a8eda41e..761814b4 100644
--- a/t/lei-q-save.t
+++ b/t/lei-q-save.t
@@ -55,5 +55,17 @@ test_lei(sub {
ok(-s "$home/mbcl2" > $size, 'size increased after up');
ok(!lei(qw(up -q), $home), 'up fails w/o --save');
+
+ lei_ok qw(ls-search); my @d = split(/\n/, $lei_out);
+ lei_ok qw(ls-search -z); my @z = split(/\0/, $lei_out);
+ is_deeply(\@d, \@z, '-z output matches non-z');
+ is_deeply(\@d, [ "$home/mbcl2", "$home/md/" ],
+ 'ls-search output alphabetically sorted');
+ lei_ok qw(ls-search -l);
+ my $json = PublicInbox::Config->json->decode($lei_out);
+ ok($json && $json->[0]->{output}, 'JSON has output');
+ lei_ok qw(_complete lei up);
+ like($lei_out, qr!^\Q$home/mbcl2\E$!sm, 'complete got mbcl2 output');
+ like($lei_out, qr!^\Q$home/md/\E$!sm, 'complete got maildir output');
});
done_testing;
^ permalink raw reply related [relevance 36%]
* Re: [PATCH] lei q: fix MUA spawn after reading query from stdin
2021-04-17 19:00 60% ` [PATCH] lei q: fix MUA spawn after reading query from stdin Eric Wong
@ 2021-04-17 20:13 70% ` Kyle Meyer
0 siblings, 0 replies; 200+ results
From: Kyle Meyer @ 2021-04-17 20:13 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
Eric Wong writes:
> Kyle Meyer <kyle@kyleam.com> wrote:
>> I'm not a mutt user, but I think that's because mutt sees that stdin
>> isn't attached to a tty. I haven't tried anything on my end yet, but
>> perhaps there's a clean way to make --stdin and --mua [*] work together.
>
> Thanks for the report. Yes, that's correct, fortunately we can
> reasonably expect stdout to be a terminal and the patch below
> should fix it.
Confirmed. Thanks for the quick fix.
>> [*] The only other --mua value I checked was 'mail' (and that shows a
>> similar issue), but I'm guessing other MUAs don't not work well
>> with --stdin either.
>
> Btw, since you seem to be a gnus user;
I'm a lightweight Gnus user :) I just use gnus to read NNTP, and use
Notmuch for mail. But...
> does/can gnus work via --mua= ?
... I don't see a good way to do this, no. A more natural approach
would be calling lei from Emacs.
I've been thinking a good amount about Emacs integration with lei in the
context of piem (<https://git.kyleam.com/piem/>). My initial focus will
be something closer to Notmuch's Emacs interface. I'm pretty excited
about the it, as it will be the main way I interact with lei (but sadly
it'll be at least a few weeks before I have the free time to begin any
work on it).
The above interface will probably reduce my use of Gnus to lists that
don't have public-inbox archives, but I still might try to add some Gnus
integration eventually. Gnus has a mairix backend, so that'd be the
first placed I'd study for approaching integration.
^ permalink raw reply [relevance 70%]
* [PATCH 2/] lei up: further improve Maildir canonicalization
2021-04-17 10:24 62% [PATCH] lei up: fix canonicalization of Maildirs Eric Wong
@ 2021-04-17 19:00 63% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-17 19:00 UTC (permalink / raw)
To: meta
We want to be able to use "lei up ." when inside a Maildir.
We'll also relax Maildir/mbox basenames to be any non-'/'
character after converting relative paths to absolute. The
old restriction on allowed characters was unnecessary and made
it impossible to reliably map "." when used as the sole argument
for "lei up".
---
lib/PublicInbox/LeiSavedSearch.pm | 6 ++++--
t/lei-q-save.t | 4 ++++
2 files changed, 8 insertions(+), 2 deletions(-)
diff --git a/lib/PublicInbox/LeiSavedSearch.pm b/lib/PublicInbox/LeiSavedSearch.pm
index 0f632d93..e44779ee 100644
--- a/lib/PublicInbox/LeiSavedSearch.pm
+++ b/lib/PublicInbox/LeiSavedSearch.pm
@@ -24,9 +24,10 @@ sub lss_dir_for ($$) {
$$dstref = $$uri;
@n = ($uri->mailbox);
} else { # basename
- @n = ($$dstref =~ m{([\w\-\.]+)/*\z});
$$dstref = $lei->rel2abs($$dstref);
$$dstref .= '/' if -d $$dstref;
+ $$dstref =~ tr!/!/!s;
+ @n = ($$dstref =~ m{([^/]+)/*\z});
}
push @n, sha256_hex($$dstref);
$lei->share_path . '/saved-searches/' . join('-', @n);
@@ -40,7 +41,8 @@ sub new {
my $f;
$dir = $dst;
output2lssdir($self, $lei, \$dir, \$f) or
- return $lei->fail("--save was not used with $dst");
+ return $lei->fail("--save was not used with $dst cwd=".
+ $lei->rel2abs('.'));
$self->{-cfg} //= PublicInbox::Config::git_config_dump($f);
$self->{'-f'} = $f;
} else { # new saved search "lei q --save"
diff --git a/t/lei-q-save.t b/t/lei-q-save.t
index 6389825f..a8eda41e 100644
--- a/t/lei-q-save.t
+++ b/t/lei-q-save.t
@@ -29,6 +29,8 @@ test_lei(sub {
lei_ok [qw(import -q -F eml -)], undef, { 0 => \$in, %$lei_opt };
opendir my $dh, '.' or xbail "opendir .: $!";
lei_ok qw(up -q md -C), $home;
+ lei_ok qw(up -q . -C), "$home/md";
+ lei_ok qw(up -q), "/$home/md";
chdir($dh) or xbail "fchdir . $!";
my %after = map { $_ => 1 } glob("$home/md/cur/*");
is(delete $after{(keys(%before))[0]}, 1, 'original message kept');
@@ -51,5 +53,7 @@ test_lei(sub {
lei_ok [qw(import -q -F eml -)], undef, { 0 => \$in, %$lei_opt };
lei_ok([qw(up mbcl2)], undef, { -C => $home, %$lei_opt });
ok(-s "$home/mbcl2" > $size, 'size increased after up');
+
+ ok(!lei(qw(up -q), $home), 'up fails w/o --save');
});
done_testing;
^ permalink raw reply related [relevance 63%]
* [PATCH] lei q: fix MUA spawn after reading query from stdin
2021-04-17 15:53 71% lei q: --stdin confuses --mua Kyle Meyer
2021-04-17 16:04 71% ` Kyle Meyer
@ 2021-04-17 19:00 60% ` Eric Wong
2021-04-17 20:13 70% ` Kyle Meyer
1 sibling, 1 reply; 200+ results
From: Eric Wong @ 2021-04-17 19:00 UTC (permalink / raw)
To: Kyle Meyer; +Cc: meta
Kyle Meyer <kyle@kyleam.com> wrote:
> I'm not a mutt user, but I think that's because mutt sees that stdin
> isn't attached to a tty. I haven't tried anything on my end yet, but
> perhaps there's a clean way to make --stdin and --mua [*] work together.
Thanks for the report. Yes, that's correct, fortunately we can
reasonably expect stdout to be a terminal and the patch below
should fix it.
> [*] The only other --mua value I checked was 'mail' (and that shows a
> similar issue), but I'm guessing other MUAs don't not work well
> with --stdin either.
Btw, since you seem to be a gnus user; does/can gnus work
via --mua= ?
----------8<---------
Subject: [PATCH] lei q: fix MUA spawn after reading query from stdin
Since "lei q" may read queries from stdin, we must reconnect a
known terminal before spawning terminal MUAs. Attempt to use
stdout as stdin for this purpose, since terminal MUAs tend to
expect stdout to be a terminal.
Reported-By: Kyle Meyer <kyle@kyleam.com>
Link: https://public-inbox.org/meta/87v98klxg3.fsf@kyleam.com/
---
lib/PublicInbox/LEI.pm | 10 ++++++++--
script/lei | 12 ++++++------
2 files changed, 14 insertions(+), 8 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index ebd0f154..f223b3de 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -840,11 +840,17 @@ sub start_mua {
@cmd = map { $_ eq '%f' ? ($replaced = $mfolder) : $_ } @cmd;
}
push @cmd, $mfolder unless defined($replaced);
- if (my $sock = $self->{sock}) { # lei(1) client process runs it
- send($sock, exec_buf(\@cmd, {}), MSG_EOR);
+ if ($self->{sock}) { # lei(1) client process runs it
+ # restore terminal: echo $query | lei q -stdin --mua=...
+ my $io = [];
+ $io->[0] = $self->{1} if $self->{opt}->{stdin} && -t $self->{1};
+ send_exec_cmd($self, $io, \@cmd, {});
} elsif ($self->{oneshot}) {
my $pid = fork // die "fork: $!";
if ($pid > 0) { # original process
+ if ($self->{opt}->{stdin} && -t STDOUT) {
+ open STDIN, '+<&', \*STDOUT or die "dup2: $!";
+ }
exec(@cmd);
warn "exec @cmd: $!\n";
POSIX::_exit(1);
diff --git a/script/lei b/script/lei
index 76217ab9..56e9d299 100755
--- a/script/lei
+++ b/script/lei
@@ -33,6 +33,9 @@ my $exec_cmd = sub {
push @rdr, shift(@old), $newfh;
}
my $do_exec = sub {
+ while (my ($io, $newfh) = splice(@rdr, 0, 2)) {
+ open $io, '+<&', $newfh or die "open +<&=: $!";
+ }
my %env = map { split(/=/, $_, 2) } splice(@argv, $argc);
@ENV{keys %env} = values %env;
exec(@argv);
@@ -42,20 +45,17 @@ my $exec_cmd = sub {
$SIG{CHLD} = $sigchld;
my $pid = fork // die "fork: $!";
if ($pid == 0) {
- while (my ($io, $newfh) = splice(@rdr, 0, 2)) {
- open $io, '+<&', $newfh or die "open +<&=: $!";
- }
- $do_exec->() if scalar(@$fds); # git-credential, pager
+ $do_exec->() if $fds->[1]; # git-credential, pager
# parent backgrounds on MUA
POSIX::setsid() > 0 or die "setsid: $!";
@parent = ($parent);
return; # continue $recv_cmd in background
}
- if (scalar(@$fds)) {
+ if ($fds->[1]) {
$pids{$pid} = undef;
} else {
- $do_exec->(); # MUA reuses all FDs
+ $do_exec->(); # MUA reuses stdout
}
};
^ permalink raw reply related [relevance 60%]
* Re: lei q: --stdin confuses --mua
2021-04-17 15:53 71% lei q: --stdin confuses --mua Kyle Meyer
@ 2021-04-17 16:04 71% ` Kyle Meyer
2021-04-17 19:00 60% ` [PATCH] lei q: fix MUA spawn after reading query from stdin Eric Wong
1 sibling, 0 replies; 200+ results
From: Kyle Meyer @ 2021-04-17 16:04 UTC (permalink / raw)
To: meta
Kyle Meyer writes:
> [*] The only other --mua value I checked was 'mail' (and that shows a
> similar issue), but I'm guessing other MUAs don't not work well
guh, that's a confusing typo: s/don't not/don't/
> with --stdin either.
^ permalink raw reply [relevance 71%]
* lei q: --stdin confuses --mua
@ 2021-04-17 15:53 71% Kyle Meyer
2021-04-17 16:04 71% ` Kyle Meyer
2021-04-17 19:00 60% ` [PATCH] lei q: fix MUA spawn after reading query from stdin Eric Wong
0 siblings, 2 replies; 200+ results
From: Kyle Meyer @ 2021-04-17 15:53 UTC (permalink / raw)
To: meta
I used `lei p2p' to track down an associated thread recently. It worked
nicely, though I noticed that I wasn't able to jump straight into mutt
via --mua.
Here's an example using public-inbox.git. Feeding the p2q output on
stdin, I'm told that no recipients were specified, and the command exits
rather than dropping me into mutt's interface.
$ lei p2q 8ab43c1c27c725a8ef9307f5dba3e565169d48ca | \
lei q -q -tt - -o mboxcl2:/tmp/t.mbox --mua=mutt
No recipients were specified.
I'm not a mutt user, but I think that's because mutt sees that stdin
isn't attached to a tty. I haven't tried anything on my end yet, but
perhaps there's a clean way to make --stdin and --mua [*] work together.
Thoughts?
[*] The only other --mua value I checked was 'mail' (and that shows a
similar issue), but I'm guessing other MUAs don't not work well
with --stdin either.
^ permalink raw reply [relevance 71%]
* [PATCH] lei up: fix canonicalization of Maildirs
@ 2021-04-17 10:24 62% Eric Wong
2021-04-17 19:00 63% ` [PATCH 2/] lei up: further improve Maildir canonicalization Eric Wong
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2021-04-17 10:24 UTC (permalink / raw)
To: meta
We always represent --output destination directories with a
trailing slash to disambiguate directories from mbox filenames.
Therefore, we must use the trailing slash when mapping the
destination beck from the lei/saved-search/* directory.
"lei up" now relies exclusively on the users --output pathname
or URL for updates. This ought to be less confusing since
pathnames in ~/.local/store/lei/saved-searches aren't ideal.
---
lib/PublicInbox/LeiSavedSearch.pm | 13 ++++++++-----
t/lei-q-save.t | 4 +++-
2 files changed, 11 insertions(+), 6 deletions(-)
diff --git a/lib/PublicInbox/LeiSavedSearch.pm b/lib/PublicInbox/LeiSavedSearch.pm
index 932b2aa4..0f632d93 100644
--- a/lib/PublicInbox/LeiSavedSearch.pm
+++ b/lib/PublicInbox/LeiSavedSearch.pm
@@ -26,18 +26,21 @@ sub lss_dir_for ($$) {
} else { # basename
@n = ($$dstref =~ m{([\w\-\.]+)/*\z});
$$dstref = $lei->rel2abs($$dstref);
+ $$dstref .= '/' if -d $$dstref;
}
push @n, sha256_hex($$dstref);
$lei->share_path . '/saved-searches/' . join('-', @n);
}
sub new {
- my ($cls, $lei, $dir) = @_;
+ my ($cls, $lei, $dst) = @_;
my $self = bless { ale => $lei->ale }, $cls;
- if (defined $dir) { # updating existing saved search via "lei up"
- my $f = "$dir/lei.saved-search";
- ((-f $f && -r _) || output2lssdir($self, $lei, \$dir, \$f)) or
- return $lei->fail("$f non-existent or unreadable");
+ my $dir;
+ if (defined $dst) { # updating existing saved search via "lei up"
+ my $f;
+ $dir = $dst;
+ output2lssdir($self, $lei, \$dir, \$f) or
+ return $lei->fail("--save was not used with $dst");
$self->{-cfg} //= PublicInbox::Config::git_config_dump($f);
$self->{'-f'} = $f;
} else { # new saved search "lei q --save"
diff --git a/t/lei-q-save.t b/t/lei-q-save.t
index d43f508b..6389825f 100644
--- a/t/lei-q-save.t
+++ b/t/lei-q-save.t
@@ -27,7 +27,9 @@ test_lei(sub {
# ensure "lei up" works, since it compliments "lei q --save"
$in = $doc2->as_string;
lei_ok [qw(import -q -F eml -)], undef, { 0 => \$in, %$lei_opt };
- lei_ok qw(up -q), $s[0];
+ opendir my $dh, '.' or xbail "opendir .: $!";
+ lei_ok qw(up -q md -C), $home;
+ chdir($dh) or xbail "fchdir . $!";
my %after = map { $_ => 1 } glob("$home/md/cur/*");
is(delete $after{(keys(%before))[0]}, 1, 'original message kept');
is(scalar(keys %after), 1, 'one new message added');
^ permalink raw reply related [relevance 62%]
* [PATCH 8/9] lei q --save: avoid lei.q.format
2021-04-16 23:10 71% [PATCH 0/9] lei saved search usability improvements Eric Wong
` (4 preceding siblings ...)
2021-04-16 23:10 50% ` [PATCH 7/9] lei up: support output destination as arg Eric Wong
@ 2021-04-16 23:10 90% ` Eric Wong
2021-04-16 23:10 61% ` [PATCH 9/9] lei q --save: clobber config file on repeats Eric Wong
6 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-16 23:10 UTC (permalink / raw)
To: meta
It is redundant since we stuff everything into the lei.q.output
config key.
---
lib/PublicInbox/LeiSavedSearch.pm | 2 --
lib/PublicInbox/LeiUp.pm | 1 -
2 files changed, 3 deletions(-)
diff --git a/lib/PublicInbox/LeiSavedSearch.pm b/lib/PublicInbox/LeiSavedSearch.pm
index 93b1b23a..a8bf470b 100644
--- a/lib/PublicInbox/LeiSavedSearch.pm
+++ b/lib/PublicInbox/LeiSavedSearch.pm
@@ -51,8 +51,6 @@ sub new {
} else {
cfg_set($self, 'lei.q', $q);
}
- my $fmt = $lei->{opt}->{'format'};
- cfg_set($self, 'lei.q.format', $fmt) if defined $fmt;
$dst = "$lei->{ovv}->{fmt}:$dst" if $dst !~ m!\Aimaps?://!i;
cfg_set($self, 'lei.q.output', $dst);
for my $k (qw(only include exclude)) {
diff --git a/lib/PublicInbox/LeiUp.pm b/lib/PublicInbox/LeiUp.pm
index 7ddb1dd0..9fe4901b 100644
--- a/lib/PublicInbox/LeiUp.pm
+++ b/lib/PublicInbox/LeiUp.pm
@@ -24,7 +24,6 @@ sub lei_up {
}
$lei->{opt}->{output} = $lss->{-cfg}->{'lei.q.output'} //
return $lei->fail("lei.q.output unset in $lss->{-f}");
- $lei->{opt}->{'format'} //= $lss->{-cfg}->{'lei.q.format'}; # optional
my $to_avref = $lss->{-cfg}->can('_array');
for my $k (qw(only include exclude)) {
^ permalink raw reply related [relevance 90%]
* [PATCH 6/9] lei: fix rel2abs
2021-04-16 23:10 71% [PATCH 0/9] lei saved search usability improvements Eric Wong
` (2 preceding siblings ...)
2021-04-16 23:10 69% ` [PATCH 3/9] lei: saved searches keyed only by path/URL and format Eric Wong
@ 2021-04-16 23:10 71% ` Eric Wong
2021-04-16 23:10 50% ` [PATCH 7/9] lei up: support output destination as arg Eric Wong
` (2 subsequent siblings)
6 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-16 23:10 UTC (permalink / raw)
To: meta
We don't want pathnames with "GLOB(0xADD12355)" in them.
---
lib/PublicInbox/LEI.pm | 13 +++++++------
1 file changed, 7 insertions(+), 6 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 52b588a2..ebd0f154 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -68,18 +68,19 @@ sub rel2abs ($$) {
my ($self, $p) = @_;
return $p if index($p, '/') == 0; # already absolute
my $pwd = $self->{env}->{PWD};
+ my $cwd;
if (defined $pwd) {
- my $cwd = $self->{3} // getcwd() // die "getcwd(PWD=$pwd): $!";
+ my $xcwd = $self->{3} //
+ ($cwd = getcwd() // die "getcwd(PWD=$pwd): $!");
if (my @st_pwd = stat($pwd)) {
- my @st_cwd = stat($cwd) or die "stat($cwd): $!";
+ my @st_cwd = stat($xcwd) or die "stat($xcwd): $!";
"@st_pwd[1,0]" eq "@st_cwd[1,0]" or
- $self->{env}->{PWD} = $pwd = $cwd;
+ $self->{env}->{PWD} = $pwd = undef;
} else { # PWD was invalid
- delete $self->{env}->{PWD};
- undef $pwd;
+ $self->{env}->{PWD} = $pwd = undef;
}
}
- $pwd //= $self->{env}->{PWD} = getcwd() // die "getcwd(PWD=$pwd): $!";
+ $pwd //= $self->{env}->{PWD} = $cwd // getcwd() // die "getcwd: $!";
File::Spec->rel2abs($p, $pwd);
}
^ permalink raw reply related [relevance 71%]
* [PATCH 2/9] lei: expose share_path as a method
2021-04-16 23:10 71% [PATCH 0/9] lei saved search usability improvements Eric Wong
2021-04-16 23:10 53% ` [PATCH 1/9] lei q: --save preserves relative time queries Eric Wong
@ 2021-04-16 23:10 69% ` Eric Wong
2021-04-16 23:10 69% ` [PATCH 3/9] lei: saved searches keyed only by path/URL and format Eric Wong
` (4 subsequent siblings)
6 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-16 23:10 UTC (permalink / raw)
To: meta
Since saved-searches aren't a part of lei/store, nor
could it be considered cache data... (or can it? it
is discardable, after all).
---
lib/PublicInbox/LEI.pm | 6 ++++--
lib/PublicInbox/LeiSavedSearch.pm | 2 +-
2 files changed, 5 insertions(+), 3 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 4b87c104..52b588a2 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -83,13 +83,15 @@ sub rel2abs ($$) {
File::Spec->rel2abs($p, $pwd);
}
-sub store_path ($) {
+sub share_path ($) { # $HOME/.local/share/lei/$FOO
my ($self) = @_;
rel2abs($self, ($self->{env}->{XDG_DATA_HOME} //
($self->{env}->{HOME} // '/nonexistent').'/.local/share')
- .'/lei/store');
+ .'/lei');
}
+sub store_path ($) { share_path($_[0]) . '/store' }
+
sub _config_path ($) {
my ($self) = @_;
rel2abs($self, ($self->{env}->{XDG_CONFIG_HOME} //
diff --git a/lib/PublicInbox/LeiSavedSearch.pm b/lib/PublicInbox/LeiSavedSearch.pm
index e79cf76a..fe8301d6 100644
--- a/lib/PublicInbox/LeiSavedSearch.pm
+++ b/lib/PublicInbox/LeiSavedSearch.pm
@@ -23,7 +23,7 @@ sub new {
return $lei->fail("$f non-existent or unreadable");
$self->{-cfg} = PublicInbox::Config::git_config_dump($f);
} else { # new saved search "lei q --save"
- my $saved_dir = $lei->store_path . '/../saved-searches/';
+ my $saved_dir = $lei->share_path . '/saved-searches/';
my (@name) = ($lei->{ovv}->{dst} =~ m{([\w\-\.]+)/*\z});
my $q = $lei->{mset_opt}->{q_raw} // die 'BUG: {q_raw} missing';
my $q_raw_str = ref($q) ? "@$q" : $q;
^ permalink raw reply related [relevance 69%]
* [PATCH 3/9] lei: saved searches keyed only by path/URL and format
2021-04-16 23:10 71% [PATCH 0/9] lei saved search usability improvements Eric Wong
2021-04-16 23:10 53% ` [PATCH 1/9] lei q: --save preserves relative time queries Eric Wong
2021-04-16 23:10 69% ` [PATCH 2/9] lei: expose share_path as a method Eric Wong
@ 2021-04-16 23:10 69% ` Eric Wong
2021-04-16 23:10 71% ` [PATCH 6/9] lei: fix rel2abs Eric Wong
` (3 subsequent siblings)
6 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-16 23:10 UTC (permalink / raw)
To: meta
We want users to be able to edit and refine the query over
time while using the same output destination.
---
lib/PublicInbox/LeiSavedSearch.pm | 13 ++++++-------
1 file changed, 6 insertions(+), 7 deletions(-)
diff --git a/lib/PublicInbox/LeiSavedSearch.pm b/lib/PublicInbox/LeiSavedSearch.pm
index fe8301d6..ebc63091 100644
--- a/lib/PublicInbox/LeiSavedSearch.pm
+++ b/lib/PublicInbox/LeiSavedSearch.pm
@@ -10,9 +10,8 @@ use PublicInbox::OverIdx;
use PublicInbox::LeiSearch;
use PublicInbox::Config;
use PublicInbox::Spawn qw(run_die);
-use PublicInbox::ContentHash qw(content_hash git_sha);
-use PublicInbox::Eml;
-use PublicInbox::Hval qw(to_filename);
+use PublicInbox::ContentHash qw(git_sha);
+use Digest::SHA qw(sha256_hex);
sub new {
my ($cls, $lei, $dir) = @_;
@@ -24,11 +23,11 @@ sub new {
$self->{-cfg} = PublicInbox::Config::git_config_dump($f);
} else { # new saved search "lei q --save"
my $saved_dir = $lei->share_path . '/saved-searches/';
- my (@name) = ($lei->{ovv}->{dst} =~ m{([\w\-\.]+)/*\z});
+ my (@n) = ($lei->{ovv}->{dst} =~ m{([\w\-\.]+)/*\z});
my $q = $lei->{mset_opt}->{q_raw} // die 'BUG: {q_raw} missing';
- my $q_raw_str = ref($q) ? "@$q" : $q;
- push @name, to_filename($q_raw_str);
- $dir = $saved_dir . join('-', @name);
+ push @n, sha256_hex("$lei->{ovv}->{fmt}\0$lei->{ovv}->{dst}");
+
+ $dir = $saved_dir . join('-', @n);
require File::Path;
File::Path::make_path($dir); # raises on error
$self->{'-f'} = "$dir/lei.saved-search";
^ permalink raw reply related [relevance 69%]
* [PATCH 9/9] lei q --save: clobber config file on repeats
2021-04-16 23:10 71% [PATCH 0/9] lei saved search usability improvements Eric Wong
` (5 preceding siblings ...)
2021-04-16 23:10 90% ` [PATCH 8/9] lei q --save: avoid lei.q.format Eric Wong
@ 2021-04-16 23:10 61% ` Eric Wong
6 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-16 23:10 UTC (permalink / raw)
To: meta
A user may wish to clobber/refine existing search parameters
by issuing "lei q --save" again. Support that by overwriting
the lei.saved-search state file entirely.
We continue to preserve over.sqlite3 for deduplication purposes.
This way, we don't get something redundant like:
[lei]
q = term1
q = term2
q = term1
q = term2
q = term3
...whenever a user wants to refine their search. Instead,
we'll just have:
[lei]
q = term1
q = term2
q = term3
On the second go.
---
lib/PublicInbox/Config.pm | 9 +++++++++
lib/PublicInbox/LeiSavedSearch.pm | 10 +++++++++-
lib/PublicInbox/Reply.pm | 10 ++--------
3 files changed, 20 insertions(+), 9 deletions(-)
diff --git a/lib/PublicInbox/Config.pm b/lib/PublicInbox/Config.pm
index 26ac298e..603dad98 100644
--- a/lib/PublicInbox/Config.pm
+++ b/lib/PublicInbox/Config.pm
@@ -559,4 +559,13 @@ sub json {
};
}
+sub squote_maybe ($) {
+ my ($val) = @_;
+ if ($val =~ m{([^\w@\./,\%\+\-])}) {
+ $val =~ s/(['!])/'\\$1'/g; # '!' for csh
+ return "'$val'";
+ }
+ $val;
+}
+
1;
diff --git a/lib/PublicInbox/LeiSavedSearch.pm b/lib/PublicInbox/LeiSavedSearch.pm
index a8bf470b..932b2aa4 100644
--- a/lib/PublicInbox/LeiSavedSearch.pm
+++ b/lib/PublicInbox/LeiSavedSearch.pm
@@ -13,6 +13,8 @@ use PublicInbox::Spawn qw(run_die);
use PublicInbox::ContentHash qw(git_sha);
use Digest::SHA qw(sha256_hex);
+*squote_maybe = \&PublicInbox::Config::squote_maybe;
+
sub lss_dir_for ($$) {
my ($lei, $dstref) = @_;
my @n;
@@ -44,7 +46,13 @@ sub new {
require File::Path;
File::Path::make_path($dir); # raises on error
$self->{-cfg} = {};
- $self->{'-f'} = "$dir/lei.saved-search";
+ my $f = $self->{'-f'} = "$dir/lei.saved-search";
+ open my $fh, '>', $f or return $lei->fail("open $f: $!");
+ my $sq_dst = squote_maybe($dst);
+ print $fh <<EOM or return $lei->fail("print $f: $!");
+; to refresh with new results, run: lei up $sq_dst
+EOM
+ close $fh or return $lei->fail("close $f: $!");
my $q = $lei->{mset_opt}->{q_raw} // die 'BUG: {q_raw} missing';
if (ref $q) {
cfg_set($self, '--add', 'lei.q', $_) for @$q;
diff --git a/lib/PublicInbox/Reply.pm b/lib/PublicInbox/Reply.pm
index 2a1066d2..79dd46a7 100644
--- a/lib/PublicInbox/Reply.pm
+++ b/lib/PublicInbox/Reply.pm
@@ -9,15 +9,9 @@ use URI::Escape qw/uri_escape_utf8/;
use PublicInbox::Hval qw(ascii_html obfuscate_addrs mid_href);
use PublicInbox::Address;
use PublicInbox::MID qw(mid_clean);
+use PublicInbox::Config;
-sub squote_maybe ($) {
- my ($val) = @_;
- if ($val =~ m{([^\w@\./,\%\+\-])}) {
- $val =~ s/(['!])/'\\$1'/g; # '!' for csh
- return "'$val'";
- }
- $val;
-}
+*squote_maybe = \&PublicInbox::Config::squote_maybe;
sub add_addrs {
my ($to, $cc, @addrs) = @_;
^ permalink raw reply related [relevance 61%]
* [PATCH 1/9] lei q: --save preserves relative time queries
2021-04-16 23:10 71% [PATCH 0/9] lei saved search usability improvements Eric Wong
@ 2021-04-16 23:10 53% ` Eric Wong
2021-04-16 23:10 69% ` [PATCH 2/9] lei: expose share_path as a method Eric Wong
` (5 subsequent siblings)
6 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-16 23:10 UTC (permalink / raw)
To: meta
Somebody may want a saved search which consistently asks for
messages within a rolling time period window. In other words,
we want to support using "lei q --save dt:last.week.." and keeps
the "dt:last.week.." relative to whenever "lei up" is run. This
ensures relative date-time specifications get used in the future
rather than converting into an absolute date-time from the
initial "lei q" invocation.
---
lib/PublicInbox/LeiQuery.pm | 2 +-
lib/PublicInbox/LeiSavedSearch.pm | 5 +++--
t/lei-q-save.t | 25 +++++++++++++++++++++----
3 files changed, 25 insertions(+), 7 deletions(-)
diff --git a/lib/PublicInbox/LeiQuery.pm b/lib/PublicInbox/LeiQuery.pm
index 7456f7f9..7ddba4cf 100644
--- a/lib/PublicInbox/LeiQuery.pm
+++ b/lib/PublicInbox/LeiQuery.pm
@@ -143,7 +143,7 @@ no query allowed on command-line with --stdin
PublicInbox::InputPipe::consume($self->{0}, \&qstr_add, $self);
return;
}
- $mset_opt{q_raw} = \@argv;
+ $mset_opt{q_raw} = [ @argv ]; # copy
$mset_opt{qstr} =
$self->{lse}->query_argv_to_string($self->{lse}->git, \@argv);
_start_query($self);
diff --git a/lib/PublicInbox/LeiSavedSearch.pm b/lib/PublicInbox/LeiSavedSearch.pm
index 815008fd..e79cf76a 100644
--- a/lib/PublicInbox/LeiSavedSearch.pm
+++ b/lib/PublicInbox/LeiSavedSearch.pm
@@ -25,12 +25,13 @@ sub new {
} else { # new saved search "lei q --save"
my $saved_dir = $lei->store_path . '/../saved-searches/';
my (@name) = ($lei->{ovv}->{dst} =~ m{([\w\-\.]+)/*\z});
- push @name, to_filename($lei->{mset_opt}->{qstr});
+ my $q = $lei->{mset_opt}->{q_raw} // die 'BUG: {q_raw} missing';
+ my $q_raw_str = ref($q) ? "@$q" : $q;
+ push @name, to_filename($q_raw_str);
$dir = $saved_dir . join('-', @name);
require File::Path;
File::Path::make_path($dir); # raises on error
$self->{'-f'} = "$dir/lei.saved-search";
- my $q = $lei->{mset_opt}->{q_raw};
if (ref $q) {
cfg_set($self, '--add', 'lei.q', $_) for @$q;
} else {
diff --git a/t/lei-q-save.t b/t/lei-q-save.t
index a6d579cf..6cfac20b 100644
--- a/t/lei-q-save.t
+++ b/t/lei-q-save.t
@@ -2,24 +2,41 @@
# Copyright (C) 2021 all contributors <meta@public-inbox.org>
# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
use strict; use v5.10.1; use PublicInbox::TestCommon;
+use PublicInbox::Smsg;
my $doc1 = eml_load('t/plack-qp.eml');
+$doc1->header_set('Date', PublicInbox::Smsg::date({ds => time - (86400 * 5)}));
my $doc2 = eml_load('t/utf8.eml');
+$doc2->header_set('Date', PublicInbox::Smsg::date({ds => time - (86400 * 4)}));
+
test_lei(sub {
my $home = $ENV{HOME};
- lei_ok qw(import -q t/plack-qp.eml);
- lei_ok qw(q -q --save z:0..), '-o', "$home/md/";
+ my $in = $doc1->as_string;
+ lei_ok [qw(import -q -F eml -)], undef, { 0 => \$in, %$lei_opt };
+ lei_ok qw(q -q --save z:0.. d:last.week..), '-o', "$home/md/";
my %before = map { $_ => 1 } glob("$home/md/cur/*");
is_deeply(eml_load((keys %before)[0]), $doc1, 'doc1 matches');
my @s = glob("$home/.local/share/lei/saved-searches/md-*");
is(scalar(@s), 1, 'got one saved search');
+ my $cfg = PublicInbox::Config->new("$s[0]/lei.saved-search");
+ is_deeply($cfg->{'lei.q'}, ['z:0..', 'd:last.week..'],
+ 'store relative time, not parsed (absolute) timestamp');
# ensure "lei up" works, since it compliments "lei q --save"
- lei_ok qw(import t/utf8.eml);
- lei_ok qw(up), $s[0];
+ $in = $doc2->as_string;
+ lei_ok [qw(import -q -F eml -)], undef, { 0 => \$in, %$lei_opt };
+ lei_ok qw(up -q), $s[0];
my %after = map { $_ => 1 } glob("$home/md/cur/*");
is(delete $after{(keys(%before))[0]}, 1, 'original message kept');
is(scalar(keys %after), 1, 'one new message added');
is_deeply(eml_load((keys %after)[0]), $doc2, 'doc2 matches');
+
+ # check stdin
+ lei_ok [qw(q --save - -o), "mboxcl2:mbcl2" ],
+ undef, { -C => $home, %$lei_opt, 0 => \'d:last.week..'};
+ @s = glob("$home/.local/share/lei/saved-searches/mbcl2-*");
+ $cfg = PublicInbox::Config->new("$s[0]/lei.saved-search");
+ is_deeply $cfg->{'lei.q'}, 'd:last.week..',
+ 'q --stdin stores relative time';
});
done_testing;
^ permalink raw reply related [relevance 53%]
* [PATCH 7/9] lei up: support output destination as arg
2021-04-16 23:10 71% [PATCH 0/9] lei saved search usability improvements Eric Wong
` (3 preceding siblings ...)
2021-04-16 23:10 71% ` [PATCH 6/9] lei: fix rel2abs Eric Wong
@ 2021-04-16 23:10 50% ` Eric Wong
2021-04-16 23:10 90% ` [PATCH 8/9] lei q --save: avoid lei.q.format Eric Wong
2021-04-16 23:10 61% ` [PATCH 9/9] lei q --save: clobber config file on repeats Eric Wong
6 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-16 23:10 UTC (permalink / raw)
To: meta
Specifying a directory in ~/.local/share/lei/saved-searches/
is painful, so support (and start encouraging) the use of
the output.
---
lib/PublicInbox/LeiSavedSearch.pm | 55 ++++++++++++++++++++++++-------
lib/PublicInbox/LeiUp.pm | 4 +--
t/lei-q-save.t | 11 +++++++
3 files changed, 57 insertions(+), 13 deletions(-)
diff --git a/lib/PublicInbox/LeiSavedSearch.pm b/lib/PublicInbox/LeiSavedSearch.pm
index ebc63091..93b1b23a 100644
--- a/lib/PublicInbox/LeiSavedSearch.pm
+++ b/lib/PublicInbox/LeiSavedSearch.pm
@@ -13,24 +13,39 @@ use PublicInbox::Spawn qw(run_die);
use PublicInbox::ContentHash qw(git_sha);
use Digest::SHA qw(sha256_hex);
+sub lss_dir_for ($$) {
+ my ($lei, $dstref) = @_;
+ my @n;
+ if ($$dstref =~ m,\Aimaps?://,i) { # already canonicalized
+ require PublicInbox::URIimap;
+ my $uri = PublicInbox::URIimap->new($$dstref)->canonical;
+ $$dstref = $$uri;
+ @n = ($uri->mailbox);
+ } else { # basename
+ @n = ($$dstref =~ m{([\w\-\.]+)/*\z});
+ $$dstref = $lei->rel2abs($$dstref);
+ }
+ push @n, sha256_hex($$dstref);
+ $lei->share_path . '/saved-searches/' . join('-', @n);
+}
+
sub new {
my ($cls, $lei, $dir) = @_;
- my $self = bless { ale => $lei->ale, -cfg => {} }, $cls;
+ my $self = bless { ale => $lei->ale }, $cls;
if (defined $dir) { # updating existing saved search via "lei up"
- my $f = $self->{'-f'} = "$dir/lei.saved-search";
- -f $f && -r _ or
+ my $f = "$dir/lei.saved-search";
+ ((-f $f && -r _) || output2lssdir($self, $lei, \$dir, \$f)) or
return $lei->fail("$f non-existent or unreadable");
- $self->{-cfg} = PublicInbox::Config::git_config_dump($f);
+ $self->{-cfg} //= PublicInbox::Config::git_config_dump($f);
+ $self->{'-f'} = $f;
} else { # new saved search "lei q --save"
- my $saved_dir = $lei->share_path . '/saved-searches/';
- my (@n) = ($lei->{ovv}->{dst} =~ m{([\w\-\.]+)/*\z});
- my $q = $lei->{mset_opt}->{q_raw} // die 'BUG: {q_raw} missing';
- push @n, sha256_hex("$lei->{ovv}->{fmt}\0$lei->{ovv}->{dst}");
-
- $dir = $saved_dir . join('-', @n);
+ my $dst = $lei->{ovv}->{dst};
+ $dir = lss_dir_for($lei, \$dst);
require File::Path;
File::Path::make_path($dir); # raises on error
+ $self->{-cfg} = {};
$self->{'-f'} = "$dir/lei.saved-search";
+ my $q = $lei->{mset_opt}->{q_raw} // die 'BUG: {q_raw} missing';
if (ref $q) {
cfg_set($self, '--add', 'lei.q', $_) for @$q;
} else {
@@ -38,7 +53,8 @@ sub new {
}
my $fmt = $lei->{opt}->{'format'};
cfg_set($self, 'lei.q.format', $fmt) if defined $fmt;
- cfg_set($self, 'lei.q.output', $lei->{opt}->{output});
+ $dst = "$lei->{ovv}->{fmt}:$dst" if $dst !~ m!\Aimaps?://!i;
+ cfg_set($self, 'lei.q.output', $dst);
for my $k (qw(only include exclude)) {
my $ary = $lei->{opt}->{$k} // next;
for my $x (@$ary) {
@@ -127,6 +143,23 @@ sub mm { undef }
sub altid_map { {} }
sub cloneurl { [] }
+
+# find existing directory containing a `lei.saved-search' file based on
+# $dir_ref which is an output
+sub output2lssdir {
+ my ($self, $lei, $dir_ref, $fn_ref) = @_;
+ my $dst = $$dir_ref; # imap://$MAILBOX, /path/to/maildir, /path/to/mbox
+ my $dir = lss_dir_for($lei, \$dst);
+ my $f = "$dir/lei.saved-search";
+ if (-f $f && -r _) {
+ $self->{-cfg} = PublicInbox::Config::git_config_dump($f);
+ $$dir_ref = $dir;
+ $$fn_ref = $f;
+ return 1;
+ }
+ undef;
+}
+
no warnings 'once';
*nntp_url = \&cloneurl;
*base_url = \&PublicInbox::Inbox::base_url;
diff --git a/lib/PublicInbox/LeiUp.pm b/lib/PublicInbox/LeiUp.pm
index 386a7566..7ddb1dd0 100644
--- a/lib/PublicInbox/LeiUp.pm
+++ b/lib/PublicInbox/LeiUp.pm
@@ -9,9 +9,9 @@ use PublicInbox::LeiSavedSearch;
use PublicInbox::LeiOverview;
sub lei_up {
- my ($lei, $dir) = @_;
+ my ($lei, $out) = @_;
$lei->{lse} = $lei->_lei_store(1)->search;
- my $lss = PublicInbox::LeiSavedSearch->new($lei, $dir) or return;
+ my $lss = PublicInbox::LeiSavedSearch->new($lei, $out) or return;
my $mset_opt = $lei->{mset_opt} = { relevance => -2 };
$mset_opt->{limit} = $lei->{opt}->{limit} // 10000;
my $q = $mset_opt->{q_raw} = $lss->{-cfg}->{'lei.q'} //
diff --git a/t/lei-q-save.t b/t/lei-q-save.t
index 6cfac20b..d43f508b 100644
--- a/t/lei-q-save.t
+++ b/t/lei-q-save.t
@@ -7,6 +7,8 @@ my $doc1 = eml_load('t/plack-qp.eml');
$doc1->header_set('Date', PublicInbox::Smsg::date({ds => time - (86400 * 5)}));
my $doc2 = eml_load('t/utf8.eml');
$doc2->header_set('Date', PublicInbox::Smsg::date({ds => time - (86400 * 4)}));
+my $doc3 = eml_load('t/msg_iter-order.eml');
+$doc3->header_set('Date', PublicInbox::Smsg::date({ds => time - (86400 * 4)}));
test_lei(sub {
my $home = $ENV{HOME};
@@ -38,5 +40,14 @@ test_lei(sub {
$cfg = PublicInbox::Config->new("$s[0]/lei.saved-search");
is_deeply $cfg->{'lei.q'}, 'd:last.week..',
'q --stdin stores relative time';
+ my $size = -s "$home/mbcl2";
+ ok(defined($size) && $size > 0, 'results written');
+ lei_ok([qw(up mbcl2)], undef, { -C => $home, %$lei_opt });
+ is(-s "$home/mbcl2", $size, 'size unchanged on noop up');
+
+ $in = $doc3->as_string;
+ lei_ok [qw(import -q -F eml -)], undef, { 0 => \$in, %$lei_opt };
+ lei_ok([qw(up mbcl2)], undef, { -C => $home, %$lei_opt });
+ ok(-s "$home/mbcl2" > $size, 'size increased after up');
});
done_testing;
^ permalink raw reply related [relevance 50%]
* [PATCH 0/9] lei saved search usability improvements
@ 2021-04-16 23:10 71% Eric Wong
2021-04-16 23:10 53% ` [PATCH 1/9] lei q: --save preserves relative time queries Eric Wong
` (6 more replies)
0 siblings, 7 replies; 200+ results
From: Eric Wong @ 2021-04-16 23:10 UTC (permalink / raw)
To: meta
Found a few bugfixes along the way, but after thinking it over,
I think "lei up /path/to/maildir/or/mbox/or/IMAP-URI" makes the
most sense.
Eric Wong (9):
lei q: --save preserves relative time queries
lei: expose share_path as a method
lei: saved searches keyed only by path/URL and format
lei_to_mail: cast to URIimap object early
test_common: handle '-C' (chdir) spawn option properly
lei: fix rel2abs
lei up: support output destination as arg
lei q --save: avoid lei.q.format
lei q --save: clobber config file on repeats
lib/PublicInbox/Config.pm | 9 ++++
lib/PublicInbox/LEI.pm | 19 +++++----
lib/PublicInbox/LeiQuery.pm | 2 +-
lib/PublicInbox/LeiSavedSearch.pm | 71 ++++++++++++++++++++++++-------
lib/PublicInbox/LeiToMail.pm | 12 +++---
lib/PublicInbox/LeiUp.pm | 5 +--
lib/PublicInbox/Reply.pm | 10 +----
lib/PublicInbox/TestCommon.pm | 7 +++
t/lei-q-save.t | 36 ++++++++++++++--
9 files changed, 126 insertions(+), 45 deletions(-)
^ permalink raw reply [relevance 71%]
* Re: [PATCH 4/5] lei q: start wiring up saved search
2021-04-13 11:25 71% ` Eric Wong
@ 2021-04-13 19:13 71% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-13 19:13 UTC (permalink / raw)
To: meta
Eric Wong <e@80x24.org> wrote:
> Eh, that should probably be "maxuid" since "num" is more
> strongly associated with "NNTP article number". extindex may
> only be easily exposed via IMAP and HTTP (since we Message-IDs
> may conflict with different list trailers).
Pushed with the following squashed in:
diff --git a/lib/PublicInbox/LeiXSearch.pm b/lib/PublicInbox/LeiXSearch.pm
index 7c540c1c..e1538391 100644
--- a/lib/PublicInbox/LeiXSearch.pm
+++ b/lib/PublicInbox/LeiXSearch.pm
@@ -159,7 +159,7 @@ sub query_one_mset { # for --threads and l2m w/o sort
my $fl = $threads > 1 ? 1 : undef;
my $lss = $lei->{dedupe};
$lss = undef unless $lss && $lss->can('cfg_set'); # saved search
- my $maxk = "external.$dir.maxnum";
+ my $maxk = "external.$dir.maxuid";
my $stop_at = $lss ? $lss->{-cfg}->{$maxk} : undef;
if (defined $stop_at) {
die "$maxk=$stop_at has multiple values" if ref $stop_at;
^ permalink raw reply related [relevance 71%]
* Re: [PATCH 4/5] lei q: start wiring up saved search
2021-04-13 10:54 32% ` [PATCH 4/5] lei q: start wiring up saved search Eric Wong
@ 2021-04-13 11:25 71% ` Eric Wong
2021-04-13 19:13 71% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2021-04-13 11:25 UTC (permalink / raw)
To: meta
Eric Wong <e@80x24.org> wrote:
> --- a/lib/PublicInbox/LeiXSearch.pm
> +++ b/lib/PublicInbox/LeiXSearch.pm
> @@ -149,22 +149,38 @@ sub query_one_mset { # for --threads and l2m w/o sort
> local $0 = "$0 query_one_mset";
> my $lei = $self->{lei};
> my ($srch, $over) = ($ibxish->search, $ibxish->over);
> - my $desc = $ibxish->{inboxdir} // $ibxish->{topdir};
> - return warn("$desc not indexed by Xapian\n") unless ($srch && $over);
> - my $mo = { %{$lei->{mset_opt}} };
> + my $dir = $ibxish->{inboxdir} // $ibxish->{topdir};
> + return warn("$dir not indexed by Xapian\n") unless ($srch && $over);
> + my $mo = { %{$lei->{mset_opt}} }; # copy
> my $mset;
> my $each_smsg = $lei->{ovv}->ovv_each_smsg_cb($lei);
> my $can_kw = !!$ibxish->can('msg_keywords');
> my $threads = $lei->{opt}->{threads} // 0;
> my $fl = $threads > 1 ? 1 : undef;
> + my $lss = $lei->{dedupe};
> + $lss = undef unless $lss && $lss->can('cfg_set'); # saved search
> + my $maxk = "external.$dir.maxnum";
Eh, that should probably be "maxuid" since "num" is more
strongly associated with "NNTP article number". extindex may
only be easily exposed via IMAP and HTTP (since we Message-IDs
may conflict with different list trailers).
^ permalink raw reply [relevance 71%]
* [PATCH 5/5] lei: add "lei up" to complement "lei q --save"
2021-04-13 10:54 90% [PATCH 0/5] "lei q --save" + "lei up" Eric Wong
2021-04-13 10:54 32% ` [PATCH 4/5] lei q: start wiring up saved search Eric Wong
@ 2021-04-13 10:54 75% ` Eric Wong
1 sibling, 0 replies; 200+ results
From: Eric Wong @ 2021-04-13 10:54 UTC (permalink / raw)
To: meta
The command isn't finalized, yet, but it's intended to update
an existing saved search.
---
MANIFEST | 1 +
lib/PublicInbox/LEI.pm | 2 ++
lib/PublicInbox/LeiQuery.pm | 2 +-
lib/PublicInbox/LeiSavedSearch.pm | 24 ++++++++++------
lib/PublicInbox/LeiToMail.pm | 12 ++++----
lib/PublicInbox/LeiUp.pm | 46 +++++++++++++++++++++++++++++++
t/lei-q-save.t | 17 ++++++++++--
7 files changed, 88 insertions(+), 16 deletions(-)
create mode 100644 lib/PublicInbox/LeiUp.pm
diff --git a/MANIFEST b/MANIFEST
index 20615abc..1b7d16ee 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -208,6 +208,7 @@ lib/PublicInbox/LeiStoreErr.pm
lib/PublicInbox/LeiSucks.pm
lib/PublicInbox/LeiTag.pm
lib/PublicInbox/LeiToMail.pm
+lib/PublicInbox/LeiUp.pm
lib/PublicInbox/LeiXSearch.pm
lib/PublicInbox/Linkify.pm
lib/PublicInbox/Listener.pm
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 7292d0f2..4b87c104 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -135,6 +135,8 @@ our %CMD = ( # sorted in order of importance/use:
sort|s=s reverse|r offset=i pretty jobs|j=s globoff|g augment|a
import-before! lock=s@ rsyncable alert=s@ mua=s verbose|v+), @c_opt,
opt_dash('limit|n=i', '[0-9]+') ],
+'up' => [ 'SEARCH_TERMS...', 'update saved search',
+ qw(jobs|j=s lock=s@ alert=s@ mua=s verbose|v+), @c_opt ],
'blob' => [ 'OID', 'show a git blob, reconstructing from mail if necessary',
qw(git-dir=s@ cwd! verbose|v+ mail! oid-a|A=s path-a|a=s path-b|b=s),
diff --git a/lib/PublicInbox/LeiQuery.pm b/lib/PublicInbox/LeiQuery.pm
index 8bca1020..7456f7f9 100644
--- a/lib/PublicInbox/LeiQuery.pm
+++ b/lib/PublicInbox/LeiQuery.pm
@@ -12,7 +12,7 @@ sub prep_ext { # externals_each callback
$lxs->prepare_external($loc) unless $exclude->{$loc};
}
-sub _start_query {
+sub _start_query { # used by "lei q" and "lei up"
my ($self) = @_;
PublicInbox::LeiOverview->new($self) or return;
my $opt = $self->{opt};
diff --git a/lib/PublicInbox/LeiSavedSearch.pm b/lib/PublicInbox/LeiSavedSearch.pm
index ab9f393b..815008fd 100644
--- a/lib/PublicInbox/LeiSavedSearch.pm
+++ b/lib/PublicInbox/LeiSavedSearch.pm
@@ -17,18 +17,12 @@ use PublicInbox::Hval qw(to_filename);
sub new {
my ($cls, $lei, $dir) = @_;
my $self = bless { ale => $lei->ale, -cfg => {} }, $cls;
- if (defined $dir) { # updating existing saved search
+ if (defined $dir) { # updating existing saved search via "lei up"
my $f = $self->{'-f'} = "$dir/lei.saved-search";
-f $f && -r _ or
return $lei->fail("$f non-existent or unreadable");
$self->{-cfg} = PublicInbox::Config::git_config_dump($f);
- my $q = $lei->{mset_opt}->{q_raw} = $self->{-cfg}->{'lei.q'} //
- return $lei->fail("lei.q unset in $f");
- my $lse = $lei->{lse} // die 'BUG: {lse} missing';
- $lei->{mset_opt}->{qstr} = ref($q) ?
- $lse->query_argv_to_string($lse->git, $q) :
- $lse->query_approxidate($lse->git, $q);
- } else { # new saved search
+ } else { # new saved search "lei q --save"
my $saved_dir = $lei->store_path . '/../saved-searches/';
my (@name) = ($lei->{ovv}->{dst} =~ m{([\w\-\.]+)/*\z});
push @name, to_filename($lei->{mset_opt}->{qstr});
@@ -42,6 +36,20 @@ sub new {
} else {
cfg_set($self, 'lei.q', $q);
}
+ my $fmt = $lei->{opt}->{'format'};
+ cfg_set($self, 'lei.q.format', $fmt) if defined $fmt;
+ cfg_set($self, 'lei.q.output', $lei->{opt}->{output});
+ for my $k (qw(only include exclude)) {
+ my $ary = $lei->{opt}->{$k} // next;
+ for my $x (@$ary) {
+ cfg_set($self, '--add', "lei.q.$k", $x);
+ }
+ }
+ for my $k (qw(external local remote import-remote
+ import-before threads)) {
+ my $val = $lei->{opt}->{$k} // next;
+ cfg_set($self, "lei.q.$k", $val);
+ }
}
bless $self->{-cfg}, 'PublicInbox::Config';
$self->{lock_path} = "$self->{-f}.flock";
diff --git a/lib/PublicInbox/LeiToMail.pm b/lib/PublicInbox/LeiToMail.pm
index bd2b714a..4ebaf8f3 100644
--- a/lib/PublicInbox/LeiToMail.pm
+++ b/lib/PublicInbox/LeiToMail.pm
@@ -349,11 +349,13 @@ sub new {
die "bad mail --format=$fmt\n";
}
$self->{dst} = $dst;
- my $dd_cls = 'PublicInbox::'.
- ($lei->{opt}->{save} ? 'LeiSavedSearch' : 'LeiDedupe');
- eval "require $dd_cls";
- die "$dd_cls: $@" if $@;
- $lei->{dedupe} = $dd_cls->new($lei);
+ $lei->{dedupe} = $lei->{lss} // do {
+ my $dd_cls = 'PublicInbox::'.
+ ($lei->{opt}->{save} ? 'LeiSavedSearch' : 'LeiDedupe');
+ eval "require $dd_cls";
+ die "$dd_cls: $@" if $@;
+ $dd_cls->new($lei);
+ };
$self;
}
diff --git a/lib/PublicInbox/LeiUp.pm b/lib/PublicInbox/LeiUp.pm
new file mode 100644
index 00000000..386a7566
--- /dev/null
+++ b/lib/PublicInbox/LeiUp.pm
@@ -0,0 +1,46 @@
+# Copyright (C) 2021 all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+
+# "lei up" - updates the result of "lei q --save"
+package PublicInbox::LeiUp;
+use strict;
+use v5.10.1;
+use PublicInbox::LeiSavedSearch;
+use PublicInbox::LeiOverview;
+
+sub lei_up {
+ my ($lei, $dir) = @_;
+ $lei->{lse} = $lei->_lei_store(1)->search;
+ my $lss = PublicInbox::LeiSavedSearch->new($lei, $dir) or return;
+ my $mset_opt = $lei->{mset_opt} = { relevance => -2 };
+ $mset_opt->{limit} = $lei->{opt}->{limit} // 10000;
+ my $q = $mset_opt->{q_raw} = $lss->{-cfg}->{'lei.q'} //
+ return $lei->fail("lei.q unset in $lss->{-f}");
+ my $lse = $lei->{lse} // die 'BUG: {lse} missing';
+ if (ref($q)) {
+ $mset_opt->{qstr} = $lse->query_argv_to_string($lse->git, $q);
+ } else {
+ $lse->query_approxidate($lse->git, $mset_opt->{qstr} = $q);
+ }
+ $lei->{opt}->{output} = $lss->{-cfg}->{'lei.q.output'} //
+ return $lei->fail("lei.q.output unset in $lss->{-f}");
+ $lei->{opt}->{'format'} //= $lss->{-cfg}->{'lei.q.format'}; # optional
+
+ my $to_avref = $lss->{-cfg}->can('_array');
+ for my $k (qw(only include exclude)) {
+ my $v = $lss->{-cfg}->{"lei.q.$k"} // next;
+ $lei->{opt}->{$k} = $to_avref->($v);
+ }
+ for my $k (qw(external local remote
+ import-remote import-before threads)) {
+ my $v = $lss->{-cfg}->{"lei.q.$k"} // next;
+ $lei->{opt}->{$k} = $v;
+ }
+ $lei->{lss} = $lss; # for LeiOverview->new
+ my $lxs = $lei->lxs_prepare or return;
+ $lei->ale->refresh_externals($lxs);
+ $lei->{opt}->{save} = 1;
+ $lei->_start_query;
+}
+
+1;
diff --git a/t/lei-q-save.t b/t/lei-q-save.t
index 56f7cb37..a6d579cf 100644
--- a/t/lei-q-save.t
+++ b/t/lei-q-save.t
@@ -2,11 +2,24 @@
# Copyright (C) 2021 all contributors <meta@public-inbox.org>
# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
use strict; use v5.10.1; use PublicInbox::TestCommon;
+my $doc1 = eml_load('t/plack-qp.eml');
+my $doc2 = eml_load('t/utf8.eml');
test_lei(sub {
my $home = $ENV{HOME};
- lei_ok qw(import t/plack-qp.eml);
- lei_ok qw(q --save z:0..), '-o', "$home/md/";
+ lei_ok qw(import -q t/plack-qp.eml);
+ lei_ok qw(q -q --save z:0..), '-o', "$home/md/";
+ my %before = map { $_ => 1 } glob("$home/md/cur/*");
+ is_deeply(eml_load((keys %before)[0]), $doc1, 'doc1 matches');
+
my @s = glob("$home/.local/share/lei/saved-searches/md-*");
is(scalar(@s), 1, 'got one saved search');
+
+ # ensure "lei up" works, since it compliments "lei q --save"
+ lei_ok qw(import t/utf8.eml);
+ lei_ok qw(up), $s[0];
+ my %after = map { $_ => 1 } glob("$home/md/cur/*");
+ is(delete $after{(keys(%before))[0]}, 1, 'original message kept');
+ is(scalar(keys %after), 1, 'one new message added');
+ is_deeply(eml_load((keys %after)[0]), $doc2, 'doc2 matches');
});
done_testing;
^ permalink raw reply related [relevance 75%]
* [PATCH 0/5] "lei q --save" + "lei up"
@ 2021-04-13 10:54 90% Eric Wong
2021-04-13 10:54 32% ` [PATCH 4/5] lei q: start wiring up saved search Eric Wong
2021-04-13 10:54 75% ` [PATCH 5/5] lei: add "lei up" to complement "lei q --save" Eric Wong
0 siblings, 2 replies; 200+ results
From: Eric Wong @ 2021-04-13 10:54 UTC (permalink / raw)
To: meta
"--save" may become the default when writing to a
pathname or URL. "lei up" will be used to update
the results of "--save".
This only supports local externals at the moment,
remote externals won't be able to avoid excess
traffic easily.
Usability improvements are coming...
Eric Wong (5):
lei_xsearch: use per-external queries when not sorting
lei_dedupe: adjust to prepare for saved searches
lei_query: rearrange internals to capture query early
lei q: start wiring up saved search
lei: add "lei up" to complement "lei q --save"
MANIFEST | 4 +
lib/PublicInbox/LEI.pm | 6 +-
lib/PublicInbox/LeiDedupe.pm | 16 ++--
lib/PublicInbox/LeiQuery.pm | 59 +++++++------
lib/PublicInbox/LeiSavedSearch.pm | 142 ++++++++++++++++++++++++++++++
lib/PublicInbox/LeiToMail.pm | 18 ++--
lib/PublicInbox/LeiUp.pm | 46 ++++++++++
lib/PublicInbox/LeiXSearch.pm | 94 +++++++++++++-------
t/lei-q-save.t | 25 ++++++
t/lei.t | 2 +-
t/lei_dedupe.t | 11 ++-
t/lei_saved_search.t | 10 +++
12 files changed, 356 insertions(+), 77 deletions(-)
create mode 100644 lib/PublicInbox/LeiSavedSearch.pm
create mode 100644 lib/PublicInbox/LeiUp.pm
create mode 100644 t/lei-q-save.t
create mode 100644 t/lei_saved_search.t
^ permalink raw reply [relevance 90%]
* [PATCH 4/5] lei q: start wiring up saved search
2021-04-13 10:54 90% [PATCH 0/5] "lei q --save" + "lei up" Eric Wong
@ 2021-04-13 10:54 32% ` Eric Wong
2021-04-13 11:25 71% ` Eric Wong
2021-04-13 10:54 75% ` [PATCH 5/5] lei: add "lei up" to complement "lei q --save" Eric Wong
1 sibling, 1 reply; 200+ results
From: Eric Wong @ 2021-04-13 10:54 UTC (permalink / raw)
To: meta
This will have a over.sqlite3 for content-based deduplication.
It may exhibit ibxish methods, so serving a read-only (or even
R/W) IMAP or instance or displaying HTML isn't outside the realm
of possibility.
---
MANIFEST | 3 +
lib/PublicInbox/LEI.pm | 4 +-
lib/PublicInbox/LeiQuery.pm | 2 +
lib/PublicInbox/LeiSavedSearch.pm | 134 ++++++++++++++++++++++++++++++
lib/PublicInbox/LeiToMail.pm | 10 ++-
lib/PublicInbox/LeiXSearch.pm | 31 ++++++-
t/lei-q-save.t | 12 +++
t/lei.t | 2 +-
t/lei_saved_search.t | 10 +++
9 files changed, 199 insertions(+), 9 deletions(-)
create mode 100644 lib/PublicInbox/LeiSavedSearch.pm
create mode 100644 t/lei-q-save.t
create mode 100644 t/lei_saved_search.t
diff --git a/MANIFEST b/MANIFEST
index 12247ad2..20615abc 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -201,6 +201,7 @@ lib/PublicInbox/LeiOverview.pm
lib/PublicInbox/LeiP2q.pm
lib/PublicInbox/LeiQuery.pm
lib/PublicInbox/LeiRemote.pm
+lib/PublicInbox/LeiSavedSearch.pm
lib/PublicInbox/LeiSearch.pm
lib/PublicInbox/LeiStore.pm
lib/PublicInbox/LeiStoreErr.pm
@@ -393,12 +394,14 @@ t/lei-mirror.t
t/lei-p2q.t
t/lei-q-kw.t
t/lei-q-remote-import.t
+t/lei-q-save.t
t/lei-q-thread.t
t/lei-tag.t
t/lei.t
t/lei_dedupe.t
t/lei_external.t
t/lei_overview.t
+t/lei_saved_search.t
t/lei_store.t
t/lei_to_mail.t
t/lei_xsearch.t
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 475af8f0..7292d0f2 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -131,7 +131,7 @@ our %CMD = ( # sorted in order of importance/use:
'q' => [ '--stdin|SEARCH_TERMS...', 'search for messages matching terms',
'stdin|', # /|\z/ must be first for lone dash
@lxs_opt,
- qw(save-as=s output|mfolder|o=s format|f=s dedupe|d=s threads|t+
+ qw(save output|mfolder|o=s format|f=s dedupe|d=s threads|t+
sort|s=s reverse|r offset=i pretty jobs|j=s globoff|g augment|a
import-before! lock=s@ rsyncable alert=s@ mua=s verbose|v+), @c_opt,
opt_dash('limit|n=i', '[0-9]+') ],
@@ -249,7 +249,7 @@ my %OPTDESC = (
'torsocks=s' => ['VAL|auto|no|yes',
'whether or not to wrap git and curl commands with torsocks'],
'no-torsocks' => 'alias for --torsocks=no',
-'save-as=s' => ['NAME', 'save a search terms by given name'],
+'save' => "save a search for `lei up'",
'import-remote!' => 'do not memoize remote messages into local store',
'type=s' => [ 'any|mid|git', 'disambiguate type' ],
diff --git a/lib/PublicInbox/LeiQuery.pm b/lib/PublicInbox/LeiQuery.pm
index 224eba69..8bca1020 100644
--- a/lib/PublicInbox/LeiQuery.pm
+++ b/lib/PublicInbox/LeiQuery.pm
@@ -53,6 +53,7 @@ sub qstr_add { # PublicInbox::InputPipe::consume callback for --stdin
my ($self) = @_; # $_[1] = $rbuf
if (defined($_[1])) {
$_[1] eq '' and return eval {
+ $self->{mset_opt}->{q_raw} = $self->{mset_opt}->{qstr};
$self->{lse}->query_approxidate($self->{lse}->git,
$self->{mset_opt}->{qstr});
_start_query($self);
@@ -142,6 +143,7 @@ no query allowed on command-line with --stdin
PublicInbox::InputPipe::consume($self->{0}, \&qstr_add, $self);
return;
}
+ $mset_opt{q_raw} = \@argv;
$mset_opt{qstr} =
$self->{lse}->query_argv_to_string($self->{lse}->git, \@argv);
_start_query($self);
diff --git a/lib/PublicInbox/LeiSavedSearch.pm b/lib/PublicInbox/LeiSavedSearch.pm
new file mode 100644
index 00000000..ab9f393b
--- /dev/null
+++ b/lib/PublicInbox/LeiSavedSearch.pm
@@ -0,0 +1,134 @@
+# Copyright (C) 2021 all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+
+# pretends to be like LeiDedupe and also PublicInbox::Inbox
+package PublicInbox::LeiSavedSearch;
+use strict;
+use v5.10.1;
+use parent qw(PublicInbox::Lock);
+use PublicInbox::OverIdx;
+use PublicInbox::LeiSearch;
+use PublicInbox::Config;
+use PublicInbox::Spawn qw(run_die);
+use PublicInbox::ContentHash qw(content_hash git_sha);
+use PublicInbox::Eml;
+use PublicInbox::Hval qw(to_filename);
+
+sub new {
+ my ($cls, $lei, $dir) = @_;
+ my $self = bless { ale => $lei->ale, -cfg => {} }, $cls;
+ if (defined $dir) { # updating existing saved search
+ my $f = $self->{'-f'} = "$dir/lei.saved-search";
+ -f $f && -r _ or
+ return $lei->fail("$f non-existent or unreadable");
+ $self->{-cfg} = PublicInbox::Config::git_config_dump($f);
+ my $q = $lei->{mset_opt}->{q_raw} = $self->{-cfg}->{'lei.q'} //
+ return $lei->fail("lei.q unset in $f");
+ my $lse = $lei->{lse} // die 'BUG: {lse} missing';
+ $lei->{mset_opt}->{qstr} = ref($q) ?
+ $lse->query_argv_to_string($lse->git, $q) :
+ $lse->query_approxidate($lse->git, $q);
+ } else { # new saved search
+ my $saved_dir = $lei->store_path . '/../saved-searches/';
+ my (@name) = ($lei->{ovv}->{dst} =~ m{([\w\-\.]+)/*\z});
+ push @name, to_filename($lei->{mset_opt}->{qstr});
+ $dir = $saved_dir . join('-', @name);
+ require File::Path;
+ File::Path::make_path($dir); # raises on error
+ $self->{'-f'} = "$dir/lei.saved-search";
+ my $q = $lei->{mset_opt}->{q_raw};
+ if (ref $q) {
+ cfg_set($self, '--add', 'lei.q', $_) for @$q;
+ } else {
+ cfg_set($self, 'lei.q', $q);
+ }
+ }
+ bless $self->{-cfg}, 'PublicInbox::Config';
+ $self->{lock_path} = "$self->{-f}.flock";
+ $self->{-ovf} = "$dir/over.sqlite3";
+ $self;
+}
+
+sub description { $_[0]->{qstr} } # for WWW
+
+sub cfg_set {
+ my ($self, @args) = @_;
+ my $lk = $self->lock_for_scope; # git-config doesn't wait
+ run_die([qw(git config -f), $self->{'-f'}, @args]);
+}
+
+# drop-in for LeiDedupe API
+sub is_dup {
+ my ($self, $eml, $smsg) = @_;
+ my $oidx = $self->{oidx} // die 'BUG: no {oidx}';
+ my $blob = $smsg ? $smsg->{blob} : undef;
+ return 1 if $blob && $oidx->blob_exists($blob);
+ my $lk = $self->lock_for_scope_fast;
+ if (my $xoids = PublicInbox::LeiSearch::xoids_for($self, $eml, 1)) {
+ for my $docid (values %$xoids) {
+ $oidx->add_xref3($docid, -1, $blob, '.');
+ }
+ $oidx->commit_lazy;
+ 1;
+ } else {
+ # n.b. above xoids_for fills out eml->{-lei_fake_mid} if needed
+ unless ($smsg) {
+ $smsg = bless {}, 'PublicInbox::Smsg';
+ $smsg->{bytes} = 0;
+ $smsg->populate($eml);
+ }
+ $oidx->begin_lazy;
+ $smsg->{num} = $oidx->adj_counter('eidx_docid', '+');
+ $smsg->{blob} //= git_sha(1, $eml)->hexdigest;
+ $oidx->add_overview($eml, $smsg);
+ $oidx->add_xref3($smsg->{num}, -1, $smsg->{blob}, '.');
+ $oidx->commit_lazy;
+ undef;
+ }
+}
+
+sub prepare_dedupe {
+ my ($self) = @_;
+ $self->{oidx} //= do {
+ my $creat = !-f $self->{-ovf};
+ my $lk = $self->lock_for_scope; # git-config doesn't wait
+ my $oidx = PublicInbox::OverIdx->new($self->{-ovf});
+ $oidx->{-no_fsync} = 1;
+ $oidx->dbh;
+ if ($creat) {
+ $oidx->{dbh}->do('PRAGMA journal_mode = WAL');
+ $oidx->eidx_prep; # for xref3
+ }
+ $oidx
+ };
+}
+
+sub over { $_[0]->{oidx} } # for xoids_for
+
+sub git { $_[0]->{ale}->git }
+
+sub pause_dedupe {
+ my ($self) = @_;
+ $self->{ale}->git->cleanup;
+ my $oidx = delete($self->{oidx}) // return;
+ $oidx->commit_lazy;
+}
+
+sub mm { undef }
+
+sub altid_map { {} }
+
+sub cloneurl { [] }
+no warnings 'once';
+*nntp_url = \&cloneurl;
+*base_url = \&PublicInbox::Inbox::base_url;
+*smsg_eml = \&PublicInbox::Inbox::smsg_eml;
+*smsg_by_mid = \&PublicInbox::Inbox::smsg_by_mid;
+*msg_by_mid = \&PublicInbox::Inbox::msg_by_mid;
+*modified = \&PublicInbox::Inbox::modified;
+*recent = \&PublicInbox::Inbox::recent;
+*max_git_epoch = *nntp_usable = *msg_by_path = \&mm; # undef
+*isrch = *search = \&mm; # TODO
+*DESTROY = \&pause_dedupe;
+
+1;
diff --git a/lib/PublicInbox/LeiToMail.pm b/lib/PublicInbox/LeiToMail.pm
index 7adbffe7..bd2b714a 100644
--- a/lib/PublicInbox/LeiToMail.pm
+++ b/lib/PublicInbox/LeiToMail.pm
@@ -9,7 +9,6 @@ use parent qw(PublicInbox::IPC);
use PublicInbox::Eml;
use PublicInbox::ProcessPipe;
use PublicInbox::Spawn qw(spawn);
-use PublicInbox::LeiDedupe;
use PublicInbox::PktOp qw(pkt_do);
use Symbol qw(gensym);
use IO::Handle; # ->autoflush
@@ -350,7 +349,11 @@ sub new {
die "bad mail --format=$fmt\n";
}
$self->{dst} = $dst;
- $lei->{dedupe} = PublicInbox::LeiDedupe->new($lei);
+ my $dd_cls = 'PublicInbox::'.
+ ($lei->{opt}->{save} ? 'LeiSavedSearch' : 'LeiDedupe');
+ eval "require $dd_cls";
+ die "$dd_cls: $@" if $@;
+ $lei->{dedupe} = $dd_cls->new($lei);
$self;
}
@@ -368,6 +371,7 @@ sub _pre_augment_maildir {
sub _do_augment_maildir {
my ($self, $lei) = @_;
+ return if defined($lei->{opt}->{save});
my $dst = $lei->{ovv}->{dst};
my $lse = $lei->{opt}->{'import-before'} ? $lei->{lse} : undef;
my $mdr = PublicInbox::MdirReader->new;
@@ -398,6 +402,7 @@ sub _imap_augment_or_delete { # PublicInbox::NetReader::imap_each cb
sub _do_augment_imap {
my ($self, $lei) = @_;
+ return if defined($lei->{opt}->{save});
my $net = $lei->{net};
my $lse = $lei->{opt}->{'import-before'} ? $lei->{lse} : undef;
if ($lei->{opt}->{augment}) {
@@ -468,6 +473,7 @@ sub _do_augment_mbox {
my ($self, $lei) = @_;
return unless $self->{seekable};
my $opt = $lei->{opt};
+ return if defined($opt->{save});
my $out = $lei->{1};
my ($fmt, $dst) = @{$lei->{ovv}}{qw(fmt dst)};
return unless -s $out;
diff --git a/lib/PublicInbox/LeiXSearch.pm b/lib/PublicInbox/LeiXSearch.pm
index 9d367977..7c540c1c 100644
--- a/lib/PublicInbox/LeiXSearch.pm
+++ b/lib/PublicInbox/LeiXSearch.pm
@@ -149,22 +149,38 @@ sub query_one_mset { # for --threads and l2m w/o sort
local $0 = "$0 query_one_mset";
my $lei = $self->{lei};
my ($srch, $over) = ($ibxish->search, $ibxish->over);
- my $desc = $ibxish->{inboxdir} // $ibxish->{topdir};
- return warn("$desc not indexed by Xapian\n") unless ($srch && $over);
- my $mo = { %{$lei->{mset_opt}} };
+ my $dir = $ibxish->{inboxdir} // $ibxish->{topdir};
+ return warn("$dir not indexed by Xapian\n") unless ($srch && $over);
+ my $mo = { %{$lei->{mset_opt}} }; # copy
my $mset;
my $each_smsg = $lei->{ovv}->ovv_each_smsg_cb($lei);
my $can_kw = !!$ibxish->can('msg_keywords');
my $threads = $lei->{opt}->{threads} // 0;
my $fl = $threads > 1 ? 1 : undef;
+ my $lss = $lei->{dedupe};
+ $lss = undef unless $lss && $lss->can('cfg_set'); # saved search
+ my $maxk = "external.$dir.maxnum";
+ my $stop_at = $lss ? $lss->{-cfg}->{$maxk} : undef;
+ if (defined $stop_at) {
+ die "$maxk=$stop_at has multiple values" if ref $stop_at;
+ my @e;
+ local $SIG{__WARN__} = sub { push @e, @_ };
+ $stop_at += 0;
+ return warn("$maxk=$stop_at: @e") if @e;
+ }
+ my $first_ids;
do {
$mset = $srch->mset($mo->{qstr}, $mo);
- mset_progress($lei, $desc, $mset->size,
+ mset_progress($lei, $dir, $mset->size,
$mset->get_matches_estimated);
wait_startq($lei); # wait for keyword updates
my $ids = $srch->mset_to_artnums($mset, $mo);
+ @$ids = grep { $_ > $stop_at } @$ids if defined($stop_at);
my $i = 0;
if ($threads) {
+ # copy $ids if $lss since over->expand_thread
+ # shifts @{$ctx->{ids}}
+ $first_ids = [ @$ids ] if $lss;
my $ctx = { ids => $ids };
my %n2item = map { ($ids->[$i++], $_) } $mset->items;
while ($over->expand_thread($ctx)) {
@@ -183,6 +199,7 @@ sub query_one_mset { # for --threads and l2m w/o sort
@{$ctx->{xids}} = ();
}
} else {
+ $first_ids = $ids;
my @items = $mset->items;
for my $n (@$ids) {
my $mitem = $items[$i++];
@@ -193,6 +210,12 @@ sub query_one_mset { # for --threads and l2m w/o sort
}
}
} while (_mset_more($mset, $mo));
+ if ($lss && scalar(@$first_ids)) {
+ undef $stop_at;
+ my $max = $first_ids->[0];
+ $lss->cfg_set($maxk, $max);
+ undef $lss;
+ }
undef $each_smsg; # may commit
$lei->{ovv}->ovv_atexit_child($lei);
}
diff --git a/t/lei-q-save.t b/t/lei-q-save.t
new file mode 100644
index 00000000..56f7cb37
--- /dev/null
+++ b/t/lei-q-save.t
@@ -0,0 +1,12 @@
+#!perl -w
+# Copyright (C) 2021 all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+use strict; use v5.10.1; use PublicInbox::TestCommon;
+test_lei(sub {
+ my $home = $ENV{HOME};
+ lei_ok qw(import t/plack-qp.eml);
+ lei_ok qw(q --save z:0..), '-o', "$home/md/";
+ my @s = glob("$home/.local/share/lei/saved-searches/md-*");
+ is(scalar(@s), 1, 'got one saved search');
+});
+done_testing;
diff --git a/t/lei.t b/t/lei.t
index 2be9b4e8..6ade2f18 100644
--- a/t/lei.t
+++ b/t/lei.t
@@ -114,7 +114,7 @@ my $test_completion = sub {
%out = map { $_ => 1 } split(/\s+/s, $lei_out);
for my $sw (qw(-f --format -o --output --mfolder --augment -a
--mua --no-local --local --verbose -v
- --save-as --no-remote --remote --torsocks
+ --save --no-remote --remote --torsocks
--reverse -r )) {
ok($out{$sw}, "$sw offered as `lei q' completion");
}
diff --git a/t/lei_saved_search.t b/t/lei_saved_search.t
new file mode 100644
index 00000000..6d26cd2b
--- /dev/null
+++ b/t/lei_saved_search.t
@@ -0,0 +1,10 @@
+#!perl -w
+# Copyright (C) 2021 all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+use strict;
+use v5.10.1;
+use PublicInbox::TestCommon;
+require_mods(qw(DBD::SQLite));
+use_ok 'PublicInbox::LeiSavedSearch';
+
+done_testing;
^ permalink raw reply related [relevance 32%]
* [PATCH] lei blob: quiet "git rev-parse --git-dir" stderr w/o --cwd
@ 2021-04-12 17:32 68% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-12 17:32 UTC (permalink / raw)
To: meta
This seemed to be causing occasional "make check-run" failures
with errors bleeding into other tests.
---
lib/PublicInbox/LeiBlob.pm | 14 ++++++++++----
1 file changed, 10 insertions(+), 4 deletions(-)
diff --git a/lib/PublicInbox/LeiBlob.pm b/lib/PublicInbox/LeiBlob.pm
index ed0754a3..ad885306 100644
--- a/lib/PublicInbox/LeiBlob.pm
+++ b/lib/PublicInbox/LeiBlob.pm
@@ -23,12 +23,18 @@ sub sol_done { # EOF callback for main daemon
$sol->wq_wait_old(\&sol_done_wait, $lei);
}
-sub get_git_dir ($) {
- my ($d) = @_;
+sub get_git_dir ($$) {
+ my ($lei, $d) = @_;
return $d if -d "$d/objects" && -d "$d/refs" && -e "$d/HEAD";
my $cmd = [ qw(git rev-parse --git-dir) ];
- my ($r, $pid) = popen_rd($cmd, {GIT_DIR => undef}, { '-C' => $d });
+ my $opt = { '-C' => $d };
+ if (defined($lei->{opt}->{cwd})) { # --cwd used, report errors
+ $opt->{2} = $lei->{2};
+ } else { # implicit --cwd, quiet errors
+ open $opt->{2}, '>', '/dev/null' or die "open /dev/null: $!";
+ }
+ my ($r, $pid) = popen_rd($cmd, {GIT_DIR => undef}, $opt);
chomp(my $gd = do { local $/; <$r> });
waitpid($pid, 0) == $pid or die "BUG: waitpid @$cmd ($!)";
$? == 0 ? $gd : undef;
@@ -114,7 +120,7 @@ sub lei_blob {
# maybe it's a non-email (code) blob from a coderepo
my $git_dirs = $opt->{'git-dir'} //= [];
if ($opt->{'cwd'} // 1) {
- my $cgd = get_git_dir('.');
+ my $cgd = get_git_dir($lei, '.');
unshift(@$git_dirs, $cgd) if defined $cgd;
}
return $lei->fail('no --git-dir to try') unless @$git_dirs;
^ permalink raw reply related [relevance 68%]
* Re: [PATCH] script/lei: waitpid for git-credential and pager
2021-04-05 8:36 71% ` [PATCH] script/lei: waitpid for git-credential and pager Eric Wong
2021-04-05 8:59 71% ` [RFC] script/lei: don't setsid on MUA spawn Eric Wong
@ 2021-04-05 21:26 71% ` Kyle Meyer
1 sibling, 0 replies; 200+ results
From: Kyle Meyer @ 2021-04-05 21:26 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
Eric Wong writes:
> Odd, I'm not getting the exact same behavior, but I noticed
> it's not getting reaped. Does this fix things for you?
That seems to do the trick. Thank you!
^ permalink raw reply [relevance 71%]
* [PATCH 5/5] lei q: fix auth IMAP --output with remote mboxrd
2021-04-05 10:27 43% ` [PATCH 3/5] lei: maildir: move shard support to MdirReader Eric Wong
@ 2021-04-05 10:27 60% ` Eric Wong
1 sibling, 0 replies; 200+ results
From: Eric Wong @ 2021-04-05 10:27 UTC (permalink / raw)
To: meta
IMAP authentication info is only shared amongst lei2mail workers,
so we must ensure all IMAP writes go through lei2mail workers
even if we don't have to access the mail through git.
This allows us to decouple the latency of the remote mboxrd from
the latency of the IMAP --output at the expense of extra IPC
overhead within our own processes.
---
lib/PublicInbox/LeiOverview.pm | 14 ++++----------
lib/PublicInbox/LeiToMail.pm | 3 ++-
lib/PublicInbox/LeiXSearch.pm | 4 ++--
3 files changed, 8 insertions(+), 13 deletions(-)
diff --git a/lib/PublicInbox/LeiOverview.pm b/lib/PublicInbox/LeiOverview.pm
index cdd9ee04..bfb8b143 100644
--- a/lib/PublicInbox/LeiOverview.pm
+++ b/lib/PublicInbox/LeiOverview.pm
@@ -195,7 +195,7 @@ sub _json_pretty {
}
sub ovv_each_smsg_cb { # runs in wq worker usually
- my ($self, $lei, $ibxish) = @_;
+ my ($self, $lei) = @_;
my ($json, $dedupe);
if (my $pkg = $self->{json}) {
$json = $pkg->new;
@@ -208,17 +208,11 @@ sub ovv_each_smsg_cb { # runs in wq worker usually
$dedupe->prepare_dedupe;
}
$lei->{ovv_buf} = \(my $buf = '') if !$l2m;
- if ($l2m && !$ibxish) { # remote https?:// mboxrd
- my $wcb = $l2m->write_cb($lei);
- sub {
- my ($smsg, undef, $eml) = @_; # no mitem in $_[1]
- $wcb->(undef, $smsg, $eml);
- };
- } elsif ($l2m && $l2m->{-wq_s1}) {
+ if ($l2m) {
sub {
- my ($smsg, $mitem) = @_;
+ my ($smsg, $mitem, $eml) = @_;
$smsg->{pct} = get_pct($mitem) if $mitem;
- $l2m->wq_io_do('write_mail', [], $smsg);
+ $l2m->wq_io_do('write_mail', [], $smsg, $eml);
}
} elsif ($self->{fmt} =~ /\A(concat)?json\z/ && $lei->{opt}->{pretty}) {
my $EOR = ($1//'') eq 'concat' ? "\n}" : "\n},";
diff --git a/lib/PublicInbox/LeiToMail.pm b/lib/PublicInbox/LeiToMail.pm
index 9411313b..70164e40 100644
--- a/lib/PublicInbox/LeiToMail.pm
+++ b/lib/PublicInbox/LeiToMail.pm
@@ -583,7 +583,8 @@ sub poke_dst {
}
sub write_mail { # via ->wq_io_do
- my ($self, $smsg) = @_;
+ my ($self, $smsg, $eml) = @_;
+ return $self->{wcb}->(undef, $smsg, $eml) if $eml;
$self->{lei}->{ale}->git->cat_async($smsg->{blob}, \&git_to_mail,
[$self->{wcb}, $smsg]);
}
diff --git a/lib/PublicInbox/LeiXSearch.pm b/lib/PublicInbox/LeiXSearch.pm
index 2b23e8e9..692d5e54 100644
--- a/lib/PublicInbox/LeiXSearch.pm
+++ b/lib/PublicInbox/LeiXSearch.pm
@@ -153,7 +153,7 @@ sub query_thread_mset { # for --threads
return warn("$desc not indexed by Xapian\n") unless ($srch && $over);
my $mo = { %{$lei->{mset_opt}} };
my $mset;
- my $each_smsg = $lei->{ovv}->ovv_each_smsg_cb($lei, $ibxish);
+ my $each_smsg = $lei->{ovv}->ovv_each_smsg_cb($lei);
my $can_kw = !!$ibxish->can('msg_keywords');
my $fl = $lei->{opt}->{threads} > 1 ? 1 : undef;
do {
@@ -196,7 +196,7 @@ sub query_mset { # non-parallel for non-"--threads" users
for my $loc (locals($self)) {
attach_external($self, $loc);
}
- my $each_smsg = $lei->{ovv}->ovv_each_smsg_cb($lei, $self);
+ my $each_smsg = $lei->{ovv}->ovv_each_smsg_cb($lei);
do {
$mset = $self->mset($mo->{qstr}, $mo);
mset_progress($lei, 'xsearch', $mset->size,
^ permalink raw reply related [relevance 60%]
* [PATCH 3/5] lei: maildir: move shard support to MdirReader
@ 2021-04-05 10:27 43% ` Eric Wong
2021-04-05 10:27 60% ` [PATCH 5/5] lei q: fix auth IMAP --output with remote mboxrd Eric Wong
1 sibling, 0 replies; 200+ results
From: Eric Wong @ 2021-04-05 10:27 UTC (permalink / raw)
To: meta
We'll eventually want lei_input users like "lei import" and
"lei tag" to support parallel reads.
---
lib/PublicInbox/InboxWritable.pm | 4 ++--
lib/PublicInbox/LeiInput.pm | 2 +-
lib/PublicInbox/LeiToMail.pm | 29 +++++++++--------------------
lib/PublicInbox/MdirReader.pm | 25 +++++++++++++++++++++----
t/lei-convert.t | 2 +-
t/lei_to_mail.t | 8 ++++----
6 files changed, 38 insertions(+), 32 deletions(-)
diff --git a/lib/PublicInbox/InboxWritable.pm b/lib/PublicInbox/InboxWritable.pm
index eeebc485..45d8cdc7 100644
--- a/lib/PublicInbox/InboxWritable.pm
+++ b/lib/PublicInbox/InboxWritable.pm
@@ -154,8 +154,8 @@ sub import_maildir {
my $im = $self->importer(1);
my @self = $self->filter($im) ? ($self) : ();
require PublicInbox::MdirReader;
- PublicInbox::MdirReader::maildir_each_file(\&_each_maildir_fn,
- $im, @self);
+ PublicInbox::MdirReader->new->maildir_each_file(\&_each_maildir_fn,
+ $im, @self);
$im->done;
}
diff --git a/lib/PublicInbox/LeiInput.pm b/lib/PublicInbox/LeiInput.pm
index 40d71f9e..e416d3ed 100644
--- a/lib/PublicInbox/LeiInput.pm
+++ b/lib/PublicInbox/LeiInput.pm
@@ -88,7 +88,7 @@ sub input_path_url {
return $lei->fail(<<EOM) if $ifmt && $ifmt ne 'maildir';
$input appears to a be a maildir, not $ifmt
EOM
- PublicInbox::MdirReader::maildir_each_eml($input,
+ PublicInbox::MdirReader->new->maildir_each_eml($input,
$self->can('input_maildir_cb'),
$self, @args);
} else {
diff --git a/lib/PublicInbox/LeiToMail.pm b/lib/PublicInbox/LeiToMail.pm
index 76a11b0e..2e736070 100644
--- a/lib/PublicInbox/LeiToMail.pm
+++ b/lib/PublicInbox/LeiToMail.pm
@@ -14,7 +14,6 @@ use PublicInbox::PktOp qw(pkt_do);
use Symbol qw(gensym);
use IO::Handle; # ->autoflush
use Fcntl qw(SEEK_SET SEEK_END O_CREAT O_EXCL O_WRONLY);
-use Digest::SHA qw(sha256_hex);
my %kw2char = ( # Maildir characters
draft => 'D',
@@ -234,17 +233,9 @@ sub update_kw_maybe ($$$$) {
}
}
-sub _augment_or_unlink { # maildir_each_eml cb
- my ($f, $kw, $eml, $lei, $lse, $mod, $shard, $unlink) = @_;
- if ($mod) {
- # can't get dirent.d_ino w/ pure Perl readdir, so we extract
- # the OID if it looks like one instead of doing stat(2)
- my $hex = $f =~ m!\b([a-f0-9]{40,})[^/]*\z! ?
- $1 : sha256_hex($f);
- my $recno = hex(substr($hex, 0, 8));
- return if ($recno % $mod) != $shard;
- update_kw_maybe($lei, $lse, $eml, $kw);
- }
+sub _md_update { # maildir_each_eml cb
+ my ($f, $kw, $eml, $lei, $lse, $unlink) = @_;
+ update_kw_maybe($lei, $lse, $eml, $kw);
$unlink ? unlink($f) : _augment($eml, $lei);
}
@@ -392,21 +383,19 @@ sub _do_augment_maildir {
my ($self, $lei) = @_;
my $dst = $lei->{ovv}->{dst};
my $lse = $lei->{opt}->{'import-before'} ? $lei->{lse} : undef;
- my ($mod, $shard) = @{$self->{shard_info} // []};
+ my $mdr = PublicInbox::MdirReader->new;
if ($lei->{opt}->{augment}) {
my $dedupe = $lei->{dedupe};
if ($dedupe && $dedupe->prepare_dedupe) {
- PublicInbox::MdirReader::maildir_each_eml($dst,
- \&_augment_or_unlink,
- $lei, $lse, $mod, $shard);
+ $mdr->{shard_info} = $self->{shard_info};
+ $mdr->maildir_each_eml($dst, \&_md_update, $lei, $lse);
$dedupe->pause_dedupe;
}
} elsif ($lse) {
- PublicInbox::MdirReader::maildir_each_eml($dst,
- \&_augment_or_unlink,
- $lei, $lse, $mod, $shard, 1);
+ $mdr->{shard_info} = $self->{shard_info};
+ $mdr->maildir_each_eml($dst, \&_md_update, $lei, $lse, 1);
} else {# clobber existing Maildir
- PublicInbox::MdirReader::maildir_each_file($dst, \&_unlink);
+ $mdr->maildir_each_file($dst, \&_unlink);
}
}
diff --git a/lib/PublicInbox/MdirReader.pm b/lib/PublicInbox/MdirReader.pm
index 1685e4d8..b49c8ceb 100644
--- a/lib/PublicInbox/MdirReader.pm
+++ b/lib/PublicInbox/MdirReader.pm
@@ -8,6 +8,7 @@ package PublicInbox::MdirReader;
use strict;
use v5.10.1;
use PublicInbox::InboxWritable qw(eml_from_path);
+use Digest::SHA qw(sha256_hex);
# returns Maildir flags from a basename ('' for no flags, undef for invalid)
sub maildir_basename_flags {
@@ -24,14 +25,25 @@ sub maildir_path_flags {
$i >= 0 ? maildir_basename_flags(substr($f, $i + 1)) : undef;
}
-sub maildir_each_file ($$;@) {
- my ($dir, $cb, @arg) = @_;
+sub shard_ok ($$$) {
+ my ($bn, $mod, $shard) = @_;
+ # can't get dirent.d_ino w/ pure Perl readdir, so we extract
+ # the OID if it looks like one instead of doing stat(2)
+ my $hex = $bn =~ m!\A([a-f0-9]{40,})! ? $1 : sha256_hex($bn);
+ my $recno = hex(substr($hex, 0, 8));
+ ($recno % $mod) == $shard;
+}
+
+sub maildir_each_file {
+ my ($self, $dir, $cb, @arg) = @_;
$dir .= '/' unless substr($dir, -1) eq '/';
+ my ($mod, $shard) = @{$self->{shard_info} // []};
for my $d (qw(new/ cur/)) {
my $pfx = $dir.$d;
opendir my $dh, $pfx or next;
while (defined(my $bn = readdir($dh))) {
maildir_basename_flags($bn) // next;
+ next if defined($mod) && !shard_ok($bn, $mod, $shard);
$cb->($pfx.$bn, @arg);
}
}
@@ -40,15 +52,17 @@ sub maildir_each_file ($$;@) {
my %c2kw = ('D' => 'draft', F => 'flagged', P => 'forwarded',
R => 'answered', S => 'seen');
-sub maildir_each_eml ($$;@) {
- my ($dir, $cb, @arg) = @_;
+sub maildir_each_eml {
+ my ($self, $dir, $cb, @arg) = @_;
$dir .= '/' unless substr($dir, -1) eq '/';
+ my ($mod, $shard) = @{$self->{shard_info} // []};
my $pfx = $dir . 'new/';
if (opendir(my $dh, $pfx)) {
while (defined(my $bn = readdir($dh))) {
next if substr($bn, 0, 1) eq '.';
my @f = split(/:/, $bn, -1);
next if scalar(@f) != 1;
+ next if defined($mod) && !shard_ok($bn, $mod, $shard);
my $f = $pfx.$bn;
my $eml = eml_from_path($f) or next;
$cb->($f, [], $eml, @arg);
@@ -59,6 +73,7 @@ sub maildir_each_eml ($$;@) {
while (defined(my $bn = readdir($dh))) {
my $fl = maildir_basename_flags($bn) // next;
next if index($fl, 'T') >= 0;
+ next if defined($mod) && !shard_ok($bn, $mod, $shard);
my $f = $pfx.$bn;
my $eml = eml_from_path($f) or next;
my @kw = sort(map { $c2kw{$_} // () } split(//, $fl));
@@ -66,4 +81,6 @@ sub maildir_each_eml ($$;@) {
}
}
+sub new { bless {}, __PACKAGE__ }
+
1;
diff --git a/t/lei-convert.t b/t/lei-convert.t
index dc53b82c..0ea860c8 100644
--- a/t/lei-convert.t
+++ b/t/lei-convert.t
@@ -57,7 +57,7 @@ test_lei({ tmpdir => $tmpdir }, sub {
lei_ok('convert', '-o', "$d/md", "mboxrd:$d/foo.mboxrd");
ok(-d "$d/md", 'Maildir created');
my @md;
- PublicInbox::MdirReader::maildir_each_eml("$d/md", sub {
+ PublicInbox::MdirReader->new->maildir_each_eml("$d/md", sub {
push @md, $_[2];
});
is(scalar(@md), scalar(@mboxrd), 'got expected emails in Maildir') or
diff --git a/t/lei_to_mail.t b/t/lei_to_mail.t
index 75314add..51357257 100644
--- a/t/lei_to_mail.t
+++ b/t/lei_to_mail.t
@@ -253,7 +253,7 @@ SKIP: { # FIFO support
}
{ # Maildir support
- my $each_file = PublicInbox::MdirReader->can('maildir_each_file');
+ my $mdr = PublicInbox::MdirReader->new;
my $md = "$tmpdir/maildir/";
my $wcb = $wcb_get->('maildir', $md);
is(ref($wcb), 'CODE', 'got Maildir callback');
@@ -261,7 +261,7 @@ SKIP: { # FIFO support
$wcb->(\(my $x = $buf), $b4dc0ffee);
my @f;
- $each_file->($md, sub { push @f, shift });
+ $mdr->maildir_each_file($md, sub { push @f, shift });
open my $fh, $f[0] or BAIL_OUT $!;
is(do { local $/; <$fh> }, $buf, 'wrote to Maildir');
@@ -270,7 +270,7 @@ SKIP: { # FIFO support
$wcb->(\($x = $buf."\nx\n"), $deadcafe);
my @x = ();
- $each_file->($md, sub { push @x, shift });
+ $mdr->maildir_each_file($md, sub { push @x, shift });
is(scalar(@x), 1, 'wrote one new file');
ok(!-f $f[0], 'old file clobbered');
open $fh, $x[0] or BAIL_OUT $!;
@@ -281,7 +281,7 @@ SKIP: { # FIFO support
$wcb->(\($x = $buf."\ny\n"), $deadcafe);
$wcb->(\($x = $buf."\ny\n"), $b4dc0ffee); # skipped by dedupe
@f = ();
- $each_file->($md, sub { push @f, shift });
+ $mdr->maildir_each_file($md, sub { push @f, shift });
is(scalar grep(/\A\Q$x[0]\E\z/, @f), 1, 'old file still there');
my @new = grep(!/\A\Q$x[0]\E\z/, @f);
is(scalar @new, 1, '1 new file written (b4dc0ffee skipped)');
^ permalink raw reply related [relevance 43%]
* [RFC] script/lei: don't setsid on MUA spawn
2021-04-05 8:36 71% ` [PATCH] script/lei: waitpid for git-credential and pager Eric Wong
@ 2021-04-05 8:59 71% ` Eric Wong
2021-04-05 21:26 71% ` [PATCH] script/lei: waitpid for git-credential and pager Kyle Meyer
1 sibling, 0 replies; 200+ results
From: Eric Wong @ 2021-04-05 8:59 UTC (permalink / raw)
To: Kyle Meyer; +Cc: meta
Btw, the setsid call doesn't seem necessary, either, the MUA
seems to work fine without it...
------8<------
Subject: [PATCH] script/lei: don't setsid on MUA spawn
---
script/lei | 1 -
1 file changed, 1 deletion(-)
diff --git a/script/lei b/script/lei
index 76217ab9..cd535afb 100755
--- a/script/lei
+++ b/script/lei
@@ -48,7 +48,6 @@ my $exec_cmd = sub {
$do_exec->() if scalar(@$fds); # git-credential, pager
# parent backgrounds on MUA
- POSIX::setsid() > 0 or die "setsid: $!";
@parent = ($parent);
return; # continue $recv_cmd in background
}
^ permalink raw reply related [relevance 71%]
* [PATCH] script/lei: waitpid for git-credential and pager
2021-04-05 4:17 71% ` Kyle Meyer
@ 2021-04-05 8:36 71% ` Eric Wong
2021-04-05 8:59 71% ` [RFC] script/lei: don't setsid on MUA spawn Eric Wong
2021-04-05 21:26 71% ` [PATCH] script/lei: waitpid for git-credential and pager Kyle Meyer
0 siblings, 2 replies; 200+ results
From: Eric Wong @ 2021-04-05 8:36 UTC (permalink / raw)
To: Kyle Meyer; +Cc: meta
Kyle Meyer <kyle@kyleam.com> wrote:
> This seems to break the pager functionality. For example, if I do
>
> lei q -I https://public-inbox.org/meta/ s:blob
>
> I end up in a misbehaving pager (/bin/less, on my system). Navigation
> keys like 'j' and 'k' are instead passed through, and enter returns to
> the prompt.
Odd, I'm not getting the exact same behavior, but I noticed
it's not getting reaped. Does this fix things for you?
-----8<-----
Subject: [PATCH] script/lei: waitpid for git-credential and pager
We need to ensure we reap things we spawn.
---
script/lei | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/script/lei b/script/lei
index 78a7dab9..76217ab9 100755
--- a/script/lei
+++ b/script/lei
@@ -52,7 +52,11 @@ my $exec_cmd = sub {
@parent = ($parent);
return; # continue $recv_cmd in background
}
- $do_exec->() if !scalar(@$fds); # MUA reuses all FDs
+ if (scalar(@$fds)) {
+ $pids{$pid} = undef;
+ } else {
+ $do_exec->(); # MUA reuses all FDs
+ }
};
if ($send_cmd && eval {
^ permalink raw reply related [relevance 71%]
* Re: [PATCH] lei: fix git-credential handling
2021-04-02 9:42 57% [PATCH] lei: fix git-credential handling Eric Wong
@ 2021-04-05 4:17 71% ` Kyle Meyer
2021-04-05 8:36 71% ` [PATCH] script/lei: waitpid for git-credential and pager Eric Wong
0 siblings, 1 reply; 200+ results
From: Kyle Meyer @ 2021-04-05 4:17 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
Eric Wong writes:
> I completely forgot about git-credential prompting when
> making lei background the client process for MUA.
>
> Now it backgrounds itself only for the MUA when no FDs are
> passed, since the MUA is the final command run. Otherwise, it
> relies on FD passing as before.
This seems to break the pager functionality. For example, if I do
lei q -I https://public-inbox.org/meta/ s:blob
I end up in a misbehaving pager (/bin/less, on my system). Navigation
keys like 'j' and 'k' are instead passed through, and enter returns to
the prompt.
^ permalink raw reply [relevance 71%]
* [PATCH 3/5] lei tag: note message mismatches on failure
2021-04-03 10:48 71% [PATCH 0/5] lei/store: fix epoch roll, better errors Eric Wong
@ 2021-04-03 10:48 71% ` Eric Wong
2021-04-03 10:48 55% ` [PATCH 4/5] lei: improve handling of Message-ID-less draft messages Eric Wong
2021-04-03 10:48 43% ` [PATCH 5/5] lei/store: (more) synchronous non-fatal error output Eric Wong
2 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-03 10:48 UTC (permalink / raw)
To: meta
Just exiting with a failure code and no error message is
confusing :x
---
lib/PublicInbox/LeiTag.pm | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/lib/PublicInbox/LeiTag.pm b/lib/PublicInbox/LeiTag.pm
index 06313139..c7a21c87 100644
--- a/lib/PublicInbox/LeiTag.pm
+++ b/lib/PublicInbox/LeiTag.pm
@@ -121,7 +121,8 @@ sub lei_tag { # the "lei tag" method
sub note_missing {
my ($self) = @_;
- $self->{lei}->child_error(1 << 8) if $self->{missing};
+ my $n = $self->{missing} or return;
+ $self->{lei}->child_error(1 << 8, "$n missed messages");
}
sub ipc_atfork_child {
^ permalink raw reply related [relevance 71%]
* [PATCH 4/5] lei: improve handling of Message-ID-less draft messages
2021-04-03 10:48 71% [PATCH 0/5] lei/store: fix epoch roll, better errors Eric Wong
2021-04-03 10:48 71% ` [PATCH 3/5] lei tag: note message mismatches on failure Eric Wong
@ 2021-04-03 10:48 55% ` Eric Wong
2021-04-03 10:48 43% ` [PATCH 5/5] lei/store: (more) synchronous non-fatal error output Eric Wong
2 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-03 10:48 UTC (permalink / raw)
To: meta
We need a stable fallback time for digest2mid in the presence
of messages without Received/Date headers. Furthermore, we
must avoid using uninitialized smsg->{mid} when parsing
References for draft replies.
---
lib/PublicInbox/Import.pm | 6 +++---
lib/PublicInbox/LeiSearch.pm | 2 +-
lib/PublicInbox/OverIdx.pm | 6 ++++--
lib/PublicInbox/Smsg.pm | 2 +-
t/lei-import.t | 21 +++++++++++++++++++++
5 files changed, 30 insertions(+), 7 deletions(-)
diff --git a/lib/PublicInbox/Import.pm b/lib/PublicInbox/Import.pm
index 34738279..46f57e27 100644
--- a/lib/PublicInbox/Import.pm
+++ b/lib/PublicInbox/Import.pm
@@ -510,8 +510,8 @@ sub atfork_child {
}
}
-sub digest2mid ($$) {
- my ($dig, $hdr) = @_;
+sub digest2mid ($$;$) {
+ my ($dig, $hdr, $fallback_time) = @_;
my $b64 = $dig->clone->b64digest;
# Make our own URLs nicer:
# See "Base 64 Encoding with URL and Filename Safe Alphabet" in RFC4648
@@ -520,7 +520,7 @@ sub digest2mid ($$) {
# Add a date prefix to prevent a leading '-' in case that trips
# up some tools (e.g. if a Message-ID were a expected as a
# command-line arg)
- my $dt = msg_datestamp($hdr);
+ my $dt = msg_datestamp($hdr, $fallback_time);
$dt = POSIX::strftime('%Y%m%d%H%M%S', gmtime($dt));
"$dt.$b64" . '@z';
}
diff --git a/lib/PublicInbox/LeiSearch.pm b/lib/PublicInbox/LeiSearch.pm
index 69ba8303..148aa185 100644
--- a/lib/PublicInbox/LeiSearch.pm
+++ b/lib/PublicInbox/LeiSearch.pm
@@ -58,7 +58,7 @@ sub content_key ($) {
qw(Message-ID X-Alt-Message-ID Resent-Message-ID));
unless (@$mids) {
$eml->{-lei_fake_mid} = $mids->[0] =
- PublicInbox::Import::digest2mid($dig, $eml);
+ PublicInbox::Import::digest2mid($dig, $eml, 0);
}
($chash, $mids);
}
diff --git a/lib/PublicInbox/OverIdx.pm b/lib/PublicInbox/OverIdx.pm
index e1cd31b9..66dec099 100644
--- a/lib/PublicInbox/OverIdx.pm
+++ b/lib/PublicInbox/OverIdx.pm
@@ -264,8 +264,10 @@ sub add_overview {
$smsg->{lines} = $eml->body_raw =~ tr!\n!\n!;
my $mids = mids_for_index($eml);
my $refs = $smsg->parse_references($eml, $mids);
- $mids->[0] //= $smsg->{mid} //= $eml->{-lei_fake_mid};
- $smsg->{mid} //= '';
+ $mids->[0] //= do {
+ $smsg->{mid} //= '';
+ $eml->{-lei_fake_mid};
+ };
my $subj = $smsg->{subject};
my $xpath;
if ($subj ne '') {
diff --git a/lib/PublicInbox/Smsg.pm b/lib/PublicInbox/Smsg.pm
index b4cc2ecb..da8ce590 100644
--- a/lib/PublicInbox/Smsg.pm
+++ b/lib/PublicInbox/Smsg.pm
@@ -76,7 +76,7 @@ sub parse_references ($$$) {
return $refs if scalar(@$refs) == 0;
# prevent circular references here:
- my %seen = ( $smsg->{mid} => 1 );
+ my %seen = ( ($smsg->{mid} // '') => 1 );
my @keep;
foreach my $ref (@$refs) {
if (length($ref) > PublicInbox::MID::MAX_MID_SIZE) {
diff --git a/t/lei-import.t b/t/lei-import.t
index 99289748..9bb4e1fa 100644
--- a/t/lei-import.t
+++ b/t/lei-import.t
@@ -79,6 +79,27 @@ is($res->[1], undef, 'only one result');
is($res->[0]->{'m'}, 'k@y', 'got expected message');
is_deeply($res->[0]->{kw}, ['seen'], "`seen' keywords set");
+# no From, Sender, or Message-ID
+$eml_str = <<'EOM';
+Subject: draft message with no sender
+References: <y@y>
+
+No use for a name
+EOM
+lei_ok([qw(import -F eml -)], undef, { %$lei_opt, 0 => \$eml_str });
+lei_ok(['q', 's:draft message with no sender']);
+my $draft_a = json_utf8->decode($lei_out);
+ok(!exists $draft_a->[0]->{'m'}, 'no fake mid stored or exposed');
+lei_ok([qw(tag -F eml - +kw:draft)], undef, { %$lei_opt, 0 => \$eml_str });
+lei_ok(['q', 's:draft message with no sender']);
+my $draft_b = json_utf8->decode($lei_out);
+my $kw = delete $draft_b->[0]->{kw};
+is_deeply($kw, ['draft'], 'draft kw set');
+is_deeply($draft_a, $draft_b, 'fake Message-ID lookup') or
+ diag explain($draft_a, $draft_b);
+lei_ok('blob', '--mail', $draft_b->[0]->{blob});
+is($lei_out, $eml_str, 'draft retrieved by blob');
+
# see t/lei_to_mail.t for "import -F mbox*"
});
done_testing;
^ permalink raw reply related [relevance 55%]
* [PATCH 5/5] lei/store: (more) synchronous non-fatal error output
2021-04-03 10:48 71% [PATCH 0/5] lei/store: fix epoch roll, better errors Eric Wong
2021-04-03 10:48 71% ` [PATCH 3/5] lei tag: note message mismatches on failure Eric Wong
2021-04-03 10:48 55% ` [PATCH 4/5] lei: improve handling of Message-ID-less draft messages Eric Wong
@ 2021-04-03 10:48 43% ` Eric Wong
2 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-03 10:48 UTC (permalink / raw)
To: meta
Since every command that writes to lei/store calls ->done
to commit its output, we can rely on that to return a
pathname for a readable file with errors in it.
Errors can still get crossed up if multiple lei commands
are writing to the store at once, but reduces the delay
in seeing them and ensures it won't get seen when somebody
is attempting to use shell completion.
---
MANIFEST | 1 +
lib/PublicInbox/LeiStore.pm | 69 ++++++++++++++++++++++++++++------
lib/PublicInbox/LeiStoreErr.pm | 30 +++++++++++++++
3 files changed, 89 insertions(+), 11 deletions(-)
create mode 100644 lib/PublicInbox/LeiStoreErr.pm
diff --git a/MANIFEST b/MANIFEST
index 64293bb6..b663c2a2 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -203,6 +203,7 @@ lib/PublicInbox/LeiQuery.pm
lib/PublicInbox/LeiRemote.pm
lib/PublicInbox/LeiSearch.pm
lib/PublicInbox/LeiStore.pm
+lib/PublicInbox/LeiStoreErr.pm
lib/PublicInbox/LeiSucks.pm
lib/PublicInbox/LeiTag.pm
lib/PublicInbox/LeiToMail.pm
diff --git a/lib/PublicInbox/LeiStore.pm b/lib/PublicInbox/LeiStore.pm
index 87082638..094e1555 100644
--- a/lib/PublicInbox/LeiStore.pm
+++ b/lib/PublicInbox/LeiStore.pm
@@ -3,9 +3,14 @@
#
# Local storage (cache/memo) for lei(1), suitable for personal/private
# mail iff on encrypted device/FS. Based on v2, but only deduplicates
-# based on git OID.
+# git storage based on git OID (index deduplication is done in ContentHash)
#
# for xref3, the following are constant: $eidx_key = '.', $xnum = -1
+#
+# We rely on the synchronous IPC API for this in lei-daemon and
+# multiple lei clients to write to it at once. This allows the
+# lei/store IPC process to be decoupled from network latency in
+# lei WQ workers.
package PublicInbox::LeiStore;
use strict;
use v5.10.1;
@@ -19,7 +24,10 @@ use PublicInbox::ContentHash qw(content_hash);
use PublicInbox::MID qw(mids);
use PublicInbox::LeiSearch;
use PublicInbox::MDA;
+use PublicInbox::Spawn qw(spawn);
use List::Util qw(max);
+use File::Temp ();
+use POSIX ();
sub new {
my (undef, $dir, $opt) = @_;
@@ -102,18 +110,27 @@ sub search {
PublicInbox::LeiSearch->new($_[0]->{priv_eidx}->{topdir});
}
+# follows the stderr file
+sub _tail_err {
+ my ($self) = @_;
+ print { $self->{-err_wr} } readline($self->{-tmp_err});
+}
+
sub eidx_init {
my ($self) = @_;
my $eidx = $self->{priv_eidx};
+ my $tl = wantarray && $self->{-err_wr} ?
+ PublicInbox::OnDestroy->new($$, \&_tail_err, $self) :
+ undef;
$eidx->idx_init({-private => 1});
- $eidx;
+ wantarray ? ($eidx, $tl) : $eidx;
}
sub _docids_for ($$) {
my ($self, $eml) = @_;
my %docids;
+ my $eidx = $self->{priv_eidx};
my ($chash, $mids) = PublicInbox::LeiSearch::content_key($eml);
- my $eidx = eidx_init($self);
my $oidx = $eidx->{oidx};
my $im = $self->{im};
for my $mid (@$mids) {
@@ -137,7 +154,7 @@ sub _docids_for ($$) {
sub set_eml_vmd {
my ($self, $eml, $vmd, $docids) = @_;
- my $eidx = eidx_init($self);
+ my ($eidx, $tl) = eidx_init($self);
$docids //= [ _docids_for($self, $eml) ];
for my $docid (@$docids) {
$eidx->idx_shard($docid)->ipc_do('set_vmd', $docid, $vmd);
@@ -147,7 +164,7 @@ sub set_eml_vmd {
sub add_eml_vmd {
my ($self, $eml, $vmd) = @_;
- my $eidx = eidx_init($self);
+ my ($eidx, $tl) = eidx_init($self);
my @docids = _docids_for($self, $eml);
for my $docid (@docids) {
$eidx->idx_shard($docid)->ipc_do('add_vmd', $docid, $vmd);
@@ -157,7 +174,7 @@ sub add_eml_vmd {
sub remove_eml_vmd {
my ($self, $eml, $vmd) = @_;
- my $eidx = eidx_init($self);
+ my ($eidx, $tl) = eidx_init($self);
my @docids = _docids_for($self, $eml);
for my $docid (@docids) {
$eidx->idx_shard($docid)->ipc_do('remove_vmd', $docid, $vmd);
@@ -168,7 +185,7 @@ sub remove_eml_vmd {
sub add_eml {
my ($self, $eml, $vmd, $xoids) = @_;
my $im = $self->importer; # may create new epoch
- my $eidx = eidx_init($self); # writes ALL.git/objects/info/alternates
+ my ($eidx, $tl) = eidx_init($self); # updates/writes alternates file
my $oidx = $eidx->{oidx}; # PublicInbox::Import::add checks this
my $smsg = bless { -oidx => $oidx }, 'PublicInbox::Smsg';
$im->add($eml, undef, $smsg) or return; # duplicate returns undef
@@ -257,7 +274,7 @@ sub _external_only ($$$) {
sub update_xvmd {
my ($self, $xoids, $eml, $vmd_mod) = @_;
- my $eidx = eidx_init($self);
+ my ($eidx, $tl) = eidx_init($self);
my $oidx = $eidx->{oidx};
my %seen;
for my $oid (keys %$xoids) {
@@ -294,7 +311,7 @@ sub update_xvmd {
sub set_xvmd {
my ($self, $xoids, $eml, $vmd) = @_;
- my $eidx = eidx_init($self);
+ my ($eidx, $tl) = eidx_init($self);
my $oidx = $eidx->{oidx};
my %seen;
@@ -329,6 +346,21 @@ sub checkpoint {
$self->{priv_eidx}->checkpoint($wait);
}
+sub xchg_stderr {
+ my ($self) = @_;
+ _tail_err($self) if $self->{-err_wr};
+ my $dir = $self->{priv_eidx}->{topdir};
+ return unless -e $dir;
+ my $old = delete $self->{-tmp_err};
+ my $pfx = POSIX::strftime('%Y%m%d%H%M%S', gmtime(time));
+ my $err = File::Temp->new(TEMPLATE => "$pfx.$$.lei_storeXXXX",
+ SUFFIX => '.err', DIR => $dir);
+ open STDERR, '>>', $err->filename or die "dup2: $!";
+ STDERR->autoflush(1); # shared with shard subprocesses
+ $self->{-tmp_err} = $err; # separate file description for RO access
+ undef;
+}
+
sub done {
my ($self) = @_;
my $err = '';
@@ -339,7 +371,8 @@ sub done {
warn $err;
}
}
- $self->{priv_eidx}->done;
+ $self->{priv_eidx}->done; # V2Writable::done
+ xchg_stderr($self);
die $err if $err;
}
@@ -347,6 +380,11 @@ sub ipc_atfork_child {
my ($self) = @_;
my $lei = $self->{lei};
$lei->_lei_atfork_child(1) if $lei;
+ xchg_stderr($self);
+ if (my $err = delete($self->{err_pipe})) {
+ close $err->[0];
+ $self->{-err_wr} = $err->[1];
+ }
$SIG{__WARN__} = PublicInbox::Eml::warn_ignore_cb();
$self->SUPER::ipc_atfork_child;
}
@@ -357,11 +395,20 @@ sub write_prepare {
my $d = $lei->store_path;
$self->ipc_lock_init("$d/ipc.lock");
substr($d, -length('/lei/store'), 10, '');
+ my $err_pipe;
+ unless ($lei->{oneshot}) {
+ pipe(my ($r, $w)) or die "pipe: $!";
+ $err_pipe = [ $r, $w ];
+ }
# Mail we import into lei are private, so headers filtered out
# by -mda for public mail are not appropriate
local @PublicInbox::MDA::BAD_HEADERS = ();
$self->ipc_worker_spawn("lei/store $d", $lei->oldset,
- { lei => $lei });
+ { lei => $lei, err_pipe => $err_pipe });
+ if ($err_pipe) {
+ require PublicInbox::LeiStoreErr;
+ PublicInbox::LeiStoreErr->new($err_pipe->[0], $lei);
+ }
}
$lei->{sto} = $self;
}
diff --git a/lib/PublicInbox/LeiStoreErr.pm b/lib/PublicInbox/LeiStoreErr.pm
new file mode 100644
index 00000000..68ce96d6
--- /dev/null
+++ b/lib/PublicInbox/LeiStoreErr.pm
@@ -0,0 +1,30 @@
+# Copyright (C) 2021 all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+
+# forwards stderr from lei/store process to any lei clients using
+# the same store
+package PublicInbox::LeiStoreErr;
+use strict;
+use v5.10.1;
+use parent qw(PublicInbox::DS);
+use PublicInbox::Syscall qw(EPOLLIN EPOLLONESHOT);
+
+sub new {
+ my ($cls, $rd, $lei) = @_;
+ my $self = bless { sock => $rd, store_path => $lei->store_path }, $cls;
+ $self->SUPER::new($rd, EPOLLIN | EPOLLONESHOT);
+}
+
+sub event_step {
+ my ($self) = @_;
+ $self->do_read(\(my $rbuf), 4096) or return;
+ my $cb;
+ for my $lei (values %PublicInbox::DS::DescriptorMap) {
+ $cb = $lei->can('store_path') // next;
+ next if $cb->($lei) ne $self->{store_path};
+ my $err = $lei->{2} // next;
+ print $err $rbuf;
+ }
+}
+
+1;
^ permalink raw reply related [relevance 43%]
* [PATCH 0/5] lei/store: fix epoch roll, better errors
@ 2021-04-03 10:48 71% Eric Wong
2021-04-03 10:48 71% ` [PATCH 3/5] lei tag: note message mismatches on failure Eric Wong
` (2 more replies)
0 siblings, 3 replies; 200+ results
From: Eric Wong @ 2021-04-03 10:48 UTC (permalink / raw)
To: meta
Some things I noticed importing my personal IMAP mailboxes
Eric Wong (5):
lei_store: update alternates on new epoch
test_common: lei_ok: improve diagnostics
lei tag: note message mismatches on failure
lei: improve handling of Message-ID-less draft messages
lei/store: (more) synchronous non-fatal error output
MANIFEST | 1 +
lib/PublicInbox/Import.pm | 6 +--
lib/PublicInbox/LeiSearch.pm | 2 +-
lib/PublicInbox/LeiStore.pm | 74 ++++++++++++++++++++++++++++------
lib/PublicInbox/LeiStoreErr.pm | 30 ++++++++++++++
lib/PublicInbox/LeiTag.pm | 3 +-
lib/PublicInbox/OverIdx.pm | 6 ++-
lib/PublicInbox/Smsg.pm | 2 +-
lib/PublicInbox/TestCommon.pm | 3 +-
t/lei-import.t | 21 ++++++++++
10 files changed, 127 insertions(+), 21 deletions(-)
create mode 100644 lib/PublicInbox/LeiStoreErr.pm
^ permalink raw reply [relevance 71%]
* [PATCH 5/6] net_reader: fix read-only "lei convert" auth failures
2021-04-03 2:24 71% [PATCH 0/6] lei auth-related fixes Eric Wong
2021-04-03 2:24 57% ` [PATCH 2/6] lei q: ensure wq workers shutdown on IMAP auth failures Eric Wong
2021-04-03 2:24 68% ` [PATCH 3/6] lei tag: fix tagging of IMAP inputs Eric Wong
@ 2021-04-03 2:24 71% ` Eric Wong
2021-04-03 2:24 69% ` [PATCH 6/6] xt/lei-auth-fail: test more failure cases Eric Wong
3 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-03 2:24 UTC (permalink / raw)
To: meta
"convert" is actually a bit more complicated than "lei import"
since it may need auth for either input or output.
---
lib/PublicInbox/NetReader.pm | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/lib/PublicInbox/NetReader.pm b/lib/PublicInbox/NetReader.pm
index c269d841..821e5d7f 100644
--- a/lib/PublicInbox/NetReader.pm
+++ b/lib/PublicInbox/NetReader.pm
@@ -264,10 +264,11 @@ sub imap_common_init ($;$) {
my $mics = {}; # schema://authority => IMAPClient obj
for my $uri (@{$self->{imap_order}}) {
my $sec = uri_section($uri);
- $mics->{$sec} //= mic_for($self, "$sec/", $mic_args, $lei);
+ my $mic = $mics->{$sec} //=
+ mic_for($self, "$sec/", $mic_args, $lei) //
+ die "Unable to continue\n";
next unless $self->isa('PublicInbox::NetWriter');
my $dst = $uri->mailbox // next;
- my $mic = $mics->{$sec} // die "Unable to continue\n";
next if $mic->exists($dst); # already exists
$mic->create($dst) or die "CREATE $dst failed <$uri>: $@";
}
^ permalink raw reply related [relevance 71%]
* [PATCH 6/6] xt/lei-auth-fail: test more failure cases
2021-04-03 2:24 71% [PATCH 0/6] lei auth-related fixes Eric Wong
` (2 preceding siblings ...)
2021-04-03 2:24 71% ` [PATCH 5/6] net_reader: fix read-only "lei convert" auth failures Eric Wong
@ 2021-04-03 2:24 69% ` Eric Wong
3 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-03 2:24 UTC (permalink / raw)
To: meta
Because failures are often overlooked, unfortunately.
---
xt/lei-auth-fail.t | 11 +++++++----
1 file changed, 7 insertions(+), 4 deletions(-)
diff --git a/xt/lei-auth-fail.t b/xt/lei-auth-fail.t
index e352aab3..06cb8533 100644
--- a/xt/lei-auth-fail.t
+++ b/xt/lei-auth-fail.t
@@ -2,17 +2,20 @@
# Copyright (C) 2021 all contributors <meta@public-inbox.org>
# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
use strict; use v5.10.1; use PublicInbox::TestCommon;
-require_mods(qw(Mail::IMAPClient));
+require_mods(qw(Mail::IMAPClient lei));
# TODO: mock IMAP server which fails at authentication so we don't
# have to make external connections to test this:
my $imap_fail = $ENV{TEST_LEI_IMAP_FAIL_URL} //
'imaps://AzureDiamond:Hunter2@public-inbox.org:994/INBOX';
+my ($ro_home, $cfg_path) = setup_public_inboxes;
test_lei(sub {
- for my $pfx ([qw(convert -o mboxrd:/dev/stdout)], ['import'],
- [qw(tag +L:INBOX)]) {
+ for my $pfx ([qw(q z:0.. --only), "$ro_home/t1", '-o'],
+ [qw(convert -o mboxrd:/dev/stdout)],
+ [qw(convert t/utf8.eml -o), $imap_fail],
+ ['import'], [qw(tag +L:INBOX)]) {
ok(!lei(@$pfx, $imap_fail), "IMAP auth failure on @$pfx");
- like($lei_err, qr!\bE:.*?imaps://.*?!sm, 'error shown');
+ like($lei_err, qr!\bE:.*?imaps?://.*?!sm, 'error shown');
unlike($lei_err, qr!Hunter2!s, 'password not shown');
is($lei_out, '', 'nothing output');
}
^ permalink raw reply related [relevance 69%]
* [PATCH 0/6] lei auth-related fixes
@ 2021-04-03 2:24 71% Eric Wong
2021-04-03 2:24 57% ` [PATCH 2/6] lei q: ensure wq workers shutdown on IMAP auth failures Eric Wong
` (3 more replies)
0 siblings, 4 replies; 200+ results
From: Eric Wong @ 2021-04-03 2:24 UTC (permalink / raw)
To: meta
Auth failure handling should be more complete, but it's still
horrifically slow to test because the dovecot instance I run has
rate-limits for auth failures.
Perhaps public-inbox-imapd should learn to reject certain
username + password combos, since the -imapd instance on
public-inbox.org sees a fair amount of automated traffic
looking to steal info off leaked credentials.
3/6 is me still learning to use APIs I invent :x
Eric Wong (6):
URInntps: add URI 5.08 release note
lei q: ensure wq workers shutdown on IMAP auth failures
lei tag: fix tagging of IMAP inputs
lei_auth: rename {net_merge} to {net_merge_continue}
net_reader: fix read-only "lei convert" auth failures
xt/lei-auth-fail: test more failure cases
lib/PublicInbox/LEI.pm | 24 ++++++++----------------
lib/PublicInbox/LeiAuth.pm | 17 ++++++++++-------
lib/PublicInbox/LeiTag.pm | 6 +++++-
lib/PublicInbox/NetReader.pm | 5 +++--
lib/PublicInbox/URInntps.pm | 1 +
t/lei-import-imap.t | 1 +
xt/lei-auth-fail.t | 11 +++++++----
7 files changed, 35 insertions(+), 30 deletions(-)
^ permalink raw reply [relevance 71%]
* [PATCH 3/6] lei tag: fix tagging of IMAP inputs
2021-04-03 2:24 71% [PATCH 0/6] lei auth-related fixes Eric Wong
2021-04-03 2:24 57% ` [PATCH 2/6] lei q: ensure wq workers shutdown on IMAP auth failures Eric Wong
@ 2021-04-03 2:24 68% ` Eric Wong
2021-04-03 2:24 71% ` [PATCH 5/6] net_reader: fix read-only "lei convert" auth failures Eric Wong
2021-04-03 2:24 69% ` [PATCH 6/6] xt/lei-auth-fail: test more failure cases Eric Wong
3 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-03 2:24 UTC (permalink / raw)
To: meta
We need net_merge_all and to lock the number of worker jobs.
Parallel inputs are not supported, yet (is it needed?, I don't
expect this to be used for multiple files very often...).
---
lib/PublicInbox/LeiTag.pm | 6 +++++-
t/lei-import-imap.t | 1 +
2 files changed, 6 insertions(+), 1 deletion(-)
diff --git a/lib/PublicInbox/LeiTag.pm b/lib/PublicInbox/LeiTag.pm
index d572a84a..06313139 100644
--- a/lib/PublicInbox/LeiTag.pm
+++ b/lib/PublicInbox/LeiTag.pm
@@ -112,7 +112,8 @@ sub lei_tag { # the "lei tag" method
my $ops = { '' => [ \&tag_done, $lei ] };
$lei->{auth}->op_merge($ops, $self) if $lei->{auth};
$self->{vmd_mod} = $vmd_mod;
- (my $op_c, $ops) = $lei->workers_start($self, 'lei_tag', 1, $ops);
+ my $j = $self->{-wq_nr_workers} = 1; # locked for now
+ (my $op_c, $ops) = $lei->workers_start($self, 'lei_tag', $j, $ops);
$lei->{tag} = $self;
net_merge_complete($self) unless $lei->{auth};
$op_c->op_wait_event($ops);
@@ -175,4 +176,7 @@ sub _complete_tag {
} grep(/$re\Q$cur/, @all);
}
+no warnings 'once'; # the following works even when LeiAuth is lazy-loaded
+*net_merge_all = \&PublicInbox::LeiAuth::net_merge_all;
+
1;
diff --git a/t/lei-import-imap.t b/t/lei-import-imap.t
index fd38037a..7e4d44b9 100644
--- a/t/lei-import-imap.t
+++ b/t/lei-import-imap.t
@@ -23,5 +23,6 @@ test_lei({ tmpdir => $tmpdir }, sub {
my %r;
for (@$out) { $r{ref($_)}++ }
is_deeply(\%r, { 'HASH' => scalar(@$out) }, 'all hashes');
+ lei_ok([qw(tag +kw:seen), "imap://$host_port/t.v2.0"], undef, undef);
});
done_testing;
^ permalink raw reply related [relevance 68%]
* [PATCH 2/6] lei q: ensure wq workers shutdown on IMAP auth failures
2021-04-03 2:24 71% [PATCH 0/6] lei auth-related fixes Eric Wong
@ 2021-04-03 2:24 57% ` Eric Wong
2021-04-03 2:24 68% ` [PATCH 3/6] lei tag: fix tagging of IMAP inputs Eric Wong
` (2 subsequent siblings)
3 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-03 2:24 UTC (permalink / raw)
To: meta
Leaving workers running on after auth failures is bad and messy,
cleanup our process management to have consistent worker
teardowns. Improve error reporting, too, instead of letting
Mail::IMAPClient->exists fail due to undef.
---
lib/PublicInbox/LEI.pm | 24 ++++++++----------------
lib/PublicInbox/LeiAuth.pm | 15 +++++++++------
lib/PublicInbox/NetReader.pm | 2 +-
3 files changed, 18 insertions(+), 23 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index f9361c68..cdb4b347 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -350,6 +350,11 @@ my %CONFIG_KEYS = (
my @WQ_KEYS = qw(lxs l2m imp mrr cnv p2q tag sol); # internal workers
+sub _drop_wq {
+ my ($self) = @_;
+ for my $wq (grep(defined, delete(@$self{@WQ_KEYS}))) { $wq->DESTROY }
+}
+
# pronounced "exit": x_it(1 << 8) => exit(1); x_it(13) => SIGPIPE
sub x_it ($$) {
my ($self, $code) = @_;
@@ -360,10 +365,7 @@ sub x_it ($$) {
send($s, "x_it $code", MSG_EOR);
} elsif ($self->{oneshot}) {
# don't want to end up using $? from child processes
- for my $f (@WQ_KEYS) {
- my $wq = delete $self->{$f} or next;
- $wq->DESTROY;
- }
+ _drop_wq($self);
# cleanup anything that has tempfiles or open file handles
%PATH2CFG = ();
delete @$self{qw(ovv dedupe sto cfg)};
@@ -392,11 +394,8 @@ sub qerr ($;@) { $_[0]->{opt}->{quiet} or err(shift, @_) }
sub fail_handler ($;$$) {
my ($lei, $code, $io) = @_;
- for my $f (@WQ_KEYS) {
- my $wq = delete $lei->{$f} or next;
- $wq->wq_wait_old(undef, $lei) if $wq->wq_kill_old; # lei-daemon
- }
close($io) if $io; # needed to avoid warnings on SIGPIPE
+ _drop_wq($lei);
x_it($lei, $code // (1 << 8));
}
@@ -983,14 +982,7 @@ sub accept_dispatch { # Listener {post_accept} callback
sub dclose {
my ($self) = @_;
delete $self->{-progress};
- for my $f (@WQ_KEYS) {
- my $wq = delete $self->{$f} or next;
- if ($wq->wq_kill) {
- $wq->wq_close(0, undef, $self);
- } elsif ($wq->wq_kill_old) {
- $wq->wq_wait_old(undef, $self);
- }
- }
+ _drop_wq($self);
close(delete $self->{1}) if $self->{1}; # may reap_compress
$self->close if $self->{-event_init_done}; # PublicInbox::DS::close
}
diff --git a/lib/PublicInbox/LeiAuth.pm b/lib/PublicInbox/LeiAuth.pm
index 927fe550..48deca93 100644
--- a/lib/PublicInbox/LeiAuth.pm
+++ b/lib/PublicInbox/LeiAuth.pm
@@ -13,12 +13,15 @@ sub do_auth_atfork { # used by IPC WQ workers
return if $wq->{-wq_worker_nr} != 0;
my $lei = $wq->{lei};
my $net = $lei->{net};
- my $mics = $net->imap_common_init($lei);
- my $nn = $net->nntp_common_init($lei);
- pkt_do($lei->{pkt_op_p}, 'net_merge', $net) or
- die "pkt_do net_merge: $!";
- $net->{mics_cached} = $mics if $mics;
- $net->{nn_cached} = $nn if $nn;
+ eval {
+ my $mics = $net->imap_common_init($lei);
+ my $nn = $net->nntp_common_init($lei);
+ pkt_do($lei->{pkt_op_p}, 'net_merge', $net) or
+ die "pkt_do net_merge: $!";
+ $net->{mics_cached} = $mics if $mics;
+ $net->{nn_cached} = $nn if $nn;
+ };
+ $lei->fail($@) if $@;
}
sub net_merge_done1 { # bump merge-count in top-level lei-daemon
diff --git a/lib/PublicInbox/NetReader.pm b/lib/PublicInbox/NetReader.pm
index 6a52b479..c269d841 100644
--- a/lib/PublicInbox/NetReader.pm
+++ b/lib/PublicInbox/NetReader.pm
@@ -267,7 +267,7 @@ sub imap_common_init ($;$) {
$mics->{$sec} //= mic_for($self, "$sec/", $mic_args, $lei);
next unless $self->isa('PublicInbox::NetWriter');
my $dst = $uri->mailbox // next;
- my $mic = $mics->{$sec};
+ my $mic = $mics->{$sec} // die "Unable to continue\n";
next if $mic->exists($dst); # already exists
$mic->create($dst) or die "CREATE $dst failed <$uri>: $@";
}
^ permalink raw reply related [relevance 57%]
* [PATCH 2/2] lei: allow progress to non-TTY after MUA spawn
2021-04-03 1:37 71% [PATCH 0/2] lei MUA UX fixes Eric Wong
2021-04-03 1:37 71% ` [PATCH 1/2] lei q: don't show remote progress if MUA is running Eric Wong
@ 2021-04-03 1:37 71% ` Eric Wong
1 sibling, 0 replies; 200+ results
From: Eric Wong @ 2021-04-03 1:37 UTC (permalink / raw)
To: meta
Sometimes I want to save debug info to a file or pipe even when
spawning an MUA.
---
lib/PublicInbox/LEI.pm | 1 +
1 file changed, 1 insertion(+)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index f9361c68..96a27102 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -850,6 +850,7 @@ sub start_mua {
if ($self->{lxs} && $self->{au_done}) { # kick wait_startq
syswrite($self->{au_done}, 'q' x ($self->{lxs}->{jobs} // 0));
}
+ return unless -t $self->{2}; # XXX how to determine non-TUI MUAs?
$self->{opt}->{quiet} = 1;
delete $self->{-progress};
delete $self->{opt}->{verbose};
^ permalink raw reply related [relevance 71%]
* [PATCH 0/2] lei MUA UX fixes
@ 2021-04-03 1:37 71% Eric Wong
2021-04-03 1:37 71% ` [PATCH 1/2] lei q: don't show remote progress if MUA is running Eric Wong
2021-04-03 1:37 71% ` [PATCH 2/2] lei: allow progress to non-TTY after MUA spawn Eric Wong
0 siblings, 2 replies; 200+ results
From: Eric Wong @ 2021-04-03 1:37 UTC (permalink / raw)
To: meta
More stuff around auth coming...
Eric Wong (2):
lei q: don't show remote progress if MUA is running
lei: allow progress to non-TTY after MUA spawn
lib/PublicInbox/LEI.pm | 1 +
lib/PublicInbox/LeiXSearch.pm | 2 +-
2 files changed, 2 insertions(+), 1 deletion(-)
^ permalink raw reply [relevance 71%]
* [PATCH 1/2] lei q: don't show remote progress if MUA is running
2021-04-03 1:37 71% [PATCH 0/2] lei MUA UX fixes Eric Wong
@ 2021-04-03 1:37 71% ` Eric Wong
2021-04-03 1:37 71% ` [PATCH 2/2] lei: allow progress to non-TTY after MUA spawn Eric Wong
1 sibling, 0 replies; 200+ results
From: Eric Wong @ 2021-04-03 1:37 UTC (permalink / raw)
To: meta
Remote results can safely use the same mset progress reporting
as local results, despite not knowing the size of the result
set. We're assuming terminal MUAs, for now.
---
lib/PublicInbox/LeiXSearch.pm | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/lib/PublicInbox/LeiXSearch.pm b/lib/PublicInbox/LeiXSearch.pm
index f3b8cc25..2b23e8e9 100644
--- a/lib/PublicInbox/LeiXSearch.pm
+++ b/lib/PublicInbox/LeiXSearch.pm
@@ -232,7 +232,7 @@ sub each_remote_eml { # callback for MboxReader->mboxrd
if ($now > $next) {
$lei->{-next_progress} = $now + 1;
my $nr = $lei->{-nr_remote_eml};
- $lei->err("# $lei->{-current_url} $nr/?");
+ mset_progress($lei, $lei->{-current_url}, $nr, '?');
}
}
$each_smsg->($smsg, undef, $eml);
^ permalink raw reply related [relevance 71%]
* [PATCH] lei: fix git-credential handling
@ 2021-04-02 9:42 57% Eric Wong
2021-04-05 4:17 71% ` Kyle Meyer
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2021-04-02 9:42 UTC (permalink / raw)
To: meta
I completely forgot about git-credential prompting when
making lei background the client process for MUA.
Now it backgrounds itself only for the MUA when no FDs are
passed, since the MUA is the final command run. Otherwise, it
relies on FD passing as before.
Fixes: c790a75439f3a1db ("script/lei: background ourselves on MUA/pager exec")
---
lib/PublicInbox/LEI.pm | 8 +++++++-
script/lei | 46 ++++++++++++++++++++++++++++--------------
2 files changed, 38 insertions(+), 16 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 69d48bd1..f9361c68 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -839,7 +839,13 @@ sub start_mua {
if (my $sock = $self->{sock}) { # lei(1) client process runs it
send($sock, exec_buf(\@cmd, {}), MSG_EOR);
} elsif ($self->{oneshot}) {
- $self->{"pid.$self.$$"}->{spawn(\@cmd)} = \@cmd;
+ my $pid = fork // die "fork: $!";
+ if ($pid > 0) { # original process
+ exec(@cmd);
+ warn "exec @cmd: $!\n";
+ POSIX::_exit(1);
+ }
+ POSIX::setsid() > 0 or die "setsid: $!";
}
if ($self->{lxs} && $self->{au_done}) { # kick wait_startq
syswrite($self->{au_done}, 'q' x ($self->{lxs}->{jobs} // 0));
diff --git a/script/lei b/script/lei
index bea8dcde..78a7dab9 100755
--- a/script/lei
+++ b/script/lei
@@ -14,31 +14,45 @@ my $send_cmd = PublicInbox::CmdIPC4->can('send_cmd4') // do {
PublicInbox::Spawn->can('send_cmd4');
};
-my @orig_pid;
+my %pids;
+my $sigchld = sub {
+ my $flags = scalar(@_) ? POSIX::WNOHANG() : 0;
+ for my $pid (keys %pids) {
+ delete($pids{$pid}) if waitpid($pid, $flags) == $pid;
+ }
+};
+my @parent;
my $exec_cmd = sub {
my ($fds, $argc, @argv) = @_;
- die "BUG: already exec-ed\n" if @orig_pid;
- @orig_pid = ($$);
- require POSIX; # WNOHANG
+ my $parent = $$;
+ require POSIX;
my @old = (*STDIN{IO}, *STDOUT{IO}, *STDERR{IO});
my @rdr;
for my $fd (@$fds) {
- open(my $tmpfh, '+<&=', $fd) or die "open +<&=$fd: $!";
- push @rdr, shift(@old), $tmpfh;
+ open(my $newfh, '+<&=', $fd) or die "open +<&=$fd: $!";
+ push @rdr, shift(@old), $newfh;
}
+ my $do_exec = sub {
+ my %env = map { split(/=/, $_, 2) } splice(@argv, $argc);
+ @ENV{keys %env} = values %env;
+ exec(@argv);
+ warn "exec: @argv: $!\n";
+ POSIX::_exit(1);
+ };
+ $SIG{CHLD} = $sigchld;
my $pid = fork // die "fork: $!";
if ($pid == 0) {
+ while (my ($io, $newfh) = splice(@rdr, 0, 2)) {
+ open $io, '+<&', $newfh or die "open +<&=: $!";
+ }
+ $do_exec->() if scalar(@$fds); # git-credential, pager
+
+ # parent backgrounds on MUA
POSIX::setsid() > 0 or die "setsid: $!";
+ @parent = ($parent);
return; # continue $recv_cmd in background
}
- my %env = map { split(/=/, $_, 2) } splice(@argv, $argc);
- while (my ($old_io, $tmpfh) = splice(@rdr, 0, 2)) {
- open $old_io, '+<&', $tmpfh or die "open +<&=: $!";
- }
- @ENV{keys %env} = values %env;
- exec(@argv);
- warn "exec: @argv: $!\n";
- POSIX::_exit(1);
+ $do_exec->() if !scalar(@$fds); # MUA reuses all FDs
};
if ($send_cmd && eval {
@@ -95,16 +109,18 @@ Falling back to (slow) one-shot mode
if ($buf =~ /\Aexec (.+)\z/) {
$exec_cmd->(\@fds, split(/\0/, $1));
} elsif ($buf eq '-WINCH') {
- kill($buf, @orig_pid); # for MUA
+ kill($buf, @parent); # for MUA
} elsif ($buf =~ /\Ax_it ([0-9]+)\z/) {
$x_it_code = $1 + 0;
last;
} elsif ($buf =~ /\Achild_error ([0-9]+)\z/) {
$x_it_code = $1 + 0;
} else {
+ $sigchld->();
die $buf;
}
}
+ $sigchld->();
if (my $sig = ($x_it_code & 127)) {
kill $sig, $$;
sleep(1) while 1;
^ permalink raw reply related [relevance 57%]
* [PATCH 2/5] lei q: reduce lei/store work for kw changes to stored mail
@ 2021-04-01 12:10 84% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-01 12:10 UTC (permalink / raw)
To: meta
We can tweak lse->kw_changed to return docids and reduce IPC
traffic and reduce work the lei/store worker needs to do.
---
lib/PublicInbox/LeiSearch.pm | 9 +++++----
lib/PublicInbox/LeiStore.pm | 8 ++++----
lib/PublicInbox/LeiToMail.pm | 6 +++---
3 files changed, 12 insertions(+), 11 deletions(-)
diff --git a/lib/PublicInbox/LeiSearch.pm b/lib/PublicInbox/LeiSearch.pm
index 07d570ec..69ba8303 100644
--- a/lib/PublicInbox/LeiSearch.pm
+++ b/lib/PublicInbox/LeiSearch.pm
@@ -100,10 +100,11 @@ sub xoids_for {
# returns true if $eml is indexed by lei/store and keywords don't match
sub kw_changed {
- my ($self, $eml, $new_kw_sorted) = @_;
- my $xoids = xoids_for($self, $eml, 1) // return;
- my ($num) = values %$xoids;
- my @cur_kw = msg_keywords($self, $num);
+ my ($self, $eml, $new_kw_sorted, $docids) = @_;
+ my $xoids = xoids_for($self, $eml) // return;
+ $docids //= [];
+ @$docids = sort { $a <=> $b } values %$xoids;
+ my @cur_kw = msg_keywords($self, $docids->[0]);
join("\0", @$new_kw_sorted) eq join("\0", @cur_kw) ? 0 : 1;
}
diff --git a/lib/PublicInbox/LeiStore.pm b/lib/PublicInbox/LeiStore.pm
index b76af4d3..48ab1d76 100644
--- a/lib/PublicInbox/LeiStore.pm
+++ b/lib/PublicInbox/LeiStore.pm
@@ -131,13 +131,13 @@ sub _docids_for ($$) {
}
sub set_eml_vmd {
- my ($self, $eml, $vmd) = @_;
+ my ($self, $eml, $vmd, $docids) = @_;
my $eidx = eidx_init($self);
- my @docids = _docids_for($self, $eml);
- for my $docid (@docids) {
+ $docids //= [ _docids_for($self, $eml) ];
+ for my $docid (@$docids) {
$eidx->idx_shard($docid)->ipc_do('set_vmd', $docid, $vmd);
}
- \@docids;
+ $docids;
}
sub add_eml_vmd {
diff --git a/lib/PublicInbox/LeiToMail.pm b/lib/PublicInbox/LeiToMail.pm
index da633da4..0364d8ef 100644
--- a/lib/PublicInbox/LeiToMail.pm
+++ b/lib/PublicInbox/LeiToMail.pm
@@ -228,10 +228,10 @@ sub _mbox_write_cb ($$) {
sub update_kw_maybe ($$$$) {
my ($lei, $lse, $eml, $kw) = @_;
return unless $lse;
- my $lse_oids = $lse->kw_changed($eml, $kw);
+ my $c = $lse->kw_changed($eml, $kw, my $docids = []);
my $vmd = { kw => $kw };
- if ($lse_oids) { # already in lei/store
- $lei->{sto}->ipc_do('set_eml', $eml, $vmd);
+ if (scalar @$docids) { # already in lei/store
+ $lei->{sto}->ipc_do('set_eml_vmd', undef, $vmd, $docids) if $c;
} elsif (my $xoids = $lei->{ale}->xoids_for($eml)) {
# it's in an external, only set kw, here
$lei->{sto}->ipc_do('set_xvmd', $xoids, $eml, $vmd);
^ permalink raw reply related [relevance 84%]
* [PATCH] lei: maildir: handle "forwarded" keyword as "P"
@ 2021-04-01 10:12 71% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-01 10:12 UTC (permalink / raw)
To: meta
mbox and IMAP seem to have no way of describing this keyword.
but Maildir does with the "P" flagged (for "passed").
---
lib/PublicInbox/LeiToMail.pm | 3 ++-
lib/PublicInbox/MdirReader.pm | 3 ++-
2 files changed, 4 insertions(+), 2 deletions(-)
diff --git a/lib/PublicInbox/LeiToMail.pm b/lib/PublicInbox/LeiToMail.pm
index 4c33c752..da633da4 100644
--- a/lib/PublicInbox/LeiToMail.pm
+++ b/lib/PublicInbox/LeiToMail.pm
@@ -27,8 +27,9 @@ $PublicInbox::GitAsyncCat::GCF2C = 0;
my %kw2char = ( # Maildir characters
draft => 'D',
flagged => 'F',
+ forwarded => 'P', # passed
answered => 'R',
- seen => 'S'
+ seen => 'S',
);
my %kw2status = (
diff --git a/lib/PublicInbox/MdirReader.pm b/lib/PublicInbox/MdirReader.pm
index 30e6f8ad..1685e4d8 100644
--- a/lib/PublicInbox/MdirReader.pm
+++ b/lib/PublicInbox/MdirReader.pm
@@ -37,7 +37,8 @@ sub maildir_each_file ($$;@) {
}
}
-my %c2kw = ('D' => 'draft', F => 'flagged', R => 'answered', S => 'seen');
+my %c2kw = ('D' => 'draft', F => 'flagged', P => 'forwarded',
+ R => 'answered', S => 'seen');
sub maildir_each_eml ($$;@) {
my ($dir, $cb, @arg) = @_;
^ permalink raw reply related [relevance 71%]
* [PATCH 0/2] lei sucks
@ 2021-04-01 9:32 71% Eric Wong
2021-04-01 9:32 55% ` [PATCH 2/2] lei sucks: sub-command to aid bug reporting Eric Wong
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2021-04-01 9:32 UTC (permalink / raw)
To: meta
Inspired by the mutt slogan.
Eric Wong (2):
build: generate PublicInbox.pm with $VERSION
lei sucks: sub-command to aid bug reporting
.gitignore | 1 +
MANIFEST | 2 ++
Makefile.PL | 9 +++--
lib/PublicInbox/LeiSucks.pm | 71 +++++++++++++++++++++++++++++++++++++
t/config.t | 7 ++--
t/lei.t | 1 +
version-gen.perl | 29 +++++++++++++++
7 files changed, 115 insertions(+), 5 deletions(-)
create mode 100644 lib/PublicInbox/LeiSucks.pm
create mode 100644 version-gen.perl
^ permalink raw reply [relevance 71%]
* [PATCH 2/2] lei sucks: sub-command to aid bug reporting
2021-04-01 9:32 71% [PATCH 0/2] lei sucks Eric Wong
@ 2021-04-01 9:32 55% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-01 9:32 UTC (permalink / raw)
To: meta
It's a bit of an Easter egg, though it's not possible to hide those
in Free Software... Anyways, it doesn't cost us an entry in %CMD
of LEI.pm and anybody frustrated enough with lei just might type
"lei sucks" on the command-line :>
---
MANIFEST | 1 +
lib/PublicInbox/LeiSucks.pm | 71 +++++++++++++++++++++++++++++++++++++
t/lei.t | 1 +
3 files changed, 73 insertions(+)
create mode 100644 lib/PublicInbox/LeiSucks.pm
diff --git a/MANIFEST b/MANIFEST
index 5e3b4aec..64293bb6 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -203,6 +203,7 @@ lib/PublicInbox/LeiQuery.pm
lib/PublicInbox/LeiRemote.pm
lib/PublicInbox/LeiSearch.pm
lib/PublicInbox/LeiStore.pm
+lib/PublicInbox/LeiSucks.pm
lib/PublicInbox/LeiTag.pm
lib/PublicInbox/LeiToMail.pm
lib/PublicInbox/LeiXSearch.pm
diff --git a/lib/PublicInbox/LeiSucks.pm b/lib/PublicInbox/LeiSucks.pm
new file mode 100644
index 00000000..d364a856
--- /dev/null
+++ b/lib/PublicInbox/LeiSucks.pm
@@ -0,0 +1,71 @@
+# Copyright (C) 2021 all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+
+# Undocumented hidden command somebody might discover if they're
+# frustrated and need to report a bug. There's no manpage and
+# it won't show up in tab completions or help.
+package PublicInbox::LeiSucks;
+use strict;
+use v5.10.1;
+use Digest::SHA ();
+use Config;
+use POSIX ();
+use PublicInbox::Config;
+use PublicInbox::Search;
+
+sub lei_sucks {
+ my ($lei, @argv) = @_;
+ $lei->start_pager if -t $lei->{1};
+ my ($os, undef, $rel, undef, $mac)= POSIX::uname();
+ if ($mac eq 'x86_64' && $Config{ptrsize} == 4) {
+ $mac = $Config{cppsymbols} =~ /\b__ILP32__=1\b/ ? 'x32' : 'i386'
+ }
+ eval { require PublicInbox };
+ my $pi_ver = eval('$PublicInbox::VERSION') // '(???)';
+ my $daemon = $lei->{oneshot} ? 'oneshot' : 'daemon';
+ my @out = ("lei $pi_ver mode=$daemon\n",
+ "perl $Config{version} / $os $rel / $mac ".
+ "ptrsize=$Config{ptrsize}\n");
+ chomp(my $gv = `git --version` || "git missing");
+ $gv =~ s/ version / /;
+ my $json = ref(PublicInbox::Config->json);
+ $json .= ' ' . eval('$'.$json.'::VERSION') if $json;
+ $json ||= '(no JSON)';
+ push @out, "$gv / $json\n";
+ if (eval { require PublicInbox::Over }) {
+ push @out, 'SQLite '.
+ (eval('$DBD::SQLite::sqlite_version') // '(undef)') .
+ ', DBI '.(eval('$DBI::VERSION') // '(undef)') .
+ ', DBD::SQLite '.
+ (eval('$DBD::SQLite::VERSION') // '(undef)')."\n";
+ } else {
+ push @out, "Unable to load DBI / DBD::SQLite: $@\n";
+ }
+ if (PublicInbox::Search::load_xapian()) {
+ push @out, 'Xapian '.
+ join('.', map {
+ $PublicInbox::Search::Xap->can($_)->();
+ } qw(major_version minor_version revision)) .
+ ", bindings: $PublicInbox::Search::Xap";
+ my $xs_ver = eval '$'."$PublicInbox::Search::Xap".'::VERSION';
+ push @out, $xs_ver ? " $xs_ver\n" : " SWIG\n";
+ } else {
+ push @out, "Xapian not available: $@\n";
+ }
+ my $dig = Digest::SHA->new(1);
+ push @out, "public-inbox blob OIDs of loaded features:\n";
+ for my $m (grep(m{^PublicInbox/}, sort keys %INC)) {
+ my $f = $INC{$m};
+ $dig->add('blob '.(-s $f)."\0");
+ $dig->addfile($f);
+ push @out, ' '.$dig->hexdigest.' '.$m."\n";
+ }
+ push @out, <<'EOM';
+Let us know how it sucks! Please include the above and any other
+relevant information when sending plain-text mail to us at:
+meta@public-inbox.org -- archives: https://public-inbox.org/meta/
+EOM
+ $lei->out(@out);
+}
+
+1;
diff --git a/t/lei.t b/t/lei.t
index 0cf97866..2be9b4e8 100644
--- a/t/lei.t
+++ b/t/lei.t
@@ -154,6 +154,7 @@ my $test_fail = sub {
"error noted with q $fl");
}
}
+ lei_ok('sucks', \'yes, but hopefully less every day');
SKIP: {
skip 'no curl', 3 unless which('curl');
lei(qw(q --only http://127.0.0.1:99999/bogus/ t:m));
^ permalink raw reply related [relevance 55%]
* [PATCH] script/lei: background ourselves on MUA/pager exec
@ 2021-03-31 23:29 62% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-31 23:29 UTC (permalink / raw)
To: meta
This ought to gives the MUA or pager exclusive access to the
controlling terminal. The downside is we can only exec the
pager or MUA once per invocation, but I can't imagine a valid
case for running those things multiple times, either.
Note: I'm no expert when it comes to terminal control matters,
but this allows Ctrl-Z-ed mutt instance to come back and is
a nice code reduction, as well.
---
script/lei | 37 +++++++++++++++----------------------
1 file changed, 15 insertions(+), 22 deletions(-)
diff --git a/script/lei b/script/lei
index cb605e2e..bea8dcde 100755
--- a/script/lei
+++ b/script/lei
@@ -14,36 +14,31 @@ my $send_cmd = PublicInbox::CmdIPC4->can('send_cmd4') // do {
PublicInbox::Spawn->can('send_cmd4');
};
-my %pids;
-my $sigchld = sub {
- my $flags = scalar(@_) ? POSIX::WNOHANG() : 0;
- for my $pid (keys %pids) {
- delete($pids{$pid}) if waitpid($pid, $flags) == $pid;
- }
-};
-
+my @orig_pid;
my $exec_cmd = sub {
my ($fds, $argc, @argv) = @_;
+ die "BUG: already exec-ed\n" if @orig_pid;
+ @orig_pid = ($$);
+ require POSIX; # WNOHANG
my @old = (*STDIN{IO}, *STDOUT{IO}, *STDERR{IO});
my @rdr;
for my $fd (@$fds) {
open(my $tmpfh, '+<&=', $fd) or die "open +<&=$fd: $!";
push @rdr, shift(@old), $tmpfh;
}
- require POSIX; # WNOHANG
- $SIG{CHLD} = $sigchld;
my $pid = fork // die "fork: $!";
if ($pid == 0) {
- my %env = map { split(/=/, $_, 2) } splice(@argv, $argc);
- while (my ($old_io, $tmpfh) = splice(@rdr, 0, 2)) {
- open $old_io, '+<&', $tmpfh or die "open +<&=: $!";
- }
- %ENV = (%ENV, %env);
- exec(@argv);
- warn "exec: @argv: $!\n";
- POSIX::_exit(1);
+ POSIX::setsid() > 0 or die "setsid: $!";
+ return; # continue $recv_cmd in background
+ }
+ my %env = map { split(/=/, $_, 2) } splice(@argv, $argc);
+ while (my ($old_io, $tmpfh) = splice(@rdr, 0, 2)) {
+ open $old_io, '+<&', $tmpfh or die "open +<&=: $!";
}
- $pids{$pid} = 1;
+ @ENV{keys %env} = values %env;
+ exec(@argv);
+ warn "exec: @argv: $!\n";
+ POSIX::_exit(1);
};
if ($send_cmd && eval {
@@ -100,18 +95,16 @@ Falling back to (slow) one-shot mode
if ($buf =~ /\Aexec (.+)\z/) {
$exec_cmd->(\@fds, split(/\0/, $1));
} elsif ($buf eq '-WINCH') {
- kill($buf, $$); # for MUA
+ kill($buf, @orig_pid); # for MUA
} elsif ($buf =~ /\Ax_it ([0-9]+)\z/) {
$x_it_code = $1 + 0;
last;
} elsif ($buf =~ /\Achild_error ([0-9]+)\z/) {
$x_it_code = $1 + 0;
} else {
- $sigchld->();
die $buf;
}
}
- $sigchld->();
if (my $sig = ($x_it_code & 127)) {
kill $sig, $$;
sleep(1) while 1;
^ permalink raw reply related [relevance 62%]
* Re: [PATCH 1/2] doc: add lei-mail-formats(5) manpage
2021-03-31 0:41 46% ` [PATCH 1/2] doc: add lei-mail-formats(5) manpage Eric Wong
@ 2021-03-31 3:15 71% ` Kyle Meyer
0 siblings, 0 replies; 200+ results
From: Kyle Meyer @ 2021-03-31 3:15 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
Eric Wong writes:
> While plenty of online documentation exists, it's good to have
> a locally-available summary for users to look at offline.
That's great. Thanks!
(The other patch makes sense to me too.)
^ permalink raw reply [relevance 71%]
* [PATCH] lei blob: "--mail" disables solver, use --include/only
@ 2021-03-31 1:53 54% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-31 1:53 UTC (permalink / raw)
To: meta
Assume a user specifying --mail doesn't want to spend cycles
reconstructing a blob from a code repo. Also, don't require
users to use add-external or a previous -I or --only to ready an
external for use with ale.git.
---
lib/PublicInbox/LeiBlob.pm | 18 +++++++++++++++---
t/solver_git.t | 21 +++++++++++++++------
2 files changed, 30 insertions(+), 9 deletions(-)
diff --git a/lib/PublicInbox/LeiBlob.pm b/lib/PublicInbox/LeiBlob.pm
index 6195b368..ed0754a3 100644
--- a/lib/PublicInbox/LeiBlob.pm
+++ b/lib/PublicInbox/LeiBlob.pm
@@ -90,15 +90,25 @@ sub lei_blob {
$lei->start_pager if -t $lei->{1};
my $opt = $lei->{opt};
my $has_hints = grep(defined, @$opt{qw(oid-a path-a path-b)});
+ my $lxs;
# first, see if it's a blob returned by "lei q" JSON output:k
if ($opt->{mail} // ($has_hints ? 0 : 1)) {
+ if (grep(defined, @$opt{qw(include only)})) {
+ $lxs = $lei->lxs_prepare;
+ $lei->ale->refresh_externals($lxs);
+ }
my $rdr = { 1 => $lei->{1} };
- open $rdr->{2}, '>', '/dev/null' or die "open: $!";
+ if ($opt->{mail}) {
+ $rdr->{2} = $lei->{2};
+ } else {
+ open $rdr->{2}, '>', '/dev/null' or die "open: $!";
+ }
my $cmd = [ 'git', '--git-dir='.$lei->ale->git->{git_dir},
'cat-file', 'blob', $blob ];
waitpid(spawn($cmd, $lei->{env}, $rdr), 0);
return if $? == 0;
+ return $lei->child_error($?) if $opt->{mail};
}
# maybe it's a non-email (code) blob from a coderepo
@@ -108,7 +118,10 @@ sub lei_blob {
unshift(@$git_dirs, $cgd) if defined $cgd;
}
return $lei->fail('no --git-dir to try') unless @$git_dirs;
- my $lxs = $lei->lxs_prepare or return;
+ unless ($lxs) {
+ $lxs = $lei->lxs_prepare or return;
+ $lei->ale->refresh_externals($lxs);
+ }
if ($lxs->remotes) {
require PublicInbox::LeiRemote;
$lei->{curl} //= which('curl') or return
@@ -117,7 +130,6 @@ sub lei_blob {
}
require PublicInbox::SolverGit;
my $self = bless { lxs => $lxs, oid_b => $blob }, __PACKAGE__;
- $lei->ale;
my ($op_c, $ops) = $lei->workers_start($self, 'lei_solve', 1,
{ '' => [ \&sol_done, $lei ] });
$lei->{sol} = $self;
diff --git a/t/solver_git.t b/t/solver_git.t
index 2d803d47..8acf907f 100644
--- a/t/solver_git.t
+++ b/t/solver_git.t
@@ -6,7 +6,8 @@ use v5.10.1;
use PublicInbox::TestCommon;
use Cwd qw(abs_path);
require_git(2.6);
-use Digest::SHA qw(sha1_hex);
+use PublicInbox::ContentHash qw(git_sha);
+use PublicInbox::Eml;
use PublicInbox::Spawn qw(popen_rd which);
require_mods(qw(DBD::SQLite Search::Xapian Plack::Util));
my $git_dir = xqx([qw(git rev-parse --git-dir)], undef, {2 => \(my $null)});
@@ -17,14 +18,15 @@ chomp $git_dir;
$git_dir = abs_path($git_dir);
use_ok "PublicInbox::$_" for (qw(Inbox V2Writable Git SolverGit WWW));
+my $patch2 = eml_load 't/solve/0002-rename-with-modifications.patch';
+my $patch2_oid = git_sha(1, $patch2)->hexdigest;
my ($tmpdir, $for_destroy) = tmpdir();
my $ibx = create_inbox 'v2', version => 2,
indexlevel => 'medium', sub {
my ($im) = @_;
$im->add(eml_load 't/solve/0001-simple-mod.patch') or BAIL_OUT;
- $im->add(eml_load 't/solve/0002-rename-with-modifications.patch') or
- BAIL_OUT;
+ $im->add($patch2) or BAIL_OUT;
};
my $v1_0_0_tag = 'cb7c42b1e15577ed2215356a2bf925aef59cdd8d';
my $v1_0_0_tag_short = substr($v1_0_0_tag, 0, 16);
@@ -32,8 +34,14 @@ my $expect = '69df7d565d49fbaaeb0a067910f03dc22cd52bd0';
my $non_existent = 'ee5e32211bf62ab6531bdf39b84b6920d0b6775a';
test_lei({tmpdir => $tmpdir}, sub {
+ lei_ok('blob', '--mail', $patch2_oid, '-I', $ibx->{inboxdir},
+ \'--mail works for existing oid');
+ is($lei_out, $patch2->as_string, 'blob matches');
+ ok(!lei('blob', '--mail', '69df7d5', '-I', $ibx->{inboxdir}),
+ "--mail won't run solver");
+
lei_ok('blob', '69df7d5', '-I', $ibx->{inboxdir});
- is(sha1_hex("blob ".length($lei_out)."\0".$lei_out),
+ is(git_sha(1, PublicInbox::Eml->new($lei_out))->hexdigest,
$expect, 'blob contents output');
my $prev = $lei_out;
lei_ok(qw(blob --no-mail 69df7d5 -I), $ibx->{inboxdir});
@@ -236,8 +244,9 @@ EOF
test_lei({tmpdir => "$tmpdir/ext"}, sub {
my $rurl = "$url/$name";
lei_ok(qw(blob --no-mail 69df7d5 -I), $rurl);
- is(sha1_hex("blob ".length($lei_out)."\0".$lei_out),
- $expect, 'blob contents output');
+ my $eml = PublicInbox::Eml->new($lei_out);
+ is(git_sha(1, $eml)->hexdigest, $expect,
+ 'blob contents output');
ok(!lei(qw(blob -I), $rurl, $non_existent),
'non-existent blob fails');
});
^ permalink raw reply related [relevance 54%]
* [PATCH 0/2] lei doc: mail formats
@ 2021-03-31 0:41 71% Eric Wong
2021-03-31 0:41 46% ` [PATCH 1/2] doc: add lei-mail-formats(5) manpage Eric Wong
2021-03-31 0:41 66% ` [PATCH 2/2] doc: lei-overview: favor Maildir for mutt examples Eric Wong
0 siblings, 2 replies; 200+ results
From: Eric Wong @ 2021-03-31 0:41 UTC (permalink / raw)
To: meta
Eric Wong (2):
doc: add lei-mail-formats(5) manpage
doc: lei-overview: favor Maildir for mutt examples
Documentation/lei-mail-formats.pod | 101 +++++++++++++++++++++++++++++
Documentation/lei-overview.pod | 11 ++--
MANIFEST | 1 +
Makefile.PL | 4 +-
lib/PublicInbox/Watch.pm | 2 +-
5 files changed, 112 insertions(+), 7 deletions(-)
create mode 100644 Documentation/lei-mail-formats.pod
^ permalink raw reply [relevance 71%]
* [PATCH 2/2] doc: lei-overview: favor Maildir for mutt examples
2021-03-31 0:41 71% [PATCH 0/2] lei doc: mail formats Eric Wong
2021-03-31 0:41 46% ` [PATCH 1/2] doc: add lei-mail-formats(5) manpage Eric Wong
@ 2021-03-31 0:41 66% ` Eric Wong
1 sibling, 0 replies; 200+ results
From: Eric Wong @ 2021-03-31 0:41 UTC (permalink / raw)
To: meta
mboxes are generally horrible for interactive read-write use due
to locking. Describe our parallel behavior with mutt, since
writing mail can take a long while and being able to read
results as they're written is nice.
We'll also use a gzipped mboxrd for the import example, since
we can decompress gzipped mboxrds automatically, now.
---
Documentation/lei-overview.pod | 11 ++++++-----
1 file changed, 6 insertions(+), 5 deletions(-)
diff --git a/Documentation/lei-overview.pod b/Documentation/lei-overview.pod
index c1f952c9..70dbf2b5 100644
--- a/Documentation/lei-overview.pod
+++ b/Documentation/lei-overview.pod
@@ -23,9 +23,9 @@ to memoize messages from remotes.
=over
-=item $ lei import mboxrd:t.mbox
+=item $ lei import mboxrd:t.mbox.gz
-Import the messages from an mbox into the local storage.
+Import the messages from a gzipped mboxrd into the local storage.
=item $ lei blob 59ec517f9
@@ -82,10 +82,11 @@ Search for messages whose subject includes "lei" and "skeleton".
Do the same, but also report unmatched messages that are in the same
thread as a matched message.
-=item $ lei q -t -o mboxcl2:t.mbox --mua=mutt s:lei s:skeleton
+=item $ lei q -t -o mdir --mua=mutt s:lei s:skeleton
-Write mboxcl2-formatted results to t.mbox and enter mutt to view the
-file by invoking C<mutt -f %f>.
+Write results to a Maildir at "mdir". Mutt will be invoked
+to open mfolder (C<mutt -f %f>) while results are being fetched
+and written.
=item $ lei q kw:flagged L:next
^ permalink raw reply related [relevance 66%]
* [PATCH 1/2] doc: add lei-mail-formats(5) manpage
2021-03-31 0:41 71% [PATCH 0/2] lei doc: mail formats Eric Wong
@ 2021-03-31 0:41 46% ` Eric Wong
2021-03-31 3:15 71% ` Kyle Meyer
2021-03-31 0:41 66% ` [PATCH 2/2] doc: lei-overview: favor Maildir for mutt examples Eric Wong
1 sibling, 1 reply; 200+ results
From: Eric Wong @ 2021-03-31 0:41 UTC (permalink / raw)
To: meta
While plenty of online documentation exists, it's good to have
a locally-available summary for users to look at offline.
Fix a URL in Watch.pm while we're at it, too.
---
Documentation/lei-mail-formats.pod | 101 +++++++++++++++++++++++++++++
MANIFEST | 1 +
Makefile.PL | 4 +-
lib/PublicInbox/Watch.pm | 2 +-
4 files changed, 106 insertions(+), 2 deletions(-)
create mode 100644 Documentation/lei-mail-formats.pod
diff --git a/Documentation/lei-mail-formats.pod b/Documentation/lei-mail-formats.pod
new file mode 100644
index 00000000..8accedb4
--- /dev/null
+++ b/Documentation/lei-mail-formats.pod
@@ -0,0 +1,101 @@
+=head1 NAME
+
+lei-mail-formats - description of mail formats supported by lei
+
+=head1 DESCRIPTION
+
+L<lei-q(1)> supports writing to several existing mail formats
+for interoperability with existing mail user agents (MUA);
+below is an overview of them to help users choose.
+
+=head1 Maildir
+
+The default output format when given a filesystem path, it supports
+parallel read-write access. Performance is acceptable for smaller
+directories, but degrades as mailboxes get larger. Speed and
+scalability are limited by kernel and filesystem performance
+due to the use of small files and large number of syscalls.
+
+See also: L<https://cr.yp.to/proto/maildir.html> and
+L<https://wiki2.dovecot.org/MailboxFormat/Maildir>
+
+=head1 Mbox family
+
+The mbox family consists of several incompatible formats.
+Locking for parallel access is supported, but may not be
+compatible across tools. With compression (e.g. L<gzip(1)>),
+they require the least amount of space while offering good
+read-only performance.
+
+Keyword updates (C<Status:> and/or C<X-Status:> headers)
+generally require rewriting the entire mbox.
+
+See also:
+L<https://www.loc.gov/preservation/digital/formats/fdd/fdd000383.shtml>,
+L<mbox(5)>
+
+=head2 mboxo
+
+The traditional BSD format. It quotes C<From > to C<E<gt>From >,
+but lines already beginning with C<E<gt>From > do not get quoted,
+thus automatic reversibility is not guaranteed. MUAs which favor
+L</mboxcl> or L</mboxcl2> may convert these automatically to their
+preferred format.
+
+Truncation is undetectable unless compressed with gzip or similar.
+
+=head2 mboxrd
+
+An evolution of L</mboxo>, but quotes C<From > lines prefixed
+with any number of C<E<gt>> characters and is thus fully
+reversible.
+
+This format is emitted by L<PublicInbox::WWW(3pm)> with gzip.
+It is supported by L<git-am(1)> since git 2.10.
+
+As with uncompressed L</mboxo>, uncompressed mboxrd are vulnerable
+to undetectable truncation.
+
+It gracefully degrades to being treated as L</mboxo> by MUAs
+unaware of the format as excessive C<E<gt>From > quoting is
+recognizable to humans.
+
+=head2 mboxcl
+
+L</mboxo> with a C<Content-Length:> header, C<From > lines
+remain quoted to retain readability with L</mboxo> and L</mboxrd> MUAs.
+However, it is easy to corrupt these files when using tools
+which are not aware of C<Content-Length:> and write out updates
+as L</mboxo>.
+
+L<mutt(1)> will convert L</mboxo> and L</mboxrd> to mboxcl upon opening.
+
+See also: L<https://www.jwz.org/doc/content-length.html>
+
+=head2 mboxcl2
+
+Like L</mboxcl>, but without C<From > any quoting. It is wholly
+incompatible with MUAs which only handle L</mboxo> and/or L</mboxrd>.
+This is format is generated by L<mutt(1)> when writing to a new
+mbox.
+
+=head1 MH
+
+Not yet supported, locking semantics (or lack thereof) appear to
+make it unsuitable for parallel access.
+
+=head1 IMAP
+
+Depending on the IMAP server software and configuration, IMAP
+servers may use any (or combination) of the aforementioned
+formats or a non-standard database backend.
+
+=head1 COPYRIGHT
+
+Copyright 2021 all contributors L<mailto:meta@public-inbox.org>
+
+License: AGPL-3.0+ L<http://www.gnu.org/licenses/agpl-3.0.txt>
+
+=head1 SEE ALSO
+
+L<lei(1)>, L<lei-q(1)>, L<lei-convert(1)>, L<lei-overview(7)>
diff --git a/MANIFEST b/MANIFEST
index f3cb0147..49d273fc 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -31,6 +31,7 @@ Documentation/lei-import.pod
Documentation/lei-init.pod
Documentation/lei-ls-external.pod
Documentation/lei-ls-label.pod
+Documentation/lei-mail-formats.pod
Documentation/lei-overview.pod
Documentation/lei-p2q.pod
Documentation/lei-q.pod
diff --git a/Makefile.PL b/Makefile.PL
index 27b49c53..feb89ec1 100644
--- a/Makefile.PL
+++ b/Makefile.PL
@@ -48,7 +48,9 @@ $v->{-m1} = [ map {
lei-forget-external lei-import lei-init lei-ls-external lei-ls-label
lei-tag lei-p2q lei-q)];
$v->{-m5} = [ qw(public-inbox-config public-inbox-v1-format
- public-inbox-v2-format public-inbox-extindex-format) ];
+ public-inbox-v2-format public-inbox-extindex-format
+ lei-mail-formats
+ ) ];
$v->{-m7} = [ qw(lei-overview public-inbox-overview public-inbox-tuning
public-inbox-glossary) ];
$v->{-m8} = [ qw(public-inbox-daemon) ];
diff --git a/lib/PublicInbox/Watch.pm b/lib/PublicInbox/Watch.pm
index 4fbc9640..05956cbb 100644
--- a/lib/PublicInbox/Watch.pm
+++ b/lib/PublicInbox/Watch.pm
@@ -2,7 +2,7 @@
# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
#
# ref: https://cr.yp.to/proto/maildir.html
-# httsp://wiki2.dovecot.org/MailboxFormat/Maildir
+# https://wiki2.dovecot.org/MailboxFormat/Maildir
package PublicInbox::Watch;
use strict;
use v5.10.1;
^ permalink raw reply related [relevance 46%]
* [PATCH] lei: fix IMAP auth failure handling
@ 2021-03-30 22:29 62% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-30 22:29 UTC (permalink / raw)
To: meta
We must use the $ops hashref returned by lei->workers_start,
since it's modified to include extra handlers for auth failures
and whatnot.
Fixes: 954581b8e575966a ("lei: simplify PktOp callers")
---
lib/PublicInbox/LeiImport.pm | 2 +-
lib/PublicInbox/LeiTag.pm | 2 +-
xt/lei-auth-fail.t | 15 +++++++--------
3 files changed, 9 insertions(+), 10 deletions(-)
diff --git a/lib/PublicInbox/LeiImport.pm b/lib/PublicInbox/LeiImport.pm
index 227a2a21..dbf655b6 100644
--- a/lib/PublicInbox/LeiImport.pm
+++ b/lib/PublicInbox/LeiImport.pm
@@ -76,7 +76,7 @@ sub lei_import { # the main "lei import" method
my $ops = { '' => [ \&import_done, $lei ] };
$lei->{auth}->op_merge($ops, $self) if $lei->{auth};
$self->{-wq_nr_workers} = $j // 1; # locked
- my ($op_c, undef) = $lei->workers_start($self, 'lei_import', $j, $ops);
+ (my $op_c, $ops) = $lei->workers_start($self, 'lei_import', $j, $ops);
$lei->{imp} = $self;
net_merge_complete($self) unless $lei->{auth};
$op_c->op_wait_event($ops);
diff --git a/lib/PublicInbox/LeiTag.pm b/lib/PublicInbox/LeiTag.pm
index 56ac25fa..8b012b16 100644
--- a/lib/PublicInbox/LeiTag.pm
+++ b/lib/PublicInbox/LeiTag.pm
@@ -116,7 +116,7 @@ sub lei_tag { # the "lei tag" method
my $ops = { '' => [ \&tag_done, $lei ] };
$lei->{auth}->op_merge($ops, $self) if $lei->{auth};
$self->{vmd_mod} = $vmd_mod;
- my ($op_c, undef) = $lei->workers_start($self, 'lei_tag', 1, $ops);
+ (my $op_c, $ops) = $lei->workers_start($self, 'lei_tag', 1, $ops);
$lei->{tag} = $self;
net_merge_complete($self) unless $lei->{auth};
$op_c->op_wait_event($ops);
diff --git a/xt/lei-auth-fail.t b/xt/lei-auth-fail.t
index 78f8466d..e352aab3 100644
--- a/xt/lei-auth-fail.t
+++ b/xt/lei-auth-fail.t
@@ -9,13 +9,12 @@ require_mods(qw(Mail::IMAPClient));
my $imap_fail = $ENV{TEST_LEI_IMAP_FAIL_URL} //
'imaps://AzureDiamond:Hunter2@public-inbox.org:994/INBOX';
test_lei(sub {
- ok(!lei(qw(convert -o mboxrd:/dev/stdout), $imap_fail),
- 'IMAP auth failure on convert');
- like($lei_err, qr!\bE:.*?imaps://.*?!sm, 'error shown');
- unlike($lei_err, qr!Hunter2!s, 'password not shown');
- is($lei_out, '', 'nothing output');
- ok(!lei(qw(import), $imap_fail), 'IMAP auth failure on import');
- like($lei_err, qr!\bE:.*?imaps://.*?!sm, 'error shown');
- unlike($lei_err, qr!Hunter2!s, 'password not shown');
+ for my $pfx ([qw(convert -o mboxrd:/dev/stdout)], ['import'],
+ [qw(tag +L:INBOX)]) {
+ ok(!lei(@$pfx, $imap_fail), "IMAP auth failure on @$pfx");
+ like($lei_err, qr!\bE:.*?imaps://.*?!sm, 'error shown');
+ unlike($lei_err, qr!Hunter2!s, 'password not shown');
+ is($lei_out, '', 'nothing output');
+ }
});
done_testing;
^ permalink raw reply related [relevance 62%]
* [PATCH] lei tag: rename from "lei mark"
@ 2021-03-30 9:39 51% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-30 9:39 UTC (permalink / raw)
To: meta
I've decided "tag" is a better verb since it seems more
widely-used term for associating metadata with data.
Not only is it analogous to the "notmuch tag" command, but
also makes sense when compared to tooling for manipulating
metadata for non-mail data (e.g. audio metadata tags).
There's even a Wikipedia entry for it:
https://en.wikipedia.org/wiki/Tag_(metadata)
whereas "mark" is used in the description, but has no
entry of its own with regards to metadata.
---
Documentation/lei-overview.pod | 2 +-
Documentation/{lei-mark.pod => lei-tag.pod} | 6 ++---
Documentation/lei.pod | 2 +-
Documentation/txt2pre | 2 +-
MANIFEST | 6 ++---
Makefile.PL | 2 +-
lib/PublicInbox/LEI.pm | 6 ++---
lib/PublicInbox/{LeiMark.pm => LeiTag.pm} | 26 ++++++++++-----------
t/{lei-mark.t => lei-tag.t} | 22 ++++++++---------
9 files changed, 37 insertions(+), 37 deletions(-)
rename Documentation/{lei-mark.pod => lei-tag.pod} (89%)
rename lib/PublicInbox/{LeiMark.pm => LeiTag.pm} (90%)
rename t/{lei-mark.t => lei-tag.t} (83%)
diff --git a/Documentation/lei-overview.pod b/Documentation/lei-overview.pod
index f74a228a..c1f952c9 100644
--- a/Documentation/lei-overview.pod
+++ b/Documentation/lei-overview.pod
@@ -33,7 +33,7 @@ Show message with the git blob OID of 59ec517f9. If a message with
that OID isn't found, check if the current git repository has the
blob, trying to reconstruct it from a message if needed.
-=item $ lei blob 59ec517f9 | lei mark - -F eml +kw:flagged +L:next
+=item $ lei blob 59ec517f9 | lei tag - -F eml +kw:flagged +L:next
Set the "flagged" keyword and "next" label on the message with the
blob OID of 59ec517f9.
diff --git a/Documentation/lei-mark.pod b/Documentation/lei-tag.pod
similarity index 89%
rename from Documentation/lei-mark.pod
rename to Documentation/lei-tag.pod
index 8ef1dce2..a07738d7 100644
--- a/Documentation/lei-mark.pod
+++ b/Documentation/lei-tag.pod
@@ -1,12 +1,12 @@
=head1 NAME
-lei-mark - set/unset metadata on messages
+lei-tag - set/unset metadata on messages
=head1 SYNOPSIS
-lei mark [OPTIONS] FILE [FILE...] METADATA [METADATA...]
+lei tag [OPTIONS] FILE [FILE...] METADATA [METADATA...]
-lei mark [OPTIONS] (-|--stdin) METADATA [METADATA...]
+lei tag [OPTIONS] (-|--stdin) METADATA [METADATA...]
=head1 DESCRIPTION
diff --git a/Documentation/lei.pod b/Documentation/lei.pod
index 248e5931..805e5a75 100644
--- a/Documentation/lei.pod
+++ b/Documentation/lei.pod
@@ -48,7 +48,7 @@ Subcommands for initializing and managing local, writable storage:
=item * L<lei-import(1)>
-=item * L<lei-mark(1)>
+=item * L<lei-tag(1)>
=back
diff --git a/Documentation/txt2pre b/Documentation/txt2pre
index bfffdef1..7b9d7853 100755
--- a/Documentation/txt2pre
+++ b/Documentation/txt2pre
@@ -21,7 +21,7 @@ for (qw[lei(1)
lei-init(1)
lei-ls-external(1)
lei-ls-label(1)
- lei-mark(1)
+ lei-tag(1)
lei-overview(7)
lei-p2q(1)
lei-q(1)
diff --git a/MANIFEST b/MANIFEST
index 3d521a64..f3cb0147 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -31,10 +31,10 @@ Documentation/lei-import.pod
Documentation/lei-init.pod
Documentation/lei-ls-external.pod
Documentation/lei-ls-label.pod
-Documentation/lei-mark.pod
Documentation/lei-overview.pod
Documentation/lei-p2q.pod
Documentation/lei-q.pod
+Documentation/lei-tag.pod
Documentation/lei.pod
Documentation/marketing.txt
Documentation/mknews.perl
@@ -195,7 +195,6 @@ lib/PublicInbox/LeiImport.pm
lib/PublicInbox/LeiInit.pm
lib/PublicInbox/LeiInput.pm
lib/PublicInbox/LeiLsLabel.pm
-lib/PublicInbox/LeiMark.pm
lib/PublicInbox/LeiMirror.pm
lib/PublicInbox/LeiOverview.pm
lib/PublicInbox/LeiP2q.pm
@@ -203,6 +202,7 @@ lib/PublicInbox/LeiQuery.pm
lib/PublicInbox/LeiRemote.pm
lib/PublicInbox/LeiSearch.pm
lib/PublicInbox/LeiStore.pm
+lib/PublicInbox/LeiTag.pm
lib/PublicInbox/LeiToMail.pm
lib/PublicInbox/LeiXSearch.pm
lib/PublicInbox/Linkify.pm
@@ -386,12 +386,12 @@ t/lei-import-imap.t
t/lei-import-maildir.t
t/lei-import-nntp.t
t/lei-import.t
-t/lei-mark.t
t/lei-mirror.t
t/lei-p2q.t
t/lei-q-kw.t
t/lei-q-remote-import.t
t/lei-q-thread.t
+t/lei-tag.t
t/lei.t
t/lei_dedupe.t
t/lei_external.t
diff --git a/Makefile.PL b/Makefile.PL
index cdb67214..27b49c53 100644
--- a/Makefile.PL
+++ b/Makefile.PL
@@ -46,7 +46,7 @@ $v->{-m1} = [ map {
qw(
lei-add-external lei-blob lei-config lei-daemon-kill lei-daemon-pid
lei-forget-external lei-import lei-init lei-ls-external lei-ls-label
- lei-mark lei-p2q lei-q)];
+ lei-tag lei-p2q lei-q)];
$v->{-m5} = [ qw(public-inbox-config public-inbox-v1-format
public-inbox-v2-format public-inbox-extindex-format) ];
$v->{-m7} = [ qw(lei-overview public-inbox-overview public-inbox-tuning
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 8a07a4c8..69d48bd1 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -160,8 +160,8 @@ our %CMD = ( # sorted in order of importance/use:
'plonk' => [ '--threads|--from=IDENT',
'exclude mail matching From: or threads from non-Message-ID searches',
qw(stdin| threads|t from|f=s mid=s oid=s), @c_opt ],
-'mark' => [ 'KEYWORDS...',
- 'set/unset keywords on message(s)',
+'tag' => [ 'KEYWORDS...',
+ 'set/unset keywords and/or labels on message(s)',
qw(stdin| in-format|F=s input|i=s@ oid=s@ mid=s@), @c_opt,
pass_through('-kw:foo for delete') ],
'forget' => [ '[--stdin|--oid=OID|--by-mid=MID]',
@@ -348,7 +348,7 @@ my %CONFIG_KEYS = (
'leistore.dir' => 'top-level storage location',
);
-my @WQ_KEYS = qw(lxs l2m imp mrr cnv p2q mark sol); # internal workers
+my @WQ_KEYS = qw(lxs l2m imp mrr cnv p2q tag sol); # internal workers
# pronounced "exit": x_it(1 << 8) => exit(1); x_it(13) => SIGPIPE
sub x_it ($$) {
diff --git a/lib/PublicInbox/LeiMark.pm b/lib/PublicInbox/LeiTag.pm
similarity index 90%
rename from lib/PublicInbox/LeiMark.pm
rename to lib/PublicInbox/LeiTag.pm
index b187d6e7..56ac25fa 100644
--- a/lib/PublicInbox/LeiMark.pm
+++ b/lib/PublicInbox/LeiTag.pm
@@ -1,8 +1,8 @@
# Copyright (C) 2021 all contributors <meta@public-inbox.org>
# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
-# handles "lei mark" command
-package PublicInbox::LeiMark;
+# handles "lei tag" command
+package PublicInbox::LeiTag;
use strict;
use v5.10.1;
use parent qw(PublicInbox::IPC PublicInbox::LeiInput);
@@ -69,19 +69,19 @@ sub input_eml_cb { # used by PublicInbox::LeiInput::input_fh
sub input_mbox_cb { input_eml_cb($_[1], $_[0]) }
-sub mark_done_wait { # dwaitpid callback
+sub tag_done_wait { # dwaitpid callback
my ($arg, $pid) = @_;
- my ($mark, $lei) = @$arg;
- $lei->child_error($?, 'non-fatal errors during mark') if $?;
+ my ($tag, $lei) = @$arg;
+ $lei->child_error($?, 'non-fatal errors during tag') if $?;
my $sto = delete $lei->{sto};
my $wait = $sto->ipc_do('done') if $sto; # PublicInbox::LeiStore::done
$lei->dclose;
}
-sub mark_done { # EOF callback for main daemon
+sub tag_done { # EOF callback for main daemon
my ($lei) = @_;
- my $mark = delete $lei->{mark} or return;
- $mark->wq_wait_old(\&mark_done_wait, $lei);
+ my $tag = delete $lei->{tag} or return;
+ $tag->wq_wait_old(\&tag_done_wait, $lei);
}
sub net_merge_complete { # callback used by LeiAuth
@@ -102,7 +102,7 @@ sub input_net_cb { # imap_each, nntp_each cb
input_eml_cb($self, $eml);
}
-sub lei_mark { # the "lei mark" method
+sub lei_tag { # the "lei tag" method
my ($lei, @argv) = @_;
my $sto = $lei->_lei_store(1);
$sto->write_prepare($lei);
@@ -113,11 +113,11 @@ sub lei_mark { # the "lei mark" method
$self->prepare_inputs($lei, \@argv) or return;
grep(defined, @$vmd_mod{qw(+kw +L -L -kw)}) or
return $lei->fail('no keywords or labels specified');
- my $ops = { '' => [ \&mark_done, $lei ] };
+ my $ops = { '' => [ \&tag_done, $lei ] };
$lei->{auth}->op_merge($ops, $self) if $lei->{auth};
$self->{vmd_mod} = $vmd_mod;
- my ($op_c, undef) = $lei->workers_start($self, 'lei_mark', 1, $ops);
- $lei->{mark} = $self;
+ my ($op_c, undef) = $lei->workers_start($self, 'lei_tag', 1, $ops);
+ $lei->{tag} = $self;
net_merge_complete($self) unless $lei->{auth};
$op_c->op_wait_event($ops);
}
@@ -165,7 +165,7 @@ sub _complete_mark_common ($) {
}
# FIXME: same problems as _complete_forget_external and similar
-sub _complete_mark {
+sub _complete_tag {
my ($self, @argv) = @_;
my @L = eval { $self->_lei_store->search->all_terms('L') };
my @all = ((map { ("+kw:$_", "-kw:$_") } @KW),
diff --git a/t/lei-mark.t b/t/lei-tag.t
similarity index 83%
rename from t/lei-mark.t
rename to t/lei-tag.t
index 98652c85..5cb6d9ce 100644
--- a/t/lei-mark.t
+++ b/t/lei-tag.t
@@ -27,13 +27,13 @@ my $check_kw = sub {
test_lei(sub {
lei_ok(qw(ls-label)); is($lei_out, '', 'no labels, yet');
lei_ok(qw(import t/utf8.eml));
- lei_ok(qw(mark t/utf8.eml +kw:flagged +L:urgent));
+ lei_ok(qw(tag t/utf8.eml +kw:flagged +L:urgent));
$check_kw->(['flagged'], L => ['urgent']);
lei_ok(qw(ls-label)); is($lei_out, "urgent\n", 'label found');
- ok(!lei(qw(mark -F eml t/utf8.eml +kw:seeen)), 'bad kw rejected');
+ ok(!lei(qw(tag -F eml t/utf8.eml +kw:seeen)), 'bad kw rejected');
like($lei_err, qr/`seeen' is not one of/, 'got helpful error');
- ok(!lei(qw(mark -F eml t/utf8.eml +k:seen)), 'bad prefix rejected');
- ok(!lei(qw(mark -F eml t/utf8.eml)), 'no keywords');
+ ok(!lei(qw(tag -F eml t/utf8.eml +k:seen)), 'bad prefix rejected');
+ ok(!lei(qw(tag -F eml t/utf8.eml)), 'no keywords');
my $mb = "$ENV{HOME}/mb";
my $md = "$ENV{HOME}/md";
lei_ok(qw(q m:testmessage@example.com -o), "mboxrd:$mb");
@@ -43,15 +43,15 @@ test_lei(sub {
scalar(@fn) == 1 or xbail $lei_err, 'no mail', \@fn;
rename($fn[0], "$fn[0]S") or BAIL_OUT "rename $!";
$check_kw->(['flagged'], msg => 'after bad request');
- lei_ok(qw(mark -F eml t/utf8.eml -kw:flagged));
+ lei_ok(qw(tag -F eml t/utf8.eml -kw:flagged));
$check_kw->(undef, msg => 'keyword cleared');
- lei_ok(qw(mark -F mboxrd +kw:seen), $mb);
+ lei_ok(qw(tag -F mboxrd +kw:seen), $mb);
$check_kw->(['seen'], msg => 'mbox Status ignored');
- lei_ok(qw(mark -kw:seen +kw:answered), $md);
+ lei_ok(qw(tag -kw:seen +kw:answered), $md);
$check_kw->(['answered'], msg => 'Maildir Status ignored');
open my $in, '<', 't/utf8.eml' or BAIL_OUT $!;
- lei_ok([qw(mark -F eml - +kw:seen +L:nope)],
+ lei_ok([qw(tag -F eml - +kw:seen +L:nope)],
undef, { %$lei_opt, 0 => $in });
$check_kw->(['answered', 'seen'], msg => 'stdin works');
lei_ok(qw(q L:urgent));
@@ -62,7 +62,7 @@ test_lei(sub {
is_deeply($r2, $res, 'kw: query works, too') or
diag explain([$r2, $res]);
- lei_ok(qw(_complete lei mark));
+ lei_ok(qw(_complete lei tag));
my %c = map { $_ => 1 } split(/\s+/, $lei_out);
ok($c{'+L:urgent'} && $c{'-L:urgent'} &&
$c{'+L:nope'} && $c{'-L:nope'}, 'completed with labels');
@@ -70,7 +70,7 @@ test_lei(sub {
my $mid = 'qp@example.com';
lei_ok qw(q -f mboxrd --only), "$ro_home/t2", "mid:$mid";
$in = $lei_out;
- lei_ok [qw(mark -F mboxrd --stdin +kw:seen +L:qp)],
+ lei_ok [qw(tag -F mboxrd --stdin +kw:seen +L:qp)],
undef, { %$lei_opt, 0 => \$in };
$check_kw->(['seen'], L => ['qp'], mid => $mid,
args => [ '--only', "$ro_home/t2" ],
@@ -78,7 +78,7 @@ test_lei(sub {
lei_ok(qw(ls-label));
is($lei_out, "nope\nqp\nurgent\n", 'ls-label shows qp');
- lei_ok qw(mark -F eml t/utf8.eml +L:INBOX +L:x); diag $lei_err;
+ lei_ok qw(tag -F eml t/utf8.eml +L:INBOX +L:x); diag $lei_err;
lei_ok qw(q m:testmessage@example.com);
$check_kw->([qw(answered seen)], L => [qw(INBOX nope urgent x)]);
lei_ok(qw(ls-label));
^ permalink raw reply related [relevance 51%]
* [PATCH] lei q: avoid redundant default setting for sort with l2m
@ 2021-03-30 9:10 70% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-30 9:10 UTC (permalink / raw)
To: meta
No point in munging user-supplied $lei->{opt} when %mset_opt
exists. We'll be depending on docid being in descending order
for saved search support.
---
lib/PublicInbox/LeiOverview.pm | 2 --
lib/PublicInbox/LeiQuery.pm | 4 ++--
2 files changed, 2 insertions(+), 4 deletions(-)
diff --git a/lib/PublicInbox/LeiOverview.pm b/lib/PublicInbox/LeiOverview.pm
index 68f6c792..cdd9ee04 100644
--- a/lib/PublicInbox/LeiOverview.pm
+++ b/lib/PublicInbox/LeiOverview.pm
@@ -101,8 +101,6 @@ sub new {
if ($json) {
$lei->{dedupe} //= PublicInbox::LeiDedupe->new($lei);
} else {
- # default to the cheapest sort since MUA usually resorts
- $opt->{'sort'} //= 'docid' if $devfd < 0;
$lei->{l2m} = eval { PublicInbox::LeiToMail->new($lei) };
return $lei->fail($@) if $@;
if ($opt->{mua} && $lei->{l2m}->lock_free) {
diff --git a/lib/PublicInbox/LeiQuery.pm b/lib/PublicInbox/LeiQuery.pm
index 5376c7f8..3a437bf0 100644
--- a/lib/PublicInbox/LeiQuery.pm
+++ b/lib/PublicInbox/LeiQuery.pm
@@ -128,8 +128,8 @@ sub lei_q {
die "unrecognized --sort=$sort\n";
}
}
- # descending docid order
- $mset_opt{relevance} //= -2 if $opt->{threads};
+ # descending docid order is cheapest, MUA controls sorting order
+ $mset_opt{relevance} //= -2 if $self->{l2m} || $opt->{threads};
$self->{mset_opt} = \%mset_opt;
if ($opt->{stdin}) {
^ permalink raw reply related [relevance 70%]
* [PATCH] lei blob: cleanup solver tmpdir on failure
@ 2021-03-29 17:47 71% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-29 17:47 UTC (permalink / raw)
To: meta
$lei->fail sends SIGTERM which prevents the File::Temp::Dir in
$solver->{tmp} from being cleaned up, so use $lei->child_error
instead.
---
lib/PublicInbox/LeiBlob.pm | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/lib/PublicInbox/LeiBlob.pm b/lib/PublicInbox/LeiBlob.pm
index 91098a90..6195b368 100644
--- a/lib/PublicInbox/LeiBlob.pm
+++ b/lib/PublicInbox/LeiBlob.pm
@@ -39,7 +39,7 @@ sub solver_user_cb { # called by solver when done
my $lei = $self->{lei};
my $log_buf = delete $lei->{'log_buf'};
$$log_buf =~ s/^/# /sgm;
- ref($res) eq 'ARRAY' or return $lei->fail($$log_buf);
+ ref($res) eq 'ARRAY' or return $lei->child_error(1 << 8, $$log_buf);
$lei->qerr($$log_buf);
my ($git, $oid, $type, $size, $di) = @$res;
my $gd = $git->{git_dir};
^ permalink raw reply related [relevance 71%]
* [PATCH 1/4] doc: lei q: drop NNTP from --output description
@ 2021-03-29 8:04 71% ` Eric Wong
2021-03-29 8:04 71% ` [PATCH 2/4] doc: lei q: add warning for --output clobbering Eric Wong
` (2 subsequent siblings)
3 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-29 8:04 UTC (permalink / raw)
To: meta
We only support NNTP as inputs for convert, import, and
mark|tag. I'm not sure if supporting NNTP output is worth
it, nor do we have a good way to test it.
---
Documentation/lei-q.pod | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/Documentation/lei-q.pod b/Documentation/lei-q.pod
index 787c51bf..8668533a 100644
--- a/Documentation/lei-q.pod
+++ b/Documentation/lei-q.pod
@@ -31,7 +31,7 @@ Read search terms from stdin.
Destination for results (e.g., C<path/to/Maildir>,
C<imaps://user@mail.example.com/INBOX.test>, or
C<mboxcl2:path/to/mbox>). The prefix may be a supported protocol:
-C<imap://>, C<imaps://>, C<nntp://>, or C<nntps://>. URLs requiring
+C<imap://> or C<imaps://>. URLs requiring
authentication must use L<netrc(5)> and/or L<git-credential(1)> to
fill in the username and password.
^ permalink raw reply related [relevance 71%]
* [PATCH 2/4] doc: lei q: add warning for --output clobbering
2021-03-29 8:04 71% ` [PATCH 1/4] doc: lei q: drop NNTP from --output description Eric Wong
@ 2021-03-29 8:04 71% ` Eric Wong
2021-03-29 8:04 71% ` [PATCH 3/4] doc: lei q: clarify default output as stdout Eric Wong
2021-03-29 8:04 67% ` [PATCH 4/4] doc: lei: update description, add warnings Eric Wong
3 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-29 8:04 UTC (permalink / raw)
To: meta
The behavior matching mairix still frightens me a bit when it
comes to supporting new users. On the other hand, I've rarely
ever used --augment with mairix, so I still think the current
(dangerous) behavior makes sense in the context of search results.
---
Documentation/lei-q.pod | 3 +++
1 file changed, 3 insertions(+)
diff --git a/Documentation/lei-q.pod b/Documentation/lei-q.pod
index 8668533a..8d5053cd 100644
--- a/Documentation/lei-q.pod
+++ b/Documentation/lei-q.pod
@@ -28,6 +28,9 @@ Read search terms from stdin.
=item -o MFOLDER, --output=MFOLDER, --mfolder=MFOLDER
+Warning: this clobbers and overwrites the output destination unless
+L</-a, --augment> is specified.
+
Destination for results (e.g., C<path/to/Maildir>,
C<imaps://user@mail.example.com/INBOX.test>, or
C<mboxcl2:path/to/mbox>). The prefix may be a supported protocol:
^ permalink raw reply related [relevance 71%]
* [PATCH 3/4] doc: lei q: clarify default output as stdout
2021-03-29 8:04 71% ` [PATCH 1/4] doc: lei q: drop NNTP from --output description Eric Wong
2021-03-29 8:04 71% ` [PATCH 2/4] doc: lei q: add warning for --output clobbering Eric Wong
@ 2021-03-29 8:04 71% ` Eric Wong
2021-03-29 8:04 67% ` [PATCH 4/4] doc: lei: update description, add warnings Eric Wong
3 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-29 8:04 UTC (permalink / raw)
To: meta
Seeing a plain "-" may be confusing, especially when we also
support it for --stdin. Use the C<> POD directive to denote it
as code, too.
---
Documentation/lei-q.pod | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/Documentation/lei-q.pod b/Documentation/lei-q.pod
index 8d5053cd..a84fc440 100644
--- a/Documentation/lei-q.pod
+++ b/Documentation/lei-q.pod
@@ -48,7 +48,7 @@ non-existing path.
=for comment
TODO: Provide description of formats?
-Default: -
+Default: C<-> (stdout)
=item -f FORMAT, --format=FORMAT
^ permalink raw reply related [relevance 71%]
* [PATCH 4/4] doc: lei: update description, add warnings
` (2 preceding siblings ...)
2021-03-29 8:04 71% ` [PATCH 3/4] doc: lei q: clarify default output as stdout Eric Wong
@ 2021-03-29 8:04 67% ` Eric Wong
3 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-29 8:04 UTC (permalink / raw)
To: meta
Describing the design details and architecture doesn't seem
appropriate for a section 1 manpage.
We'll also add a warning about it being in the early stages.
---
Documentation/lei.pod | 12 +++++++-----
1 file changed, 7 insertions(+), 5 deletions(-)
diff --git a/Documentation/lei.pod b/Documentation/lei.pod
index 6c8cfc3d..248e5931 100644
--- a/Documentation/lei.pod
+++ b/Documentation/lei.pod
@@ -1,6 +1,6 @@
=head1 NAME
-lei - local email interface for public-inbox
+lei - local email interface
=head1 SYNOPSIS
@@ -8,15 +8,17 @@ lei [OPTIONS] COMMAND
=head1 DESCRIPTION
-Unlike the C10K-oriented L<public-inbox-daemon(8)>, lei is designed
-exclusively to handle trusted local clients with read/write access to
-the file system, using as many system resources as the local user has
-access to. lei supports a local, writable store built on top of
+lei is a command-line tool for importing and searching email,
+regardless of whether it is from a personal mailbox or a public-inbox.
+lei supports a local, writable store built on top of
L<public-inbox-v2-format(5)> and L<public-inbox-extindex(1)>.
L<lei-q(1)> provides an interface for querying messages across the lei
store and read-only local and remote "externals" (inboxes and external
indices).
+Warning: lei is still in its early stages and may destroy mail.
+Be sure to have backups of destinations lei writes to.
+
Available in public-inbox 1.7.0+.
=head1 OPTIONS
^ permalink raw reply related [relevance 67%]
* [PATCH 2/3] lei: use IO::Uncompress::Gunzip MultiStream
2021-03-29 7:08 71% [PATCH 0/3] lei input improvements Eric Wong
@ 2021-03-29 7:08 70% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-29 7:08 UTC (permalink / raw)
To: meta
This is compatible with default gunzip(1) behavior and
future-proofs us against potential changes in PublicInbox::WWW
to save memory on public-inbox-httpd instances.
---
lib/PublicInbox/LeiRemote.pm | 2 +-
lib/PublicInbox/LeiXSearch.pm | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/lib/PublicInbox/LeiRemote.pm b/lib/PublicInbox/LeiRemote.pm
index 399fc936..945d9990 100644
--- a/lib/PublicInbox/LeiRemote.pm
+++ b/lib/PublicInbox/LeiRemote.pm
@@ -50,7 +50,7 @@ sub mset {
my ($fh, $pid) = popen_rd($cmd, undef, $rdr);
my $reap = PublicInbox::OnDestroy->new($lei->can('sigint_reap'), $pid);
$self->{smsg} = [];
- $fh = IO::Uncompress::Gunzip->new($fh);
+ $fh = IO::Uncompress::Gunzip->new($fh, MultiStream => 1);
PublicInbox::MboxReader->mboxrd($fh, \&_each_mboxrd_eml, $self);
my $err = waitpid($pid, 0) == $pid ? undef
: "BUG: waitpid($cmd): $!";
diff --git a/lib/PublicInbox/LeiXSearch.pm b/lib/PublicInbox/LeiXSearch.pm
index 1a194f1c..f3b8cc25 100644
--- a/lib/PublicInbox/LeiXSearch.pm
+++ b/lib/PublicInbox/LeiXSearch.pm
@@ -272,7 +272,7 @@ sub query_remote_mboxrd {
$lei->qerr("# $cmd");
my ($fh, $pid) = popen_rd($cmd, undef, $rdr);
$reap_curl = PublicInbox::OnDestroy->new($sigint_reap, $pid);
- $fh = IO::Uncompress::Gunzip->new($fh);
+ $fh = IO::Uncompress::Gunzip->new($fh, MultiStream => 1);
PublicInbox::MboxReader->mboxrd($fh, \&each_remote_eml, $self,
$lei, $each_smsg);
my $err = waitpid($pid, 0) == $pid ? undef
^ permalink raw reply related [relevance 70%]
* [PATCH 0/3] lei input improvements
@ 2021-03-29 7:08 71% Eric Wong
2021-03-29 7:08 70% ` [PATCH 2/3] lei: use IO::Uncompress::Gunzip MultiStream Eric Wong
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2021-03-29 7:08 UTC (permalink / raw)
To: meta
These affect the convert, import, mark sub-commands.
Eric Wong (3):
lei_input: avoid special case sub for --stdin
lei: use IO::Uncompress::Gunzip MultiStream
lei_input: treat ".eml" and ".patch" suffix as "eml"
lib/PublicInbox/LeiConvert.pm | 1 -
lib/PublicInbox/LeiImport.pm | 1 -
lib/PublicInbox/LeiInput.pm | 33 +++++++++++++++++++--------------
lib/PublicInbox/LeiMark.pm | 1 -
lib/PublicInbox/LeiRemote.pm | 2 +-
lib/PublicInbox/LeiXSearch.pm | 2 +-
t/lei-import.t | 2 +-
t/lei-mark.t | 4 ++--
8 files changed, 24 insertions(+), 22 deletions(-)
^ permalink raw reply [relevance 71%]
* Re: [PATCH 6/8] doc lei: add manpages for new commands
2021-03-29 3:25 71% ` Eric Wong
@ 2021-03-29 3:35 71% ` Kyle Meyer
0 siblings, 0 replies; 200+ results
From: Kyle Meyer @ 2021-03-29 3:35 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
Eric Wong writes:
> Kyle Meyer <kyle@kyleam.com> wrote:
>> +=for comment
>
>> +TODO: The below options are shared with lei-q. Any good approaches to
>> +not repeating the text?
>
> I don't know about natively in POD. Perhaps just a list of
> switches and a pointer to lei-q(1)?
Yeah, this sounds like the way to go to me. I'll make a note to do it
for the next round.
Thanks.
^ permalink raw reply [relevance 71%]
* Re: [PATCH 6/8] doc lei: add manpages for new commands
2021-03-29 3:11 30% ` [PATCH 6/8] doc lei: add manpages for new commands Kyle Meyer
@ 2021-03-29 3:25 71% ` Eric Wong
2021-03-29 3:35 71% ` Kyle Meyer
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2021-03-29 3:25 UTC (permalink / raw)
To: Kyle Meyer; +Cc: meta
Kyle Meyer <kyle@kyleam.com> wrote:
> +=for comment
> +TODO: The below options are shared with lei-q. Any good approaches to
> +not repeating the text?
I don't know about natively in POD. Perhaps just a list of
switches and a pointer to lei-q(1)?
Or maybe we generate some .pod files via cat + sh
cat lei-blob.pod.in lei-q-common.pod.in >lei-blob.pod
I don't know if depending on cpp or some other widely-installed
preprocessor is worth it.
> +=for comment
> +TODO: Put a table of prefixes somewhere and reference that (at least
> +here and in lei-q)?
Yes, probably.
^ permalink raw reply [relevance 71%]
* Re: [PATCH 0/8] doc: lei manpages, round 4
2021-03-29 3:11 69% [PATCH 0/8] doc: lei manpages, round 4 Kyle Meyer
` (7 preceding siblings ...)
2021-03-29 3:11 71% ` [PATCH 8/8] doc lei overview: better explain routes into local store Kyle Meyer
@ 2021-03-29 3:18 71% ` Eric Wong
8 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-29 3:18 UTC (permalink / raw)
To: Kyle Meyer; +Cc: meta
Thanks, pushed as commit 165eabbffd15866f581ca374cb37fbf3c07989a5
^ permalink raw reply [relevance 71%]
* [PATCH 8/8] doc lei overview: better explain routes into local store
2021-03-29 3:11 69% [PATCH 0/8] doc: lei manpages, round 4 Kyle Meyer
` (6 preceding siblings ...)
2021-03-29 3:11 90% ` [PATCH 7/8] doc lei overview: note that lei-init is usually unnecessary Kyle Meyer
@ 2021-03-29 3:11 71% ` Kyle Meyer
2021-03-29 3:18 71% ` [PATCH 0/8] doc: lei manpages, round 4 Eric Wong
8 siblings, 0 replies; 200+ results
From: Kyle Meyer @ 2021-03-29 3:11 UTC (permalink / raw)
To: meta
---
Documentation/lei-overview.pod | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/Documentation/lei-overview.pod b/Documentation/lei-overview.pod
index 55e84254..f74a228a 100644
--- a/Documentation/lei-overview.pod
+++ b/Documentation/lei-overview.pod
@@ -14,6 +14,11 @@ Commands will automatically initialize the store behind the scenes if
needed, but you can call L<lei-init(1)> directly if you want to use a
store location other than the default C<$XDG_DATA_HOME/lei/store>.
+The L<lei-import(1)> command provides the primary interface for
+importing messages into the local storage. In addition, other
+commands, such as L<lei-q(1)> and L<lei-blob(1)>, use the local store
+to memoize messages from remotes.
+
=head2 EXAMPLES
=over
--
2.31.0
^ permalink raw reply related [relevance 71%]
* [PATCH 7/8] doc lei overview: note that lei-init is usually unnecessary
2021-03-29 3:11 69% [PATCH 0/8] doc: lei manpages, round 4 Kyle Meyer
` (5 preceding siblings ...)
2021-03-29 3:11 30% ` [PATCH 6/8] doc lei: add manpages for new commands Kyle Meyer
@ 2021-03-29 3:11 90% ` Kyle Meyer
2021-03-29 3:11 71% ` [PATCH 8/8] doc lei overview: better explain routes into local store Kyle Meyer
2021-03-29 3:18 71% ` [PATCH 0/8] doc: lei manpages, round 4 Eric Wong
8 siblings, 0 replies; 200+ results
From: Kyle Meyer @ 2021-03-29 3:11 UTC (permalink / raw)
To: meta
cf. https://public-inbox.org/meta/20210325083207.GA30551@dcvr
---
Documentation/lei-overview.pod | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/Documentation/lei-overview.pod b/Documentation/lei-overview.pod
index 7c7337ab..55e84254 100644
--- a/Documentation/lei-overview.pod
+++ b/Documentation/lei-overview.pod
@@ -9,8 +9,10 @@ provides some basic examples.
=head1 LEI STORE
-L<lei-init(1)> initializes writable local storage based on
-L<public-inbox-v2-format(5)>.
+lei has writable local storage based on L<public-inbox-v2-format(5)>.
+Commands will automatically initialize the store behind the scenes if
+needed, but you can call L<lei-init(1)> directly if you want to use a
+store location other than the default C<$XDG_DATA_HOME/lei/store>.
=head2 EXAMPLES
--
2.31.0
^ permalink raw reply related [relevance 90%]
* [PATCH 6/8] doc lei: add manpages for new commands
2021-03-29 3:11 69% [PATCH 0/8] doc: lei manpages, round 4 Kyle Meyer
` (4 preceding siblings ...)
2021-03-29 3:11 71% ` [PATCH 5/8] doc lei: update manpages with new options Kyle Meyer
@ 2021-03-29 3:11 30% ` Kyle Meyer
2021-03-29 3:25 71% ` Eric Wong
2021-03-29 3:11 90% ` [PATCH 7/8] doc lei overview: note that lei-init is usually unnecessary Kyle Meyer
` (2 subsequent siblings)
8 siblings, 1 reply; 200+ results
From: Kyle Meyer @ 2021-03-29 3:11 UTC (permalink / raw)
To: meta
---
Documentation/lei-blob.pod | 109 +++++++++++++++++++++++++++++++++
Documentation/lei-ls-label.pod | 43 +++++++++++++
Documentation/lei-mark.pod | 58 ++++++++++++++++++
Documentation/lei-overview.pod | 21 +++++++
Documentation/lei-p2q.pod | 79 ++++++++++++++++++++++++
Documentation/lei.pod | 8 +++
Documentation/txt2pre | 4 ++
MANIFEST | 4 ++
Makefile.PL | 5 +-
9 files changed, 329 insertions(+), 2 deletions(-)
create mode 100644 Documentation/lei-blob.pod
create mode 100644 Documentation/lei-ls-label.pod
create mode 100644 Documentation/lei-mark.pod
create mode 100644 Documentation/lei-p2q.pod
diff --git a/Documentation/lei-blob.pod b/Documentation/lei-blob.pod
new file mode 100644
index 00000000..ecdd1e99
--- /dev/null
+++ b/Documentation/lei-blob.pod
@@ -0,0 +1,109 @@
+=head1 NAME
+
+lei-blob - display a git blob, reconstructing from mail if necessary
+
+=head1 SYNOPSIS
+
+lei blob [OPTIONS] OID
+
+=head1 DESCRIPTION
+
+Display a git blob. The blob may correspond to a message from the
+local store, an existing blob in the current repository, or a
+not-yet-created blob in the current repository that can be
+reconstructed from a message.
+
+=head1 OPTIONS
+
+=over
+
+=item --git-dir=DIR
+
+Specify an additional .git/ directory to scan. This option may be
+given multiple times.
+
+=item --no-cwd
+
+Do not look in the git repository of the current working directory.
+
+=item --no-mail
+
+Do not look in mail storage for C<OID>. This is implied by
+C<--oid-a>, C<--path-a>, and C<--path-b>.
+
+=item -A OID-A, --oid-a=OID-A
+
+=item -a PATH-A, --path-a=PATH-A
+
+=item -b PATH-B, --path-b=PATH-B
+
+Provide pre-image object ID, pre-image pathname, or post-image
+pathname as a hint for reconstructing C<OID>.
+
+=for comment
+TODO: The below options are shared with lei-q. Any good approaches to
+not repeating the text?
+
+=item --[no-]remote
+
+Whether to include results requiring network access. When local
+externals are configured, C<--remote> must be explicitly passed to
+enable reporting of results from remote externals.
+
+=item --no-local
+
+Limit operations to those requiring network access.
+
+=item --no-external
+
+Don't include results from externals.
+
+=item -I LOCATION, --include=LOCATION
+
+Include specified external in search. This option may be given
+multiple times.
+
+=item --exclude=LOCATION
+
+Exclude specified external from search. This option may be given
+multiple times.
+
+=item --only=LOCATION
+
+Use only the specified external for search. This option may be given
+multiple times, in which case the search uses only the specified set.
+
+=item --no-import-remote
+
+Disable the default behavior of memoizing remote messages into the
+local store.
+
+=item -v, --verbose
+
+Provide more feedback on stderr.
+
+=item --torsocks=auto|no|yes, --no-torsocks
+
+Whether to wrap L<git(1)> and L<curl(1)> commands with torsocks.
+
+Default: C<auto>
+
+=back
+
+=head1 CONTACT
+
+Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org>
+
+The mail archives are hosted at L<https://public-inbox.org/meta/>
+and L<http://hjrcffqmbrq6wope.onion/meta/>
+
+=head1 COPYRIGHT
+
+Copyright 2021 all contributors L<mailto:meta@public-inbox.org>
+
+License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>
+
+
+=head1 SEE ALSO
+
+L<lei-add-external(1)>
diff --git a/Documentation/lei-ls-label.pod b/Documentation/lei-ls-label.pod
new file mode 100644
index 00000000..0b4e8769
--- /dev/null
+++ b/Documentation/lei-ls-label.pod
@@ -0,0 +1,43 @@
+=head1 NAME
+
+lei-ls-label - list labels
+
+=head1 SYNOPSIS
+
+lei ls-label [OPTIONS]
+
+=head1 DESCRIPTION
+
+List all known message labels ("mailboxes" in JMAP terminology).
+
+=head1 OPTIONS
+
+=over
+
+=item -z, -0
+
+Use C<\0> (NUL) instead of newline (CR) to delimit lines.
+
+=item -q, --quiet
+
+Suppress feedback messages.
+
+=back
+
+=head1 CONTACT
+
+Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org>
+
+The mail archives are hosted at L<https://public-inbox.org/meta/>
+and L<http://hjrcffqmbrq6wope.onion/meta/>
+
+=head1 COPYRIGHT
+
+Copyright 2021 all contributors L<mailto:meta@public-inbox.org>
+
+License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>
+
+
+=head1 SEE ALSO
+
+L<lei-add-external(1)>
diff --git a/Documentation/lei-mark.pod b/Documentation/lei-mark.pod
new file mode 100644
index 00000000..8ef1dce2
--- /dev/null
+++ b/Documentation/lei-mark.pod
@@ -0,0 +1,58 @@
+=head1 NAME
+
+lei-mark - set/unset metadata on messages
+
+=head1 SYNOPSIS
+
+lei mark [OPTIONS] FILE [FILE...] METADATA [METADATA...]
+
+lei mark [OPTIONS] (-|--stdin) METADATA [METADATA...]
+
+=head1 DESCRIPTION
+
+Set or unset volatile metadata on messages. In JMAP terms, "volatile
+metadata" includes "mailboxes" (analogous to a folder or label) and a
+restricted set of "keywords". This supported keywords are the
+combination of system keywords (seen, answered, flagged, and draft),
+which map to Maildir flags and mbox Status/X-Status headers, as well
+as reserved keywords (forwarded, phishing, junk, and notjunk).
+
+To add a label or keyword, prefix it with "+L:" and "+kw:",
+respectively. To remove a label or keyword, use "-L:" or "-kw:". For
+example, "+kw:flagged" would set the "flagged" keyword for the
+specified messages, and "-L:INBOX" would remove the "INBOX" label.
+
+=head1 OPTIONS
+
+=over
+
+=item -F MAIL_FORMAT, --in-format=MAIL_FORMAT
+
+Message input format: C<eml>, C<mboxrd>, C<mboxcl2>, C<mboxcl>, or
+C<mboxo>.
+
+Default: C<eml>
+
+=item -q, --quiet
+
+Suppress feedback messages.
+
+=back
+
+=head1 CONTACT
+
+Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org>
+
+The mail archives are hosted at L<https://public-inbox.org/meta/>
+and L<http://hjrcffqmbrq6wope.onion/meta/>
+
+=head1 COPYRIGHT
+
+Copyright 2021 all contributors L<mailto:meta@public-inbox.org>
+
+License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>
+
+
+=head1 SEE ALSO
+
+L<lei-add-external(1)>
diff --git a/Documentation/lei-overview.pod b/Documentation/lei-overview.pod
index c3379caa..7c7337ab 100644
--- a/Documentation/lei-overview.pod
+++ b/Documentation/lei-overview.pod
@@ -20,6 +20,17 @@ L<public-inbox-v2-format(5)>.
Import the messages from an mbox into the local storage.
+=item $ lei blob 59ec517f9
+
+Show message with the git blob OID of 59ec517f9. If a message with
+that OID isn't found, check if the current git repository has the
+blob, trying to reconstruct it from a message if needed.
+
+=item $ lei blob 59ec517f9 | lei mark - -F eml +kw:flagged +L:next
+
+Set the "flagged" keyword and "next" label on the message with the
+blob OID of 59ec517f9.
+
=back
=head1 EXTERNALS
@@ -69,6 +80,16 @@ thread as a matched message.
Write mboxcl2-formatted results to t.mbox and enter mutt to view the
file by invoking C<mutt -f %f>.
+=item $ lei q kw:flagged L:next
+
+Search for all flagged messages that also have a "next" label.
+
+=item $ lei p2q HEAD | lei q --stdin -tt -o mdir
+
+Search for messages that have post-image git blob IDs that match those
+of the current repository's HEAD commit, writing them to the Maildir
+directory "mdir" and flagging the messages that were an exact match.
+
=back
=head1 PERFORMANCE NOTES
diff --git a/Documentation/lei-p2q.pod b/Documentation/lei-p2q.pod
new file mode 100644
index 00000000..cc342bd5
--- /dev/null
+++ b/Documentation/lei-p2q.pod
@@ -0,0 +1,79 @@
+=head1 NAME
+
+lei-p2q - use a patch to generate a lei-q query
+
+=head1 SYNOPSIS
+
+lei p2q [OPTIONS] (FILE|COMMIT)
+
+lei p2q [OPTIONS] (--stdin|-)
+
+=head1 DESCRIPTION
+
+Given a patch, create a query that can be fed on stdin to L<lei-q(1)>.
+This is useful for mapping the patch to associated messages of an
+inbox.
+
+The patch can be provided on stdin or as a file. Alternatively, when
+an argument is given that does not point to an existing file, it is
+taken as a reference to a commit in the current repository, and
+L<git-format-patch(1)> is used to generate the patch.
+
+=head1 OPTIONS
+
+=over
+
+=item -w PREFIX[,PREFIX], --want=PREFIX[,PREFIX]
+
+Search prefixes to use. C<dfpost> (post-image git blob ID) and C<dfn>
+(file names from the diff) are the most useful. Other available
+values are C<dfa>, C<dfb>, C<dfctx>, C<dfhh>, and C<dfpre>.
+
+=for comment
+TODO: Put a table of prefixes somewhere and reference that (at least
+here and in lei-q)?
+
+Appending an integer to C<dfpost> or C<dfpre> indicates a minimum ID
+length, and the generated query will be for that value up through the
+default abbreviation length. For example, if the repository's
+C<core.abbrev> is set to C<auto> and git calculates the default
+abbreviation length as 7, C<dfpost6> will expand a post-image blob ID
+of e7b4b32 (seven characters) into C<dfpost:e7b4b32 OR dfpost:e7b4b3>.
+
+This option may be given multiple times.
+
+Default: C<dfpost7>
+
+=item --stdin
+
+Read patch from stdin.
+
+=item --debug
+
+Dump output that shows the information collected for every prefix.
+This information can be useful for seeing how a patch is processed,
+but the format should not be considered stable.
+
+=item -q, --quiet
+
+Suppress feedback messages.
+
+=back
+
+=head1 CONTACT
+
+Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org>
+
+The mail archives are hosted at L<https://public-inbox.org/meta/>
+and L<http://hjrcffqmbrq6wope.onion/meta/>
+
+=head1 COPYRIGHT
+
+Copyright 2021 all contributors L<mailto:meta@public-inbox.org>
+
+License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>
+
+
+=head1 SEE ALSO
+
+L<lei-add-external(1)>
diff --git a/Documentation/lei.pod b/Documentation/lei.pod
index 0b81b9f3..6c8cfc3d 100644
--- a/Documentation/lei.pod
+++ b/Documentation/lei.pod
@@ -46,6 +46,8 @@ Subcommands for initializing and managing local, writable storage:
=item * L<lei-import(1)>
+=item * L<lei-mark(1)>
+
=back
The following subcommands can be used to manage and inspect external
@@ -66,6 +68,10 @@ store and configured externals are
=over
+=item * L<lei-blob(1)>
+
+=item * L<lei-p2q(1)>
+
=item * L<lei-q(1)>
=back
@@ -80,6 +86,8 @@ Other subcommands include
=item * L<lei-daemon-pid(1)>
+=item * L<lei-ls-label(1)>
+
=back
=head1 FILES
diff --git a/Documentation/txt2pre b/Documentation/txt2pre
index 244dc50c..bfffdef1 100755
--- a/Documentation/txt2pre
+++ b/Documentation/txt2pre
@@ -12,6 +12,7 @@ use PublicInbox::Hval qw(ascii_html);
my %xurls;
for (qw[lei(1)
lei-add-external(1)
+ lei-blob(1)
lei-config(1)
lei-daemon-kill(1)
lei-daemon-pid(1)
@@ -19,7 +20,10 @@ for (qw[lei(1)
lei-import(1)
lei-init(1)
lei-ls-external(1)
+ lei-ls-label(1)
+ lei-mark(1)
lei-overview(7)
+ lei-p2q(1)
lei-q(1)
public-inbox.cgi(1)
public-inbox-compact(1)
diff --git a/MANIFEST b/MANIFEST
index 913ce55c..3d521a64 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -22,6 +22,7 @@ Documentation/flow.txt
Documentation/hosted.txt
Documentation/include.mk
Documentation/lei-add-external.pod
+Documentation/lei-blob.pod
Documentation/lei-config.pod
Documentation/lei-daemon-kill.pod
Documentation/lei-daemon-pid.pod
@@ -29,7 +30,10 @@ Documentation/lei-forget-external.pod
Documentation/lei-import.pod
Documentation/lei-init.pod
Documentation/lei-ls-external.pod
+Documentation/lei-ls-label.pod
+Documentation/lei-mark.pod
Documentation/lei-overview.pod
+Documentation/lei-p2q.pod
Documentation/lei-q.pod
Documentation/lei.pod
Documentation/marketing.txt
diff --git a/Makefile.PL b/Makefile.PL
index 8165e601..cdb67214 100644
--- a/Makefile.PL
+++ b/Makefile.PL
@@ -44,8 +44,9 @@ $v->{-m1} = [ map {
}
} @EXE_FILES,
qw(
- lei-add-external lei-config lei-daemon-kill lei-daemon-pid
- lei-forget-external lei-import lei-init lei-ls-external lei-q)];
+ lei-add-external lei-blob lei-config lei-daemon-kill lei-daemon-pid
+ lei-forget-external lei-import lei-init lei-ls-external lei-ls-label
+ lei-mark lei-p2q lei-q)];
$v->{-m5} = [ qw(public-inbox-config public-inbox-v1-format
public-inbox-v2-format public-inbox-extindex-format) ];
$v->{-m7} = [ qw(lei-overview public-inbox-overview public-inbox-tuning
--
2.31.0
^ permalink raw reply related [relevance 30%]
* [PATCH 5/8] doc lei: update manpages with new options
2021-03-29 3:11 69% [PATCH 0/8] doc: lei manpages, round 4 Kyle Meyer
` (3 preceding siblings ...)
2021-03-29 3:11 68% ` [PATCH 4/8] doc lei: don't render most to-do comments Kyle Meyer
@ 2021-03-29 3:11 71% ` Kyle Meyer
2021-03-29 3:11 30% ` [PATCH 6/8] doc lei: add manpages for new commands Kyle Meyer
` (3 subsequent siblings)
8 siblings, 0 replies; 200+ results
From: Kyle Meyer @ 2021-03-29 3:11 UTC (permalink / raw)
To: meta
---
Documentation/lei-q.pod | 5 +++++
Documentation/lei.pod | 4 ++++
2 files changed, 9 insertions(+)
diff --git a/Documentation/lei-q.pod b/Documentation/lei-q.pod
index 6f67c277..787c51bf 100644
--- a/Documentation/lei-q.pod
+++ b/Documentation/lei-q.pod
@@ -82,6 +82,11 @@ doesn't point to stdout, nothing otherwise.
Augment output destination instead of clobbering it.
+=item --no-import-before
+
+Do not importing keywords before writing to an existing output
+destination.
+
=item -t, --threads
Return all messages in the same thread as the actual match(es).
diff --git a/Documentation/lei.pod b/Documentation/lei.pod
index 36abe513..0b81b9f3 100644
--- a/Documentation/lei.pod
+++ b/Documentation/lei.pod
@@ -23,6 +23,10 @@ Available in public-inbox 1.7.0+.
=over
+=item -c NAME=VALUE
+
+Override configuration C<NAME> to C<VALUE>.
+
=item -C DIR
Change current working directory to the specified directory before
--
2.31.0
^ permalink raw reply related [relevance 71%]
* [PATCH 4/8] doc lei: don't render most to-do comments
2021-03-29 3:11 69% [PATCH 0/8] doc: lei manpages, round 4 Kyle Meyer
` (2 preceding siblings ...)
2021-03-29 3:11 71% ` [PATCH 3/8] doc lei: drop an unnecessary to-do comment Kyle Meyer
@ 2021-03-29 3:11 68% ` Kyle Meyer
2021-03-29 3:11 71% ` [PATCH 5/8] doc lei: update manpages with new options Kyle Meyer
` (4 subsequent siblings)
8 siblings, 0 replies; 200+ results
From: Kyle Meyer @ 2021-03-29 3:11 UTC (permalink / raw)
To: meta
The lei manpages have a number of to-dos, but with the exception of
the lei-q's -tt warning, none of them seem worth displaying to the
reader (and some might not be worth addressing at all).
---
Documentation/lei-add-external.pod | 1 +
Documentation/lei-q.pod | 4 ++++
2 files changed, 5 insertions(+)
diff --git a/Documentation/lei-add-external.pod b/Documentation/lei-add-external.pod
index 3bc0ba83..47158146 100644
--- a/Documentation/lei-add-external.pod
+++ b/Documentation/lei-add-external.pod
@@ -15,6 +15,7 @@ C<extindex.<name>.topdir> value in ~/.public-inbox/config.
=head1 OPTIONS
+=for comment
TODO: mention curl options?
=over
diff --git a/Documentation/lei-q.pod b/Documentation/lei-q.pod
index cb1227fb..6f67c277 100644
--- a/Documentation/lei-q.pod
+++ b/Documentation/lei-q.pod
@@ -12,10 +12,12 @@ lei q [OPTIONS] (--stdin|-)
Search for messages across the lei store and externals.
+=for comment
TODO: Give common prefixes, or at least a description/reference.
=head1 OPTIONS
+=for comment
TODO: mention curl options?
=over
@@ -40,6 +42,7 @@ the destination. C<json> is used for the default destination
(stdout), and C<maildir> is used for an existing directory or
non-existing path.
+=for comment
TODO: Provide description of formats?
Default: -
@@ -98,6 +101,7 @@ C<none>.
Default: C<content>
+=for comment
TODO: Provide description of strategies?
=item --[no-]remote
--
2.31.0
^ permalink raw reply related [relevance 68%]
* [PATCH 3/8] doc lei: drop an unnecessary to-do comment
2021-03-29 3:11 69% [PATCH 0/8] doc: lei manpages, round 4 Kyle Meyer
2021-03-29 3:11 71% ` [PATCH 1/8] doc lei-q: fix typo in -tt description Kyle Meyer
2021-03-29 3:11 71% ` [PATCH 2/8] doc lei: note --stdin shortcut in synopses Kyle Meyer
@ 2021-03-29 3:11 71% ` Kyle Meyer
2021-03-29 3:11 68% ` [PATCH 4/8] doc lei: don't render most to-do comments Kyle Meyer
` (5 subsequent siblings)
8 siblings, 0 replies; 200+ results
From: Kyle Meyer @ 2021-03-29 3:11 UTC (permalink / raw)
To: meta
When a new command is implemented, it is probably clear that it should
be added to lei.pod, but either way, having a to-do comment in lei.pod
isn't likely to help.
---
Documentation/lei.pod | 2 --
1 file changed, 2 deletions(-)
diff --git a/Documentation/lei.pod b/Documentation/lei.pod
index e1502122..36abe513 100644
--- a/Documentation/lei.pod
+++ b/Documentation/lei.pod
@@ -66,8 +66,6 @@ store and configured externals are
=back
-TODO: Add lei-show (and perhaps others) once implemented.
-
Other subcommands include
=over
--
2.31.0
^ permalink raw reply related [relevance 71%]
* [PATCH 1/8] doc lei-q: fix typo in -tt description
2021-03-29 3:11 69% [PATCH 0/8] doc: lei manpages, round 4 Kyle Meyer
@ 2021-03-29 3:11 71% ` Kyle Meyer
2021-03-29 3:11 71% ` [PATCH 2/8] doc lei: note --stdin shortcut in synopses Kyle Meyer
` (7 subsequent siblings)
8 siblings, 0 replies; 200+ results
From: Kyle Meyer @ 2021-03-29 3:11 UTC (permalink / raw)
To: meta
---
Documentation/lei-q.pod | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/Documentation/lei-q.pod b/Documentation/lei-q.pod
index e878157d..446a82c6 100644
--- a/Documentation/lei-q.pod
+++ b/Documentation/lei-q.pod
@@ -84,7 +84,7 @@ Augment output destination instead of clobbering it.
Return all messages in the same thread as the actual match(es).
Using this twice (C<-tt>) sets the C<flagged> (AKA "important")
-on messages which were actual messages. This is useful to distinguish
+on messages which were actual matches. This is useful to distinguish
messages which were direct hits from messages which were merely part
of the same thread.
--
2.31.0
^ permalink raw reply related [relevance 71%]
* [PATCH 2/8] doc lei: note --stdin shortcut in synopses
2021-03-29 3:11 69% [PATCH 0/8] doc: lei manpages, round 4 Kyle Meyer
2021-03-29 3:11 71% ` [PATCH 1/8] doc lei-q: fix typo in -tt description Kyle Meyer
@ 2021-03-29 3:11 71% ` Kyle Meyer
2021-03-29 3:11 71% ` [PATCH 3/8] doc lei: drop an unnecessary to-do comment Kyle Meyer
` (6 subsequent siblings)
8 siblings, 0 replies; 200+ results
From: Kyle Meyer @ 2021-03-29 3:11 UTC (permalink / raw)
To: meta
---
Documentation/lei-import.pod | 2 +-
Documentation/lei-q.pod | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/Documentation/lei-import.pod b/Documentation/lei-import.pod
index 7f4e3745..acc4f776 100644
--- a/Documentation/lei-import.pod
+++ b/Documentation/lei-import.pod
@@ -6,7 +6,7 @@ lei-import - one-time import of messages into local store
lei import [OPTIONS] LOCATION [LOCATION...]
-lei import [OPTIONS] --stdin
+lei import [OPTIONS] (--stdin|-)
=head1 DESCRIPTION
diff --git a/Documentation/lei-q.pod b/Documentation/lei-q.pod
index 446a82c6..cb1227fb 100644
--- a/Documentation/lei-q.pod
+++ b/Documentation/lei-q.pod
@@ -6,7 +6,7 @@ lei-q - search for messages matching terms
lei q [OPTIONS] TERM [TERM...]
-lei q [OPTIONS] --stdin
+lei q [OPTIONS] (--stdin|-)
=head1 DESCRIPTION
--
2.31.0
^ permalink raw reply related [relevance 71%]
* [PATCH 0/8] doc: lei manpages, round 4
@ 2021-03-29 3:11 69% Kyle Meyer
2021-03-29 3:11 71% ` [PATCH 1/8] doc lei-q: fix typo in -tt description Kyle Meyer
` (8 more replies)
0 siblings, 9 replies; 200+ results
From: Kyle Meyer @ 2021-03-29 3:11 UTC (permalink / raw)
To: meta
This series updates the lei manpages, continuing from
<20210227180328.28057-1-kyle@kyleam.com>. It covers changes up to the
current tip of master (c9ff20cbef45d32e..80f4192574065106).
[1/8] doc lei-q: fix typo in -tt description
[2/8] doc lei: note --stdin shortcut in synopses
[3/8] doc lei: drop an unnecessary to-do comment
[4/8] doc lei: don't render most to-do comments
[5/8] doc lei: update manpages with new options
[6/8] doc lei: add manpages for new commands
[7/8] doc lei overview: note that lei-init is usually unnecessary
[8/8] doc lei overview: better explain routes into local store
Documentation/lei-add-external.pod | 1 +
Documentation/lei-blob.pod | 109 +++++++++++++++++++++++++++++
Documentation/lei-import.pod | 2 +-
Documentation/lei-ls-label.pod | 43 ++++++++++++
Documentation/lei-mark.pod | 58 +++++++++++++++
Documentation/lei-overview.pod | 32 ++++++++-
Documentation/lei-p2q.pod | 79 +++++++++++++++++++++
Documentation/lei-q.pod | 13 +++-
Documentation/lei.pod | 14 +++-
Documentation/txt2pre | 4 ++
MANIFEST | 4 ++
Makefile.PL | 5 +-
12 files changed, 355 insertions(+), 9 deletions(-)
create mode 100644 Documentation/lei-blob.pod
create mode 100644 Documentation/lei-ls-label.pod
create mode 100644 Documentation/lei-mark.pod
create mode 100644 Documentation/lei-p2q.pod
base-commit: 80f4192574065106ae72a7a73ee0f02ebd86708a
--
2.31.0
^ permalink raw reply [relevance 69%]
* Re: [PATCH 11/12] lei: drop coderepo placeholders, submodule TODO
2021-03-28 9:01 66% ` [PATCH 11/12] lei: drop coderepo placeholders, submodule TODO Eric Wong
@ 2021-03-28 9:31 71% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-28 9:31 UTC (permalink / raw)
To: meta
Eric Wong <e@80x24.org> wrote:
> "lei blob" supports --git-dir and -C, and checks if the
> current directory has a git directory associated with it.
> It will likely support submodules in the future.
>
> I'm inclined to believe declaring coderepos in a command-line
> tool is needless clutter and users will rarely want to search
> for blobs across different projects when on the command-line.
Fwiw, we may we end up supporting read-write JMAP AND expose
blob reconstruction as a vendor-specific JMAP function.
But that's not happening for 1.7...
^ permalink raw reply [relevance 71%]
* [PATCH 11/12] lei: drop coderepo placeholders, submodule TODO
2021-03-28 9:01 67% [PATCH 00/12] lei blob and some yak-shaving Eric Wong
` (8 preceding siblings ...)
2021-03-28 9:01 44% ` [PATCH 10/12] lei blob: add remote external support Eric Wong
@ 2021-03-28 9:01 66% ` Eric Wong
2021-03-28 9:31 71% ` Eric Wong
9 siblings, 1 reply; 200+ results
From: Eric Wong @ 2021-03-28 9:01 UTC (permalink / raw)
To: meta
"lei blob" supports --git-dir and -C, and checks if the
current directory has a git directory associated with it.
It will likely support submodules in the future.
I'm inclined to believe declaring coderepos in a command-line
tool is needless clutter and users will rarely want to search
for blobs across different projects when on the command-line.
---
lib/PublicInbox/LEI.pm | 9 ---------
lib/PublicInbox/LeiBlob.pm | 1 +
2 files changed, 1 insertion(+), 9 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index a94941a9..8a07a4c8 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -172,15 +172,6 @@ our %CMD = ( # sorted in order of importance/use:
'remove imported messages from IMAP, Maildirs, and MH',
qw(exact! all jobs:i indexed), @c_opt ],
-# code repos are used for `show' to solve blobs from patch mails
-'add-coderepo' => [ 'DIRNAME', 'add or set priority of a git code repo',
- qw(boost=i), @c_opt ],
-'ls-coderepo' => [ '[FILTER_TERMS...]',
- 'list known code repos', qw(format|f=s z), @c_opt ],
-'forget-coderepo' => [ 'DIRNAME',
- 'stop using repo to solve blobs from patches',
- qw(prune), @c_opt ],
-
'add-watch' => [ 'LOCATION', 'watch for new messages and flag changes',
qw(import! kw|keywords|flags! interval=s recursive|r
exclude=s include=s), @c_opt ],
diff --git a/lib/PublicInbox/LeiBlob.pm b/lib/PublicInbox/LeiBlob.pm
index 8e610efd..91098a90 100644
--- a/lib/PublicInbox/LeiBlob.pm
+++ b/lib/PublicInbox/LeiBlob.pm
@@ -2,6 +2,7 @@
# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
# "lei blob $OID" command
+# TODO: this doesn't scan submodules, but maybe it should
package PublicInbox::LeiBlob;
use strict;
use v5.10.1;
^ permalink raw reply related [relevance 66%]
* [PATCH 10/12] lei blob: add remote external support
2021-03-28 9:01 67% [PATCH 00/12] lei blob and some yak-shaving Eric Wong
` (7 preceding siblings ...)
2021-03-28 9:01 69% ` [PATCH 08/12] lei blob: flesh out help text Eric Wong
@ 2021-03-28 9:01 44% ` Eric Wong
2021-03-28 9:01 66% ` [PATCH 11/12] lei: drop coderepo placeholders, submodule TODO Eric Wong
9 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-28 9:01 UTC (permalink / raw)
To: meta
Introduce a new LeiRemote wrapper to provide an internal API
which SolverGit expects. This lets us use HTTP/HTTPS endpoints
to reconstruct blobs off patches as we would with local
endpoints, just more slowly...
---
MANIFEST | 1 +
lib/PublicInbox/LEI.pm | 2 +-
lib/PublicInbox/LeiBlob.pm | 16 +++++--
lib/PublicInbox/LeiRemote.pm | 81 ++++++++++++++++++++++++++++++++++++
t/solver_git.t | 16 ++++++-
5 files changed, 110 insertions(+), 6 deletions(-)
create mode 100644 lib/PublicInbox/LeiRemote.pm
diff --git a/MANIFEST b/MANIFEST
index 9048b900..913ce55c 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -196,6 +196,7 @@ lib/PublicInbox/LeiMirror.pm
lib/PublicInbox/LeiOverview.pm
lib/PublicInbox/LeiP2q.pm
lib/PublicInbox/LeiQuery.pm
+lib/PublicInbox/LeiRemote.pm
lib/PublicInbox/LeiSearch.pm
lib/PublicInbox/LeiStore.pm
lib/PublicInbox/LeiToMail.pm
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index a4f4e58c..a94941a9 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -121,7 +121,7 @@ sub index_opt {
my @c_opt = qw(c=s@ C=s@ quiet|q);
my @lxs_opt = (qw(remote! local! external! include|I=s@ exclude=s@ only=s@
- import-remote! no-torsocks torsocks=s),
+ import-remote! no-torsocks torsocks=s),
PublicInbox::LeiQuery::curl_opt());
# we generate shell completion + help using %CMD and %OPTDESC,
diff --git a/lib/PublicInbox/LeiBlob.pm b/lib/PublicInbox/LeiBlob.pm
index f44d8af1..8e610efd 100644
--- a/lib/PublicInbox/LeiBlob.pm
+++ b/lib/PublicInbox/LeiBlob.pm
@@ -6,7 +6,7 @@ package PublicInbox::LeiBlob;
use strict;
use v5.10.1;
use parent qw(PublicInbox::IPC);
-use PublicInbox::Spawn qw(spawn popen_rd);
+use PublicInbox::Spawn qw(spawn popen_rd which);
use PublicInbox::DS;
sub sol_done_wait { # dwaitpid callback
@@ -66,7 +66,10 @@ sub do_solve_blob { # via wq_do
}
open my $log, '+>', \(my $log_buf = '') or die "PerlIO::scalar: $!";
$lei->{log_buf} = \$log_buf;
- my $git = $lei->ale->git;
+ my $git = $lei->{ale}->git;
+ my @rmt = map {
+ PublicInbox::LeiRemote->new($lei, $_)
+ } $self->{lxs}->remotes;
my $solver = bless {
gits => [ map {
PublicInbox::Git->new($lei->rel2abs($_))
@@ -74,7 +77,7 @@ sub do_solve_blob { # via wq_do
user_cb => \&solver_user_cb,
uarg => $self,
# -cur_di, -qsp, -msg => temporary fields for Qspawn callbacks
- inboxes => [ $self->{lxs}->locals ],
+ inboxes => [ $self->{lxs}->locals, @rmt ],
}, 'PublicInbox::SolverGit';
$lei->{env}->{'psgi.errors'} = $lei->{2}; # ugh...
local $PublicInbox::DS::in_loop = 0; # waitpid synchronously
@@ -105,8 +108,15 @@ sub lei_blob {
}
return $lei->fail('no --git-dir to try') unless @$git_dirs;
my $lxs = $lei->lxs_prepare or return;
+ if ($lxs->remotes) {
+ require PublicInbox::LeiRemote;
+ $lei->{curl} //= which('curl') or return
+ $lei->fail('curl needed for', $lxs->remotes);
+ $lei->_lei_store(1)->write_prepare($lei);
+ }
require PublicInbox::SolverGit;
my $self = bless { lxs => $lxs, oid_b => $blob }, __PACKAGE__;
+ $lei->ale;
my ($op_c, $ops) = $lei->workers_start($self, 'lei_solve', 1,
{ '' => [ \&sol_done, $lei ] });
$lei->{sol} = $self;
diff --git a/lib/PublicInbox/LeiRemote.pm b/lib/PublicInbox/LeiRemote.pm
new file mode 100644
index 00000000..399fc936
--- /dev/null
+++ b/lib/PublicInbox/LeiRemote.pm
@@ -0,0 +1,81 @@
+# Copyright (C) 2021 all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+
+# Make remote externals HTTP(S) inboxes behave like
+# PublicInbox::Inbox and PublicInbox::Search/ExtSearch.
+# This exists solely for SolverGit. It is a high-latency a
+# synchronous API that is not at all fast.
+package PublicInbox::LeiRemote;
+use v5.10.1;
+use strict;
+use IO::Uncompress::Gunzip;
+use PublicInbox::OnDestroy;
+use PublicInbox::MboxReader;
+use PublicInbox::Spawn qw(popen_rd);
+use PublicInbox::LeiCurl;
+use PublicInbox::ContentHash qw(git_sha);
+
+sub new {
+ my ($cls, $lei, $uri) = @_;
+ bless { uri => $uri, lei => $lei }, $cls;
+}
+
+sub isrch { $_[0] } # SolverGit expcets this
+
+sub _each_mboxrd_eml { # callback for MboxReader->mboxrd
+ my ($eml, $self) = @_;
+ my $lei = $self->{lei};
+ my $xoids = $lei->{ale}->xoids_for($eml, 1);
+ if ($lei->{sto} && !$xoids) { # memoize locally
+ $lei->{sto}->ipc_do('add_eml', $eml);
+ }
+ my $smsg = bless {}, 'PublicInbox::Smsg';
+ $smsg->{blob} = $xoids ? (keys(%$xoids))[0]
+ : git_sha(1, $eml)->hexdigest;
+ $smsg->populate($eml);
+ $smsg->{mid} //= '(none)';
+ push @{$self->{smsg}}, $smsg;
+}
+
+sub mset {
+ my ($self, $qstr, undef) = @_; # $opt ($_[2]) ignored
+ my $lei = $self->{lei};
+ my $curl = PublicInbox::LeiCurl->new($lei, $lei->{curl});
+ push @$curl, '-s', '-d', '';
+ my $uri = $self->{uri}->clone;
+ $uri->query_form(q => $qstr, x => 'm', r => 1); # r=1: relevance
+ my $cmd = $curl->for_uri($self->{lei}, $uri);
+ $self->{lei}->qerr("# $cmd");
+ my $rdr = { 2 => $lei->{2}, pgid => 0 };
+ my ($fh, $pid) = popen_rd($cmd, undef, $rdr);
+ my $reap = PublicInbox::OnDestroy->new($lei->can('sigint_reap'), $pid);
+ $self->{smsg} = [];
+ $fh = IO::Uncompress::Gunzip->new($fh);
+ PublicInbox::MboxReader->mboxrd($fh, \&_each_mboxrd_eml, $self);
+ my $err = waitpid($pid, 0) == $pid ? undef
+ : "BUG: waitpid($cmd): $!";
+ @$reap = (); # cancel OnDestroy
+ my $wait = $self->{lei}->{sto}->ipc_do('done');
+ die $err if $err;
+ $self; # we are the mset (and $ibx, and $self)
+}
+
+sub size { scalar @{$_[0]->{smsg}} } # size of previous results
+
+sub mset_to_smsg {
+ my ($self, $ibx, $mset) = @_; # all 3 are $self
+ wantarray ? ($self->size, @{$self->{smsg}}) : $self->{smsg};
+}
+
+sub base_url { "$_[0]->{uri}" }
+
+sub smsg_eml {
+ my ($self, $smsg) = @_;
+ if (my $bref = $self->{lei}->ale->git->cat_file($smsg->{blob})) {
+ return PublicInbox::Eml->new($bref);
+ }
+ $self->{lei}->err("E: $self->{uri} $smsg->{blob} gone <$smsg->{mid}>");
+ undef;
+}
+
+1;
diff --git a/t/solver_git.t b/t/solver_git.t
index 6d4b93c7..2d803d47 100644
--- a/t/solver_git.t
+++ b/t/solver_git.t
@@ -7,7 +7,7 @@ use PublicInbox::TestCommon;
use Cwd qw(abs_path);
require_git(2.6);
use Digest::SHA qw(sha1_hex);
-use PublicInbox::Spawn qw(popen_rd);
+use PublicInbox::Spawn qw(popen_rd which);
require_mods(qw(DBD::SQLite Search::Xapian Plack::Util));
my $git_dir = xqx([qw(git rev-parse --git-dir)], undef, {2 => \(my $null)});
$? == 0 or plan skip_all => "$0 must be run from a git working tree";
@@ -227,8 +227,20 @@ EOF
my $cmd = [ qw(-httpd -W0), "--stdout=$out", "--stderr=$err" ];
my $td = start_script($cmd, $env, { 3 => $sock });
my ($h, $p) = tcp_host_port($sock);
- local $ENV{PLACK_TEST_EXTERNALSERVER_URI} = "http://$h:$p";
+ my $url = "http://$h:$p";
+ local $ENV{PLACK_TEST_EXTERNALSERVER_URI} = $url;
Plack::Test::ExternalServer::test_psgi(client => $client);
+ skip 'no curl', 1 unless which('curl');
+
+ mkdir "$tmpdir/ext" // xbail "mkdir $!";
+ test_lei({tmpdir => "$tmpdir/ext"}, sub {
+ my $rurl = "$url/$name";
+ lei_ok(qw(blob --no-mail 69df7d5 -I), $rurl);
+ is(sha1_hex("blob ".length($lei_out)."\0".$lei_out),
+ $expect, 'blob contents output');
+ ok(!lei(qw(blob -I), $rurl, $non_existent),
+ 'non-existent blob fails');
+ });
}
}
^ permalink raw reply related [relevance 44%]
* [PATCH 03/12] lei blob: dclose if already failed
2021-03-28 9:01 67% [PATCH 00/12] lei blob and some yak-shaving Eric Wong
2021-03-28 9:01 43% ` [PATCH 01/12] lei: simplify PktOp callers Eric Wong
2021-03-28 9:01 58% ` [PATCH 02/12] lei init: split out into separate file Eric Wong
@ 2021-03-28 9:01 71% ` Eric Wong
2021-03-28 9:01 57% ` [PATCH 04/12] lei blob: support --no-mail switch Eric Wong
` (6 subsequent siblings)
9 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-28 9:01 UTC (permalink / raw)
To: meta
We must close the socket to trigger pager exit if blob
reconstruction fails. Not sure how to test this in the
test suite...
---
lib/PublicInbox/LeiBlob.pm | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/lib/PublicInbox/LeiBlob.pm b/lib/PublicInbox/LeiBlob.pm
index 97747220..9b4c4f30 100644
--- a/lib/PublicInbox/LeiBlob.pm
+++ b/lib/PublicInbox/LeiBlob.pm
@@ -19,7 +19,7 @@ sub sol_done_wait { # dwaitpid callback
sub sol_done { # EOF callback for main daemon
my ($lei) = @_;
- my $sol = delete $lei->{sol} or return;
+ my $sol = delete $lei->{sol} // return $lei->dclose; # already failed
$sol->wq_wait_old(\&sol_done_wait, $lei);
}
^ permalink raw reply related [relevance 71%]
* [PATCH 05/12] lei blob: fail early if no git dirs
2021-03-28 9:01 67% [PATCH 00/12] lei blob and some yak-shaving Eric Wong
` (3 preceding siblings ...)
2021-03-28 9:01 57% ` [PATCH 04/12] lei blob: support --no-mail switch Eric Wong
@ 2021-03-28 9:01 71% ` Eric Wong
2021-03-28 9:01 67% ` [PATCH 06/12] lei blob: some extra tests Eric Wong
` (4 subsequent siblings)
9 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-28 9:01 UTC (permalink / raw)
To: meta
This avoids triggering a "BUG:" message in the solver code.
---
lib/PublicInbox/LeiBlob.pm | 1 +
1 file changed, 1 insertion(+)
diff --git a/lib/PublicInbox/LeiBlob.pm b/lib/PublicInbox/LeiBlob.pm
index 4bd86253..f44d8af1 100644
--- a/lib/PublicInbox/LeiBlob.pm
+++ b/lib/PublicInbox/LeiBlob.pm
@@ -103,6 +103,7 @@ sub lei_blob {
my $cgd = get_git_dir('.');
unshift(@$git_dirs, $cgd) if defined $cgd;
}
+ return $lei->fail('no --git-dir to try') unless @$git_dirs;
my $lxs = $lei->lxs_prepare or return;
require PublicInbox::SolverGit;
my $self = bless { lxs => $lxs, oid_b => $blob }, __PACKAGE__;
^ permalink raw reply related [relevance 71%]
* [PATCH 07/12] lei help: show "NAME=VALUE" properly for -c
2021-03-28 9:01 67% [PATCH 00/12] lei blob and some yak-shaving Eric Wong
` (5 preceding siblings ...)
2021-03-28 9:01 67% ` [PATCH 06/12] lei blob: some extra tests Eric Wong
@ 2021-03-28 9:01 71% ` Eric Wong
2021-03-28 9:01 69% ` [PATCH 08/12] lei blob: flesh out help text Eric Wong
` (2 subsequent siblings)
9 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-28 9:01 UTC (permalink / raw)
To: meta
---
lib/PublicInbox/LeiHelp.pm | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/lib/PublicInbox/LeiHelp.pm b/lib/PublicInbox/LeiHelp.pm
index 9c1b30a1..fa0e7866 100644
--- a/lib/PublicInbox/LeiHelp.pm
+++ b/lib/PublicInbox/LeiHelp.pm
@@ -49,7 +49,7 @@ sub call {
length($_) > 1 ? push(@l, "--$_") : push(@s, "-$_");
}
if (!scalar(@vals)) { # no args 'threads|t'
- } elsif ($arg_vals =~ s/\A([A-Z_]+)\b//) { # "NAME"
+ } elsif ($arg_vals =~ s/\A([A-Z_=]+)\b//) { # "NAME"
$vals[1] = $1;
} else {
$vals[1] = uc(substr($l[0], 2)); # "--type" => "TYPE"
^ permalink raw reply related [relevance 71%]
* [PATCH 08/12] lei blob: flesh out help text
2021-03-28 9:01 67% [PATCH 00/12] lei blob and some yak-shaving Eric Wong
` (6 preceding siblings ...)
2021-03-28 9:01 71% ` [PATCH 07/12] lei help: show "NAME=VALUE" properly for -c Eric Wong
@ 2021-03-28 9:01 69% ` Eric Wong
2021-03-28 9:01 44% ` [PATCH 10/12] lei blob: add remote external support Eric Wong
2021-03-28 9:01 66% ` [PATCH 11/12] lei: drop coderepo placeholders, submodule TODO Eric Wong
9 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-28 9:01 UTC (permalink / raw)
To: meta
This means "lei blob" gets shell completion, too.
---
lib/PublicInbox/LEI.pm | 9 +++++++--
1 file changed, 7 insertions(+), 2 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index bad7fad9..a4f4e58c 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -249,7 +249,12 @@ my %OPTDESC = (
"and\xa0'[]'\x{a0}ranges",
'verbose|v+' => 'be more verbose',
'external!' => 'do not use externals',
-'solve!' => 'do not attempt to reconstruct blobs from emails',
+'mail!' => 'do not look in mail storage for OID',
+'cwd!' => 'do not look in git repo of current working directory',
+'oid-a|A=s' => 'pre-image OID',
+'path-a|a=s' => 'pre-image pathname associated with OID',
+'path-b|b=s' => 'post-image pathname associated with OID',
+'git-dir=s@' => 'additional git repository to scan',
'torsocks=s' => ['VAL|auto|no|yes',
'whether or not to wrap git and curl commands with torsocks'],
'no-torsocks' => 'alias for --torsocks=no',
@@ -786,7 +791,7 @@ sub lei__complete {
if (s/[:=].+\z//) { # req/optional args, e.g output|o=i
} elsif (s/\+\z//) { # verbose|v+
} elsif (s/!\z//) {
- # negation: solve! => no-solve|solve
+ # negation: mail! => no-mail|mail
s/([\w\-]+)/$1|no-$1/g
}
map {
^ permalink raw reply related [relevance 69%]
* [PATCH 06/12] lei blob: some extra tests
2021-03-28 9:01 67% [PATCH 00/12] lei blob and some yak-shaving Eric Wong
` (4 preceding siblings ...)
2021-03-28 9:01 71% ` [PATCH 05/12] lei blob: fail early if no git dirs Eric Wong
@ 2021-03-28 9:01 67% ` Eric Wong
2021-03-28 9:01 71% ` [PATCH 07/12] lei help: show "NAME=VALUE" properly for -c Eric Wong
` (3 subsequent siblings)
9 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-28 9:01 UTC (permalink / raw)
To: meta
Most of it already gets tested since most of the logic is in
SolverGit, but make sure it's all wired up properly to lei.
---
t/solver_git.t | 21 ++++++++++++++++++++-
1 file changed, 20 insertions(+), 1 deletion(-)
diff --git a/t/solver_git.t b/t/solver_git.t
index 7bf3ba21..6d4b93c7 100644
--- a/t/solver_git.t
+++ b/t/solver_git.t
@@ -29,6 +29,7 @@ my $ibx = create_inbox 'v2', version => 2,
my $v1_0_0_tag = 'cb7c42b1e15577ed2215356a2bf925aef59cdd8d';
my $v1_0_0_tag_short = substr($v1_0_0_tag, 0, 16);
my $expect = '69df7d565d49fbaaeb0a067910f03dc22cd52bd0';
+my $non_existent = 'ee5e32211bf62ab6531bdf39b84b6920d0b6775a';
test_lei({tmpdir => $tmpdir}, sub {
lei_ok('blob', '69df7d5', '-I', $ibx->{inboxdir});
@@ -37,6 +38,25 @@ test_lei({tmpdir => $tmpdir}, sub {
my $prev = $lei_out;
lei_ok(qw(blob --no-mail 69df7d5 -I), $ibx->{inboxdir});
is($lei_out, $prev, '--no-mail works');
+ ok(!lei(qw(blob -I), $ibx->{inboxdir}, $non_existent),
+ 'non-existent blob fails');
+ SKIP: {
+ skip '/.git exists', 1 if -e '/.git';
+ require PublicInbox::OnDestroy;
+ opendir my $dh, '.' or xbail "opendir: $!";
+ my $end = PublicInbox::OnDestroy->new($$, sub {
+ chdir $dh or xbail "chdir: $!";
+ });
+ lei_ok(qw(-C / blob 69df7d5 -I), $ibx->{inboxdir},
+ "--git-dir=$git_dir");
+ is($lei_out, $prev, '--git-dir works');
+
+ ok(!lei(qw(-C / blob --no-cwd 69df7d5 -I), $ibx->{inboxdir}),
+ '--no-cwd works');
+
+ ok(!lei(qw(-C / blob -I), $ibx->{inboxdir}, $non_existent),
+ 'non-existent blob fails');
+ }
# fallbacks
lei_ok('blob', $v1_0_0_tag, '-I', $ibx->{inboxdir});
@@ -163,7 +183,6 @@ EOF
close $cfgfh or die;
my $cfg = PublicInbox::Config->new($cfgpath);
my $www = PublicInbox::WWW->new($cfg);
- my $non_existent = 'ee5e32211bf62ab6531bdf39b84b6920d0b6775a';
my $client = sub {
my ($cb) = @_;
my $mid = '20190401081523.16213-1-BOFH@YHBT.net';
^ permalink raw reply related [relevance 67%]
* [PATCH 02/12] lei init: split out into separate file
2021-03-28 9:01 67% [PATCH 00/12] lei blob and some yak-shaving Eric Wong
2021-03-28 9:01 43% ` [PATCH 01/12] lei: simplify PktOp callers Eric Wong
@ 2021-03-28 9:01 58% ` Eric Wong
2021-03-28 9:01 71% ` [PATCH 03/12] lei blob: dclose if already failed Eric Wong
` (7 subsequent siblings)
9 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-28 9:01 UTC (permalink / raw)
To: meta
This is a rarely-needed command, so keep it separate file
so it's easier-to-find and maybe saves a bit of RAM.
---
MANIFEST | 1 +
lib/PublicInbox/LEI.pm | 32 -----------------------------
lib/PublicInbox/LeiInit.pm | 41 ++++++++++++++++++++++++++++++++++++++
3 files changed, 42 insertions(+), 32 deletions(-)
create mode 100644 lib/PublicInbox/LeiInit.pm
diff --git a/MANIFEST b/MANIFEST
index 64b3626f..9048b900 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -188,6 +188,7 @@ lib/PublicInbox/LeiDedupe.pm
lib/PublicInbox/LeiExternal.pm
lib/PublicInbox/LeiHelp.pm
lib/PublicInbox/LeiImport.pm
+lib/PublicInbox/LeiInit.pm
lib/PublicInbox/LeiInput.pm
lib/PublicInbox/LeiLsLabel.pm
lib/PublicInbox/LeiMark.pm
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 9cacb142..fdb0bbcf 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -744,38 +744,6 @@ sub lei_config {
x_it($self, $?) if $?;
}
-sub lei_init {
- my ($self, $dir) = @_;
- my $cfg = _lei_cfg($self, 1);
- my $cur = $cfg->{'leistore.dir'};
- $dir //= store_path($self);
- $dir = rel2abs($self, $dir);
- my @cur = stat($cur) if defined($cur);
- $cur = File::Spec->canonpath($cur // $dir);
- my @dir = stat($dir);
- my $exists = "# leistore.dir=$cur already initialized" if @dir;
- if (@cur) {
- if ($cur eq $dir) {
- _lei_store($self, 1)->done;
- return qerr($self, $exists);
- }
-
- # some folks like symlinks and bind mounts :P
- if (@dir && "@cur[1,0]" eq "@dir[1,0]") {
- lei_config($self, 'leistore.dir', $dir);
- _lei_store($self, 1)->done;
- return qerr($self, "$exists (as $cur)");
- }
- return fail($self, <<"");
-E: leistore.dir=$cur already initialized and it is not $dir
-
- }
- lei_config($self, 'leistore.dir', $dir);
- _lei_store($self, 1)->done;
- $exists //= "# leistore.dir=$dir newly initialized";
- return qerr($self, $exists);
-}
-
sub lei_daemon_pid { puts shift, $$ }
sub lei_daemon_kill {
diff --git a/lib/PublicInbox/LeiInit.pm b/lib/PublicInbox/LeiInit.pm
new file mode 100644
index 00000000..c6c0c01b
--- /dev/null
+++ b/lib/PublicInbox/LeiInit.pm
@@ -0,0 +1,41 @@
+# Copyright (C) 2021 all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+
+# for the "lei init" command, not sure if it's even needed...
+package PublicInbox::LeiInit;
+use v5.10.1;
+use File::Spec;
+
+sub lei_init {
+ my ($self, $dir) = @_;
+ my $cfg = $self->_lei_cfg(1);
+ my $cur = $cfg->{'leistore.dir'};
+ $dir //= $self->store_path;
+ $dir = $self->rel2abs($dir);
+ my @cur = stat($cur) if defined($cur);
+ $cur = File::Spec->canonpath($cur // $dir);
+ my @dir = stat($dir);
+ my $exists = "# leistore.dir=$cur already initialized" if @dir;
+ if (@cur) {
+ if ($cur eq $dir) {
+ $self->_lei_store(1)->done;
+ return $self->qerr($exists);
+ }
+
+ # some folks like symlinks and bind mounts :P
+ if (@dir && "@cur[1,0]" eq "@dir[1,0]") {
+ $self->lei_config('leistore.dir', $dir);
+ $self->_lei_store(1)->done;
+ return $self->qerr("$exists (as $cur)");
+ }
+ return $self->fail(<<"");
+E: leistore.dir=$cur already initialized and it is not $dir
+
+ }
+ $self->lei_config('leistore.dir', $dir);
+ $self->_lei_store(1)->done;
+ $exists //= "# leistore.dir=$dir newly initialized";
+ $self->qerr($exists);
+}
+
+1;
^ permalink raw reply related [relevance 58%]
* [PATCH 04/12] lei blob: support --no-mail switch
2021-03-28 9:01 67% [PATCH 00/12] lei blob and some yak-shaving Eric Wong
` (2 preceding siblings ...)
2021-03-28 9:01 71% ` [PATCH 03/12] lei blob: dclose if already failed Eric Wong
@ 2021-03-28 9:01 57% ` Eric Wong
2021-03-28 9:01 71% ` [PATCH 05/12] lei blob: fail early if no git dirs Eric Wong
` (5 subsequent siblings)
9 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-28 9:01 UTC (permalink / raw)
To: meta
It's possible for a abbreviated OID to be resolved unambiguously
to an email before we attempt to look at externals via xsearch;
so provide a way for a user to force searching coderepos.
If hints (--oid-a, --path-a, --path-b) are present, we'll
assume --no-mail by default, otherwise we'll assume the
user wants to look through mail for a matching blob.
---
lib/PublicInbox/LEI.pm | 4 ++--
lib/PublicInbox/LeiBlob.pm | 23 +++++++++++++----------
t/solver_git.t | 3 +++
3 files changed, 18 insertions(+), 12 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index fdb0bbcf..bad7fad9 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -136,8 +136,8 @@ our %CMD = ( # sorted in order of importance/use:
import-before! lock=s@ rsyncable alert=s@ mua=s verbose|v+), @c_opt,
opt_dash('limit|n=i', '[0-9]+') ],
-'blob' => [ 'OID', 'display a git blob object, solving if necessary',
- qw(git-dir=s@ cwd! verbose|v+ oid-a|A=s path-a|a=s path-b|b=s),
+'blob' => [ 'OID', 'show a git blob, reconstructing from mail if necessary',
+ qw(git-dir=s@ cwd! verbose|v+ mail! oid-a|A=s path-a|a=s path-b|b=s),
@lxs_opt, @c_opt ],
'add-external' => [ 'LOCATION',
diff --git a/lib/PublicInbox/LeiBlob.pm b/lib/PublicInbox/LeiBlob.pm
index 9b4c4f30..4bd86253 100644
--- a/lib/PublicInbox/LeiBlob.pm
+++ b/lib/PublicInbox/LeiBlob.pm
@@ -8,7 +8,6 @@ use v5.10.1;
use parent qw(PublicInbox::IPC);
use PublicInbox::Spawn qw(spawn popen_rd);
use PublicInbox::DS;
-use PublicInbox::Eml;
sub sol_done_wait { # dwaitpid callback
my ($arg, $pid) = @_;
@@ -85,18 +84,22 @@ sub do_solve_blob { # via wq_do
sub lei_blob {
my ($lei, $blob) = @_;
$lei->start_pager if -t $lei->{1};
+ my $opt = $lei->{opt};
+ my $has_hints = grep(defined, @$opt{qw(oid-a path-a path-b)});
- # first, see if it's a blob returned by "lei q" JSON output:
- my $rdr = { 1 => $lei->{1} };
- open $rdr->{2}, '>', '/dev/null' or die "open: $!";
- my $cmd = [ 'git', '--git-dir='.$lei->ale->git->{git_dir},
- 'cat-file', 'blob', $blob ];
- waitpid(spawn($cmd, $lei->{env}, $rdr), 0);
- return if $? == 0;
+ # first, see if it's a blob returned by "lei q" JSON output:k
+ if ($opt->{mail} // ($has_hints ? 0 : 1)) {
+ my $rdr = { 1 => $lei->{1} };
+ open $rdr->{2}, '>', '/dev/null' or die "open: $!";
+ my $cmd = [ 'git', '--git-dir='.$lei->ale->git->{git_dir},
+ 'cat-file', 'blob', $blob ];
+ waitpid(spawn($cmd, $lei->{env}, $rdr), 0);
+ return if $? == 0;
+ }
# maybe it's a non-email (code) blob from a coderepo
- my $git_dirs = $lei->{opt}->{'git-dir'} //= [];
- if ($lei->{opt}->{'cwd'} //= 1) {
+ my $git_dirs = $opt->{'git-dir'} //= [];
+ if ($opt->{'cwd'} // 1) {
my $cgd = get_git_dir('.');
unshift(@$git_dirs, $cgd) if defined $cgd;
}
diff --git a/t/solver_git.t b/t/solver_git.t
index 22714ae5..7bf3ba21 100644
--- a/t/solver_git.t
+++ b/t/solver_git.t
@@ -34,6 +34,9 @@ test_lei({tmpdir => $tmpdir}, sub {
lei_ok('blob', '69df7d5', '-I', $ibx->{inboxdir});
is(sha1_hex("blob ".length($lei_out)."\0".$lei_out),
$expect, 'blob contents output');
+ my $prev = $lei_out;
+ lei_ok(qw(blob --no-mail 69df7d5 -I), $ibx->{inboxdir});
+ is($lei_out, $prev, '--no-mail works');
# fallbacks
lei_ok('blob', $v1_0_0_tag, '-I', $ibx->{inboxdir});
^ permalink raw reply related [relevance 57%]
* [PATCH 01/12] lei: simplify PktOp callers
2021-03-28 9:01 67% [PATCH 00/12] lei blob and some yak-shaving Eric Wong
@ 2021-03-28 9:01 43% ` Eric Wong
2021-03-28 9:01 58% ` [PATCH 02/12] lei init: split out into separate file Eric Wong
` (8 subsequent siblings)
9 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-28 9:01 UTC (permalink / raw)
To: meta
Provide a consistent ->op_wait_event method instead of
forcing callers to loop (or not) at each callsite.
This also avoid a leak possibility by avoiding circular
references.
---
lib/PublicInbox/LEI.pm | 11 +++++------
lib/PublicInbox/LeiBlob.pm | 4 ++--
lib/PublicInbox/LeiConvert.pm | 4 ++--
lib/PublicInbox/LeiImport.pm | 4 ++--
lib/PublicInbox/LeiMark.pm | 4 ++--
lib/PublicInbox/LeiMirror.pm | 4 ++--
lib/PublicInbox/LeiP2q.pm | 4 ++--
lib/PublicInbox/LeiXSearch.pm | 8 +++-----
lib/PublicInbox/PktOp.pm | 20 +++++++++++++++-----
9 files changed, 35 insertions(+), 28 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 478912cd..9cacb142 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -494,11 +494,11 @@ sub _delete_pkt_op { # OnDestroy callback to prevent leaks on die
}
sub pkt_op_pair {
- my ($self, $ops) = @_;
+ my ($self) = @_;
require PublicInbox::OnDestroy;
require PublicInbox::PktOp;
my $end = PublicInbox::OnDestroy->new($$, \&_delete_pkt_op, $self);
- @$self{qw(pkt_op_c pkt_op_p)} = PublicInbox::PktOp->pair($ops);
+ @$self{qw(pkt_op_c pkt_op_p)} = PublicInbox::PktOp->pair;
$end;
}
@@ -512,14 +512,13 @@ sub workers_start {
($ops ? %$ops : ()),
};
$ops->{''} //= [ \&dclose, $lei ];
- my $end = $lei->pkt_op_pair($ops);
+ my $end = $lei->pkt_op_pair;
$wq->wq_workers_start($ident, $jobs, $lei->oldset, { lei => $lei });
delete $lei->{pkt_op_p};
- my $op = delete $lei->{pkt_op_c};
+ my $op_c = delete $lei->{pkt_op_c};
@$end = ();
$lei->event_step_init;
- # oneshot needs $op, daemon-mode uses DS->EventLoop to handle $op
- $lei->{oneshot} ? $op : undef;
+ ($op_c, $ops);
}
sub _help {
diff --git a/lib/PublicInbox/LeiBlob.pm b/lib/PublicInbox/LeiBlob.pm
index 2facbad3..97747220 100644
--- a/lib/PublicInbox/LeiBlob.pm
+++ b/lib/PublicInbox/LeiBlob.pm
@@ -103,12 +103,12 @@ sub lei_blob {
my $lxs = $lei->lxs_prepare or return;
require PublicInbox::SolverGit;
my $self = bless { lxs => $lxs, oid_b => $blob }, __PACKAGE__;
- my $op = $lei->workers_start($self, 'lei_solve', 1,
+ my ($op_c, $ops) = $lei->workers_start($self, 'lei_solve', 1,
{ '' => [ \&sol_done, $lei ] });
$lei->{sol} = $self;
$self->wq_io_do('do_solve_blob', []);
$self->wq_close(1);
- while ($op && $op->{sock}) { $op->event_step }
+ $op_c->op_wait_event($ops);
}
sub ipc_atfork_child {
diff --git a/lib/PublicInbox/LeiConvert.pm b/lib/PublicInbox/LeiConvert.pm
index 083ecc33..5d0adb14 100644
--- a/lib/PublicInbox/LeiConvert.pm
+++ b/lib/PublicInbox/LeiConvert.pm
@@ -53,11 +53,11 @@ sub lei_convert { # the main "lei convert" method
my $devfd = $lei->path_to_fd($ovv->{dst}) // return;
$lei->{opt}->{augment} = 1 if $devfd < 0;
$self->prepare_inputs($lei, \@inputs) or return;
- my $op = $lei->workers_start($self, 'lei_convert', 1);
+ my ($op_c, $ops) = $lei->workers_start($self, 'lei_convert', 1);
$lei->{cnv} = $self;
$self->wq_io_do('do_convert', []);
$self->wq_close(1);
- while ($op && $op->{sock}) { $op->event_step }
+ $op_c->op_wait_event($ops);
}
sub ipc_atfork_child {
diff --git a/lib/PublicInbox/LeiImport.pm b/lib/PublicInbox/LeiImport.pm
index 7c5b7d09..803b5cda 100644
--- a/lib/PublicInbox/LeiImport.pm
+++ b/lib/PublicInbox/LeiImport.pm
@@ -76,11 +76,11 @@ sub lei_import { # the main "lei import" method
my $ops = { '' => [ \&import_done, $lei ] };
$lei->{auth}->op_merge($ops, $self) if $lei->{auth};
$self->{-wq_nr_workers} = $j // 1; # locked
- my $op = $lei->workers_start($self, 'lei_import', undef, $ops);
+ my ($op_c, undef) = $lei->workers_start($self, 'lei_import', $j, $ops);
$lei->{imp} = $self;
$self->wq_io_do('input_stdin', []) if $self->{0};
net_merge_complete($self) unless $lei->{auth};
- while ($op && $op->{sock}) { $op->event_step }
+ $op_c->op_wait_event($ops);
}
no warnings 'once';
diff --git a/lib/PublicInbox/LeiMark.pm b/lib/PublicInbox/LeiMark.pm
index 34846b84..6e611318 100644
--- a/lib/PublicInbox/LeiMark.pm
+++ b/lib/PublicInbox/LeiMark.pm
@@ -116,11 +116,11 @@ sub lei_mark { # the "lei mark" method
my $ops = { '' => [ \&mark_done, $lei ] };
$lei->{auth}->op_merge($ops, $self) if $lei->{auth};
$self->{vmd_mod} = $vmd_mod;
- my $op = $lei->workers_start($self, 'lei_mark', 1, $ops);
+ my ($op_c, undef) = $lei->workers_start($self, 'lei_mark', 1, $ops);
$lei->{mark} = $self;
$self->wq_io_do('input_stdin', []) if $self->{0};
net_merge_complete($self) unless $lei->{auth};
- while ($op && $op->{sock}) { $op->event_step }
+ $op_c->op_wait_event($ops);
}
sub note_missing {
diff --git a/lib/PublicInbox/LeiMirror.pm b/lib/PublicInbox/LeiMirror.pm
index c83386c6..89574d28 100644
--- a/lib/PublicInbox/LeiMirror.pm
+++ b/lib/PublicInbox/LeiMirror.pm
@@ -282,13 +282,13 @@ sub start {
require PublicInbox::Inbox;
require PublicInbox::Admin;
require PublicInbox::InboxWritable;
- my $op = $lei->workers_start($self, 'lei_mirror', 1, {
+ my ($op, $ops) = $lei->workers_start($self, 'lei_mirror', 1, {
'' => [ \&mirror_done, $lei ]
});
$lei->{mrr} = $self;
$self->wq_io_do('do_mirror', []);
$self->wq_close(1);
- while ($op && $op->{sock}) { $op->event_step }
+ $op->op_wait_event($ops);
}
sub ipc_atfork_child {
diff --git a/lib/PublicInbox/LeiP2q.pm b/lib/PublicInbox/LeiP2q.pm
index 25f63a10..a8a3dd2c 100644
--- a/lib/PublicInbox/LeiP2q.pm
+++ b/lib/PublicInbox/LeiP2q.pm
@@ -185,11 +185,11 @@ sub lei_p2q { # the "lei patch-to-query" entry point
} else {
$self->{input} = $input;
}
- my $op = $lei->workers_start($self, 'lei_p2q', 1);
+ my ($op, $ops) = $lei->workers_start($self, 'lei_p2q', 1);
$lei->{p2q} = $self;
$self->wq_io_do('do_p2q', []);
$self->wq_close(1);
- while ($op && $op->{sock}) { $op->event_step }
+ $op->op_wait_event($ops);
}
sub ipc_atfork_child {
diff --git a/lib/PublicInbox/LeiXSearch.pm b/lib/PublicInbox/LeiXSearch.pm
index b41daffe..1a194f1c 100644
--- a/lib/PublicInbox/LeiXSearch.pm
+++ b/lib/PublicInbox/LeiXSearch.pm
@@ -427,7 +427,7 @@ sub do_query {
'incr_start_query' => [ \&incr_start_query, $self, $l2m ],
};
$lei->{auth}->op_merge($ops, $l2m) if $l2m && $lei->{auth};
- my $end = $lei->pkt_op_pair($ops);
+ my $end = $lei->pkt_op_pair;
$lei->{1}->autoflush(1);
$lei->start_pager if delete $lei->{need_pager};
$lei->{ovv}->ovv_begin($lei);
@@ -445,7 +445,7 @@ sub do_query {
}
$self->wq_workers_start('lei_xsearch', undef,
$lei->oldset, { lei => $lei });
- my $op = delete $lei->{pkt_op_c};
+ my $op_c = delete $lei->{pkt_op_c};
delete $lei->{pkt_op_p};
@$end = ();
$self->{threads} = $lei->{opt}->{threads};
@@ -455,9 +455,7 @@ sub do_query {
start_query($self);
}
$lei->event_step_init; # wait for shutdowns
- if ($lei->{oneshot}) {
- while ($op->{sock}) { $op->event_step }
- }
+ $op_c->op_wait_event($ops);
}
sub add_uri {
diff --git a/lib/PublicInbox/PktOp.pm b/lib/PublicInbox/PktOp.pm
index 5d8e78ea..c3221735 100644
--- a/lib/PublicInbox/PktOp.pm
+++ b/lib/PublicInbox/PktOp.pm
@@ -16,21 +16,23 @@ use PublicInbox::IPC qw(ipc_freeze ipc_thaw);
our @EXPORT_OK = qw(pkt_do);
sub new {
- my ($cls, $r, $ops) = @_;
- my $self = bless { sock => $r, ops => $ops }, $cls;
+ my ($cls, $r) = @_;
+ my $self = bless { sock => $r }, $cls;
if ($PublicInbox::DS::in_loop) { # iff using DS->EventLoop
$r->blocking(0);
$self->SUPER::new($r, EPOLLIN|EPOLLET);
+ } else {
+ $self->{blocking} = 1;
}
$self;
}
# returns a blessed object as the consumer, and a GLOB/IO for the producer
sub pair {
- my ($cls, $ops) = @_;
+ my ($cls) = @_;
my ($c, $p);
socketpair($c, $p, AF_UNIX, SOCK_SEQPACKET, 0) or die "socketpair: $!";
- (new($cls, $c, $ops), $p);
+ (new($cls, $c), $p);
}
sub pkt_do { # for the producer to trigger event_step in consumer
@@ -41,7 +43,7 @@ sub pkt_do { # for the producer to trigger event_step in consumer
sub close {
my ($self) = @_;
my $c = $self->{sock} or return;
- $c->blocking ? delete($self->{sock}) : $self->SUPER::close;
+ $self->{blocking} ? delete($self->{sock}) : $self->SUPER::close;
}
sub event_step {
@@ -73,4 +75,12 @@ sub event_step {
}
}
+# call this when we're ready to wait on events,
+# returns immediately if non-blocking
+sub op_wait_event {
+ my ($self, $ops) = @_;
+ $self->{ops} = $ops;
+ while ($self->{blocking} && $self->{sock}) { event_step($self) }
+}
+
1;
^ permalink raw reply related [relevance 43%]
* [PATCH 00/12] lei blob and some yak-shaving
@ 2021-03-28 9:01 67% Eric Wong
2021-03-28 9:01 43% ` [PATCH 01/12] lei: simplify PktOp callers Eric Wong
` (9 more replies)
0 siblings, 10 replies; 200+ results
From: Eric Wong @ 2021-03-28 9:01 UTC (permalink / raw)
To: meta
"lei blob" manages to work with HTTPS and HTTP .onion endpoints.
Eric Wong (12):
lei: simplify PktOp callers
lei init: split out into separate file
lei blob: dclose if already failed
lei blob: support --no-mail switch
lei blob: fail early if no git dirs
lei blob: some extra tests
lei help: show "NAME=VALUE" properly for -c
lei blob: flesh out help text
t/lei_store: ensure LeiSearch responds to ->isrch
lei blob: add remote external support
lei: drop coderepo placeholders, submodule TODO
treewide: shorten temporary filename
MANIFEST | 2 +
lib/PublicInbox/LEI.pm | 67 +++++++---------------------
lib/PublicInbox/LeiBlob.pm | 47 +++++++++++++-------
lib/PublicInbox/LeiConvert.pm | 4 +-
lib/PublicInbox/LeiHelp.pm | 2 +-
lib/PublicInbox/LeiImport.pm | 4 +-
lib/PublicInbox/LeiInit.pm | 41 +++++++++++++++++
lib/PublicInbox/LeiMark.pm | 4 +-
lib/PublicInbox/LeiMirror.pm | 4 +-
lib/PublicInbox/LeiOverview.pm | 2 +-
lib/PublicInbox/LeiP2q.pm | 4 +-
lib/PublicInbox/LeiRemote.pm | 81 ++++++++++++++++++++++++++++++++++
lib/PublicInbox/LeiXSearch.pm | 8 ++--
lib/PublicInbox/Msgmap.pm | 2 +-
lib/PublicInbox/PktOp.pm | 20 ++++++---
lib/PublicInbox/SolverGit.pm | 2 +-
lib/PublicInbox/TestCommon.pm | 2 +-
lib/PublicInbox/V2Writable.pm | 6 +--
lib/PublicInbox/Xapcmd.pm | 9 ++--
script/public-inbox-edit | 2 +-
script/public-inbox-init | 2 +-
scripts/ssoma-replay | 2 +-
t/check-www-inbox.perl | 2 +-
t/inbox.t | 2 +-
t/lei_store.t | 1 +
t/nodatacow.t | 2 +-
t/solver_git.t | 40 +++++++++++++++--
27 files changed, 254 insertions(+), 110 deletions(-)
create mode 100644 lib/PublicInbox/LeiInit.pm
create mode 100644 lib/PublicInbox/LeiRemote.pm
^ permalink raw reply [relevance 67%]
* Re: is "lei mark" a good name?
2021-03-26 9:48 68% ` Eric Wong
@ 2021-03-28 2:21 71% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-28 2:21 UTC (permalink / raw)
To: Kyle Meyer; +Cc: meta
Eric Wong <e@80x24.org> wrote:
> Kyle Meyer <kyle@kyleam.com> wrote:
> > Eric Wong writes:
> >
> > > It can set/unset volatile metadata for any number of messages.
> > >
> > > "Volatile metadata" being "labels" (aka "mailboxes" in
> > > JMAP-speak) and "keywords" (seen|flagged|answered|...),
> > > (aka "flags" in IMAP/Maildir-speak).
> > >
> > > "lei mark +kw:seen" # makes sense
> >
> > > "lei mark +L:some-folder-name" # might sound odd...
> > >
> > >
> > > AFAIK, notmuch uses "notmuch tag" which combines both labels
> > > and keywords into one thing: "tags". But I'm also not a
> > > notmuch user...
> >
> > (I am, though I still might be wrong.) Notmuch allows arbitrary tags to
> > be associated with message IDs. It has some automatic tags like
> > "attachment", "encrypted", and "signed" that it adds when indexing.
>
> Hmm..., JMAP has a "hasAttachment" property to search on.
>
> > By default, it also syncs some maildir flags to its tags (e.g., "F ->
> > flagged", "R -> replied", "No S -> unread").
> >
> > As far as I understand, tags are never connected up to the maildir
> > folder/location, though Notmuch does support a separate "folder:..."
> > search term.
>
> I've been thinking about storing source location data, too.
> Having the source of an email in lei/store would make it easier
> and faster to export keywords back to IMAP/Maildir.
>
> > So, I guess in JMAP terms, Notmuch tags would be "mailboxes" (a subset
> > of which are synced with maildir flags).
>
> The arbitrary nm tags (and "attachment|signed|encrypted) would be
> "mailboxes", yes. But the Maildir flags are JMAP keywords:
>
> # LeiToMail.pm:
> my %kw2char = ( # Maildir characters
> draft => 'D',
> flagged => 'F',
> answered => 'R',
> seen => 'S'
> );
>
> > > Would "lei tag" be better?
> >
> > Neither one really jumps out to me as better. "mark" sounds fine to me,
> > and I don't find "tag" any more or less odd for the +L case above.
>
> OK. I'm still on the fence...
>
> On one hand, "tag" is probably a more familiar term to users of
> existing software. Not just notmuch, but also other software
> for music metadata, software (debtags), etc...
>
> On the other hand, lei will support "lei show" to wrap "git show"
> with SolverGit support. Thus I'm worried "lei tag" might
> be misconstrued as a wrapper for "git tag"...
>
> But now that I think about it more; "lei show" probably won't
> treat git tree/tag/commit objects any differently than "git show".
> Perhaps "lei show" can be named "lei blob", instead.
OK, so I've gone ahead with "lei blob" so far...
"git show" has a plethora of options affecting diff generation
which would be redundant to have in lei. "lei blob" can focus
on reconstructing and showing blobs. The name also corresponds
to the JSON output field of "lei q".
I like "lei tag", more than "lei mark", since I think "tag" is
more commonly used for attaching metadata labels to stuff.
"lei tag" might still give people the idea that it's for
operating on git tags (because "lei blob" operates on git blobs);
but I don't think it's a huge risk since our code base does
nothing with git tags
^ permalink raw reply [relevance 71%]
* [PATCH] lei mark: relax label requirements
@ 2021-03-27 23:22 69% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-27 23:22 UTC (permalink / raw)
To: meta
It seems safe to use ALLCAPS labels like "INBOX" with Xapian.
We'll also allow single-character labels.
---
lib/PublicInbox/LeiMark.pm | 2 +-
t/lei-mark.t | 6 ++++++
2 files changed, 7 insertions(+), 1 deletion(-)
diff --git a/lib/PublicInbox/LeiMark.pm b/lib/PublicInbox/LeiMark.pm
index 6d236411..34846b84 100644
--- a/lib/PublicInbox/LeiMark.pm
+++ b/lib/PublicInbox/LeiMark.pm
@@ -23,7 +23,7 @@ my %ERR = (
my ($label) = @_;
length($label) >= $L_MAX and
return "`$label' too long (must be <= $L_MAX)";
- $label =~ m{\A[a-z0-9_][a-z0-9_\-\./\@,]*[a-z0-9]\z} ?
+ $label =~ m{\A[a-z0-9_](?:[a-z0-9_\-\./\@,]*[a-z0-9])?\z}i ?
undef : "`$label' is invalid";
},
kw => sub {
diff --git a/t/lei-mark.t b/t/lei-mark.t
index 23f5002e..7855839e 100644
--- a/t/lei-mark.t
+++ b/t/lei-mark.t
@@ -78,6 +78,12 @@ test_lei(sub {
lei_ok(qw(ls-label));
is($lei_out, "nope\nqp\nurgent\n", 'ls-label shows qp');
+ lei_ok qw(mark -F eml t/utf8.eml +L:INBOX +L:x); diag $lei_err;
+ lei_ok qw(q m:testmessage@example.com);
+ $check_kw->([qw(answered seen)], L => [qw(INBOX nope urgent x)]);
+ lei_ok(qw(ls-label));
+ is($lei_out, "INBOX\nnope\nqp\nurgent\nx\n", 'ls-label shows qp');
+
if (0) { # TODO label+kw search w/ externals
lei_ok(qw(q L:qp), "mid:$mid", '--only', "$ro_home/t2");
}
^ permalink raw reply related [relevance 69%]
* [SQUASH] lei blob: use absolute path
2021-03-27 11:45 38% ` [PATCH 4/4] lei blob: aka "git-show-harder" for blobs Eric Wong
@ 2021-03-27 20:20 71% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-27 20:20 UTC (permalink / raw)
To: meta
I found this bug because I /wasn't/ in an alternate git worktree :x
---
lib/PublicInbox/LeiBlob.pm | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/lib/PublicInbox/LeiBlob.pm b/lib/PublicInbox/LeiBlob.pm
index a50255aa..2facbad3 100644
--- a/lib/PublicInbox/LeiBlob.pm
+++ b/lib/PublicInbox/LeiBlob.pm
@@ -69,7 +69,9 @@ sub do_solve_blob { # via wq_do
$lei->{log_buf} = \$log_buf;
my $git = $lei->ale->git;
my $solver = bless {
- gits => [ map { PublicInbox::Git->new($_) } @$git_dirs ],
+ gits => [ map {
+ PublicInbox::Git->new($lei->rel2abs($_))
+ } @$git_dirs ],
user_cb => \&solver_user_cb,
uarg => $self,
# -cur_di, -qsp, -msg => temporary fields for Qspawn callbacks
^ permalink raw reply related [relevance 71%]
* [PATCH 0/4] lei blob (formerly known as "lei show")
@ 2021-03-27 11:45 90% Eric Wong
2021-03-27 11:45 90% ` [PATCH 2/4] lei help: move "lei help" into LeiHelp.pm Eric Wong
2021-03-27 11:45 38% ` [PATCH 4/4] lei blob: aka "git-show-harder" for blobs Eric Wong
0 siblings, 2 replies; 200+ results
From: Eric Wong @ 2021-03-27 11:45 UTC (permalink / raw)
To: meta
It's still a "git show" with blob reconstruction (aka "solver")
I'm not sure if lei <add|ls|forget>-coderepo is necessary,
since this supports multiple --git-dir args (and uses
the current working directory).
Maybe submodule directories could be scanned...
Eric Wong (4):
lei_ale: do not create store unnecessarily
lei help: move "lei help" into LeiHelp.pm
lei_query: hoist out lxs_prepare
lei blob: aka "git-show-harder" for blobs
MANIFEST | 1 +
lib/PublicInbox/LEI.pm | 26 ++++----
lib/PublicInbox/LeiALE.pm | 3 +-
lib/PublicInbox/LeiBlob.pm | 119 ++++++++++++++++++++++++++++++++++++
lib/PublicInbox/LeiHelp.pm | 3 +
lib/PublicInbox/LeiQuery.pm | 38 +++++++-----
t/lei-import.t | 2 +
t/solver_git.t | 14 ++++-
8 files changed, 173 insertions(+), 33 deletions(-)
create mode 100644 lib/PublicInbox/LeiBlob.pm
^ permalink raw reply [relevance 90%]
* [PATCH 2/4] lei help: move "lei help" into LeiHelp.pm
2021-03-27 11:45 90% [PATCH 0/4] lei blob (formerly known as "lei show") Eric Wong
@ 2021-03-27 11:45 90% ` Eric Wong
2021-03-27 11:45 38% ` [PATCH 4/4] lei blob: aka "git-show-harder" for blobs Eric Wong
1 sibling, 0 replies; 200+ results
From: Eric Wong @ 2021-03-27 11:45 UTC (permalink / raw)
To: meta
We need to load LeiHelp.pm anyways if somebody calls "lei help",
so save a few kB RAM for users who don't need help.
---
lib/PublicInbox/LEI.pm | 2 --
lib/PublicInbox/LeiHelp.pm | 3 +++
2 files changed, 3 insertions(+), 2 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index eb3ad9e2..e680f5f0 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -787,8 +787,6 @@ sub lei_daemon_kill {
kill($sig, $$) or fail($self, "kill($sig, $$): $!");
}
-sub lei_help { _help($_[0]) }
-
# Shell completion helper. Used by lei-completion.bash and hopefully
# other shells. Try to do as much here as possible to avoid redundancy
# and improve maintainability.
diff --git a/lib/PublicInbox/LeiHelp.pm b/lib/PublicInbox/LeiHelp.pm
index be31c2a8..9c1b30a1 100644
--- a/lib/PublicInbox/LeiHelp.pm
+++ b/lib/PublicInbox/LeiHelp.pm
@@ -97,4 +97,7 @@ EOF
undef;
}
+# the "lei help" command
+sub lei_help { $_[0]->_help }
+
1;
^ permalink raw reply related [relevance 90%]
* [PATCH 4/4] lei blob: aka "git-show-harder" for blobs
2021-03-27 11:45 90% [PATCH 0/4] lei blob (formerly known as "lei show") Eric Wong
2021-03-27 11:45 90% ` [PATCH 2/4] lei help: move "lei help" into LeiHelp.pm Eric Wong
@ 2021-03-27 11:45 38% ` Eric Wong
2021-03-27 20:20 71% ` [SQUASH] lei blob: use absolute path Eric Wong
1 sibling, 1 reply; 200+ results
From: Eric Wong @ 2021-03-27 11:45 UTC (permalink / raw)
To: meta
This implements blob reconstruction via SolverGit,
emulating the functionality of /$INBOX/$OID/s/ endpoint
in PublicInbox::WWW.
It uses the current working tree as a coderepo, and
accepts any number of --git-dir=$PATH args.
Remote externals are not yet supported.
---
MANIFEST | 1 +
lib/PublicInbox/LEI.pm | 24 ++++----
lib/PublicInbox/LeiBlob.pm | 119 +++++++++++++++++++++++++++++++++++++
t/lei-import.t | 2 +
t/solver_git.t | 14 ++++-
5 files changed, 146 insertions(+), 14 deletions(-)
create mode 100644 lib/PublicInbox/LeiBlob.pm
diff --git a/MANIFEST b/MANIFEST
index 6b2b33ac..64b3626f 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -181,6 +181,7 @@ lib/PublicInbox/KQNotify.pm
lib/PublicInbox/LEI.pm
lib/PublicInbox/LeiALE.pm
lib/PublicInbox/LeiAuth.pm
+lib/PublicInbox/LeiBlob.pm
lib/PublicInbox/LeiConvert.pm
lib/PublicInbox/LeiCurl.pm
lib/PublicInbox/LeiDedupe.pm
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index e680f5f0..478912cd 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -120,6 +120,9 @@ sub index_opt {
}
my @c_opt = qw(c=s@ C=s@ quiet|q);
+my @lxs_opt = (qw(remote! local! external! include|I=s@ exclude=s@ only=s@
+ import-remote! no-torsocks torsocks=s),
+ PublicInbox::LeiQuery::curl_opt());
# we generate shell completion + help using %CMD and %OPTDESC,
# see lei__complete() and PublicInbox::LeiHelp
@@ -127,16 +130,15 @@ my @c_opt = qw(c=s@ C=s@ quiet|q);
our %CMD = ( # sorted in order of importance/use:
'q' => [ '--stdin|SEARCH_TERMS...', 'search for messages matching terms',
'stdin|', # /|\z/ must be first for lone dash
+ @lxs_opt,
qw(save-as=s output|mfolder|o=s format|f=s dedupe|d=s threads|t+
- sort|s=s reverse|r offset=i remote! local! external! pretty
- include|I=s@ exclude=s@ only=s@ jobs|j=s globoff|g augment|a
- import-remote! import-before! lock=s@ rsyncable
- alert=s@ mua=s no-torsocks torsocks=s verbose|v+), @c_opt,
- PublicInbox::LeiQuery::curl_opt(), opt_dash('limit|n=i', '[0-9]+') ],
+ sort|s=s reverse|r offset=i pretty jobs|j=s globoff|g augment|a
+ import-before! lock=s@ rsyncable alert=s@ mua=s verbose|v+), @c_opt,
+ opt_dash('limit|n=i', '[0-9]+') ],
-'show' => [ 'MID|OID', 'show a given object (Message-ID or object ID)',
- qw(type=s solve! format|f=s dedupe|d=s threads|t remote local!
- verbose|v+), @c_opt, pass_through('git show') ],
+'blob' => [ 'OID', 'display a git blob object, solving if necessary',
+ qw(git-dir=s@ cwd! verbose|v+ oid-a|A=s path-a|a=s path-b|b=s),
+ @lxs_opt, @c_opt ],
'add-external' => [ 'LOCATION',
'add/set priority of a publicinbox|extindex for extra matches',
@@ -350,7 +352,7 @@ my %CONFIG_KEYS = (
'leistore.dir' => 'top-level storage location',
);
-my @WQ_KEYS = qw(lxs l2m imp mrr cnv p2q mark); # internal workers
+my @WQ_KEYS = qw(lxs l2m imp mrr cnv p2q mark sol); # internal workers
# pronounced "exit": x_it(1 << 8) => exit(1); x_it(13) => SIGPIPE
sub x_it ($$) {
@@ -726,10 +728,6 @@ sub _lei_store ($;$) {
};
}
-sub lei_show {
- my ($self, @argv) = @_;
-}
-
sub _config {
my ($self, @argv) = @_;
my %env = (%{$self->{env}}, GIT_CONFIG => undef);
diff --git a/lib/PublicInbox/LeiBlob.pm b/lib/PublicInbox/LeiBlob.pm
new file mode 100644
index 00000000..a50255aa
--- /dev/null
+++ b/lib/PublicInbox/LeiBlob.pm
@@ -0,0 +1,119 @@
+# Copyright (C) 2021 all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+
+# "lei blob $OID" command
+package PublicInbox::LeiBlob;
+use strict;
+use v5.10.1;
+use parent qw(PublicInbox::IPC);
+use PublicInbox::Spawn qw(spawn popen_rd);
+use PublicInbox::DS;
+use PublicInbox::Eml;
+
+sub sol_done_wait { # dwaitpid callback
+ my ($arg, $pid) = @_;
+ my (undef, $lei) = @$arg;
+ $lei->child_error($?) if $?;
+ $lei->dclose;
+}
+
+sub sol_done { # EOF callback for main daemon
+ my ($lei) = @_;
+ my $sol = delete $lei->{sol} or return;
+ $sol->wq_wait_old(\&sol_done_wait, $lei);
+}
+
+sub get_git_dir ($) {
+ my ($d) = @_;
+ return $d if -d "$d/objects" && -d "$d/refs" && -e "$d/HEAD";
+
+ my $cmd = [ qw(git rev-parse --git-dir) ];
+ my ($r, $pid) = popen_rd($cmd, {GIT_DIR => undef}, { '-C' => $d });
+ chomp(my $gd = do { local $/; <$r> });
+ waitpid($pid, 0) == $pid or die "BUG: waitpid @$cmd ($!)";
+ $? == 0 ? $gd : undef;
+}
+
+sub solver_user_cb { # called by solver when done
+ my ($res, $self) = @_;
+ my $lei = $self->{lei};
+ my $log_buf = delete $lei->{'log_buf'};
+ $$log_buf =~ s/^/# /sgm;
+ ref($res) eq 'ARRAY' or return $lei->fail($$log_buf);
+ $lei->qerr($$log_buf);
+ my ($git, $oid, $type, $size, $di) = @$res;
+ my $gd = $git->{git_dir};
+
+ # don't try to support all the git-show(1) options for non-blob,
+ # this is just a convenience:
+ $type ne 'blob' and
+ $lei->err("# $oid is a $type of $size bytes in:\n#\t$gd");
+
+ my $cmd = [ 'git', "--git-dir=$gd", 'show', $oid ];
+ my $rdr = { 1 => $lei->{1}, 2 => $lei->{2} };
+ waitpid(spawn($cmd, $lei->{env}, $rdr), 0);
+ $lei->child_error($?) if $?;
+}
+
+sub do_solve_blob { # via wq_do
+ my ($self) = @_;
+ my $lei = $self->{lei};
+ my $git_dirs = $lei->{opt}->{'git-dir'};
+ my $hints = {};
+ for my $x (qw(oid-a path-a path-b)) {
+ my $v = $lei->{opt}->{$x} // next;
+ $x =~ tr/-/_/;
+ $hints->{$x} = $v;
+ }
+ open my $log, '+>', \(my $log_buf = '') or die "PerlIO::scalar: $!";
+ $lei->{log_buf} = \$log_buf;
+ my $git = $lei->ale->git;
+ my $solver = bless {
+ gits => [ map { PublicInbox::Git->new($_) } @$git_dirs ],
+ user_cb => \&solver_user_cb,
+ uarg => $self,
+ # -cur_di, -qsp, -msg => temporary fields for Qspawn callbacks
+ inboxes => [ $self->{lxs}->locals ],
+ }, 'PublicInbox::SolverGit';
+ $lei->{env}->{'psgi.errors'} = $lei->{2}; # ugh...
+ local $PublicInbox::DS::in_loop = 0; # waitpid synchronously
+ $solver->solve($lei->{env}, $log, $self->{oid_b}, $hints);
+}
+
+sub lei_blob {
+ my ($lei, $blob) = @_;
+ $lei->start_pager if -t $lei->{1};
+
+ # first, see if it's a blob returned by "lei q" JSON output:
+ my $rdr = { 1 => $lei->{1} };
+ open $rdr->{2}, '>', '/dev/null' or die "open: $!";
+ my $cmd = [ 'git', '--git-dir='.$lei->ale->git->{git_dir},
+ 'cat-file', 'blob', $blob ];
+ waitpid(spawn($cmd, $lei->{env}, $rdr), 0);
+ return if $? == 0;
+
+ # maybe it's a non-email (code) blob from a coderepo
+ my $git_dirs = $lei->{opt}->{'git-dir'} //= [];
+ if ($lei->{opt}->{'cwd'} //= 1) {
+ my $cgd = get_git_dir('.');
+ unshift(@$git_dirs, $cgd) if defined $cgd;
+ }
+ my $lxs = $lei->lxs_prepare or return;
+ require PublicInbox::SolverGit;
+ my $self = bless { lxs => $lxs, oid_b => $blob }, __PACKAGE__;
+ my $op = $lei->workers_start($self, 'lei_solve', 1,
+ { '' => [ \&sol_done, $lei ] });
+ $lei->{sol} = $self;
+ $self->wq_io_do('do_solve_blob', []);
+ $self->wq_close(1);
+ while ($op && $op->{sock}) { $op->event_step }
+}
+
+sub ipc_atfork_child {
+ my ($self) = @_;
+ $self->{lei}->_lei_atfork_child;
+ $SIG{__WARN__} = PublicInbox::Eml::warn_ignore_cb();
+ $self->SUPER::ipc_atfork_child;
+}
+
+1;
diff --git a/t/lei-import.t b/t/lei-import.t
index fa40ad01..33ce490d 100644
--- a/t/lei-import.t
+++ b/t/lei-import.t
@@ -54,6 +54,8 @@ is($res->[0]->{'m'}, 'x@y', 'got expected message');
is($res->[0]->{kw}, undef, 'Status ignored for eml');
lei_ok(qw(q -f mboxrd m:x@y));
unlike($lei_out, qr/^Status:/, 'no Status: in imported message');
+lei_ok('blob', $res->[0]->{blob});
+is($lei_out, "From: a\@b\nMessage-ID: <x\@y>\n", 'got blob back');
$eml->header_set('Message-ID', '<v@y>');
diff --git a/t/solver_git.t b/t/solver_git.t
index 99ffb9e3..22714ae5 100644
--- a/t/solver_git.t
+++ b/t/solver_git.t
@@ -6,6 +6,7 @@ use v5.10.1;
use PublicInbox::TestCommon;
use Cwd qw(abs_path);
require_git(2.6);
+use Digest::SHA qw(sha1_hex);
use PublicInbox::Spawn qw(popen_rd);
require_mods(qw(DBD::SQLite Search::Xapian Plack::Util));
my $git_dir = xqx([qw(git rev-parse --git-dir)], undef, {2 => \(my $null)});
@@ -27,6 +28,18 @@ my $ibx = create_inbox 'v2', version => 2,
};
my $v1_0_0_tag = 'cb7c42b1e15577ed2215356a2bf925aef59cdd8d';
my $v1_0_0_tag_short = substr($v1_0_0_tag, 0, 16);
+my $expect = '69df7d565d49fbaaeb0a067910f03dc22cd52bd0';
+
+test_lei({tmpdir => $tmpdir}, sub {
+ lei_ok('blob', '69df7d5', '-I', $ibx->{inboxdir});
+ is(sha1_hex("blob ".length($lei_out)."\0".$lei_out),
+ $expect, 'blob contents output');
+
+ # fallbacks
+ lei_ok('blob', $v1_0_0_tag, '-I', $ibx->{inboxdir});
+ lei_ok('blob', $v1_0_0_tag_short, '-I', $ibx->{inboxdir});
+});
+
my $git = PublicInbox::Git->new($git_dir);
$ibx->{-repo_objs} = [ $git ];
my $res;
@@ -38,7 +51,6 @@ $solver->solve($psgi_env, $log, '69df7d5', {});
ok($res, 'solved a blob!');
my $wt_git = $res->[0];
is(ref($wt_git), 'PublicInbox::Git', 'got a git object for the blob');
-my $expect = '69df7d565d49fbaaeb0a067910f03dc22cd52bd0';
is($res->[1], $expect, 'resolved blob to unabbreviated identifier');
is($res->[2], 'blob', 'type specified');
is($res->[3], 4405, 'size returned');
^ permalink raw reply related [relevance 38%]
* labels for externals [was: lei labels support]
2021-03-26 4:29 71% [PATCH 0/3] lei labels support Eric Wong
2021-03-26 4:29 71% ` [PATCH 2/3] lei: _lei_store: use default even if unconfigured Eric Wong
2021-03-26 4:29 30% ` [PATCH 3/3] lei: add some labels support Eric Wong
@ 2021-03-26 10:31 69% ` Eric Wong
2 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-26 10:31 UTC (permalink / raw)
To: meta
Eric Wong <e@80x24.org> wrote:
> Unfortunately, it's limited to imported messages and
> searching for labeled external messages doesn't work,
> yet. I'm not sure how to best go about this...
I'm thinking adding a label to an external-only message
should cause it to be imported into lei/store.
Related to "lei: per-message keywords and externals" at
https://public-inbox.org/meta/20210224204950.GA2076@dcvr/
in that it's technically the same problem, but not semantically.
The key difference between keywords and labels is labels require
explicit, concious effort on a user's part to "lei mark" a
message. Even within an MUA, copying/moving a message to a new
mailbox requires explicit user action.
Keyword updates are largely transparent from the MUA, with
"seen" is the most common keyword. Default MUA behavior is to
set the "seen" keyword on any read message. "seen" keyword
updates will be even more transparent when inotify/IMAP IDLE
support is added to lei.
So again, I think users should get an automatic import when they
label. Maybe implementing indexlevel=medium on a per-message level
could be used to save space.
git storage shouldn't be too expensive, and alternates can be
used safely unless the external was is subject to
-purge/edit-prone
^ permalink raw reply [relevance 69%]
* [PATCH 2/4] lei: do not blindly commit to lei/store on close
2021-03-26 9:51 71% [PATCH 0/4] lei minor things Eric Wong
2021-03-26 9:51 68% ` [PATCH 1/4] lei q: skip lei/store->write_prepare for JSON outputs Eric Wong
@ 2021-03-26 9:51 90% ` Eric Wong
2021-03-26 9:51 42% ` [PATCH 3/4] lei: support /dev/fd/[0-2] inputs and outputs in daemon Eric Wong
2021-03-26 9:51 71% ` [PATCH 4/4] lei mark: disallow '!' in labels Eric Wong
3 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-26 9:51 UTC (permalink / raw)
To: meta
It may hide errors/bugs, instead do it explicitly for each
worker that writes to it. For lei_xsearch, it will be better
to close before spawning the MUA for future use since we may
need it again once the user starts changing keywords.
---
lib/PublicInbox/LEI.pm | 3 ---
lib/PublicInbox/LeiXSearch.pm | 1 +
2 files changed, 1 insertion(+), 3 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 6a5c32b3..59715633 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -1030,9 +1030,6 @@ sub dclose {
}
}
close(delete $self->{1}) if $self->{1}; # may reap_compress
- if (my $sto = delete $self->{sto}) {
- $sto->ipc_do('done');
- }
$self->close if $self->{-event_init_done}; # PublicInbox::DS::close
}
diff --git a/lib/PublicInbox/LeiXSearch.pm b/lib/PublicInbox/LeiXSearch.pm
index 6410e0ea..b41daffe 100644
--- a/lib/PublicInbox/LeiXSearch.pm
+++ b/lib/PublicInbox/LeiXSearch.pm
@@ -319,6 +319,7 @@ sub query_done { # EOF callback for main daemon
if (my $lxs = delete $lei->{lxs}) {
$lxs->wq_wait_old(\&xsearch_done_wait, $lei);
}
+ my $wait = $lei->{sto} ? $lei->{sto}->ipc_do('done') : undef;
$lei->{ovv}->ovv_end($lei);
if ($l2m) { # close() calls LeiToMail reap_compress
if (my $out = delete $lei->{old_1}) {
^ permalink raw reply related [relevance 90%]
* [PATCH 4/4] lei mark: disallow '!' in labels
2021-03-26 9:51 71% [PATCH 0/4] lei minor things Eric Wong
` (2 preceding siblings ...)
2021-03-26 9:51 42% ` [PATCH 3/4] lei: support /dev/fd/[0-2] inputs and outputs in daemon Eric Wong
@ 2021-03-26 9:51 71% ` Eric Wong
3 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-26 9:51 UTC (permalink / raw)
To: meta
'!' could wreak havoc if exposed to a shell like bash. It seems
like a rare character for use in file/directory/mailbox names.
---
lib/PublicInbox/LeiMark.pm | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/lib/PublicInbox/LeiMark.pm b/lib/PublicInbox/LeiMark.pm
index 7a2ccf77..6d236411 100644
--- a/lib/PublicInbox/LeiMark.pm
+++ b/lib/PublicInbox/LeiMark.pm
@@ -23,7 +23,7 @@ my %ERR = (
my ($label) = @_;
length($label) >= $L_MAX and
return "`$label' too long (must be <= $L_MAX)";
- $label =~ m{\A[a-z0-9_][a-z0-9_\-\./\@\!,]*[a-z0-9]\z} ?
+ $label =~ m{\A[a-z0-9_][a-z0-9_\-\./\@,]*[a-z0-9]\z} ?
undef : "`$label' is invalid";
},
kw => sub {
^ permalink raw reply related [relevance 71%]
* [PATCH 1/4] lei q: skip lei/store->write_prepare for JSON outputs
2021-03-26 9:51 71% [PATCH 0/4] lei minor things Eric Wong
@ 2021-03-26 9:51 68% ` Eric Wong
2021-03-26 9:51 90% ` [PATCH 2/4] lei: do not blindly commit to lei/store on close Eric Wong
` (2 subsequent siblings)
3 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-26 9:51 UTC (permalink / raw)
To: meta
JSON outputs won't write to lei/store at all, so there's
no point in forking the store worker if it's not already
running.
LeiSearch object ($lse) is also fork-safe until it opens a
persistent FD for Xapian/SQLite so we can unconditionally
carry it across fork.
---
lib/PublicInbox/LeiOverview.pm | 4 ++--
lib/PublicInbox/LeiQuery.pm | 16 ++++++++--------
lib/PublicInbox/LeiToMail.pm | 16 ++++++++--------
lib/PublicInbox/LeiXSearch.pm | 1 +
4 files changed, 19 insertions(+), 18 deletions(-)
diff --git a/lib/PublicInbox/LeiOverview.pm b/lib/PublicInbox/LeiOverview.pm
index b4d81328..96bfff24 100644
--- a/lib/PublicInbox/LeiOverview.pm
+++ b/lib/PublicInbox/LeiOverview.pm
@@ -223,7 +223,7 @@ sub ovv_each_smsg_cb { # runs in wq worker usually
}
} elsif ($self->{fmt} =~ /\A(concat)?json\z/ && $lei->{opt}->{pretty}) {
my $EOR = ($1//'') eq 'concat' ? "\n}" : "\n},";
- my $lse = $lei->{sto}->search;
+ my $lse = $lei->{lse};
sub { # DIY prettiness :P
my ($smsg, $mitem) = @_;
return if $dedupe->is_smsg_dup($smsg);
@@ -247,7 +247,7 @@ sub ovv_each_smsg_cb { # runs in wq worker usually
}
} elsif ($json) {
my $ORS = $self->{fmt} eq 'json' ? ",\n" : "\n"; # JSONL
- my $lse = $lei->{sto}->search;
+ my $lse = $lei->{lse};
sub {
my ($smsg, $mitem) = @_;
return if $dedupe->is_smsg_dup($smsg);
diff --git a/lib/PublicInbox/LeiQuery.pm b/lib/PublicInbox/LeiQuery.pm
index 84996e7e..65aa9e87 100644
--- a/lib/PublicInbox/LeiQuery.pm
+++ b/lib/PublicInbox/LeiQuery.pm
@@ -25,8 +25,7 @@ sub qstr_add { # PublicInbox::InputPipe::consume callback for --stdin
my ($self) = @_; # $_[1] = $rbuf
if (defined($_[1])) {
$_[1] eq '' and return eval {
- my $lse = delete $self->{lse};
- $lse->query_approxidate($lse->git,
+ $self->{lse}->query_approxidate($self->{lse}->git,
$self->{mset_opt}->{qstr});
_start_query($self);
};
@@ -50,11 +49,7 @@ sub lei_q {
# --local is enabled by default unless --only is used
# we'll allow "--only $LOCATION --local"
my $sto = $self->_lei_store(1);
- if (($opt->{'import-remote'} //= 1) |
- (($opt->{'import-before'} //= \1) ? 1 : 0)) {
- $sto->write_prepare($self);
- }
- my $lse = $sto->search;
+ my $lse = $self->{lse} = $sto->search;
if ($opt->{'local'} //= scalar(@only) ? 0 : 1) {
$lxs->prepare_external($lse);
}
@@ -103,6 +98,12 @@ sub lei_q {
return $self->fail("`$mj' writer jobs must be >= 1");
}
PublicInbox::LeiOverview->new($self) or return;
+ if ($self->{l2m} && ($opt->{'import-remote'} //= 1) |
+ # we use \1 (a ref) to distinguish between
+ # user-supplied and default value
+ (($opt->{'import-before'} //= \1) ? 1 : 0)) {
+ $sto->write_prepare($self);
+ }
$self->{l2m} and $self->{l2m}->{-wq_nr_workers} = $mj // do {
$mj = POSIX::lround($nproc * 3 / 4); # keep some CPU for git
$mj <= 0 ? 1 : $mj;
@@ -131,7 +132,6 @@ sub lei_q {
no query allowed on command-line with --stdin
require PublicInbox::InputPipe;
- $self->{lse} = $lse; # for query_approxidate
PublicInbox::InputPipe::consume($self->{0}, \&qstr_add, $self);
return;
}
diff --git a/lib/PublicInbox/LeiToMail.pm b/lib/PublicInbox/LeiToMail.pm
index 1be15707..f71f74cc 100644
--- a/lib/PublicInbox/LeiToMail.pm
+++ b/lib/PublicInbox/LeiToMail.pm
@@ -263,7 +263,7 @@ sub _mbox_write_cb ($$) {
my $atomic_append = !defined($ovv->{lock_path});
my $dedupe = $lei->{dedupe};
$dedupe->prepare_dedupe;
- my $lse = $lei->{sto} ? $lei->{sto}->search : undef;
+ my $lse = $lei->{lse}; # may be undef
sub { # for git_to_mail
my ($buf, $smsg, $eml) = @_;
$eml //= PublicInbox::Eml->new($buf);
@@ -352,7 +352,7 @@ sub _maildir_write_cb ($$) {
my $dedupe = $lei->{dedupe};
$dedupe->prepare_dedupe if $dedupe;
my $dst = $lei->{ovv}->{dst};
- my $lse = $lei->{sto} ? $lei->{sto}->search : undef;
+ my $lse = $lei->{lse}; # may be undef
sub { # for git_to_mail
my ($buf, $smsg, $eml) = @_;
$dst // return $lei->fail; # dst may be undef-ed in last run
@@ -373,7 +373,7 @@ sub _imap_write_cb ($$) {
my $imap_append = $lei->{net}->can('imap_append');
my $mic = $lei->{net}->mic_get($self->{uri});
my $folder = $self->{uri}->mailbox;
- my $lse = $lei->{sto} ? $lei->{sto}->search : undef;
+ my $lse = $lei->{lse}; # may be undef
sub { # for git_to_mail
my ($bref, $smsg, $eml) = @_;
$mic // return $lei->fail; # dst may be undef-ed in last run
@@ -449,7 +449,7 @@ sub _pre_augment_maildir {
sub _do_augment_maildir {
my ($self, $lei) = @_;
my $dst = $lei->{ovv}->{dst};
- my $lse = $lei->{sto}->search if $lei->{opt}->{'import-before'};
+ my $lse = $lei->{opt}->{'import-before'} ? $lei->{lse} : undef;
my ($mod, $shard) = @{$self->{shard_info} // []};
if ($lei->{opt}->{augment}) {
my $dedupe = $lei->{dedupe};
@@ -481,7 +481,7 @@ sub _imap_augment_or_delete { # PublicInbox::NetReader::imap_each cb
sub _do_augment_imap {
my ($self, $lei) = @_;
my $net = $lei->{net};
- my $lse = $lei->{sto}->search if $lei->{opt}->{'import-before'};
+ my $lse = $lei->{opt}->{'import-before'} ? $lei->{lse} : undef;
if ($lei->{opt}->{augment}) {
my $dedupe = $lei->{dedupe};
if ($dedupe && $dedupe->prepare_dedupe) {
@@ -523,9 +523,9 @@ sub _pre_augment_mbox {
die "seek($dst): $!\n";
}
if (!$self->{seekable}) {
- my $ia = $lei->{opt}->{'import-before'};
+ my $imp_before = $lei->{opt}->{'import-before'};
die "--import-before specified but $dst is not seekable\n"
- if $ia && !ref($ia);
+ if $imp_before && !ref($imp_before);
die "--augment specified but $dst is not seekable\n" if
$lei->{opt}->{augment};
}
@@ -562,7 +562,7 @@ sub _do_augment_mbox {
$dedupe->prepare_dedupe if $dedupe;
}
if ($opt->{'import-before'}) { # the default
- my $lse = $lei->{sto}->search;
+ my $lse = $lei->{lse};
PublicInbox::MboxReader->$fmt($rd, \&_mbox_augment_kw_maybe,
$lei, $lse, $opt->{augment});
if (!$opt->{augment} and !truncate($out, 0)) {
diff --git a/lib/PublicInbox/LeiXSearch.pm b/lib/PublicInbox/LeiXSearch.pm
index f64b2c62..6410e0ea 100644
--- a/lib/PublicInbox/LeiXSearch.pm
+++ b/lib/PublicInbox/LeiXSearch.pm
@@ -430,6 +430,7 @@ sub do_query {
$lei->{1}->autoflush(1);
$lei->start_pager if delete $lei->{need_pager};
$lei->{ovv}->ovv_begin($lei);
+ die 'BUG: xdb|over open' if $lei->{lse}->{xdb} || $lei->{lse}->{over};
if ($l2m) {
$l2m->pre_augment($lei);
if ($lei->{opt}->{augment} && delete $lei->{early_mua}) {
^ permalink raw reply related [relevance 68%]
* [PATCH 3/4] lei: support /dev/fd/[0-2] inputs and outputs in daemon
2021-03-26 9:51 71% [PATCH 0/4] lei minor things Eric Wong
2021-03-26 9:51 68% ` [PATCH 1/4] lei q: skip lei/store->write_prepare for JSON outputs Eric Wong
2021-03-26 9:51 90% ` [PATCH 2/4] lei: do not blindly commit to lei/store on close Eric Wong
@ 2021-03-26 9:51 42% ` Eric Wong
2021-03-26 9:51 71% ` [PATCH 4/4] lei mark: disallow '!' in labels Eric Wong
3 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-26 9:51 UTC (permalink / raw)
To: meta
Since lei-daemon won't have the same FDs as the client, we
need to special-case thse mappings and won't be able to open
arbitrary, non-standard FDs.
We also won't attempt to support /proc/self/fd/[0-2] since
that's a Linux-ism. /dev/fd/[0-2] and /dev/std{in,out,err}
are portable to FreeBSD, at least. mawk(1) also supports
/dev/std{out,err}, as does gawk(1) (which supports everything
we can support, and arbitrary /dev/fd/$FD).
---
lib/PublicInbox/LEI.pm | 23 ++++++++++-------------
lib/PublicInbox/LeiConvert.pm | 3 ++-
lib/PublicInbox/LeiInput.pm | 5 ++++-
lib/PublicInbox/LeiOverview.pm | 13 +++++++------
lib/PublicInbox/LeiP2q.pm | 7 +++++--
lib/PublicInbox/LeiToMail.pm | 12 ++++++++----
t/lei-convert.t | 2 +-
t/lei-import.t | 2 +-
8 files changed, 38 insertions(+), 29 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 59715633..eb3ad9e2 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -464,7 +464,6 @@ sub _lei_atfork_child {
}
} else { # worker, Net::NNTP (Net::Cmd) uses STDERR directly
open STDERR, '+>&='.fileno($self->{2}) or warn "open $!";
- delete $self->{0};
}
for (delete @$self{qw(3 old_1 au_done)}) {
close($_) if defined($_);
@@ -929,19 +928,17 @@ sub poke_mua { # forces terminal MUAs to wake up and hopefully notice new mail
}
my %path_to_fd = ('/dev/stdin' => 0, '/dev/stdout' => 1, '/dev/stderr' => 2);
-$path_to_fd{"/dev/fd/$_"} = $path_to_fd{"/proc/self/fd/$_"} for (0..2);
-sub fopen {
- my ($self, $mode, $path) = @_;
- rel2abs($self, $path);
+$path_to_fd{"/dev/fd/$_"} = $_ for (0..2);
+
+# this also normalizes the path
+sub path_to_fd {
+ my ($self, $path) = @_;
+ $path = rel2abs($self, $path);
$path =~ tr!/!/!s;
- if (defined(my $fd = $path_to_fd{$path})) {
- return $self->{$fd};
- }
- if ($path =~ m!\A/(?:dev|proc/self)/fd/[0-9]+\z!) {
- return fail($self, "cannot open $path from daemon");
- }
- open my $fh, $mode, $path or return;
- $fh;
+ $path_to_fd{$path} // (
+ ($path =~ m!\A/(?:dev|proc/self)/fd/[0-9]+\z!) ?
+ fail($self, "cannot open $path from daemon") : -1
+ );
}
# caller needs to "-t $self->{1}" to check if tty
diff --git a/lib/PublicInbox/LeiConvert.pm b/lib/PublicInbox/LeiConvert.pm
index 0cc65108..083ecc33 100644
--- a/lib/PublicInbox/LeiConvert.pm
+++ b/lib/PublicInbox/LeiConvert.pm
@@ -50,7 +50,8 @@ sub lei_convert { # the main "lei convert" method
my $ovv = PublicInbox::LeiOverview->new($lei, 'out-format');
$lei->{l2m} or return
$lei->fail("output not specified or is not a mail destination");
- $lei->{opt}->{augment} = 1 unless $ovv->{dst} eq '/dev/stdout';
+ my $devfd = $lei->path_to_fd($ovv->{dst}) // return;
+ $lei->{opt}->{augment} = 1 if $devfd < 0;
$self->prepare_inputs($lei, \@inputs) or return;
my $op = $lei->workers_start($self, 'lei_convert', 1);
$lei->{cnv} = $self;
diff --git a/lib/PublicInbox/LeiInput.pm b/lib/PublicInbox/LeiInput.pm
index b059ecda..eed0eed7 100644
--- a/lib/PublicInbox/LeiInput.pm
+++ b/lib/PublicInbox/LeiInput.pm
@@ -67,7 +67,10 @@ sub input_path_url {
return;
}
$input =~ s!\A([a-z0-9]+):!!i and $ifmt = lc($1);
- if (-f $input) {
+ my $devfd = $lei->path_to_fd($input) // return;
+ if ($devfd >= 0) {
+ $self->input_fh($ifmt, $lei->{$devfd}, $input, @args);
+ } elsif (-f $input) {
my $m = $lei->{opt}->{'lock'} // ($ifmt eq 'eml' ? ['none'] :
PublicInbox::MboxLock->defaults);
my $mbl = PublicInbox::MboxLock->acq($input, 0, $m);
diff --git a/lib/PublicInbox/LeiOverview.pm b/lib/PublicInbox/LeiOverview.pm
index 96bfff24..8e26cba4 100644
--- a/lib/PublicInbox/LeiOverview.pm
+++ b/lib/PublicInbox/LeiOverview.pm
@@ -71,8 +71,9 @@ sub new {
--$ofmt_key=$fmt and --output=$ofmt conflict
}
- $fmt //= 'json' if $dst eq '/dev/stdout';
- $fmt //= detect_fmt($lei, $dst) or return;
+
+ my $devfd = $lei->path_to_fd($dst) // return;
+ $fmt //= $devfd >= 0 ? 'json' : (detect_fmt($lei, $dst) or return);
if (index($dst, '://') < 0) { # not a URL, so assume path
$dst = File::Spec->canonpath($dst);
@@ -84,11 +85,11 @@ sub new {
if ($fmt =~ /\A($JSONL|(?:concat)?json)\z/) {
$json = $self->{json} = ref(PublicInbox::Config->json);
}
- if ($dst eq '/dev/stdout') {
- my $isatty = $lei->{need_pager} = -t $lei->{1};
+ if ($devfd >= 0) {
+ my $isatty = $lei->{need_pager} = -t $lei->{$devfd};
$opt->{pretty} //= $isatty;
if (!$isatty && -f _) {
- my $fl = fcntl($lei->{1}, F_GETFL, 0) //
+ my $fl = fcntl($lei->{$devfd}, F_GETFL, 0) //
return $lei->fail("fcntl(stdout): $!");
ovv_out_lk_init($self) unless ($fl & O_APPEND);
} else {
@@ -101,7 +102,7 @@ sub new {
$lei->{dedupe} //= PublicInbox::LeiDedupe->new($lei);
} else {
# default to the cheapest sort since MUA usually resorts
- $opt->{'sort'} //= 'docid' if $dst ne '/dev/stdout';
+ $opt->{'sort'} //= 'docid' if $devfd < 0;
$lei->{l2m} = eval { PublicInbox::LeiToMail->new($lei) };
return $lei->fail($@) if $@;
if ($opt->{mua} && $lei->{l2m}->lock_free) {
diff --git a/lib/PublicInbox/LeiP2q.pm b/lib/PublicInbox/LeiP2q.pm
index fda055fe..25f63a10 100644
--- a/lib/PublicInbox/LeiP2q.pm
+++ b/lib/PublicInbox/LeiP2q.pm
@@ -107,8 +107,11 @@ sub do_p2q { # via wq_do
my $in = $self->{0};
unless ($in) {
my $input = $self->{input};
- if (-e $input) {
- $in = $lei->fopen('<', $input) or
+ my $devfd = $lei->path_to_fd($input) // return;
+ if ($devfd >= 0) {
+ $in = $lei->{$devfd};
+ } elsif (-e $input) {
+ open($in, '<', $input) or
return $lei->fail("open < $input: $!");
} else {
my @cmd = (qw(git format-patch --stdout -1), $input);
diff --git a/lib/PublicInbox/LeiToMail.pm b/lib/PublicInbox/LeiToMail.pm
index f71f74cc..88468c34 100644
--- a/lib/PublicInbox/LeiToMail.pm
+++ b/lib/PublicInbox/LeiToMail.pm
@@ -503,10 +503,12 @@ sub _do_augment_imap {
sub _pre_augment_mbox {
my ($self, $lei) = @_;
my $dst = $lei->{ovv}->{dst};
- my $out = $lei->{1};
- if ($dst ne '/dev/stdout') {
+ my $out;
+ my $devfd = $lei->path_to_fd($dst) // die "bad $dst";
+ if ($devfd >= 0) {
+ $out = $lei->{$devfd};
+ } else { # normal-looking path
if (-p $dst) {
- $out = undef;
open $out, '>', $dst or die "open($dst): $!";
} elsif (-f _ || !-e _) {
require PublicInbox::MboxLock;
@@ -514,12 +516,14 @@ sub _pre_augment_mbox {
PublicInbox::MboxLock->defaults;
$self->{mbl} = PublicInbox::MboxLock->acq($dst, 1, $m);
$out = $self->{mbl}->{fh};
+ } else {
+ die "$dst is not a file or FIFO\n";
}
$lei->{old_1} = $lei->{1}; # keep for spawning MUA
}
# Perl does SEEK_END even with O_APPEND :<
$self->{seekable} = seek($out, 0, SEEK_SET);
- if (!$self->{seekable} && $! != ESPIPE && $dst ne '/dev/stdout') {
+ if (!$self->{seekable} && $! != ESPIPE && !defined($devfd)) {
die "seek($dst): $!\n";
}
if (!$self->{seekable}) {
diff --git a/t/lei-convert.t b/t/lei-convert.t
index e147715d..9b430d8e 100644
--- a/t/lei-convert.t
+++ b/t/lei-convert.t
@@ -87,7 +87,7 @@ test_lei({ tmpdir => $tmpdir }, sub {
my $exp = do { local $/; <$fh> };
is($out, $exp, 'stdin => stdout');
- lei_ok qw(convert -F eml -o mboxcl2:/dev/stdout t/plack-qp.eml);
+ lei_ok qw(convert -F eml -o mboxcl2:/dev/fd/1 t/plack-qp.eml);
open $fh, '<', \$lei_out or BAIL_OUT;
@bar = ();
PublicInbox::MboxReader->mboxcl2($fh, sub {
diff --git a/t/lei-import.t b/t/lei-import.t
index a697d756..fa40ad01 100644
--- a/t/lei-import.t
+++ b/t/lei-import.t
@@ -69,7 +69,7 @@ is($res->[0]->{kw}, undef, 'no keywords set');
$eml->header_set('Message-ID', '<k@y>');
$in = 'From k@y Fri Oct 2 00:00:00 1993'."\n".$eml->as_string;
-lei_ok([qw(import -F mboxrd -)], undef, { %$lei_opt, 0 => \$in },
+lei_ok([qw(import -F mboxrd /dev/fd/0)], undef, { %$lei_opt, 0 => \$in },
\'import single file with --kw (default) from stdin');
lei(qw(q m:k@y));
$res = json_utf8->decode($lei_out);
^ permalink raw reply related [relevance 42%]
* [PATCH 0/4] lei minor things
@ 2021-03-26 9:51 71% Eric Wong
2021-03-26 9:51 68% ` [PATCH 1/4] lei q: skip lei/store->write_prepare for JSON outputs Eric Wong
` (3 more replies)
0 siblings, 4 replies; 200+ results
From: Eric Wong @ 2021-03-26 9:51 UTC (permalink / raw)
To: meta
Been putting off 3/4 off for a bit, still thinking about
labels a bit...
Eric Wong (4):
lei q: skip lei/store->write_prepare for JSON outputs
lei: do not blindly commit to lei/store on close
lei: support /dev/fd/[0-2] inputs and outputs in daemon
lei mark: disallow '!' in labels
lib/PublicInbox/LEI.pm | 26 ++++++++++----------------
lib/PublicInbox/LeiConvert.pm | 3 ++-
lib/PublicInbox/LeiInput.pm | 5 ++++-
lib/PublicInbox/LeiMark.pm | 2 +-
lib/PublicInbox/LeiOverview.pm | 17 +++++++++--------
lib/PublicInbox/LeiP2q.pm | 7 +++++--
lib/PublicInbox/LeiQuery.pm | 16 ++++++++--------
lib/PublicInbox/LeiToMail.pm | 28 ++++++++++++++++------------
lib/PublicInbox/LeiXSearch.pm | 2 ++
t/lei-convert.t | 2 +-
t/lei-import.t | 2 +-
11 files changed, 59 insertions(+), 51 deletions(-)
^ permalink raw reply [relevance 71%]
* Re: is "lei mark" a good name?
2021-03-26 1:54 71% ` Kyle Meyer
@ 2021-03-26 9:48 68% ` Eric Wong
2021-03-28 2:21 71% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2021-03-26 9:48 UTC (permalink / raw)
To: Kyle Meyer; +Cc: meta
Kyle Meyer <kyle@kyleam.com> wrote:
> Eric Wong writes:
>
> > It can set/unset volatile metadata for any number of messages.
> >
> > "Volatile metadata" being "labels" (aka "mailboxes" in
> > JMAP-speak) and "keywords" (seen|flagged|answered|...),
> > (aka "flags" in IMAP/Maildir-speak).
> >
> > "lei mark +kw:seen" # makes sense
>
> > "lei mark +L:some-folder-name" # might sound odd...
> >
> >
> > AFAIK, notmuch uses "notmuch tag" which combines both labels
> > and keywords into one thing: "tags". But I'm also not a
> > notmuch user...
>
> (I am, though I still might be wrong.) Notmuch allows arbitrary tags to
> be associated with message IDs. It has some automatic tags like
> "attachment", "encrypted", and "signed" that it adds when indexing.
Hmm..., JMAP has a "hasAttachment" property to search on.
> By default, it also syncs some maildir flags to its tags (e.g., "F ->
> flagged", "R -> replied", "No S -> unread").
>
> As far as I understand, tags are never connected up to the maildir
> folder/location, though Notmuch does support a separate "folder:..."
> search term.
I've been thinking about storing source location data, too.
Having the source of an email in lei/store would make it easier
and faster to export keywords back to IMAP/Maildir.
> So, I guess in JMAP terms, Notmuch tags would be "mailboxes" (a subset
> of which are synced with maildir flags).
The arbitrary nm tags (and "attachment|signed|encrypted) would be
"mailboxes", yes. But the Maildir flags are JMAP keywords:
# LeiToMail.pm:
my %kw2char = ( # Maildir characters
draft => 'D',
flagged => 'F',
answered => 'R',
seen => 'S'
);
> > Would "lei tag" be better?
>
> Neither one really jumps out to me as better. "mark" sounds fine to me,
> and I don't find "tag" any more or less odd for the +L case above.
OK. I'm still on the fence...
On one hand, "tag" is probably a more familiar term to users of
existing software. Not just notmuch, but also other software
for music metadata, software (debtags), etc...
On the other hand, lei will support "lei show" to wrap "git show"
with SolverGit support. Thus I'm worried "lei tag" might
be misconstrued as a wrapper for "git tag"...
But now that I think about it more; "lei show" probably won't
treat git tree/tag/commit objects any differently than "git show".
Perhaps "lei show" can be named "lei blob", instead.
Originally, I intended "lei show" to do something with commit
objects; but we already have "lei p2q" which lends itself
better to composability.
^ permalink raw reply [relevance 68%]
* [SQUASH 4/3] lei: account for unconfigured leistore.dir
2021-03-26 4:29 71% ` [PATCH 2/3] lei: _lei_store: use default even if unconfigured Eric Wong
@ 2021-03-26 5:01 71% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-26 5:01 UTC (permalink / raw)
To: meta
We just use the default directory if it's not in the
config.
---
lib/PublicInbox/LEI.pm | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index fab2af90..6a5c32b3 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -702,8 +702,9 @@ sub _lei_cfg ($;$) {
bless $cfg, 'PublicInbox::Config';
$cfg->{-st} = $cur_st;
$cfg->{'-f'} = $f;
- if ($sto && File::Spec->canonpath($sto_dir) eq
- File::Spec->canonpath($cfg->{'leistore.dir'})) {
+ if ($sto && File::Spec->canonpath($sto_dir // store_path($self))
+ eq File::Spec->canonpath($cfg->{'leistore.dir'} //
+ store_path($self))) {
$cfg->{-lei_store} = $sto;
}
if (scalar(keys %PATH2CFG) > 5) {
^ permalink raw reply related [relevance 71%]
* [PATCH 0/3] lei labels support
@ 2021-03-26 4:29 71% Eric Wong
2021-03-26 4:29 71% ` [PATCH 2/3] lei: _lei_store: use default even if unconfigured Eric Wong
` (2 more replies)
0 siblings, 3 replies; 200+ results
From: Eric Wong @ 2021-03-26 4:29 UTC (permalink / raw)
To: meta
Unfortunately, it's limited to imported messages and
searching for labeled external messages doesn't work,
yet. I'm not sure how to best go about this...
Eric Wong (3):
lei_xsearch: wait for kw updates for non-threaded case, too
lei: _lei_store: use default even if unconfigured
lei: add some labels support
MANIFEST | 1 +
lib/PublicInbox/LEI.pm | 5 +--
lib/PublicInbox/LeiLsLabel.pm | 17 ++++++++++
lib/PublicInbox/LeiMark.pm | 6 ++--
lib/PublicInbox/LeiOverview.pm | 4 +--
lib/PublicInbox/LeiSearch.pm | 37 ++++++++++++++++++---
lib/PublicInbox/LeiStore.pm | 59 +++++++++++++++++++++++++---------
lib/PublicInbox/LeiXSearch.pm | 17 ++++++----
lib/PublicInbox/Search.pm | 6 ++--
lib/PublicInbox/SearchIdx.pm | 2 +-
t/lei-mark.t | 46 +++++++++++++++++++++++---
11 files changed, 160 insertions(+), 40 deletions(-)
create mode 100644 lib/PublicInbox/LeiLsLabel.pm
^ permalink raw reply [relevance 71%]
* [PATCH 2/3] lei: _lei_store: use default even if unconfigured
2021-03-26 4:29 71% [PATCH 0/3] lei labels support Eric Wong
@ 2021-03-26 4:29 71% ` Eric Wong
2021-03-26 5:01 71% ` [SQUASH 4/3] lei: account for unconfigured leistore.dir Eric Wong
2021-03-26 4:29 30% ` [PATCH 3/3] lei: add some labels support Eric Wong
2021-03-26 10:31 69% ` labels for externals [was: lei labels support] Eric Wong
2 siblings, 1 reply; 200+ results
From: Eric Wong @ 2021-03-26 4:29 UTC (permalink / raw)
To: meta
Perhaps leistore.dir doesn't need to have a config file
entry if we're using the default location.
---
lib/PublicInbox/LEI.pm | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index d534f1d0..b42ba0ae 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -719,8 +719,8 @@ sub _lei_store ($;$) {
my $cfg = _lei_cfg($self, $creat);
$cfg->{-lei_store} //= do {
require PublicInbox::LeiStore;
- my $dir = $cfg->{'leistore.dir'};
- $dir //= $creat ? store_path($self) : return;
+ my $dir = $cfg->{'leistore.dir'} // store_path($self);
+ return unless $creat || -d $dir;
PublicInbox::LeiStore->new($dir, { creat => $creat });
};
}
^ permalink raw reply related [relevance 71%]
* [PATCH 3/3] lei: add some labels support
2021-03-26 4:29 71% [PATCH 0/3] lei labels support Eric Wong
2021-03-26 4:29 71% ` [PATCH 2/3] lei: _lei_store: use default even if unconfigured Eric Wong
@ 2021-03-26 4:29 30% ` Eric Wong
2021-03-26 10:31 69% ` labels for externals [was: lei labels support] Eric Wong
2 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-26 4:29 UTC (permalink / raw)
To: meta
"lei q" now displays labels in JSON output, "lei mark"
can add or remove labels for any messages.
"lei ls-label" is supported, too.
Unfortunately, "lei q" won't hande "kw:" or "L:" for
external messages, they must be imported, first.
---
MANIFEST | 1 +
lib/PublicInbox/LEI.pm | 1 +
lib/PublicInbox/LeiLsLabel.pm | 17 ++++++++++
lib/PublicInbox/LeiMark.pm | 6 ++--
lib/PublicInbox/LeiOverview.pm | 4 +--
lib/PublicInbox/LeiSearch.pm | 37 ++++++++++++++++++---
lib/PublicInbox/LeiStore.pm | 59 +++++++++++++++++++++++++---------
lib/PublicInbox/LeiXSearch.pm | 13 +++++---
lib/PublicInbox/Search.pm | 6 ++--
lib/PublicInbox/SearchIdx.pm | 2 +-
t/lei-mark.t | 46 +++++++++++++++++++++++---
11 files changed, 156 insertions(+), 36 deletions(-)
create mode 100644 lib/PublicInbox/LeiLsLabel.pm
diff --git a/MANIFEST b/MANIFEST
index 87e4b616..6b2b33ac 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -188,6 +188,7 @@ lib/PublicInbox/LeiExternal.pm
lib/PublicInbox/LeiHelp.pm
lib/PublicInbox/LeiImport.pm
lib/PublicInbox/LeiInput.pm
+lib/PublicInbox/LeiLsLabel.pm
lib/PublicInbox/LeiMark.pm
lib/PublicInbox/LeiMirror.pm
lib/PublicInbox/LeiOverview.pm
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index b42ba0ae..fab2af90 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -145,6 +145,7 @@ our %CMD = ( # sorted in order of importance/use:
PublicInbox::LeiQuery::curl_opt() ],
'ls-external' => [ '[FILTER]', 'list publicinbox|extindex locations',
qw(format|f=s z|0 globoff|g invert-match|v local remote), @c_opt ],
+'ls-label' => [ '', 'list labels', qw(z|0 stats:s), @c_opt ],
'forget-external' => [ 'LOCATION...|--prune',
'exclude further results from a publicinbox|extindex',
qw(prune), @c_opt ],
diff --git a/lib/PublicInbox/LeiLsLabel.pm b/lib/PublicInbox/LeiLsLabel.pm
new file mode 100644
index 00000000..474224d4
--- /dev/null
+++ b/lib/PublicInbox/LeiLsLabel.pm
@@ -0,0 +1,17 @@
+# Copyright (C) 2021 all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+
+# "lei ls-label" command
+package PublicInbox::LeiLsLabel;
+use strict;
+use v5.10.1;
+
+sub lei_ls_label { # the "lei ls-label" method
+ my ($lei, @argv) = @_;
+ # TODO: document stats/counts (expensive)
+ my @L = eval { $lei->_lei_store->search->all_terms('L') };
+ my $ORS = $lei->{opt}->{z} ? "\0" : "\n";
+ $lei->out(map { $_.$ORS } @L);
+}
+
+1;
diff --git a/lib/PublicInbox/LeiMark.pm b/lib/PublicInbox/LeiMark.pm
index 9d77f4b4..7a2ccf77 100644
--- a/lib/PublicInbox/LeiMark.pm
+++ b/lib/PublicInbox/LeiMark.pm
@@ -60,7 +60,7 @@ sub vmd_mod_extract {
sub input_eml_cb { # used by PublicInbox::LeiInput::input_fh
my ($self, $eml) = @_;
if (my $xoids = $self->{lei}->{ale}->xoids_for($eml)) {
- $self->{lei}->{sto}->ipc_do('update_xvmd', $xoids,
+ $self->{lei}->{sto}->ipc_do('update_xvmd', $xoids, $eml,
$self->{vmd_mod});
} else {
++$self->{missing};
@@ -168,7 +168,9 @@ sub _complete_mark_common ($) {
# FIXME: same problems as _complete_forget_external and similar
sub _complete_mark {
my ($self, @argv) = @_;
- my @all = map { ("+kw:$_", "-kw:$_") } @KW;
+ my @L = eval { $self->_lei_store->search->all_terms('L') };
+ my @all = ((map { ("+kw:$_", "-kw:$_") } @KW),
+ (map { ("+L:$_", "-L:$_") } @L));
return @all if !@argv;
my ($cur, $re) = _complete_mark_common(\@argv);
map {
diff --git a/lib/PublicInbox/LeiOverview.pm b/lib/PublicInbox/LeiOverview.pm
index 1ce2a098..b4d81328 100644
--- a/lib/PublicInbox/LeiOverview.pm
+++ b/lib/PublicInbox/LeiOverview.pm
@@ -227,7 +227,7 @@ sub ovv_each_smsg_cb { # runs in wq worker usually
sub { # DIY prettiness :P
my ($smsg, $mitem) = @_;
return if $dedupe->is_smsg_dup($smsg);
- $lse->xsmsg_vmd($smsg);
+ $lse->xsmsg_vmd($smsg, $smsg->{L} ? undef : 1);
$smsg = _unbless_smsg($smsg, $mitem);
$buf .= "{\n";
$buf .= join(",\n", map {
@@ -251,7 +251,7 @@ sub ovv_each_smsg_cb { # runs in wq worker usually
sub {
my ($smsg, $mitem) = @_;
return if $dedupe->is_smsg_dup($smsg);
- $lse->xsmsg_vmd($smsg);
+ $lse->xsmsg_vmd($smsg, $smsg->{L} ? undef : 1);
$buf .= $json->encode(_unbless_smsg(@_)) . $ORS;
return if length($buf) < 65536;
my $lk = $self->lock_for_scope;
diff --git a/lib/PublicInbox/LeiSearch.pm b/lib/PublicInbox/LeiSearch.pm
index bbb00661..07d570ec 100644
--- a/lib/PublicInbox/LeiSearch.pm
+++ b/lib/PublicInbox/LeiSearch.pm
@@ -27,18 +27,25 @@ sub msg_keywords {
wantarray ? sort(keys(%$kw)) : $kw;
}
+# lookup keywords+labels for external messages
sub xsmsg_vmd {
- my ($self, $smsg) = @_;
+ my ($self, $smsg, $want_label) = @_;
return if $smsg->{kw};
my $xdb = $self->xdb; # set {nshard};
- my %kw;
+ my (%kw, %L, $doc, $x);
$kw{flagged} = 1 if delete($smsg->{lei_q_tt_flagged});
my @num = $self->over->blob_exists($smsg->{blob});
for my $num (@num) { # there should only be one...
- my $kw = xap_terms('K', $xdb, num2docid($self, $num));
- %kw = (%kw, %$kw);
+ $doc = $xdb->get_document(num2docid($self, $num));
+ $x = xap_terms('K', $doc);
+ %kw = (%kw, %$x);
+ if ($want_label) { # JSON/JMAP only
+ $x = xap_terms('L', $doc);
+ %L = (%L, %$x);
+ }
}
$smsg->{kw} = [ sort keys %kw ] if scalar(keys(%kw));
+ $smsg->{L} = [ sort keys %L ] if scalar(keys(%L));
}
# when a message has no Message-IDs at all, this is needed for
@@ -100,4 +107,26 @@ sub kw_changed {
join("\0", @$new_kw_sorted) eq join("\0", @cur_kw) ? 0 : 1;
}
+sub all_terms {
+ my ($self, $pfx) = @_;
+ my $xdb = $self->xdb;
+ my $cur = $xdb->allterms_begin($pfx);
+ my $end = $xdb->allterms_end($pfx);
+ my %ret;
+ for (; $cur != $end; $cur++) {
+ my $tn = $cur->get_termname;
+ index($tn, $pfx) == 0 and
+ $ret{substr($tn, length($pfx))} = undef;
+ }
+ wantarray ? (sort keys %ret) : \%ret;
+}
+
+sub qparse_new {
+ my ($self) = @_;
+ my $qp = $self->SUPER::qparse_new; # PublicInbox::Search
+ $qp->add_boolean_prefix('kw', 'K');
+ $qp->add_boolean_prefix('L', 'L');
+ $qp
+}
+
1;
diff --git a/lib/PublicInbox/LeiStore.pm b/lib/PublicInbox/LeiStore.pm
index 1311ad46..b76af4d3 100644
--- a/lib/PublicInbox/LeiStore.pm
+++ b/lib/PublicInbox/LeiStore.pm
@@ -228,8 +228,30 @@ sub set_eml {
set_eml_vmd($self, $eml, $vmd);
}
+sub _external_only ($$$) {
+ my ($self, $xoids, $eml) = @_;
+ my $eidx = $self->{priv_eidx};
+ my $oidx = $eidx->{oidx} // die 'BUG: {oidx} missing';
+ my $smsg = bless { blob => '' }, 'PublicInbox::Smsg';
+ $smsg->{num} = $oidx->adj_counter('eidx_docid', '+');
+ # save space for an externals-only message
+ my $hdr = $eml->header_obj;
+ $smsg->populate($hdr); # sets lines == 0
+ $smsg->{bytes} = 0;
+ delete @$smsg{qw(From Subject)};
+ $smsg->{to} = $smsg->{cc} = $smsg->{from} = '';
+ $oidx->add_overview($hdr, $smsg); # subject+references for threading
+ $smsg->{subject} = '';
+ for my $oid (keys %$xoids) {
+ $oidx->add_xref3($smsg->{num}, -1, $oid, '.');
+ }
+ my $idx = $eidx->idx_shard($smsg->{num});
+ $idx->index_eml(PublicInbox::Eml->new("\n\n"), $smsg);
+ ($smsg, $idx);
+}
+
sub update_xvmd {
- my ($self, $xoids, $vmd_mod) = @_;
+ my ($self, $xoids, $eml, $vmd_mod) = @_;
my $eidx = eidx_init($self);
my $oidx = $eidx->{oidx};
my %seen;
@@ -242,7 +264,25 @@ sub update_xvmd {
my $idx = $eidx->idx_shard($docid);
$idx->ipc_do('update_vmd', $docid, $vmd_mod);
}
+ delete $xoids->{$oid};
}
+ return unless scalar(keys(%$xoids));
+
+ # see if it was indexed, but with different OID(s)
+ if (my @docids = _docids_for($self, $eml)) {
+ for my $docid (@docids) {
+ next if $seen{$docid};
+ for my $oid (keys %$xoids) {
+ $oidx->add_xref3($docid, -1, $oid, '.');
+ }
+ my $idx = $eidx->idx_shard($docid);
+ $idx->ipc_do('update_vmd', $docid, $vmd_mod);
+ }
+ return;
+ }
+ # totally unseen
+ my ($smsg, $idx) = _external_only($self, $xoids, $eml);
+ $idx->ipc_do('update_vmd', $smsg->{num}, $vmd_mod);
}
# set or update keywords for external message, called via ipc_do
@@ -270,6 +310,7 @@ sub set_xvmd {
# see if it was indexed, but with different OID(s)
if (my @docids = _docids_for($self, $eml)) {
for my $docid (@docids) {
+ next if $seen{$docid};
for my $oid (keys %$xoids) {
$oidx->add_xref3($docid, -1, $oid, '.');
}
@@ -279,21 +320,7 @@ sub set_xvmd {
return;
}
# totally unseen
- my $smsg = bless { blob => '' }, 'PublicInbox::Smsg';
- $smsg->{num} = $oidx->adj_counter('eidx_docid', '+');
- # save space for an externals-only message
- my $hdr = $eml->header_obj;
- $smsg->populate($hdr); # sets lines == 0
- $smsg->{bytes} = 0;
- delete @$smsg{qw(From Subject)};
- $smsg->{to} = $smsg->{cc} = $smsg->{from} = '';
- $oidx->add_overview($hdr, $smsg); # subject+references for threading
- $smsg->{subject} = '';
- for my $oid (keys %$xoids) {
- $oidx->add_xref3($smsg->{num}, -1, $oid, '.');
- }
- my $idx = $eidx->idx_shard($smsg->{num});
- $idx->index_eml(PublicInbox::Eml->new("\n\n"), $smsg);
+ my ($smsg, $idx) = _external_only($self, $xoids, $eml);
$idx->ipc_do('add_vmd', $smsg->{num}, $vmd);
}
diff --git a/lib/PublicInbox/LeiXSearch.pm b/lib/PublicInbox/LeiXSearch.pm
index 386c4eba..f64b2c62 100644
--- a/lib/PublicInbox/LeiXSearch.pm
+++ b/lib/PublicInbox/LeiXSearch.pm
@@ -69,11 +69,13 @@ sub xdb_shards_flat { @{$_[0]->{shards_flat} // []} }
sub mitem_kw ($$;$) {
my ($smsg, $mitem, $flagged) = @_;
- my $kw = xap_terms('K', $mitem->get_document);
+ my $kw = xap_terms('K', my $doc = $mitem->get_document);
$kw->{flagged} = 1 if $flagged;
- # we keep the empty array here to prevent expensive work in
+ # we keep the empty {kw} array here to prevent expensive work in
# ->xsmsg_vmd, _unbless_smsg will clobber it iff it's empty
$smsg->{kw} = [ sort keys %$kw ];
+ my $L = xap_terms('L', $doc);
+ $smsg->{L} = [ sort keys %$L ] if scalar(keys %$L);
}
# like over->get_art
@@ -86,8 +88,10 @@ sub smsg_for {
my $num = int(($docid - 1) / $nshard) + 1;
my $ibx = $self->{shard2ibx}->[$shard];
my $smsg = $ibx->over->get_art($num);
- return if $smsg->{bytes} == 0;
- mitem_kw($smsg, $mitem) if $ibx->can('msg_keywords');
+ return if $smsg->{bytes} == 0; # external message
+ if ($ibx->can('msg_keywords')) {
+ mitem_kw($smsg, $mitem);
+ }
$smsg;
}
@@ -170,6 +174,7 @@ sub query_thread_mset { # for --threads
if ($can_kw) {
mitem_kw($smsg, $mitem, $fl);
} elsif ($fl) {
+ # call ->xsmsg_vmd, later
$smsg->{lei_q_tt_flagged} = 1;
}
}
diff --git a/lib/PublicInbox/Search.pm b/lib/PublicInbox/Search.pm
index c7d52daf..ab04d430 100644
--- a/lib/PublicInbox/Search.pm
+++ b/lib/PublicInbox/Search.pm
@@ -370,7 +370,7 @@ sub query_approxidate {
sub mset {
my ($self, $query_string, $opts) = @_;
$opts ||= {};
- my $qp = $self->{qp} //= qparse_new($self);
+ my $qp = $self->{qp} //= $self->qparse_new;
my $query = $qp->parse_query($query_string, $self->{qp_flags});
_do_enquire($self, $query, $opts);
}
@@ -463,7 +463,7 @@ sub mset_to_smsg {
sub stemmer { $X{Stem}->new($LANG) }
# read-only
-sub qparse_new ($) {
+sub qparse_new {
my ($self) = @_;
my $xdb = xdb($self);
@@ -516,7 +516,7 @@ EOF
sub help {
my ($self) = @_;
- $self->{qp} //= qparse_new($self); # parse altids
+ $self->{qp} //= $self->qparse_new; # parse altids
my @ret = @HELP;
if (my $user_pfx = $self->{-user_pfx}) {
push @ret, @$user_pfx;
diff --git a/lib/PublicInbox/SearchIdx.pm b/lib/PublicInbox/SearchIdx.pm
index 7d46489c..ca1f3588 100644
--- a/lib/PublicInbox/SearchIdx.pm
+++ b/lib/PublicInbox/SearchIdx.pm
@@ -35,7 +35,7 @@ use constant DEBUG => !!$ENV{DEBUG};
my $xapianlevels = qr/\A(?:full|medium)\z/;
my $hex = '[a-f0-9]';
my $OID = $hex .'{40,}';
-my @VMD_MAP = (kw => 'K', label => 'L');
+my @VMD_MAP = (kw => 'K', L => 'L');
our $INDEXLEVELS = qr/\A(?:full|medium|basic)\z/;
sub new {
diff --git a/t/lei-mark.t b/t/lei-mark.t
index 76995589..23f5002e 100644
--- a/t/lei-mark.t
+++ b/t/lei-mark.t
@@ -4,22 +4,32 @@
use strict; use v5.10.1; use PublicInbox::TestCommon;
require_git 2.6;
require_mods(qw(json DBD::SQLite Search::Xapian));
+my ($ro_home, $cfg_path) = setup_public_inboxes;
my $check_kw = sub {
my ($exp, %opt) = @_;
+ my $args = $opt{args} // [];
my $mid = $opt{mid} // 'testmessage@example.com';
- lei_ok('q', "m:$mid");
+ lei_ok('q', "m:$mid", @$args);
my $res = json_utf8->decode($lei_out);
is($res->[1], undef, 'only got one result');
my $msg = $opt{msg} ? " $opt{msg}" : '';
($exp ? is_deeply($res->[0]->{kw}, $exp, "got @$exp$msg")
: is($res->[0]->{kw}, undef, "got undef$msg")) or
diag explain($res);
+ if (exists $opt{L}) {
+ $exp = $opt{L};
+ ($exp ? is_deeply($res->[0]->{L}, $exp, "got @$exp$msg")
+ : is($res->[0]->{L}, undef, "got undef$msg")) or
+ diag explain($res);
+ }
};
test_lei(sub {
+ lei_ok(qw(ls-label)); is($lei_out, '', 'no labels, yet');
lei_ok(qw(import -F eml t/utf8.eml));
- lei_ok(qw(mark -F eml t/utf8.eml +kw:flagged));
- $check_kw->(['flagged']);
+ lei_ok(qw(mark -F eml t/utf8.eml +kw:flagged +L:urgent));
+ $check_kw->(['flagged'], L => ['urgent']);
+ lei_ok(qw(ls-label)); is($lei_out, "urgent\n", 'label found');
ok(!lei(qw(mark -F eml t/utf8.eml +kw:seeen)), 'bad kw rejected');
like($lei_err, qr/`seeen' is not one of/, 'got helpful error');
ok(!lei(qw(mark -F eml t/utf8.eml +k:seen)), 'bad prefix rejected');
@@ -41,7 +51,35 @@ test_lei(sub {
$check_kw->(['answered'], msg => 'Maildir Status ignored');
open my $in, '<', 't/utf8.eml' or BAIL_OUT $!;
- lei_ok([qw(mark -F eml - +kw:seen)], undef, { %$lei_opt, 0 => $in });
+ lei_ok([qw(mark -F eml - +kw:seen +L:nope)],
+ undef, { %$lei_opt, 0 => $in });
$check_kw->(['answered', 'seen'], msg => 'stdin works');
+ lei_ok(qw(q L:urgent));
+ my $res = json_utf8->decode($lei_out);
+ is($res->[0]->{'m'}, 'testmessage@example.com', 'L: query works');
+ lei_ok(qw(q kw:seen));
+ my $r2 = json_utf8->decode($lei_out);
+ is_deeply($r2, $res, 'kw: query works, too') or
+ diag explain([$r2, $res]);
+
+ lei_ok(qw(_complete lei mark));
+ my %c = map { $_ => 1 } split(/\s+/, $lei_out);
+ ok($c{'+L:urgent'} && $c{'-L:urgent'} &&
+ $c{'+L:nope'} && $c{'-L:nope'}, 'completed with labels');
+
+ my $mid = 'qp@example.com';
+ lei_ok qw(q -f mboxrd --only), "$ro_home/t2", "mid:$mid";
+ $in = $lei_out;
+ lei_ok [qw(mark -F mboxrd --stdin +kw:seen +L:qp)],
+ undef, { %$lei_opt, 0 => \$in };
+ $check_kw->(['seen'], L => ['qp'], mid => $mid,
+ args => [ '--only', "$ro_home/t2" ],
+ msg => 'external-only message');
+ lei_ok(qw(ls-label));
+ is($lei_out, "nope\nqp\nurgent\n", 'ls-label shows qp');
+
+ if (0) { # TODO label+kw search w/ externals
+ lei_ok(qw(q L:qp), "mid:$mid", '--only', "$ro_home/t2");
+ }
});
done_testing;
^ permalink raw reply related [relevance 30%]
* Re: is "lei mark" a good name?
2021-03-25 5:22 71% is "lei mark" a good name? Eric Wong
@ 2021-03-26 1:54 71% ` Kyle Meyer
2021-03-26 9:48 68% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Kyle Meyer @ 2021-03-26 1:54 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
Eric Wong writes:
> It can set/unset volatile metadata for any number of messages.
>
> "Volatile metadata" being "labels" (aka "mailboxes" in
> JMAP-speak) and "keywords" (seen|flagged|answered|...),
> (aka "flags" in IMAP/Maildir-speak).
>
> "lei mark +kw:seen" # makes sense
> "lei mark +L:some-folder-name" # might sound odd...
>
>
> AFAIK, notmuch uses "notmuch tag" which combines both labels
> and keywords into one thing: "tags". But I'm also not a
> notmuch user...
(I am, though I still might be wrong.) Notmuch allows arbitrary tags to
be associated with message IDs. It has some automatic tags like
"attachment", "encrypted", and "signed" that it adds when indexing. By
default, it also syncs some maildir flags to its tags (e.g., "F ->
flagged", "R -> replied", "No S -> unread").
As far as I understand, tags are never connected up to the maildir
folder/location, though Notmuch does support a separate "folder:..."
search term.
So, I guess in JMAP terms, Notmuch tags would be "mailboxes" (a subset
of which are synced with maildir flags).
> Would "lei tag" be better?
Neither one really jumps out to me as better. "mark" sounds fine to me,
and I don't find "tag" any more or less odd for the +L case above.
> Anything else?
Nothing comes to mind.
^ permalink raw reply [relevance 71%]
* Re: does "lei init" even need to exist?
2021-03-25 8:32 71% does "lei init" even need to exist? Eric Wong
@ 2021-03-26 1:15 71% ` Kyle Meyer
0 siblings, 0 replies; 200+ results
From: Kyle Meyer @ 2021-03-26 1:15 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
Eric Wong writes:
> All the commands just initialize the config and store on an
> as-needed basis, so I'm starting to think it's an unnecessary
> thing I mindlessly copied from git...
If somebody wants to use some place other than $XDG_DATA_HOME/lei/store,
I guess `lei init /other/location' provides an easy way to do that. I'm
not really sure there's much value in that, though.
^ permalink raw reply [relevance 71%]
* does "lei init" even need to exist?
@ 2021-03-25 8:32 71% Eric Wong
2021-03-26 1:15 71% ` Kyle Meyer
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2021-03-25 8:32 UTC (permalink / raw)
To: meta
All the commands just initialize the config and store on an
as-needed basis, so I'm starting to think it's an unnecessary
thing I mindlessly copied from git...
I'd honestly forgotten the command exists until now.
^ permalink raw reply [relevance 71%]
* is "lei mark" a good name?
@ 2021-03-25 5:22 71% Eric Wong
2021-03-26 1:54 71% ` Kyle Meyer
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2021-03-25 5:22 UTC (permalink / raw)
To: meta
It can set/unset volatile metadata for any number of messages.
"Volatile metadata" being "labels" (aka "mailboxes" in
JMAP-speak) and "keywords" (seen|flagged|answered|...),
(aka "flags" in IMAP/Maildir-speak).
"lei mark +kw:seen" # makes sense
"lei mark +L:some-folder-name" # might sound odd...
AFAIK, notmuch uses "notmuch tag" which combines both labels
and keywords into one thing: "tags". But I'm also not a
notmuch user...
Would "lei tag" be better?
Anything else?
^ permalink raw reply [relevance 71%]
* [PATCH 10/10] t/lei: add more diagnostics for failures
2021-03-25 4:20 68% [PATCH 00/10] lei testing improvements Eric Wong
` (3 preceding siblings ...)
2021-03-25 4:20 52% ` [PATCH 08/10] lei import: force store, improve test diagnostics Eric Wong
@ 2021-03-25 4:20 56% ` Eric Wong
4 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-25 4:20 UTC (permalink / raw)
To: meta
This seems to error out while looping the test suite and
I'm not 100% sure why.
---
t/lei-import-maildir.t | 13 +++++++++----
t/lei-q-thread.t | 15 ++++++++-------
2 files changed, 17 insertions(+), 11 deletions(-)
diff --git a/t/lei-import-maildir.t b/t/lei-import-maildir.t
index bd89677a..6706b014 100644
--- a/t/lei-import-maildir.t
+++ b/t/lei-import-maildir.t
@@ -11,17 +11,20 @@ test_lei(sub {
symlink(abs_path('t/data/0001.patch'), "$md/cur/x:2,S") or
BAIL_OUT "symlink $md $!";
lei_ok(qw(import), $md, \'import Maildir');
+ my $imp_err = $lei_err;
lei_ok(qw(q s:boolean));
my $res = json_utf8->decode($lei_out);
- like($res->[0]->{'s'}, qr/use boolean/, 'got expected result');
+ like($res->[0]->{'s'}, qr/use boolean/, 'got expected result')
+ or diag explain($imp_err, $res);
is_deeply($res->[0]->{kw}, ['seen'], 'keyword set');
is($res->[1], undef, 'only got one result');
lei_ok(qw(import), $md, \'import Maildir again');
+ $imp_err = $lei_err;
lei_ok(qw(q -d none s:boolean), \'lei q w/o dedupe');
my $r2 = json_utf8->decode($lei_out);
- is_deeply($r2, $res, 'idempotent import');
-
+ is_deeply($r2, $res, 'idempotent import')
+ or diag explain($imp_err, $res);
rename("$md/cur/x:2,S", "$md/cur/x:2,SR") or BAIL_OUT "rename: $!";
lei_ok('import', "maildir:$md", \'import Maildir after +answered');
lei_ok(qw(q -d none s:boolean), \'lei q after +answered');
@@ -33,8 +36,10 @@ test_lei(sub {
symlink(abs_path('t/utf8.eml'), "$md/cur/u:2,ST") or
BAIL_OUT "symlink $md $!";
lei_ok('import', "maildir:$md", \'import Maildir w/ trashed message');
+ $imp_err = $lei_err;
lei_ok(qw(q -d none m:testmessage@example.com));
$res = json_utf8->decode($lei_out);
- is_deeply($res, [ undef ], 'trashed message not imported');
+ is_deeply($res, [ undef ], 'trashed message not imported')
+ or diag explain($imp_err, $res);
});
done_testing;
diff --git a/t/lei-q-thread.t b/t/lei-q-thread.t
index c999d12b..26d06eec 100644
--- a/t/lei-q-thread.t
+++ b/t/lei-q-thread.t
@@ -13,7 +13,8 @@ test_lei(sub {
lei_ok qw(q -t m:testmessage@example.com);
my $res = json_utf8->decode($lei_out);
- is_deeply($res->[0]->{kw}, [ 'seen' ], 'q -t sets keywords');
+ is_deeply($res->[0]->{kw}, [ 'seen' ], 'q -t sets keywords') or
+ diag explain($res);
$eml = eml_load('t/utf8.eml');
$eml->header_set('References', $eml->header('Message-ID'));
@@ -28,9 +29,9 @@ test_lei(sub {
pop @$res;
my %m = map { $_->{'m'} => $_ } @$res;
is_deeply($m{'testmessage@example.com'}->{kw}, ['seen'],
- 'flag set in direct hit');
- 'TODO' or is_deeply($m{'a-reply@miss'}->{kw}, ['draft'],
- 'flag set in thread hit');
+ 'flag set in direct hit') or diag explain($res);
+ is_deeply($m{'a-reply@miss'}->{kw}, ['draft'],
+ 'flag set in thread hit') or diag explain($res);
lei_ok qw(q -t -t m:testmessage@example.com);
$res = json_utf8->decode($lei_out);
@@ -38,9 +39,9 @@ test_lei(sub {
pop @$res;
%m = map { $_->{'m'} => $_ } @$res;
is_deeply($m{'testmessage@example.com'}->{kw}, ['flagged', 'seen'],
- 'flagged set in direct hit');
- 'TODO' or is_deeply($m{'testmessage@example.com'}->{kw}, ['draft'],
- 'flagged set in direct hit');
+ 'flagged set in direct hit') or diag explain($res);
+ is_deeply($m{'a-reply@miss'}->{kw}, ['draft'],
+ 'set in thread hit') or diag explain($res);
lei_ok qw(q -tt m:testmessage@example.com --only), "$ro_home/t2";
$res = json_utf8->decode($lei_out);
is_deeply($res->[0]->{kw}, [ qw(flagged seen) ],
^ permalink raw reply related [relevance 56%]
* [PATCH 02/10] lei: janky $PATH2CFG garbage collection
2021-03-25 4:20 68% [PATCH 00/10] lei testing improvements Eric Wong
@ 2021-03-25 4:20 71% ` Eric Wong
2021-03-25 4:20 71% ` [PATCH 04/10] lei add-external: do not initialize writable store Eric Wong
` (3 subsequent siblings)
4 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-25 4:20 UTC (permalink / raw)
To: meta
We need to rely on this to keep our config cache (and lei_store
pipes) under control with tests each creating a new config and
directory.
---
lib/PublicInbox/LEI.pm | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index e5211764..d534f1d0 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -705,6 +705,12 @@ sub _lei_cfg ($;$) {
File::Spec->canonpath($cfg->{'leistore.dir'})) {
$cfg->{-lei_store} = $sto;
}
+ if (scalar(keys %PATH2CFG) > 5) {
+ # FIXME: use inotify/EVFILT_VNODE to detect unlinked configs
+ for my $k (keys %PATH2CFG) {
+ delete($PATH2CFG{$k}) unless -f $k
+ }
+ }
$self->{cfg} = $PATH2CFG{$f} = $cfg;
}
^ permalink raw reply related [relevance 71%]
* [PATCH 04/10] lei add-external: do not initialize writable store
2021-03-25 4:20 68% [PATCH 00/10] lei testing improvements Eric Wong
2021-03-25 4:20 71% ` [PATCH 02/10] lei: janky $PATH2CFG garbage collection Eric Wong
@ 2021-03-25 4:20 71% ` Eric Wong
2021-03-25 4:20 57% ` [PATCH 07/10] tests: "check-run" uses persistent lei daemon Eric Wong
` (2 subsequent siblings)
4 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-25 4:20 UTC (permalink / raw)
To: meta
There's no need to create or write lei/store when adding
an external, we just need to write to the config file.
---
lib/PublicInbox/LeiExternal.pm | 2 --
t/lei-externals.t | 3 +--
2 files changed, 1 insertion(+), 4 deletions(-)
diff --git a/lib/PublicInbox/LeiExternal.pm b/lib/PublicInbox/LeiExternal.pm
index 56d6ef39..5e8dc71a 100644
--- a/lib/PublicInbox/LeiExternal.pm
+++ b/lib/PublicInbox/LeiExternal.pm
@@ -144,8 +144,6 @@ sub add_external_finish {
sub lei_add_external {
my ($self, $location) = @_;
- my $sto = $self->_lei_store(1);
- $sto->write_prepare($self);
my $opt = $self->{opt};
my $mirror = $opt->{mirror} // do {
my @fail;
diff --git a/t/lei-externals.t b/t/lei-externals.t
index 2045691f..afd90d19 100644
--- a/t/lei-externals.t
+++ b/t/lei-externals.t
@@ -93,8 +93,7 @@ test_lei(sub {
\'added external');
is($lei_out.$lei_err, '', 'no output');
});
- ok(-s $config_file && -e $store_dir,
- 'add-external created config + store');
+ ok(-s $config_file, 'add-external created config');
my $lcfg = PublicInbox::Config->new($config_file);
$cfg->each_inbox(sub {
my ($ibx) = @_;
^ permalink raw reply related [relevance 71%]
* [PATCH 07/10] tests: "check-run" uses persistent lei daemon
2021-03-25 4:20 68% [PATCH 00/10] lei testing improvements Eric Wong
2021-03-25 4:20 71% ` [PATCH 02/10] lei: janky $PATH2CFG garbage collection Eric Wong
2021-03-25 4:20 71% ` [PATCH 04/10] lei add-external: do not initialize writable store Eric Wong
@ 2021-03-25 4:20 57% ` Eric Wong
2021-03-25 4:20 52% ` [PATCH 08/10] lei import: force store, improve test diagnostics Eric Wong
2021-03-25 4:20 56% ` [PATCH 10/10] t/lei: add more diagnostics for failures Eric Wong
4 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-25 4:20 UTC (permalink / raw)
To: meta
We'll use a lei-daemon if it's already running and
TEST_LEI_DAEMON_PERSIST_DIR is set, but we can also start
one and manage it from t/run.perl
This drops "make check-run TEST_LEI_DAEMON_ONLY=1"
time by ~10% for me.
---
lib/PublicInbox/TestCommon.pm | 22 +++++++++++++++-------
t/lei-externals.t | 4 ++++
t/run.perl | 19 +++++++++++++++++++
3 files changed, 38 insertions(+), 7 deletions(-)
diff --git a/lib/PublicInbox/TestCommon.pm b/lib/PublicInbox/TestCommon.pm
index ffff5902..ca165a04 100644
--- a/lib/PublicInbox/TestCommon.pm
+++ b/lib/PublicInbox/TestCommon.pm
@@ -515,22 +515,30 @@ EOM
my ($daemon_pid, $for_destroy, $daemon_xrd);
my $tmpdir = $test_opt->{tmpdir};
($tmpdir, $for_destroy) = tmpdir unless $tmpdir;
+ state $persist_xrd = $ENV{TEST_LEI_DAEMON_PERSIST_DIR};
SKIP: {
skip 'TEST_LEI_ONESHOT set', 1 if $ENV{TEST_LEI_ONESHOT};
my $home = "$tmpdir/lei-daemon";
mkdir($home, 0700) or BAIL_OUT "mkdir: $!";
local $ENV{HOME} = $home;
- $daemon_xrd = "$home/xdg_run";
- mkdir($daemon_xrd, 0700) or BAIL_OUT "mkdir: $!";
+ my $persist;
+ if ($persist_xrd && !$test_opt->{daemon_only}) {
+ $persist = $daemon_xrd = $persist_xrd;
+ } else {
+ $daemon_xrd = "$home/xdg_run";
+ mkdir($daemon_xrd, 0700) or BAIL_OUT "mkdir: $!";
+ }
local $ENV{XDG_RUNTIME_DIR} = $daemon_xrd;
$cb->();
- lei_ok(qw(daemon-pid), \"daemon-pid after $t");
- chomp($daemon_pid = $lei_out);
- if ($daemon_pid) {
+ unless ($persist) {
+ lei_ok(qw(daemon-pid), \"daemon-pid after $t");
+ chomp($daemon_pid = $lei_out);
+ if (!$daemon_pid) {
+ fail("daemon not running after $t");
+ skip 'daemon died unexpectedly', 2;
+ }
ok(kill(0, $daemon_pid), "daemon running after $t");
lei_ok(qw(daemon-kill), \"daemon-kill after $t");
- } else {
- fail("daemon not running after $t");
}
}; # SKIP for lei_daemon
unless ($test_opt->{daemon_only}) {
diff --git a/t/lei-externals.t b/t/lei-externals.t
index afd90d19..488bf5ad 100644
--- a/t/lei-externals.t
+++ b/t/lei-externals.t
@@ -57,6 +57,8 @@ SKIP: {
chomp(my $pid_after = $lei_out);
is($pid_after, $pid_before, 'pid unchanged') or
skip 'daemon died', 1;
+ skip 'not killing persistent lei-daemon', 2 if
+ $ENV{TEST_LEI_DAEMON_PERSIST_DIR};
lei_ok 'daemon-kill';
my $alive = 1;
for (1..100) {
@@ -262,6 +264,8 @@ test_lei(sub {
}
{
+ skip 'TEST_LEI_DAEMON_PERSIST_DIR in use', 1 if
+ $ENV{TEST_LEI_DAEMON_PERSIST_DIR};
opendir my $dh, '.' or BAIL_OUT "opendir(.) $!";
my $od = PublicInbox::OnDestroy->new($$, sub {
chdir $dh or BAIL_OUT "chdir: $!"
diff --git a/t/run.perl b/t/run.perl
index e8512e18..1fb1c5f0 100755
--- a/t/run.perl
+++ b/t/run.perl
@@ -14,6 +14,7 @@ use strict;
use v5.10.1;
use IO::Handle; # ->autoflush
use PublicInbox::TestCommon;
+use PublicInbox::Spawn;
use Getopt::Long qw(:config gnu_getopt no_ignore_case auto_abbrev);
use Errno qw(EINTR);
use Fcntl qw(:seek);
@@ -40,6 +41,20 @@ $OLDERR->autoflush(1);
key2sub($_) for @tests; # precache
+my ($for_destroy, $lei_env, $lei_daemon_pid, $owner_pid);
+if (!$ENV{TEST_LEI_DAEMON_PERSIST_DIR} &&
+ (PublicInbox::Spawn->can('recv_cmd4') ||
+ eval { require Socket::MsgHdr })) {
+ $lei_env = {};
+ ($lei_env->{XDG_RUNTIME_DIR}, $for_destroy) = tmpdir;
+ $ENV{TEST_LEI_DAEMON_PERSIST_DIR} = $lei_env->{XDG_RUNTIME_DIR};
+ run_script([qw(lei daemon-pid)], $lei_env, { 1 => \$lei_daemon_pid });
+ chomp $lei_daemon_pid;
+ $lei_daemon_pid =~ /\A[0-9]+\z/ or die "no daemon pid: $lei_daemon_pid";
+ kill(0, $lei_daemon_pid) or die "kill $lei_daemon_pid: $!";
+ $owner_pid = $$;
+}
+
if ($shuffle) {
require List::Util;
} elsif (open(my $prove_state, '<', '.prove') && eval { require YAML::XS }) {
@@ -209,3 +224,7 @@ for (my $i = $repeat; $i != 0; $i--) {
}
print $OLDOUT "1..".($repeat * scalar(@tests))."\n" if $repeat >= 0;
+if ($lei_env && $$ == $owner_pid) {
+ my $opt = {}; # 1 => $OLDOUT, 2 => $OLDERR };
+ run_script([qw(lei daemon-kill)], $lei_env, $opt);
+}
^ permalink raw reply related [relevance 57%]
* [PATCH 08/10] lei import: force store, improve test diagnostics
2021-03-25 4:20 68% [PATCH 00/10] lei testing improvements Eric Wong
` (2 preceding siblings ...)
2021-03-25 4:20 57% ` [PATCH 07/10] tests: "check-run" uses persistent lei daemon Eric Wong
@ 2021-03-25 4:20 52% ` Eric Wong
2021-03-25 4:20 56% ` [PATCH 10/10] t/lei: add more diagnostics for failures Eric Wong
4 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-25 4:20 UTC (permalink / raw)
To: meta
"lei import" should never be without a {sto}, and *_done should
not be called multiple times, so ensure we can fail if it's
missing.
Update some existing tests to complain loudly by introducing a
handy "xbail" function which wraps "explain" and BAIL_OUT.
BAIL_OUT was painful to type and concatenating the result of
"explain" doesn't work as I thought it would since "explain"
always returns an array, and BAIL_OUT only accepts a single
scalar arg (unlike "die").
---
lib/PublicInbox/LeiImport.pm | 6 +++---
lib/PublicInbox/TestCommon.pm | 4 +++-
t/lei-mark.t | 2 +-
t/lei-q-kw.t | 6 +++---
4 files changed, 10 insertions(+), 8 deletions(-)
diff --git a/lib/PublicInbox/LeiImport.pm b/lib/PublicInbox/LeiImport.pm
index 9da6b7f9..7c5b7d09 100644
--- a/lib/PublicInbox/LeiImport.pm
+++ b/lib/PublicInbox/LeiImport.pm
@@ -39,14 +39,14 @@ sub import_done_wait { # dwaitpid callback
my ($arg, $pid) = @_;
my ($imp, $lei) = @$arg;
$lei->child_error($?, 'non-fatal errors during import') if $?;
- my $sto = delete $lei->{sto};
- my $wait = $sto->ipc_do('done') if $sto; # PublicInbox::LeiStore::done
+ my $sto = delete $lei->{sto} // return $lei->fail('BUG: {sto} gone');
+ my $wait = $sto->ipc_do('done'); # PublicInbox::LeiStore::done
$lei->dclose;
}
sub import_done { # EOF callback for main daemon
my ($lei) = @_;
- my $imp = delete $lei->{imp} or return;
+ my $imp = delete $lei->{imp} // return $lei->fail('BUG: {imp} gone');
$imp->wq_wait_old(\&import_done_wait, $lei);
}
diff --git a/lib/PublicInbox/TestCommon.pm b/lib/PublicInbox/TestCommon.pm
index ca165a04..72617a78 100644
--- a/lib/PublicInbox/TestCommon.pm
+++ b/lib/PublicInbox/TestCommon.pm
@@ -17,7 +17,7 @@ BEGIN {
run_script start_script key2sub xsys xsys_e xqx eml_load tick
have_xapian_compact json_utf8 setup_public_inboxes create_inbox
tcp_host_port test_lei lei lei_ok $lei_out $lei_err $lei_opt
- test_httpd);
+ test_httpd xbail);
require Test::More;
my @methods = grep(!/\W/, @Test::More::EXPORT);
eval(join('', map { "*$_=\\&Test::More::$_;" } @methods));
@@ -25,6 +25,8 @@ BEGIN {
push @EXPORT, @methods;
}
+sub xbail (@) { BAIL_OUT join(' ', map { ref ? (explain($_)) : ($_) } @_) }
+
sub eml_load ($) {
my ($path, $cb) = @_;
open(my $fh, '<', $path) or die "open $path: $!";
diff --git a/t/lei-mark.t b/t/lei-mark.t
index ddf5634c..76995589 100644
--- a/t/lei-mark.t
+++ b/t/lei-mark.t
@@ -30,7 +30,7 @@ test_lei(sub {
ok(-s $mb, 'wrote mbox result');
lei_ok(qw(q m:testmessage@example.com -o), $md);
my @fn = glob("$md/cur/*");
- scalar(@fn) == 1 or BAIL_OUT 'no mail '.explain(\@fn);
+ scalar(@fn) == 1 or xbail $lei_err, 'no mail', \@fn;
rename($fn[0], "$fn[0]S") or BAIL_OUT "rename $!";
$check_kw->(['flagged'], msg => 'after bad request');
lei_ok(qw(mark -F eml t/utf8.eml -kw:flagged));
diff --git a/t/lei-q-kw.t b/t/lei-q-kw.t
index 4db27363..c17411fb 100644
--- a/t/lei-q-kw.t
+++ b/t/lei-q-kw.t
@@ -21,7 +21,7 @@ lei_ok(qw(import -F eml t/plack-qp.eml));
my $o = "$ENV{HOME}/dst";
lei_ok(qw(q -o), "maildir:$o", qw(m:qp@example.com));
my @fn = glob("$o/cur/*:2,");
-scalar(@fn) == 1 or BAIL_OUT "wrote multiple or zero files: ".explain(\@fn);
+scalar(@fn) == 1 or xbail $lei_err, 'wrote multiple or zero files:', \@fn;
rename($fn[0], "$fn[0]S") or BAIL_OUT "rename $!";
lei_ok(qw(q -o), "maildir:$o", qw(m:bogus-noresults@example.com));
@@ -124,7 +124,7 @@ lei_ok(qw(q -o), $o, "m:$m", @inc);
# emulate MUA marking a Maildir message as read:
@fn = glob("$o/cur/*");
-scalar(@fn) == 1 or BAIL_OUT "wrote multiple or zero files: ".explain(\@fn);
+scalar(@fn) == 1 or xbail $lei_err, 'wrote multiple or zero files:', \@fn;
rename($fn[0], "$fn[0]S") or BAIL_OUT "rename $!";
lei_ok(qw(q -o), $o, 'bogus', \'clobber output dir to import keywords');
@@ -178,7 +178,7 @@ $m = 'multipart@example.com';
$o = "$ENV{HOME}/fuzz";
lei_ok('q', '-o', $o, "m:$m", @inc);
@fn = glob("$o/cur/*");
-scalar(@fn) == 1 or BAIL_OUT "wrote multiple or zero files: ".explain(\@fn);
+scalar(@fn) == 1 or xbail $lei_err, "wrote multiple or zero files", \@fn;
rename($fn[0], "$fn[0]S") or BAIL_OUT "rename $!";
lei_ok('q', '-o', $o, "m:$m");
is_deeply([glob("$o/cur/*")], [], 'clobbered output results');
^ permalink raw reply related [relevance 52%]
* [PATCH 00/10] lei testing improvements
@ 2021-03-25 4:20 68% Eric Wong
2021-03-25 4:20 71% ` [PATCH 02/10] lei: janky $PATH2CFG garbage collection Eric Wong
` (4 more replies)
0 siblings, 5 replies; 200+ results
From: Eric Wong @ 2021-03-25 4:20 UTC (permalink / raw)
To: meta
[7/10] is the centerpiece and gives a ~10% speedup for tests,
which are still slow to me. Speedup or not, it's uncovered a
bunch of subtle bugs over the past few days so I'm glad I
worked on it.
There's still some rare errors that come from looping
"make check-run TEST_LEI_ERR_LOUD" which I'm still trying
to figure out...
Eric Wong (10):
test_common: cleanup inbox objects after use
lei: janky $PATH2CFG garbage collection
test_common: TEST_LEI_ERR_LOUD does not hide path names
lei add-external: do not initialize writable store
lei_mirror: don't show success on failure
t/*: drop unnecessary v1-specific index calls
tests: "check-run" uses persistent lei daemon
lei import: force store, improve test diagnostics
t/cmd_ipc: workaround signal handling raciness
t/lei: add more diagnostics for failures
lib/PublicInbox/LEI.pm | 6 +++++
lib/PublicInbox/LeiExternal.pm | 2 --
lib/PublicInbox/LeiImport.pm | 6 ++---
lib/PublicInbox/LeiMirror.pm | 11 ++++++---
lib/PublicInbox/TestCommon.pm | 41 +++++++++++++++++++++++-----------
t/cmd_ipc.t | 28 ++++++++++++++++-------
t/inbox_idle.t | 2 --
t/lei-externals.t | 7 ++++--
t/lei-import-maildir.t | 13 +++++++----
t/lei-mark.t | 2 +-
t/lei-mirror.t | 18 +++++++++++++++
t/lei-q-kw.t | 6 ++---
t/lei-q-thread.t | 15 +++++++------
t/nntpd.t | 4 ----
t/run.perl | 19 ++++++++++++++++
t/v2mda.t | 4 ----
t/watch_filter_rubylang.t | 7 ++----
17 files changed, 130 insertions(+), 61 deletions(-)
^ permalink raw reply [relevance 68%]
* [PATCH 4/9] lei: update {3} after -C chdirs
2021-03-24 9:23 70% [PATCH 0/9] lei: various corner case leak fixes Eric Wong
2021-03-24 9:23 71% ` [PATCH 3/9] lei: drop circular reference in lei_store process Eric Wong
@ 2021-03-24 9:23 71% ` Eric Wong
2021-03-24 9:23 61% ` [PATCH 5/9] lei: clean up pkt_op consumer on exception, too Eric Wong
2021-03-24 9:23 58% ` [PATCH 9/9] lei-daemon: do not leak FDs on bogus requests Eric Wong
3 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-24 9:23 UTC (permalink / raw)
To: meta
This is necessary for lei->rel2abs correctness, and may
eventually be useful if we can use *at syscalls.
---
lib/PublicInbox/LEI.pm | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index ee991f80..74372532 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -640,6 +640,11 @@ sub dispatch {
next if $d eq ''; # same as git(1)
chdir $d or return fail($self, "cd $d: $!");
}
+ if (delete $self->{3}) { # update cwd for rel2abs
+ opendir my $dh, '.' or
+ return fail($self, "opendir . $!");
+ $self->{3} = $dh;
+ }
}
$cb->($self, @argv);
} elsif (grep(/\A-/, $cmd, @argv)) { # --help or -h only
^ permalink raw reply related [relevance 71%]
* [PATCH 5/9] lei: clean up pkt_op consumer on exception, too
2021-03-24 9:23 70% [PATCH 0/9] lei: various corner case leak fixes Eric Wong
2021-03-24 9:23 71% ` [PATCH 3/9] lei: drop circular reference in lei_store process Eric Wong
2021-03-24 9:23 71% ` [PATCH 4/9] lei: update {3} after -C chdirs Eric Wong
@ 2021-03-24 9:23 61% ` Eric Wong
2021-03-24 9:23 58% ` [PATCH 9/9] lei-daemon: do not leak FDs on bogus requests Eric Wong
3 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-24 9:23 UTC (permalink / raw)
To: meta
We need to consistently ensure pkt_op_c doesn't lead to a
long-lived circular reference if an exception is thrown in
pre_augment. Maybe the API could be better, but this fixes an
FD leak when attempting to --augment a FIFO.
Followup-to: b9524082ba39e665 ("lei_xsearch: cleanup {pkt_op_p} on exceptions")
---
lib/PublicInbox/LEI.pm | 22 ++++++++++++++++++++--
lib/PublicInbox/LeiXSearch.pm | 9 ++-------
2 files changed, 22 insertions(+), 9 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 74372532..878685f1 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -482,6 +482,24 @@ sub _lei_atfork_child {
$current_lei = $persist ? undef : $self; # for SIG{__WARN__}
}
+sub _delete_pkt_op { # OnDestroy callback to prevent leaks on die
+ my ($self) = @_;
+ if (my $op = delete $self->{pkt_op_c}) { # in case of die
+ $op->close; # PublicInbox::PktOp::close
+ }
+ my $unclosed_after_die = delete($self->{pkt_op_p}) or return;
+ close $unclosed_after_die;
+}
+
+sub pkt_op_pair {
+ my ($self, $ops) = @_;
+ require PublicInbox::OnDestroy;
+ require PublicInbox::PktOp;
+ my $end = PublicInbox::OnDestroy->new($$, \&_delete_pkt_op, $self);
+ @$self{qw(pkt_op_c pkt_op_p)} = PublicInbox::PktOp->pair($ops);
+ $end;
+}
+
sub workers_start {
my ($lei, $wq, $ident, $jobs, $ops) = @_;
$ops = {
@@ -492,11 +510,11 @@ sub workers_start {
($ops ? %$ops : ()),
};
$ops->{''} //= [ \&dclose, $lei ];
- require PublicInbox::PktOp;
- ($lei->{pkt_op_c}, $lei->{pkt_op_p}) = PublicInbox::PktOp->pair($ops);
+ my $end = $lei->pkt_op_pair($ops);
$wq->wq_workers_start($ident, $jobs, $lei->oldset, { lei => $lei });
delete $lei->{pkt_op_p};
my $op = delete $lei->{pkt_op_c};
+ @$end = ();
$lei->event_step_init;
# oneshot needs $op, daemon-mode uses DS->EventLoop to handle $op
$lei->{oneshot} ? $op : undef;
diff --git a/lib/PublicInbox/LeiXSearch.pm b/lib/PublicInbox/LeiXSearch.pm
index c6b82eeb..58b6cfc0 100644
--- a/lib/PublicInbox/LeiXSearch.pm
+++ b/lib/PublicInbox/LeiXSearch.pm
@@ -406,11 +406,6 @@ sub ipc_atfork_child {
$self->SUPER::ipc_atfork_child;
}
-sub delete_pkt_op { # OnDestroy callback
- my $unclosed_after_die = delete($_[0])->{pkt_op_p} or return;
- close $unclosed_after_die;
-}
-
sub do_query {
my ($self, $lei) = @_;
my $l2m = $lei->{l2m};
@@ -426,8 +421,7 @@ sub do_query {
'incr_start_query' => [ \&incr_start_query, $self, $l2m ],
};
$lei->{auth}->op_merge($ops, $l2m) if $l2m && $lei->{auth};
- my $od = PublicInbox::OnDestroy->new($$, \&delete_pkt_op, $lei);
- ($lei->{pkt_op_c}, $lei->{pkt_op_p}) = PublicInbox::PktOp->pair($ops);
+ my $end = $lei->pkt_op_pair($ops);
$lei->{1}->autoflush(1);
$lei->start_pager if delete $lei->{need_pager};
$lei->{ovv}->ovv_begin($lei);
@@ -446,6 +440,7 @@ sub do_query {
$lei->oldset, { lei => $lei });
my $op = delete $lei->{pkt_op_c};
delete $lei->{pkt_op_p};
+ @$end = ();
$self->{threads} = $lei->{opt}->{threads};
if ($l2m) {
$l2m->net_merge_complete unless $lei->{auth};
^ permalink raw reply related [relevance 61%]
* [PATCH 9/9] lei-daemon: do not leak FDs on bogus requests
2021-03-24 9:23 70% [PATCH 0/9] lei: various corner case leak fixes Eric Wong
` (2 preceding siblings ...)
2021-03-24 9:23 61% ` [PATCH 5/9] lei: clean up pkt_op consumer on exception, too Eric Wong
@ 2021-03-24 9:23 58% ` Eric Wong
3 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-24 9:23 UTC (permalink / raw)
To: meta
If a client passes us the incorrect number of FDs, we'll vivify
them into PerlIO objects so they can be auto-closed. Using
POSIX::close was considered, but it would've been more code to
handle an uncommon case.
---
lib/PublicInbox/LEI.pm | 15 +++++++--------
t/lei-daemon.t | 29 +++++++++++++++++++++++++++++
2 files changed, 36 insertions(+), 8 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 878685f1..e5211764 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -981,17 +981,16 @@ sub accept_dispatch { # Listener {post_accept} callback
return send($sock, 'timed out waiting to recv FDs', MSG_EOR);
# (4096 * 33) >MAX_ARG_STRLEN
my @fds = $recv_cmd->($sock, my $buf, 4096 * 33) or return; # EOF
- if (scalar(@fds) == 4) {
- for my $i (0..3) {
- my $fd = shift(@fds);
- open($self->{$i}, '+<&=', $fd) and next;
- send($sock, "open(+<&=$fd) (FD=$i): $!", MSG_EOR);
- }
- } elsif (!defined($fds[0])) {
+ if (!defined($fds[0])) {
warn(my $msg = "recv_cmd failed: $!");
return send($sock, $msg, MSG_EOR);
} else {
- return;
+ my $i = 0;
+ for my $fd (@fds) {
+ open($self->{$i++}, '+<&=', $fd) and next;
+ send($sock, "open(+<&=$fd) (FD=$i): $!", MSG_EOR);
+ }
+ return if scalar(@fds) != 4;
}
$self->{2}->autoflush(1); # keep stdout buffered until x_it|DESTROY
# $ENV_STR = join('', map { "\0$_=$ENV{$_}" } keys %ENV);
diff --git a/t/lei-daemon.t b/t/lei-daemon.t
index c30e5ac1..35e059b9 100644
--- a/t/lei-daemon.t
+++ b/t/lei-daemon.t
@@ -2,8 +2,16 @@
# Copyright (C) 2020-2021 all contributors <meta@public-inbox.org>
# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
use strict; use v5.10.1; use PublicInbox::TestCommon;
+use Socket qw(AF_UNIX SOCK_SEQPACKET MSG_EOR pack_sockaddr_un);
+use PublicInbox::Spawn qw(which);
test_lei({ daemon_only => 1 }, sub {
+ my $send_cmd = PublicInbox::Spawn->can('send_cmd4') // do {
+ require PublicInbox::CmdIPC4;
+ PublicInbox::CmdIPC4->can('send_cmd4');
+ };
+ $send_cmd or BAIL_OUT 'started testing lei-daemon w/o send_cmd4!';
+
my $sock = "$ENV{XDG_RUNTIME_DIR}/lei/5.seq.sock";
my $err_log = "$ENV{XDG_RUNTIME_DIR}/lei/errors.log";
lei_ok('daemon-pid');
@@ -22,6 +30,27 @@ test_lei({ daemon_only => 1 }, sub {
is($pid, $pid_again, 'daemon-pid idempotent');
like($lei_err, qr/phail/, 'got mock "phail" error previous run');
+ SKIP: {
+ skip 'only testing open files on Linux', 1 if $^O ne 'linux';
+ my $d = "/proc/$pid/fd";
+ skip "no $d on Linux" unless -d $d;
+ my @before = sort(glob("$d/*"));
+ my $addr = pack_sockaddr_un($sock);
+ open my $null, '<', '/dev/null' or BAIL_OUT "/dev/null: $!";
+ my @fds = map { fileno($null) } (0..2);
+ for (0..10) {
+ socket(my $c, AF_UNIX, SOCK_SEQPACKET, 0) or
+ BAIL_OUT "socket: $!";
+ connect($c, $addr) or BAIL_OUT "connect: $!";
+ $send_cmd->($c, \@fds, 'hi', MSG_EOR);
+ }
+ lei_ok('daemon-pid');
+ chomp($pid = $lei_out);
+ is($pid, $pid_again, 'pid unchanged after failed reqs');
+ my @after = sort(glob("$d/*"));
+ is_deeply(\@before, \@after, 'open files unchanged') or
+ diag explain([\@before, \@after]);;
+ }
lei_ok(qw(daemon-kill));
is($lei_out, '', 'no output from daemon-kill');
is($lei_err, '', 'no error from daemon-kill');
^ permalink raw reply related [relevance 58%]
* [PATCH 3/9] lei: drop circular reference in lei_store process
2021-03-24 9:23 70% [PATCH 0/9] lei: various corner case leak fixes Eric Wong
@ 2021-03-24 9:23 71% ` Eric Wong
2021-03-24 9:23 71% ` [PATCH 4/9] lei: update {3} after -C chdirs Eric Wong
` (2 subsequent siblings)
3 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-24 9:23 UTC (permalink / raw)
To: meta
I'm not sure if this was causing real problems, but it's sure ugly.
---
lib/PublicInbox/LEI.pm | 3 +++
1 file changed, 3 insertions(+)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 8cbaac01..ee991f80 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -458,6 +458,9 @@ sub _lei_atfork_child {
unless ($self->{oneshot}) {
close($_) for @io;
}
+ if (my $cfg = $self->{cfg}) {
+ delete $cfg->{-lei_store};
+ }
} else { # worker, Net::NNTP (Net::Cmd) uses STDERR directly
open STDERR, '+>&='.fileno($self->{2}) or warn "open $!";
delete $self->{0};
^ permalink raw reply related [relevance 71%]
* [PATCH 0/9] lei: various corner case leak fixes
@ 2021-03-24 9:23 70% Eric Wong
2021-03-24 9:23 71% ` [PATCH 3/9] lei: drop circular reference in lei_store process Eric Wong
` (3 more replies)
0 siblings, 4 replies; 200+ results
From: Eric Wong @ 2021-03-24 9:23 UTC (permalink / raw)
To: meta
Making the test suite use a single lei-daemon for all tests has
uncovered a number of bugs that wouldn't get uncovered during
normal usage. These fixes could be useful if lei-daemon is
deployed a multi-tenant server with both shared and "private"
storage support running under a single Unix user.
Eric Wong (9):
ds: improve DS->Reset fork-safety
mbox_lock: dotlock: chdir for relative lock paths
lei: drop circular reference in lei_store process
lei: update {3} after -C chdirs
lei: clean up pkt_op consumer on exception, too
lei_store: give process a better name
v2writable: cleanup SQLite handles on --xapian-only
lei_mirror: fix circular reference
lei-daemon: do not leak FDs on bogus requests
lib/PublicInbox/DS.pm | 76 +++++++++++++++++++++--------------
lib/PublicInbox/LEI.pm | 45 ++++++++++++++++-----
lib/PublicInbox/LeiMirror.pm | 2 +-
lib/PublicInbox/LeiStore.pm | 6 ++-
lib/PublicInbox/LeiXSearch.pm | 9 +----
lib/PublicInbox/MboxLock.pm | 14 +++++++
lib/PublicInbox/V2Writable.pm | 1 +
t/lei-daemon.t | 29 +++++++++++++
t/mbox_lock.t | 12 ++++++
t/v2reindex.t | 10 ++++-
10 files changed, 153 insertions(+), 51 deletions(-)
^ permalink raw reply [relevance 70%]
* [PATCH 3/5] lei: persistent workers (lei_store) run in /
2021-03-23 11:48 71% [PATCH 0/5] lei: more input + worker-related stuff Eric Wong
2021-03-23 11:48 70% ` [PATCH 2/5] test_common: check lei/errors.log Eric Wong
@ 2021-03-23 11:48 71% ` Eric Wong
2021-03-23 11:48 47% ` [PATCH 5/5] lei: improve management around short-lived workers Eric Wong
2 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-23 11:48 UTC (permalink / raw)
To: meta
Since each lei->event_step can change the directory of
lei-daemon, we need to ensure the lei_store runs in a
directory that is stable.
---
lib/PublicInbox/LEI.pm | 1 +
1 file changed, 1 insertion(+)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 17ca637e..d3ac19b2 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -453,6 +453,7 @@ sub _lei_atfork_child {
my ($self, $persist) = @_;
# we need to explicitly close things which are on stack
if ($persist) {
+ chdir '/' or die "chdir(/): $!";
my @io = delete @$self{qw(0 1 2 sock)};
unless ($self->{oneshot}) {
close($_) for @io;
^ permalink raw reply related [relevance 71%]
* [PATCH 2/5] test_common: check lei/errors.log
2021-03-23 11:48 71% [PATCH 0/5] lei: more input + worker-related stuff Eric Wong
@ 2021-03-23 11:48 70% ` Eric Wong
2021-03-23 11:48 71% ` [PATCH 3/5] lei: persistent workers (lei_store) run in / Eric Wong
2021-03-23 11:48 47% ` [PATCH 5/5] lei: improve management around short-lived workers Eric Wong
2 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-23 11:48 UTC (permalink / raw)
To: meta
This will make it easier to diagnose some large internal
rewrites.
---
lib/PublicInbox/TestCommon.pm | 13 +++++++++----
1 file changed, 9 insertions(+), 4 deletions(-)
diff --git a/lib/PublicInbox/TestCommon.pm b/lib/PublicInbox/TestCommon.pm
index e67e94ea..d4117b6c 100644
--- a/lib/PublicInbox/TestCommon.pm
+++ b/lib/PublicInbox/TestCommon.pm
@@ -507,7 +507,7 @@ SKIP: {
Socket::MsgHdr missing or Inline::C is unconfigured/missing
EOM
$lei_opt = { 1 => \$lei_out, 2 => \$lei_err };
- my ($daemon_pid, $for_destroy);
+ my ($daemon_pid, $for_destroy, $daemon_xrd);
my $tmpdir = $test_opt->{tmpdir};
($tmpdir, $for_destroy) = tmpdir unless $tmpdir;
SKIP: {
@@ -515,9 +515,9 @@ EOM
my $home = "$tmpdir/lei-daemon";
mkdir($home, 0700) or BAIL_OUT "mkdir: $!";
local $ENV{HOME} = $home;
- my $xrd = "$home/xdg_run";
- mkdir($xrd, 0700) or BAIL_OUT "mkdir: $!";
- local $ENV{XDG_RUNTIME_DIR} = $xrd;
+ $daemon_xrd = "$home/xdg_run";
+ mkdir($daemon_xrd, 0700) or BAIL_OUT "mkdir: $!";
+ local $ENV{XDG_RUNTIME_DIR} = $daemon_xrd;
$cb->();
lei_ok(qw(daemon-pid), \"daemon-pid after $t");
chomp($daemon_pid = $lei_out);
@@ -547,6 +547,11 @@ EOM
tick;
}
ok(!kill(0, $daemon_pid), "$t daemon stopped after oneshot");
+ my $f = "$daemon_xrd/lei/errors.log";
+ open my $fh, '<', $f or BAIL_OUT "$f: $!";
+ my @l = <$fh>;
+ is_deeply(\@l, [],
+ "$t daemon XDG_RUNTIME_DIR/lei/errors.log empty");
}
}; # SKIP if missing git 2.6+ || Xapian || SQLite || json
} # /test_lei
^ permalink raw reply related [relevance 70%]
* [PATCH 5/5] lei: improve management around short-lived workers
2021-03-23 11:48 71% [PATCH 0/5] lei: more input + worker-related stuff Eric Wong
2021-03-23 11:48 70% ` [PATCH 2/5] test_common: check lei/errors.log Eric Wong
2021-03-23 11:48 71% ` [PATCH 3/5] lei: persistent workers (lei_store) run in / Eric Wong
@ 2021-03-23 11:48 47% ` Eric Wong
2 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-23 11:48 UTC (permalink / raw)
To: meta
Instead of creating a short-lived circular reference,
ensure they don't exist in the first place.
Note the following changes to hold an extra ref to $sto:
- $self->_lei_store(1)->write_prepare($self);
+ my $sto = $self->_lei_store(1);
+ $sto->write_prepare($self);
I'm not a perlguts expert, but I actually wanted to switch
to the one-line version for LeiImport, but xt/lei-auth-fail.t
was getting stuck for some reason. It seems the extra ref
to the LeiStore ($sto) object is necessary.
---
lib/PublicInbox/LEI.pm | 1 -
lib/PublicInbox/LeiConvert.pm | 3 ++-
lib/PublicInbox/LeiExternal.pm | 3 ++-
lib/PublicInbox/LeiImport.pm | 20 +++++++-------------
lib/PublicInbox/LeiMark.pm | 2 +-
lib/PublicInbox/LeiMirror.pm | 2 +-
lib/PublicInbox/LeiP2q.pm | 5 +++--
lib/PublicInbox/LeiQuery.pm | 2 +-
8 files changed, 17 insertions(+), 21 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index d3ac19b2..8cbaac01 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -462,7 +462,6 @@ sub _lei_atfork_child {
open STDERR, '+>&='.fileno($self->{2}) or warn "open $!";
delete $self->{0};
}
- delete @$self{qw(cnv mark imp)};
for (delete @$self{qw(3 old_1 au_done)}) {
close($_) if defined($_);
}
diff --git a/lib/PublicInbox/LeiConvert.pm b/lib/PublicInbox/LeiConvert.pm
index bc86fe25..0cc65108 100644
--- a/lib/PublicInbox/LeiConvert.pm
+++ b/lib/PublicInbox/LeiConvert.pm
@@ -46,13 +46,14 @@ sub lei_convert { # the main "lei convert" method
my ($lei, @inputs) = @_;
$lei->{opt}->{kw} //= 1;
$lei->{opt}->{dedupe} //= 'none';
- my $self = $lei->{cnv} = bless {}, __PACKAGE__;
+ my $self = bless {}, __PACKAGE__;
my $ovv = PublicInbox::LeiOverview->new($lei, 'out-format');
$lei->{l2m} or return
$lei->fail("output not specified or is not a mail destination");
$lei->{opt}->{augment} = 1 unless $ovv->{dst} eq '/dev/stdout';
$self->prepare_inputs($lei, \@inputs) or return;
my $op = $lei->workers_start($self, 'lei_convert', 1);
+ $lei->{cnv} = $self;
$self->wq_io_do('do_convert', []);
$self->wq_close(1);
while ($op && $op->{sock}) { $op->event_step }
diff --git a/lib/PublicInbox/LeiExternal.pm b/lib/PublicInbox/LeiExternal.pm
index 9a555831..56d6ef39 100644
--- a/lib/PublicInbox/LeiExternal.pm
+++ b/lib/PublicInbox/LeiExternal.pm
@@ -144,7 +144,8 @@ sub add_external_finish {
sub lei_add_external {
my ($self, $location) = @_;
- $self->_lei_store(1)->write_prepare($self);
+ my $sto = $self->_lei_store(1);
+ $sto->write_prepare($self);
my $opt = $self->{opt};
my $mirror = $opt->{mirror} // do {
my @fail;
diff --git a/lib/PublicInbox/LeiImport.pm b/lib/PublicInbox/LeiImport.pm
index 991c84f2..9da6b7f9 100644
--- a/lib/PublicInbox/LeiImport.pm
+++ b/lib/PublicInbox/LeiImport.pm
@@ -58,9 +58,13 @@ sub net_merge_complete { # callback used by LeiAuth
$self->wq_close(1);
}
-sub import_start {
- my ($lei) = @_;
- my $self = $lei->{imp};
+sub lei_import { # the main "lei import" method
+ my ($lei, @inputs) = @_;
+ my $sto = $lei->_lei_store(1);
+ $sto->write_prepare($lei);
+ my $self = bless {}, __PACKAGE__;
+ $self->{-import_kw} = $lei->{opt}->{kw} // 1;
+ $self->prepare_inputs($lei, \@inputs) or return;
$lei->ale; # initialize for workers to read
my $j = $lei->{opt}->{jobs} // scalar(@{$self->{inputs}}) || 1;
if (my $net = $lei->{net}) {
@@ -79,16 +83,6 @@ sub import_start {
while ($op && $op->{sock}) { $op->event_step }
}
-sub lei_import { # the main "lei import" method
- my ($lei, @inputs) = @_;
- my $sto = $lei->_lei_store(1);
- $sto->write_prepare($lei);
- my $self = $lei->{imp} = bless {}, __PACKAGE__;
- $self->{-import_kw} = $lei->{opt}->{kw} // 1;
- $self->prepare_inputs($lei, \@inputs) or return;
- import_start($lei);
-}
-
no warnings 'once';
*ipc_atfork_child = \&PublicInbox::LeiInput::input_only_atfork_child;
diff --git a/lib/PublicInbox/LeiMark.pm b/lib/PublicInbox/LeiMark.pm
index 3b5e6c2c..9d77f4b4 100644
--- a/lib/PublicInbox/LeiMark.pm
+++ b/lib/PublicInbox/LeiMark.pm
@@ -105,8 +105,8 @@ sub input_net_cb { # imap_each, nntp_each cb
sub lei_mark { # the "lei mark" method
my ($lei, @argv) = @_;
my $sto = $lei->_lei_store(1);
- my $self = $lei->{mark} = bless { missing => 0 }, __PACKAGE__;
$sto->write_prepare($lei);
+ my $self = bless { missing => 0 }, __PACKAGE__;
$lei->ale; # refresh and prepare
my $vmd_mod = vmd_mod_extract(\@argv);
return $lei->fail(join("\n", @{$vmd_mod->{err}})) if $vmd_mod->{err};
diff --git a/lib/PublicInbox/LeiMirror.pm b/lib/PublicInbox/LeiMirror.pm
index c916f2d0..6e62625d 100644
--- a/lib/PublicInbox/LeiMirror.pm
+++ b/lib/PublicInbox/LeiMirror.pm
@@ -269,7 +269,6 @@ sub do_mirror { # via wq_io_do
sub start {
my ($cls, $lei, $src, $dst) = @_;
my $self = bless { lei => $lei, src => $src, dst => $dst }, $cls;
- $lei->{mrr} = $self;
if ($src =~ m!https?://!) {
require URI;
require PublicInbox::LeiCurl;
@@ -281,6 +280,7 @@ sub start {
my $op = $lei->workers_start($self, 'lei_mirror', 1, {
'' => [ \&mirror_done, $lei ]
});
+ $lei->{mrr} = $self;
$self->wq_io_do('do_mirror', []);
$self->wq_close(1);
while ($op && $op->{sock}) { $op->event_step }
diff --git a/lib/PublicInbox/LeiP2q.pm b/lib/PublicInbox/LeiP2q.pm
index 0f7ffb5f..fda055fe 100644
--- a/lib/PublicInbox/LeiP2q.pm
+++ b/lib/PublicInbox/LeiP2q.pm
@@ -176,13 +176,14 @@ sub do_p2q { # via wq_do
sub lei_p2q { # the "lei patch-to-query" entry point
my ($lei, $input) = @_;
- my $self = $lei->{p2q} = bless {}, __PACKAGE__;
+ my $self = bless {}, __PACKAGE__;
if ($lei->{opt}->{stdin}) {
$self->{0} = delete $lei->{0}; # guard from _lei_atfork_child
} else {
$self->{input} = $input;
}
- my $op = $lei->workers_start($self, 'lei patch2query', 1);
+ my $op = $lei->workers_start($self, 'lei_p2q', 1);
+ $lei->{p2q} = $self;
$self->wq_io_do('do_p2q', []);
$self->wq_close(1);
while ($op && $op->{sock}) { $op->event_step }
diff --git a/lib/PublicInbox/LeiQuery.pm b/lib/PublicInbox/LeiQuery.pm
index 148e8524..84996e7e 100644
--- a/lib/PublicInbox/LeiQuery.pm
+++ b/lib/PublicInbox/LeiQuery.pm
@@ -50,11 +50,11 @@ sub lei_q {
# --local is enabled by default unless --only is used
# we'll allow "--only $LOCATION --local"
my $sto = $self->_lei_store(1);
- my $lse = $sto->search;
if (($opt->{'import-remote'} //= 1) |
(($opt->{'import-before'} //= \1) ? 1 : 0)) {
$sto->write_prepare($self);
}
+ my $lse = $sto->search;
if ($opt->{'local'} //= scalar(@only) ? 0 : 1) {
$lxs->prepare_external($lse);
}
^ permalink raw reply related [relevance 47%]
* [PATCH 0/5] lei: more input + worker-related stuff
@ 2021-03-23 11:48 71% Eric Wong
2021-03-23 11:48 70% ` [PATCH 2/5] test_common: check lei/errors.log Eric Wong
` (2 more replies)
0 siblings, 3 replies; 200+ results
From: Eric Wong @ 2021-03-23 11:48 UTC (permalink / raw)
To: meta
Drop a bunch of redundant code, yay!
Eric Wong (5):
net_reader: nntp_each: pass keywords as `undef'
test_common: check lei/errors.log
lei: persistent workers (lei_store) run in /
lei_input: more common code between <mark|convert|import>
lei: improve management around short-lived workers
lib/PublicInbox/LEI.pm | 2 +-
lib/PublicInbox/LeiConvert.pm | 50 +++++-------------
lib/PublicInbox/LeiExternal.pm | 3 +-
lib/PublicInbox/LeiImport.pm | 94 +++++++++-------------------------
lib/PublicInbox/LeiInput.pm | 45 ++++++++++++++--
lib/PublicInbox/LeiMark.pm | 59 ++++-----------------
lib/PublicInbox/LeiMirror.pm | 2 +-
lib/PublicInbox/LeiP2q.pm | 5 +-
lib/PublicInbox/LeiQuery.pm | 2 +-
lib/PublicInbox/NetReader.pm | 5 +-
lib/PublicInbox/TestCommon.pm | 13 +++--
11 files changed, 109 insertions(+), 171 deletions(-)
^ permalink raw reply [relevance 71%]
* [PATCH] lei: hide *_atfork_child from command-line
@ 2021-03-23 6:51 56% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-23 6:51 UTC (permalink / raw)
To: meta
Otherwise we could get non-sensical results if somebody tries
running "lei atfork_child" from the command-line.
---
lib/PublicInbox/LEI.pm | 2 +-
lib/PublicInbox/LeiConvert.pm | 2 +-
lib/PublicInbox/LeiInput.pm | 2 +-
lib/PublicInbox/LeiMirror.pm | 2 +-
lib/PublicInbox/LeiP2q.pm | 4 ++--
lib/PublicInbox/LeiStore.pm | 2 +-
lib/PublicInbox/LeiToMail.pm | 2 +-
lib/PublicInbox/LeiXSearch.pm | 2 +-
8 files changed, 9 insertions(+), 9 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 0be417eb..17ca637e 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -449,7 +449,7 @@ sub note_sigpipe { # triggers sigpipe_handler
x_it($self, 13);
}
-sub lei_atfork_child {
+sub _lei_atfork_child {
my ($self, $persist) = @_;
# we need to explicitly close things which are on stack
if ($persist) {
diff --git a/lib/PublicInbox/LeiConvert.pm b/lib/PublicInbox/LeiConvert.pm
index 51a233bd..49e2b7af 100644
--- a/lib/PublicInbox/LeiConvert.pm
+++ b/lib/PublicInbox/LeiConvert.pm
@@ -86,7 +86,7 @@ sub lei_convert { # the main "lei convert" method
sub ipc_atfork_child {
my ($self) = @_;
my $lei = $self->{lei};
- $lei->lei_atfork_child;
+ $lei->_lei_atfork_child;
my $l2m = delete $lei->{l2m};
if (my $net = $lei->{net}) { # may prompt user once
$net->{mics_cached} = $net->imap_common_init($lei);
diff --git a/lib/PublicInbox/LeiInput.pm b/lib/PublicInbox/LeiInput.pm
index 6ad57772..2a4968d4 100644
--- a/lib/PublicInbox/LeiInput.pm
+++ b/lib/PublicInbox/LeiInput.pm
@@ -106,7 +106,7 @@ sub prepare_inputs { # returns undef on error
sub input_only_atfork_child {
my ($self) = @_;
my $lei = $self->{lei};
- $lei->lei_atfork_child;
+ $lei->_lei_atfork_child;
PublicInbox::IPC::ipc_atfork_child($self);
$lei->{auth}->do_auth_atfork($self) if $lei->{auth};
undef;
diff --git a/lib/PublicInbox/LeiMirror.pm b/lib/PublicInbox/LeiMirror.pm
index 65818796..c916f2d0 100644
--- a/lib/PublicInbox/LeiMirror.pm
+++ b/lib/PublicInbox/LeiMirror.pm
@@ -288,7 +288,7 @@ sub start {
sub ipc_atfork_child {
my ($self) = @_;
- $self->{lei}->lei_atfork_child;
+ $self->{lei}->_lei_atfork_child;
$SIG{TERM} = sub { exit(128 + 15) }; # trigger OnDestroy $reap
$self->SUPER::ipc_atfork_child;
}
diff --git a/lib/PublicInbox/LeiP2q.pm b/lib/PublicInbox/LeiP2q.pm
index 4abe1345..0f7ffb5f 100644
--- a/lib/PublicInbox/LeiP2q.pm
+++ b/lib/PublicInbox/LeiP2q.pm
@@ -178,7 +178,7 @@ sub lei_p2q { # the "lei patch-to-query" entry point
my ($lei, $input) = @_;
my $self = $lei->{p2q} = bless {}, __PACKAGE__;
if ($lei->{opt}->{stdin}) {
- $self->{0} = delete $lei->{0}; # guard from lei_atfork_child
+ $self->{0} = delete $lei->{0}; # guard from _lei_atfork_child
} else {
$self->{input} = $input;
}
@@ -191,7 +191,7 @@ sub lei_p2q { # the "lei patch-to-query" entry point
sub ipc_atfork_child {
my ($self) = @_;
my $lei = $self->{lei};
- $lei->lei_atfork_child;
+ $lei->_lei_atfork_child;
$SIG{__WARN__} = PublicInbox::Eml::warn_ignore_cb();
$self->SUPER::ipc_atfork_child;
}
diff --git a/lib/PublicInbox/LeiStore.pm b/lib/PublicInbox/LeiStore.pm
index b5d43b7e..fa03f93c 100644
--- a/lib/PublicInbox/LeiStore.pm
+++ b/lib/PublicInbox/LeiStore.pm
@@ -322,7 +322,7 @@ sub done {
sub ipc_atfork_child {
my ($self) = @_;
my $lei = $self->{lei};
- $lei->lei_atfork_child(1) if $lei;
+ $lei->_lei_atfork_child(1) if $lei;
$self->SUPER::ipc_atfork_child;
}
diff --git a/lib/PublicInbox/LeiToMail.pm b/lib/PublicInbox/LeiToMail.pm
index e9ab939c..1be15707 100644
--- a/lib/PublicInbox/LeiToMail.pm
+++ b/lib/PublicInbox/LeiToMail.pm
@@ -636,7 +636,7 @@ sub do_post_auth {
sub ipc_atfork_child {
my ($self) = @_;
my $lei = $self->{lei};
- $lei->lei_atfork_child;
+ $lei->_lei_atfork_child;
$lei->{auth}->do_auth_atfork($self) if $lei->{auth};
$SIG{__WARN__} = PublicInbox::Eml::warn_ignore_cb();
$self->SUPER::ipc_atfork_child;
diff --git a/lib/PublicInbox/LeiXSearch.pm b/lib/PublicInbox/LeiXSearch.pm
index b6aaf3e1..c6b82eeb 100644
--- a/lib/PublicInbox/LeiXSearch.pm
+++ b/lib/PublicInbox/LeiXSearch.pm
@@ -401,7 +401,7 @@ sub incr_start_query { # called whenever an l2m shard starts do_post_auth
sub ipc_atfork_child {
my ($self) = @_;
- $self->{lei}->lei_atfork_child;
+ $self->{lei}->_lei_atfork_child;
$SIG{__WARN__} = PublicInbox::Eml::warn_ignore_cb();
$self->SUPER::ipc_atfork_child;
}
^ permalink raw reply related [relevance 56%]
* [PATCH 0/2] lei mark: volatile metadata tagging
@ 2021-03-23 5:02 71% Eric Wong
2021-03-23 5:02 29% ` [PATCH 1/2] lei mark: command for (un)setting keywords and labels Eric Wong
2021-03-23 5:02 56% ` [PATCH 2/2] lei mark: add support for (bash) completion Eric Wong
0 siblings, 2 replies; 200+ results
From: Eric Wong @ 2021-03-23 5:02 UTC (permalink / raw)
To: meta
I'm not sure if this should be called "mark", but maybe
"lei tag" is a better name?
It allows us to set and unset keywords (aka IMAP/Maildir flags)
and labels (aka JMAP mailbox name) for messages already known
to lei/store.
We don't support reading labels, yet...
Eric Wong (2):
lei mark: command for (un)setting keywords and labels
lei mark: add support for (bash) completion
MANIFEST | 2 +
lib/PublicInbox/LEI.pm | 42 +++----
lib/PublicInbox/LeiImport.pm | 15 +--
lib/PublicInbox/LeiInput.pm | 11 +-
lib/PublicInbox/LeiMark.pm | 220 +++++++++++++++++++++++++++++++++++
lib/PublicInbox/LeiStore.pm | 19 +++
lib/PublicInbox/SearchIdx.pm | 23 ++++
t/lei-mark.t | 47 ++++++++
8 files changed, 347 insertions(+), 32 deletions(-)
create mode 100644 lib/PublicInbox/LeiMark.pm
create mode 100644 t/lei-mark.t
^ permalink raw reply [relevance 71%]
* [PATCH 2/2] lei mark: add support for (bash) completion
2021-03-23 5:02 71% [PATCH 0/2] lei mark: volatile metadata tagging Eric Wong
2021-03-23 5:02 29% ` [PATCH 1/2] lei mark: command for (un)setting keywords and labels Eric Wong
@ 2021-03-23 5:02 56% ` Eric Wong
1 sibling, 0 replies; 200+ results
From: Eric Wong @ 2021-03-23 5:02 UTC (permalink / raw)
To: meta
Only lightly tested, this seems to suffer from the same
problem as external completions for network URLs with
colons in them. In any case, its usable enough for me.
The core LEI module now supports completions for lazy-loaded
commands, too, so we'll be able to do completions for other
commands more easily.
---
lib/PublicInbox/LEI.pm | 27 ++++++++++++++----------
lib/PublicInbox/LeiMark.pm | 43 ++++++++++++++++++++++++++++++++++++++
2 files changed, 59 insertions(+), 11 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 91c95239..0be417eb 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -604,6 +604,19 @@ EOM
}
}
+sub lazy_cb ($$$) {
+ my ($self, $cmd, $pfx) = @_;
+ my $ucmd = $cmd;
+ $ucmd =~ tr/-/_/;
+ my $cb;
+ $cb = $self->can($pfx.$ucmd) and return $cb;
+ my $base = $ucmd;
+ $base =~ s/_([a-z])/\u$1/g;
+ my $pkg = "PublicInbox::Lei\u$base";
+ ($INC{"PublicInbox/Lei\u$base.pm"} // eval("require $pkg")) ?
+ $pkg->can($pfx.$ucmd) : undef;
+}
+
sub dispatch {
my ($self, $cmd, @argv) = @_;
local $current_lei = $self; # for __WARN__
@@ -616,14 +629,7 @@ sub dispatch {
push @{$self->{opt}->{substr($cmd, 1, 1)}}, $v;
$cmd = shift(@argv) // return _help($self, 'no command given');
}
- my $func = "lei_$cmd";
- $func =~ tr/-/_/;
- my $cb = __PACKAGE__->can($func) // ($CMD{$cmd} ? do {
- my $mod = "PublicInbox::Lei\u$cmd";
- ($INC{"PublicInbox/Lei\u$cmd.pm"} //
- eval("require $mod")) ? $mod->can($func) : undef;
- } : undef);
- if ($cb) {
+ if (my $cb = lazy_cb(__PACKAGE__, $cmd, 'lei_')) {
optparse($self, $cmd, \@argv) or return;
$self->{opt}->{c} and (_tmp_cfg($self) // return);
if (my $chdir = $self->{opt}->{C}) {
@@ -808,9 +814,8 @@ sub lei__complete {
@v;
} grep(/\A(?:[\w-]+\|)*$opt\b.*?(?:\t$cmd)?\z/, keys %OPTDESC);
}
- $cmd =~ tr/-/_/;
- if (my $sub = $self->can("_complete_$cmd")) {
- puts $self, $sub->($self, @argv, $cur ? ($cur) : ());
+ if (my $cb = lazy_cb($self, $cmd, '_complete_')) {
+ puts $self, $cb->($self, @argv, $cur ? ($cur) : ());
}
# TODO: URLs, pathnames, OIDs, MIDs, etc... See optparse() for
# proto parsing.
diff --git a/lib/PublicInbox/LeiMark.pm b/lib/PublicInbox/LeiMark.pm
index aa52ad5a..7b50aa51 100644
--- a/lib/PublicInbox/LeiMark.pm
+++ b/lib/PublicInbox/LeiMark.pm
@@ -174,4 +174,47 @@ sub ipc_atfork_child {
PublicInbox::OnDestroy->new($$, \¬e_missing, $self);
}
+# Workaround bash word-splitting s to ['kw', ':', 'keyword' ...]
+# Maybe there's a better way to go about this in
+# contrib/completion/lei-completion.bash
+sub _complete_mark_common ($) {
+ my ($argv) = @_;
+ # Workaround bash word-splitting URLs to ['https', ':', '//' ...]
+ # Maybe there's a better way to go about this in
+ # contrib/completion/lei-completion.bash
+ my $re = '';
+ my $cur = pop(@$argv) // '';
+ if (@$argv) {
+ my @x = @$argv;
+ if ($cur eq ':' && @x) {
+ push @x, $cur;
+ $cur = '';
+ }
+ while (@x > 2 && $x[0] !~ /\A[+\-](?:kw|L)\z/ &&
+ $x[1] ne ':') {
+ shift @x;
+ }
+ if (@x >= 2) { # qw(kw : $KEYWORD) or qw(kw :)
+ $re = join('', @x);
+ } else { # just return everything and hope for the best
+ $re = join('', @$argv);
+ }
+ $re = quotemeta($re);
+ }
+ ($cur, $re);
+}
+
+# FIXME: same problems as _complete_forget_external and similar
+sub _complete_mark {
+ my ($self, @argv) = @_;
+ my @all = map { ("+kw:$_", "-kw:$_") } @KW;
+ return @all if !@argv;
+ my ($cur, $re) = _complete_mark_common(\@argv);
+ map {
+ # only return the part specified on the CLI
+ # don't duplicate if already 100% completed
+ /\A$re(\Q$cur\E.*)/ ? ($cur eq $1 ? () : $1) : ();
+ } grep(/$re\Q$cur/, @all);
+}
+
1;
^ permalink raw reply related [relevance 56%]
* [PATCH 1/2] lei mark: command for (un)setting keywords and labels
2021-03-23 5:02 71% [PATCH 0/2] lei mark: volatile metadata tagging Eric Wong
@ 2021-03-23 5:02 29% ` Eric Wong
2021-03-23 5:02 56% ` [PATCH 2/2] lei mark: add support for (bash) completion Eric Wong
1 sibling, 0 replies; 200+ results
From: Eric Wong @ 2021-03-23 5:02 UTC (permalink / raw)
To: meta
Only tested for keywords and labels with file inputs, so far;
but it seems to do what it needs to do. There's a bit more
redundant code than I'd like, and more opportunities for code
sharing in the future
"lei import" will be expanded to support +kw:$KEYWORD and
+L:$LABEL in the future.
---
MANIFEST | 2 +
lib/PublicInbox/LEI.pm | 15 ++-
lib/PublicInbox/LeiImport.pm | 15 +--
lib/PublicInbox/LeiInput.pm | 11 ++-
lib/PublicInbox/LeiMark.pm | 177 +++++++++++++++++++++++++++++++++++
lib/PublicInbox/LeiStore.pm | 19 ++++
lib/PublicInbox/SearchIdx.pm | 23 +++++
t/lei-mark.t | 47 ++++++++++
8 files changed, 288 insertions(+), 21 deletions(-)
create mode 100644 lib/PublicInbox/LeiMark.pm
create mode 100644 t/lei-mark.t
diff --git a/MANIFEST b/MANIFEST
index df8440ef..87e4b616 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -188,6 +188,7 @@ lib/PublicInbox/LeiExternal.pm
lib/PublicInbox/LeiHelp.pm
lib/PublicInbox/LeiImport.pm
lib/PublicInbox/LeiInput.pm
+lib/PublicInbox/LeiMark.pm
lib/PublicInbox/LeiMirror.pm
lib/PublicInbox/LeiOverview.pm
lib/PublicInbox/LeiP2q.pm
@@ -377,6 +378,7 @@ t/lei-import-imap.t
t/lei-import-maildir.t
t/lei-import-nntp.t
t/lei-import.t
+t/lei-mark.t
t/lei-mirror.t
t/lei-p2q.t
t/lei-q-kw.t
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 1e720b89..91c95239 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -157,9 +157,10 @@ our %CMD = ( # sorted in order of importance/use:
'plonk' => [ '--threads|--from=IDENT',
'exclude mail matching From: or threads from non-Message-ID searches',
qw(stdin| threads|t from|f=s mid=s oid=s), @c_opt ],
-'mark' => [ 'MESSAGE_FLAGS...',
- 'set/unset keywords on message(s) from stdin',
- qw(stdin| oid=s exact by-mid|mid:s), @c_opt ],
+'mark' => [ 'KEYWORDS...',
+ 'set/unset keywords on message(s)',
+ qw(stdin| in-format|F=s input|i=s@ oid=s@ mid=s@), @c_opt,
+ pass_through('-kw:foo for delete') ],
'forget' => [ '[--stdin|--oid=OID|--by-mid=MID]',
"exclude message(s) on stdin from `q' search results",
qw(stdin| oid=s exact by-mid|mid:s), @c_opt ],
@@ -348,7 +349,7 @@ my %CONFIG_KEYS = (
'leistore.dir' => 'top-level storage location',
);
-my @WQ_KEYS = qw(lxs l2m imp mrr cnv p2q); # internal workers
+my @WQ_KEYS = qw(lxs l2m imp mrr cnv p2q mark); # internal workers
# pronounced "exit": x_it(1 << 8) => exit(1); x_it(13) => SIGPIPE
sub x_it ($$) {
@@ -460,7 +461,7 @@ sub lei_atfork_child {
open STDERR, '+>&='.fileno($self->{2}) or warn "open $!";
delete $self->{0};
}
- delete @$self{qw(cnv)};
+ delete @$self{qw(cnv mark imp)};
for (delete @$self{qw(3 old_1 au_done)}) {
close($_) if defined($_);
}
@@ -690,10 +691,6 @@ sub lei_show {
my ($self, @argv) = @_;
}
-sub lei_mark {
- my ($self, @argv) = @_;
-}
-
sub _config {
my ($self, @argv) = @_;
my %env = (%{$self->{env}}, GIT_CONFIG => undef);
diff --git a/lib/PublicInbox/LeiImport.pm b/lib/PublicInbox/LeiImport.pm
index 9ad2ff12..21af28a3 100644
--- a/lib/PublicInbox/LeiImport.pm
+++ b/lib/PublicInbox/LeiImport.pm
@@ -78,16 +78,6 @@ sub lei_import { # the main "lei import" method
import_start($lei);
}
-sub ipc_atfork_child {
- my ($self) = @_;
- my $lei = $self->{lei};
- delete $lei->{imp}; # drop circular ref
- $lei->lei_atfork_child;
- $self->SUPER::ipc_atfork_child;
- $lei->{auth}->do_auth_atfork($self) if $lei->{auth};
- undef;
-}
-
sub _import_maildir { # maildir_each_eml cb
my ($f, $kw, $eml, $sto, $set_kw) = @_;
$sto->ipc_do('set_eml', $eml, $set_kw ? { kw => $kw }: ());
@@ -137,6 +127,9 @@ sub import_stdin {
$self->input_fh($lei->{opt}->{'in-format'}, $in, '<stdin>');
}
-no warnings 'once'; # the following works even when LeiAuth is lazy-loaded
+no warnings 'once';
+*ipc_atfork_child = \&PublicInbox::LeiInput::input_only_atfork_child;
+
+# the following works even when LeiAuth is lazy-loaded
*net_merge_all = \&PublicInbox::LeiAuth::net_merge_all;
1;
diff --git a/lib/PublicInbox/LeiInput.pm b/lib/PublicInbox/LeiInput.pm
index 859fdb11..6ad57772 100644
--- a/lib/PublicInbox/LeiInput.pm
+++ b/lib/PublicInbox/LeiInput.pm
@@ -45,7 +45,7 @@ error reading $name: $!
}
}
-sub prepare_inputs {
+sub prepare_inputs { # returns undef on error
my ($self, $lei, $inputs) = @_;
my $in_fmt = $lei->{opt}->{'in-format'};
if ($lei->{opt}->{stdin}) {
@@ -103,4 +103,13 @@ sub prepare_inputs {
$self->{inputs} = $inputs;
}
+sub input_only_atfork_child {
+ my ($self) = @_;
+ my $lei = $self->{lei};
+ $lei->lei_atfork_child;
+ PublicInbox::IPC::ipc_atfork_child($self);
+ $lei->{auth}->do_auth_atfork($self) if $lei->{auth};
+ undef;
+}
+
1;
diff --git a/lib/PublicInbox/LeiMark.pm b/lib/PublicInbox/LeiMark.pm
new file mode 100644
index 00000000..aa52ad5a
--- /dev/null
+++ b/lib/PublicInbox/LeiMark.pm
@@ -0,0 +1,177 @@
+# Copyright (C) 2021 all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+
+# handles "lei mark" command
+package PublicInbox::LeiMark;
+use strict;
+use v5.10.1;
+use parent qw(PublicInbox::IPC PublicInbox::LeiInput);
+use PublicInbox::Eml;
+use PublicInbox::PktOp qw(pkt_do);
+
+# JMAP RFC 8621 4.1.1
+my @KW = (qw(seen answered flagged draft), # system
+ qw(forwarded phishing junk notjunk)); # reserved
+# note: RFC 8621 states "Users may add arbitrary keywords to an Email",
+# but is it good idea? Stick to the system and reserved ones, for now.
+# The "system" ones map to Maildir flags and mbox Status/X-Status headers.
+my %KW = map { $_ => 1 } @KW;
+my $L_MAX = 244; # Xapian term limit - length('L')
+
+# RFC 8621, sec 2 (Mailboxes) a "label" for us is a JMAP Mailbox "name"
+# "Servers MAY reject names that violate server policy"
+my %ERR = (
+ L => sub {
+ my ($label) = @_;
+ length($label) >= $L_MAX and
+ return "`$label' too long (must be <= $L_MAX)";
+ $label =~ m{\A[a-z0-9_][a-z0-9_\-\./\@\!,]*[a-z0-9]\z} ?
+ undef : "`$label' is invalid";
+ },
+ kw => sub {
+ my ($kw) = @_;
+ $KW{$kw} ? undef : <<EOM;
+`$kw' is not one of: `seen', `flagged', `answered', `draft'
+`junk', `notjunk', `phishing' or `forwarded'
+EOM
+
+ }
+);
+
+# like Getopt::Long, but for +kw:FOO and -kw:FOO to prepare
+# for update_xvmd -> update_vmd
+sub vmd_mod_extract {
+ my $argv = $_[-1];
+ my $vmd_mod = {};
+ my @new_argv;
+ for my $x (@$argv) {
+ if ($x =~ /\A(\+|\-)(kw|L):(.+)\z/) {
+ my ($op, $pfx, $val) = ($1, $2, $3);
+ if (my $err = $ERR{$pfx}->($val)) {
+ push @{$vmd_mod->{err}}, $err;
+ } else { # set "+kw", "+L", "-L", "-kw"
+ push @{$vmd_mod->{$op.$pfx}}, $val;
+ }
+ } else {
+ push @new_argv, $x;
+ }
+ }
+ @$argv = @new_argv;
+ $vmd_mod;
+}
+
+sub eml_cb { # used by PublicInbox::LeiInput::input_fh
+ my ($self, $eml) = @_;
+ if (my $xoids = $self->{lei}->{ale}->xoids_for($eml)) {
+ $self->{lei}->{sto}->ipc_do('update_xvmd', $xoids,
+ $self->{vmd_mod});
+ } else {
+ ++$self->{missing};
+ }
+}
+
+sub mbox_cb { eml_cb($_[1], $_[0]) } # used by PublicInbox::LeiInput::input_fh
+
+sub mark_done_wait { # dwaitpid callback
+ my ($arg, $pid) = @_;
+ my ($mark, $lei) = @$arg;
+ $lei->child_error($?, 'non-fatal errors during mark') if $?;
+ my $sto = delete $lei->{sto};
+ my $wait = $sto->ipc_do('done') if $sto; # PublicInbox::LeiStore::done
+ $lei->dclose;
+}
+
+sub mark_done { # EOF callback for main daemon
+ my ($lei) = @_;
+ my $mark = delete $lei->{mark} or return;
+ $mark->wq_wait_old(\&mark_done_wait, $lei);
+}
+
+sub net_merge_complete { # callback used by LeiAuth
+ my ($self) = @_;
+ for my $input (@{$self->{inputs}}) {
+ $self->wq_io_do('mark_path_url', [], $input);
+ }
+ $self->wq_close(1);
+}
+
+sub _mark_maildir { # maildir_each_eml cb
+ my ($f, $kw, $eml, $self) = @_;
+ eml_cb($self, $eml);
+}
+
+sub _mark_net { # imap_each, nntp_each cb
+ my ($url, $uid, $kw, $eml, $self) = @_;
+ eml_cb($self, $eml)
+}
+
+sub lei_mark { # the "lei mark" method
+ my ($lei, @argv) = @_;
+ my $sto = $lei->_lei_store(1);
+ my $self = $lei->{mark} = bless { missing => 0 }, __PACKAGE__;
+ $sto->write_prepare($lei);
+ $lei->ale; # refresh and prepare
+ my $vmd_mod = vmd_mod_extract(\@argv);
+ return $lei->fail(join("\n", @{$vmd_mod->{err}})) if $vmd_mod->{err};
+ $self->prepare_inputs($lei, \@argv) or return;
+ grep(defined, @$vmd_mod{qw(+kw +L -L -kw)}) or
+ return $lei->fail('no keywords or labels specified');
+ my $ops = { '' => [ \&mark_done, $lei ] };
+ $lei->{auth}->op_merge($ops, $self) if $lei->{auth};
+ $self->{vmd_mod} = $vmd_mod;
+ my $op = $lei->workers_start($self, 'lei_mark', 1, $ops);
+ $self->wq_io_do('mark_stdin', []) if $self->{0};
+ net_merge_complete($self) unless $lei->{auth};
+ while ($op && $op->{sock}) { $op->event_step }
+}
+
+sub mark_path_url {
+ my ($self, $input) = @_;
+ my $lei = $self->{lei};
+ my $ifmt = lc($lei->{opt}->{'in-format'} // '');
+ # TODO auto-detect?
+ if ($input =~ m!\Aimaps?://!i) {
+ $lei->{net}->imap_each($input, \&_mark_net, $self);
+ return;
+ } elsif ($input =~ m!\A(?:nntps?|s?news)://!i) {
+ $lei->{net}->nntp_each($input, \&_mark_net, $self);
+ return;
+ } elsif ($input =~ s!\A([a-z0-9]+):!!i) {
+ $ifmt = lc $1;
+ }
+ if (-f $input) {
+ my $m = $lei->{opt}->{'lock'} // ($ifmt eq 'eml' ? ['none'] :
+ PublicInbox::MboxLock->defaults);
+ my $mbl = PublicInbox::MboxLock->acq($input, 0, $m);
+ $self->input_fh($ifmt, $mbl->{fh}, $input);
+ } elsif (-d _ && (-d "$input/cur" || -d "$input/new")) {
+ return $lei->fail(<<EOM) if $ifmt && $ifmt ne 'maildir';
+$input appears to a be a maildir, not $ifmt
+EOM
+ PublicInbox::MdirReader::maildir_each_eml($input,
+ \&_mark_maildir, $self);
+ } else {
+ $lei->fail("$input unsupported (TODO)");
+ }
+}
+
+sub mark_stdin {
+ my ($self) = @_;
+ my $lei = $self->{lei};
+ my $in = delete $self->{0};
+ $self->input_fh($lei->{opt}->{'in-format'}, $in, '<stdin>');
+}
+
+sub note_missing {
+ my ($self) = @_;
+ $self->{lei}->child_error(1 << 8) if $self->{missing};
+}
+
+sub ipc_atfork_child {
+ my ($self) = @_;
+ PublicInbox::LeiInput::input_only_atfork_child($self);
+ # this goes out-of-scope at worker process exit:
+ PublicInbox::OnDestroy->new($$, \¬e_missing, $self);
+}
+
+1;
diff --git a/lib/PublicInbox/LeiStore.pm b/lib/PublicInbox/LeiStore.pm
index b390b318..b5d43b7e 100644
--- a/lib/PublicInbox/LeiStore.pm
+++ b/lib/PublicInbox/LeiStore.pm
@@ -228,12 +228,30 @@ sub set_eml {
set_eml_vmd($self, $eml, $vmd);
}
+sub update_xvmd {
+ my ($self, $xoids, $vmd_mod) = @_;
+ my $eidx = eidx_init($self);
+ my $oidx = $eidx->{oidx};
+ my %seen;
+ for my $oid (keys %$xoids) {
+ my @docids = $oidx->blob_exists($oid) or next;
+ scalar(@docids) > 1 and
+ warn "W: $oid indexed as multiple docids: @docids\n";
+ for my $docid (@docids) {
+ next if $seen{$docid}++;
+ my $idx = $eidx->idx_shard($docid);
+ $idx->ipc_do('update_vmd', $docid, $vmd_mod);
+ }
+ }
+}
+
# set or update keywords for external message, called via ipc_do
sub set_xvmd {
my ($self, $xoids, $eml, $vmd) = @_;
my $eidx = eidx_init($self);
my $oidx = $eidx->{oidx};
+ my %seen;
# see if we can just update existing docs
for my $oid (keys %$xoids) {
@@ -241,6 +259,7 @@ sub set_xvmd {
scalar(@docids) > 1 and
warn "W: $oid indexed as multiple docids: @docids\n";
for my $docid (@docids) {
+ next if $seen{$docid}++;
my $idx = $eidx->idx_shard($docid);
$idx->ipc_do('set_vmd', $docid, $vmd);
}
diff --git a/lib/PublicInbox/SearchIdx.pm b/lib/PublicInbox/SearchIdx.pm
index 3f933121..7d46489c 100644
--- a/lib/PublicInbox/SearchIdx.pm
+++ b/lib/PublicInbox/SearchIdx.pm
@@ -597,6 +597,29 @@ sub remove_vmd {
$self->{xdb}->replace_document($docid, $doc) if $replace;
}
+sub update_vmd {
+ my ($self, $docid, $vmd_mod) = @_;
+ begin_txn_lazy($self);
+ my $doc = _get_doc($self, $docid) or return;
+ my $updated = 0;
+ my @x = @VMD_MAP;
+ while (my ($field, $pfx) = splice(@x, 0, 2)) {
+ # field: "label" or "kw"
+ for my $val (@{$vmd_mod->{"-$field"} // []}) {
+ eval {
+ $doc->remove_term($pfx . $val);
+ ++$updated;
+ };
+ }
+ for my $val (@{$vmd_mod->{"+$field"} // []}) {
+ $doc->add_boolean_term($pfx . $val);
+ ++$updated;
+ }
+ }
+ $self->{xdb}->replace_document($docid, $doc) if $updated;
+ $updated;
+}
+
sub xdb_remove {
my ($self, @docids) = @_;
$self->begin_txn_lazy;
diff --git a/t/lei-mark.t b/t/lei-mark.t
new file mode 100644
index 00000000..ddf5634c
--- /dev/null
+++ b/t/lei-mark.t
@@ -0,0 +1,47 @@
+#!perl -w
+# Copyright (C) 2021 all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+use strict; use v5.10.1; use PublicInbox::TestCommon;
+require_git 2.6;
+require_mods(qw(json DBD::SQLite Search::Xapian));
+my $check_kw = sub {
+ my ($exp, %opt) = @_;
+ my $mid = $opt{mid} // 'testmessage@example.com';
+ lei_ok('q', "m:$mid");
+ my $res = json_utf8->decode($lei_out);
+ is($res->[1], undef, 'only got one result');
+ my $msg = $opt{msg} ? " $opt{msg}" : '';
+ ($exp ? is_deeply($res->[0]->{kw}, $exp, "got @$exp$msg")
+ : is($res->[0]->{kw}, undef, "got undef$msg")) or
+ diag explain($res);
+};
+
+test_lei(sub {
+ lei_ok(qw(import -F eml t/utf8.eml));
+ lei_ok(qw(mark -F eml t/utf8.eml +kw:flagged));
+ $check_kw->(['flagged']);
+ ok(!lei(qw(mark -F eml t/utf8.eml +kw:seeen)), 'bad kw rejected');
+ like($lei_err, qr/`seeen' is not one of/, 'got helpful error');
+ ok(!lei(qw(mark -F eml t/utf8.eml +k:seen)), 'bad prefix rejected');
+ ok(!lei(qw(mark -F eml t/utf8.eml)), 'no keywords');
+ my $mb = "$ENV{HOME}/mb";
+ my $md = "$ENV{HOME}/md";
+ lei_ok(qw(q m:testmessage@example.com -o), "mboxrd:$mb");
+ ok(-s $mb, 'wrote mbox result');
+ lei_ok(qw(q m:testmessage@example.com -o), $md);
+ my @fn = glob("$md/cur/*");
+ scalar(@fn) == 1 or BAIL_OUT 'no mail '.explain(\@fn);
+ rename($fn[0], "$fn[0]S") or BAIL_OUT "rename $!";
+ $check_kw->(['flagged'], msg => 'after bad request');
+ lei_ok(qw(mark -F eml t/utf8.eml -kw:flagged));
+ $check_kw->(undef, msg => 'keyword cleared');
+ lei_ok(qw(mark -F mboxrd +kw:seen), $mb);
+ $check_kw->(['seen'], msg => 'mbox Status ignored');
+ lei_ok(qw(mark -kw:seen +kw:answered), $md);
+ $check_kw->(['answered'], msg => 'Maildir Status ignored');
+
+ open my $in, '<', 't/utf8.eml' or BAIL_OUT $!;
+ lei_ok([qw(mark -F eml - +kw:seen)], undef, { %$lei_opt, 0 => $in });
+ $check_kw->(['answered', 'seen'], msg => 'stdin works');
+});
+done_testing;
^ permalink raw reply related [relevance 29%]
* [PATCH 4/8] lei: simplify workers_start and callers
2021-03-22 7:53 70% [PATCH 0/8] lei input handling improvements Eric Wong
2021-03-22 7:53 33% ` [PATCH 1/8] lei: support -c <name>=<value> to overrides Eric Wong
2021-03-22 7:53 38% ` [PATCH 3/8] lei: share input code between convert and import Eric Wong
@ 2021-03-22 7:53 64% ` Eric Wong
2021-03-22 7:54 55% ` [PATCH 8/8] lei import: ignore Status headers in "eml" messages Eric Wong
3 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-22 7:53 UTC (permalink / raw)
To: meta
Since workers_start is in the common PublicInbox::LEI
package, we can just use \&METHOD_NAME instead of relying
on UNIVERSAL->can to avoid a method dispatch.
Most of our worker code can just use lei->dclose, so default
to doing that unless it's been overridden.
---
lib/PublicInbox/LEI.pm | 11 ++++++-----
lib/PublicInbox/LeiConvert.pm | 4 +---
lib/PublicInbox/LeiP2q.pm | 4 +---
3 files changed, 8 insertions(+), 11 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 0bd52a46..1e720b89 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -481,12 +481,13 @@ sub lei_atfork_child {
sub workers_start {
my ($lei, $wq, $ident, $jobs, $ops) = @_;
$ops = {
- '!' => [ $lei->can('fail_handler'), $lei ],
- '|' => [ $lei->can('sigpipe_handler'), $lei ],
- 'x_it' => [ $lei->can('x_it'), $lei ],
- 'child_error' => [ $lei->can('child_error'), $lei ],
- %$ops
+ '!' => [ \&fail_handler, $lei ],
+ '|' => [ \&sigpipe_handler, $lei ],
+ 'x_it' => [ \&x_it, $lei ],
+ 'child_error' => [ \&child_error, $lei ],
+ ($ops ? %$ops : ()),
};
+ $ops->{''} //= [ \&dclose, $lei ];
require PublicInbox::PktOp;
($lei->{pkt_op_c}, $lei->{pkt_op_p}) = PublicInbox::PktOp->pair($ops);
$wq->wq_workers_start($ident, $jobs, $lei->oldset, { lei => $lei });
diff --git a/lib/PublicInbox/LeiConvert.pm b/lib/PublicInbox/LeiConvert.pm
index 0aa13229..8685c194 100644
--- a/lib/PublicInbox/LeiConvert.pm
+++ b/lib/PublicInbox/LeiConvert.pm
@@ -87,9 +87,7 @@ sub lei_convert { # the main "lei convert" method
$lei->fail("output not specified or is not a mail destination");
$lei->{opt}->{augment} = 1 unless $ovv->{dst} eq '/dev/stdout';
$self->prepare_inputs($lei, \@inputs) or return;
- my $op = $lei->workers_start($self, 'lei_convert', 1, {
- '' => [ $lei->can('dclose'), $lei ]
- });
+ my $op = $lei->workers_start($self, 'lei_convert', 1);
$self->wq_io_do('do_convert', []);
$self->wq_close(1);
while ($op && $op->{sock}) { $op->event_step }
diff --git a/lib/PublicInbox/LeiP2q.pm b/lib/PublicInbox/LeiP2q.pm
index 302d7864..4abe1345 100644
--- a/lib/PublicInbox/LeiP2q.pm
+++ b/lib/PublicInbox/LeiP2q.pm
@@ -182,9 +182,7 @@ sub lei_p2q { # the "lei patch-to-query" entry point
} else {
$self->{input} = $input;
}
- my $op = $lei->workers_start($self, 'lei patch2query', 1, {
- '' => [ $lei->{p2q_done} // $lei->can('dclose'), $lei ]
- });
+ my $op = $lei->workers_start($self, 'lei patch2query', 1);
$self->wq_io_do('do_p2q', []);
$self->wq_close(1);
while ($op && $op->{sock}) { $op->event_step }
^ permalink raw reply related [relevance 64%]
* [PATCH 8/8] lei import: ignore Status headers in "eml" messages
2021-03-22 7:53 70% [PATCH 0/8] lei input handling improvements Eric Wong
` (2 preceding siblings ...)
2021-03-22 7:53 64% ` [PATCH 4/8] lei: simplify workers_start and callers Eric Wong
@ 2021-03-22 7:54 55% ` Eric Wong
3 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-22 7:54 UTC (permalink / raw)
To: meta
Those headers only have meaning with for mboxes. Don't surprise
users by trying to make sense of a header that is defined for mboxes.
It's possible to send email with (Status|X-Status) headers and
have those headers show up in a recipient's IMAP mailbox.
This was bad because an IMAP user may want to import a single
message through their MUA and pipe its contents to "lei import"
without noticing a mischievious sender stuck "X-Status: F"
(flagged/important) in there.
---
lib/PublicInbox/LeiImport.pm | 14 +++++++-------
t/lei-import.t | 27 ++++++++++++++++++++++-----
2 files changed, 29 insertions(+), 12 deletions(-)
diff --git a/lib/PublicInbox/LeiImport.pm b/lib/PublicInbox/LeiImport.pm
index 767cae60..9ad2ff12 100644
--- a/lib/PublicInbox/LeiImport.pm
+++ b/lib/PublicInbox/LeiImport.pm
@@ -10,19 +10,19 @@ use PublicInbox::Eml;
use PublicInbox::PktOp qw(pkt_do);
sub eml_cb { # used by PublicInbox::LeiInput::input_fh
- my ($self, $eml) = @_;
- my $vmd;
- if ($self->{-import_kw}) { # FIXME
- my $kw = PublicInbox::MboxReader::mbox_keywords($eml);
- $vmd = { kw => $kw } if scalar(@$kw);
- }
+ my ($self, $eml, $vmd) = @_;
my $xoids = $self->{lei}->{ale}->xoids_for($eml);
$self->{lei}->{sto}->ipc_do('set_eml', $eml, $vmd, $xoids);
}
sub mbox_cb { # MboxReader callback used by PublicInbox::LeiInput::input_fh
my ($eml, $self) = @_;
- eml_cb($self, $eml);
+ my $vmd;
+ if ($self->{-import_kw}) {
+ my $kw = PublicInbox::MboxReader::mbox_keywords($eml);
+ $vmd = { kw => $kw } if scalar(@$kw);
+ }
+ eml_cb($self, $eml, $vmd);
}
sub import_done_wait { # dwaitpid callback
diff --git a/t/lei-import.t b/t/lei-import.t
index eef1e4e2..a697d756 100644
--- a/t/lei-import.t
+++ b/t/lei-import.t
@@ -39,27 +39,44 @@ lei_ok(qw(q m:testmessage@example.com));
is(json_utf8->decode($lei_out)->[0]->{'blob'}, $oid,
'got expected OID w/o From');
-my $str = <<'';
+my $eml_str = <<'';
From: a@b
Message-ID: <x@y>
Status: RO
-my $opt = { %$lei_opt, 0 => \$str };
+my $opt = { %$lei_opt, 0 => \$eml_str };
lei_ok([qw(import -F eml -)], undef, $opt,
\'import single file with keywords from stdin');
lei_ok(qw(q m:x@y));
my $res = json_utf8->decode($lei_out);
is($res->[1], undef, 'only one result');
-is_deeply($res->[0]->{kw}, ['seen'], "message `seen' keyword set");
+is($res->[0]->{'m'}, 'x@y', 'got expected message');
+is($res->[0]->{kw}, undef, 'Status ignored for eml');
+lei_ok(qw(q -f mboxrd m:x@y));
+unlike($lei_out, qr/^Status:/, 'no Status: in imported message');
-$str =~ tr/x/v/; # v@y
-lei_ok([qw(import --no-kw -F eml -)], undef, $opt,
+
+$eml->header_set('Message-ID', '<v@y>');
+$eml->header_set('Status', 'RO');
+$in = 'From v@y Fri Oct 2 00:00:00 1993'."\n".$eml->as_string;
+lei_ok([qw(import --no-kw -F mboxrd -)], undef, { %$lei_opt, 0 => \$in },
\'import single file with --no-kw from stdin');
lei(qw(q m:v@y));
$res = json_utf8->decode($lei_out);
is($res->[1], undef, 'only one result');
+is($res->[0]->{'m'}, 'v@y', 'got expected message');
is($res->[0]->{kw}, undef, 'no keywords set');
+$eml->header_set('Message-ID', '<k@y>');
+$in = 'From k@y Fri Oct 2 00:00:00 1993'."\n".$eml->as_string;
+lei_ok([qw(import -F mboxrd -)], undef, { %$lei_opt, 0 => \$in },
+ \'import single file with --kw (default) from stdin');
+lei(qw(q m:k@y));
+$res = json_utf8->decode($lei_out);
+is($res->[1], undef, 'only one result');
+is($res->[0]->{'m'}, 'k@y', 'got expected message');
+is_deeply($res->[0]->{kw}, ['seen'], "`seen' keywords set");
+
# see t/lei_to_mail.t for "import -F mbox*"
});
done_testing;
^ permalink raw reply related [relevance 55%]
* [PATCH 3/8] lei: share input code between convert and import
2021-03-22 7:53 70% [PATCH 0/8] lei input handling improvements Eric Wong
2021-03-22 7:53 33% ` [PATCH 1/8] lei: support -c <name>=<value> to overrides Eric Wong
@ 2021-03-22 7:53 38% ` Eric Wong
2021-03-22 7:53 64% ` [PATCH 4/8] lei: simplify workers_start and callers Eric Wong
2021-03-22 7:54 55% ` [PATCH 8/8] lei import: ignore Status headers in "eml" messages Eric Wong
3 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-22 7:53 UTC (permalink / raw)
To: meta
These commands accept mail the same way, and this forces
us to maintain consistent input format support between
commands.
We'll be using this for "lei mark", too.
---
MANIFEST | 1 +
lib/PublicInbox/LEI.pm | 17 -------
lib/PublicInbox/LeiConvert.pm | 60 +++----------------------
lib/PublicInbox/LeiImport.pm | 57 ++---------------------
lib/PublicInbox/LeiInput.pm | 85 +++++++++++++++++++++++++++++++++++
5 files changed, 94 insertions(+), 126 deletions(-)
create mode 100644 lib/PublicInbox/LeiInput.pm
diff --git a/MANIFEST b/MANIFEST
index b6b4a3ab..df8440ef 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -187,6 +187,7 @@ lib/PublicInbox/LeiDedupe.pm
lib/PublicInbox/LeiExternal.pm
lib/PublicInbox/LeiHelp.pm
lib/PublicInbox/LeiImport.pm
+lib/PublicInbox/LeiInput.pm
lib/PublicInbox/LeiMirror.pm
lib/PublicInbox/LeiOverview.pm
lib/PublicInbox/LeiP2q.pm
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 9e3bb9b7..0bd52a46 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -419,23 +419,6 @@ sub fail ($$;$) {
undef;
}
-sub check_input_format ($;$) {
- my ($self, $files) = @_;
- my $opt_key = 'in-format';
- my $fmt = $self->{opt}->{$opt_key};
- if (!$fmt) {
- my $err = $files ? "regular file(s):\n@$files" : '--stdin';
- return fail($self, "--$opt_key unset for $err");
- }
- require PublicInbox::MboxLock if $files;
- require PublicInbox::MboxReader;
- return 1 if $fmt eq 'eml';
- # XXX: should this handle {gz,bz2,xz}? that's currently in LeiToMail
- PublicInbox::MboxReader->can($fmt) or
- return fail($self, "--$opt_key=$fmt unrecognized");
- 1;
-}
-
sub out ($;@) {
my $self = shift;
return if print { $self->{1} // return } @_; # likely
diff --git a/lib/PublicInbox/LeiConvert.pm b/lib/PublicInbox/LeiConvert.pm
index 8d3b221a..0aa13229 100644
--- a/lib/PublicInbox/LeiConvert.pm
+++ b/lib/PublicInbox/LeiConvert.pm
@@ -5,7 +5,7 @@
package PublicInbox::LeiConvert;
use strict;
use v5.10.1;
-use parent qw(PublicInbox::IPC);
+use parent qw(PublicInbox::IPC PublicInbox::LeiInput);
use PublicInbox::Eml;
use PublicInbox::LeiStore;
use PublicInbox::LeiOverview;
@@ -79,64 +79,14 @@ sub do_convert { # via wq_do
sub lei_convert { # the main "lei convert" method
my ($lei, @inputs) = @_;
- my $opt = $lei->{opt};
- $opt->{kw} //= 1;
+ $lei->{opt}->{kw} //= 1;
+ $lei->{opt}->{dedupe} //= 'none';
my $self = $lei->{cnv} = bless {}, __PACKAGE__;
- my $in_fmt = $opt->{'in-format'};
- my (@f, @d);
- $opt->{dedupe} //= 'none';
my $ovv = PublicInbox::LeiOverview->new($lei, 'out-format');
$lei->{l2m} or return
$lei->fail("output not specified or is not a mail destination");
- my $net = $lei->{net}; # NetWriter may be created by l2m
- $opt->{augment} = 1 unless $ovv->{dst} eq '/dev/stdout';
- if ($opt->{stdin}) {
- @inputs and return $lei->fail("--stdin and @inputs do not mix");
- $lei->check_input_format(undef) or return;
- $self->{0} = $lei->{0};
- }
- # e.g. Maildir:/home/user/Mail/ or imaps://example.com/INBOX
- for my $input (@inputs) {
- my $input_path = $input;
- if ($input =~ m!\A(?:imaps?|nntps?|s?news)://!i) {
- require PublicInbox::NetReader;
- $net //= PublicInbox::NetReader->new;
- $net->add_url($input);
- } elsif ($input_path =~ s/\A([a-z0-9]+)://is) {
- my $ifmt = lc $1;
- if (($in_fmt // $ifmt) ne $ifmt) {
- return $lei->fail(<<"");
---in-format=$in_fmt and `$ifmt:' conflict
-
- }
- if (-f $input_path) {
- require PublicInbox::MboxLock;
- require PublicInbox::MboxReader;
- PublicInbox::MboxReader->can($ifmt) or return
- $lei->fail("$ifmt not supported");
- } elsif (-d _) {
- require PublicInbox::MdirReader;
- $ifmt eq 'maildir' or return
- $lei->fail("$ifmt not supported");
- } else {
- return $lei->fail("Unable to handle $input");
- }
- } elsif (-f $input) { push @f, $input }
- elsif (-d _) { push @d, $input }
- else { return $lei->fail("Unable to handle $input") }
- }
- if (@f) { $lei->check_input_format(\@f) or return }
- if (@d) { # TODO: check for MH vs Maildir, here
- require PublicInbox::MdirReader;
- }
- $self->{inputs} = \@inputs;
- if ($net) {
- if (my $err = $net->errors) {
- return $lei->fail($err);
- }
- $net->{quiet} = $opt->{quiet};
- $lei->{net} //= $net;
- }
+ $lei->{opt}->{augment} = 1 unless $ovv->{dst} eq '/dev/stdout';
+ $self->prepare_inputs($lei, \@inputs) or return;
my $op = $lei->workers_start($self, 'lei_convert', 1, {
'' => [ $lei->can('dclose'), $lei ]
});
diff --git a/lib/PublicInbox/LeiImport.pm b/lib/PublicInbox/LeiImport.pm
index 0e2a96e8..e769fba8 100644
--- a/lib/PublicInbox/LeiImport.pm
+++ b/lib/PublicInbox/LeiImport.pm
@@ -5,7 +5,7 @@
package PublicInbox::LeiImport;
use strict;
use v5.10.1;
-use parent qw(PublicInbox::IPC);
+use parent qw(PublicInbox::IPC PublicInbox::LeiInput);
use PublicInbox::Eml;
use PublicInbox::PktOp qw(pkt_do);
@@ -67,60 +67,9 @@ sub lei_import { # the main "lei import" method
my ($lei, @inputs) = @_;
my $sto = $lei->_lei_store(1);
$sto->write_prepare($lei);
- my ($net, @f, @d);
$lei->{opt}->{kw} //= 1;
- my $self = $lei->{imp} = bless { inputs => \@inputs }, __PACKAGE__;
- if ($lei->{opt}->{stdin}) {
- @inputs and return $lei->fail("--stdin and @inputs do not mix");
- $lei->check_input_format or return;
- $self->{0} = $lei->{0};
- }
-
- my $fmt = $lei->{opt}->{'in-format'};
- # e.g. Maildir:/home/user/Mail/ or imaps://example.com/INBOX
- for my $input (@inputs) {
- my $input_path = $input;
- if ($input =~ m!\A(?:imaps?|nntps?|s?news)://!i) {
- require PublicInbox::NetReader;
- $net //= PublicInbox::NetReader->new;
- $net->add_url($input);
- } elsif ($input_path =~ s/\A([a-z0-9]+)://is) {
- my $ifmt = lc $1;
- if (($fmt // $ifmt) ne $ifmt) {
- return $lei->fail(<<"");
---in-format=$fmt and `$ifmt:' conflict
-
- }
- if (-f $input_path) {
- require PublicInbox::MboxLock;
- require PublicInbox::MboxReader;
- PublicInbox::MboxReader->can($ifmt) or return
- $lei->fail("$ifmt not supported");
- } elsif (-d _) {
- require PublicInbox::MdirReader;
- $ifmt eq 'maildir' or return
- $lei->fail("$ifmt not supported");
- } else {
- return $lei->fail("Unable to handle $input");
- }
- } elsif (-f $input) { push @f, $input
- } elsif (-d _) { push @d, $input
- } else { return $lei->fail("Unable to handle $input") }
- }
- if (@f) { $lei->check_input_format(\@f) or return }
- if (@d) { # TODO: check for MH vs Maildir, here
- require PublicInbox::MdirReader;
- }
- $self->{inputs} = \@inputs;
- if ($net) {
- if (my $err = $net->errors) {
- return $lei->fail($err);
- }
- $net->{quiet} = $lei->{opt}->{quiet};
- $lei->{net} = $net;
- require PublicInbox::LeiAuth;
- $lei->{auth} = PublicInbox::LeiAuth->new;
- }
+ my $self = $lei->{imp} = bless {}, __PACKAGE__;
+ $self->prepare_inputs($lei, \@inputs) or return;
import_start($lei);
}
diff --git a/lib/PublicInbox/LeiInput.pm b/lib/PublicInbox/LeiInput.pm
new file mode 100644
index 00000000..89585a52
--- /dev/null
+++ b/lib/PublicInbox/LeiInput.pm
@@ -0,0 +1,85 @@
+# Copyright (C) 2021 all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+
+# parent class for LeiImport, LeiConvert
+package PublicInbox::LeiInput;
+use strict;
+use v5.10.1;
+
+sub check_input_format ($;$) {
+ my ($lei, $files) = @_;
+ my $opt_key = 'in-format';
+ my $fmt = $lei->{opt}->{$opt_key};
+ if (!$fmt) {
+ my $err = $files ? "regular file(s):\n@$files" : '--stdin';
+ return $lei->fail("--$opt_key unset for $err");
+ }
+ require PublicInbox::MboxLock if $files;
+ require PublicInbox::MboxReader;
+ return 1 if $fmt eq 'eml';
+ # XXX: should this handle {gz,bz2,xz}? that's currently in LeiToMail
+ PublicInbox::MboxReader->can($fmt) or
+ return $lei->fail("--$opt_key=$fmt unrecognized");
+ 1;
+}
+
+
+sub prepare_inputs {
+ my ($self, $lei, $inputs) = @_;
+ my $in_fmt = $lei->{opt}->{'in-format'};
+ if ($lei->{opt}->{stdin}) {
+ @$inputs and return
+ $lei->fail("--stdin and @$inputs do not mix");
+ check_input_format($lei) or return;
+ $self->{0} = $lei->{0};
+ }
+ my $net = $lei->{net}; # NetWriter may be created by l2m
+ my $fmt = $lei->{opt}->{'in-format'};
+ my (@f, @d);
+ # e.g. Maildir:/home/user/Mail/ or imaps://example.com/INBOX
+ for my $input (@$inputs) {
+ my $input_path = $input;
+ if ($input =~ m!\A(?:imaps?|nntps?|s?news)://!i) {
+ require PublicInbox::NetReader;
+ $net //= PublicInbox::NetReader->new;
+ $net->add_url($input);
+ } elsif ($input_path =~ s/\A([a-z0-9]+)://is) {
+ my $ifmt = lc $1;
+ if (($in_fmt // $ifmt) ne $ifmt) {
+ return $lei->fail(<<"");
+--in-format=$in_fmt and `$ifmt:' conflict
+
+ }
+ if (-f $input_path) {
+ require PublicInbox::MboxLock;
+ require PublicInbox::MboxReader;
+ PublicInbox::MboxReader->can($ifmt) or return
+ $lei->fail("$ifmt not supported");
+ } elsif (-d _) {
+ require PublicInbox::MdirReader;
+ $ifmt eq 'maildir' or return
+ $lei->fail("$ifmt not supported");
+ } else {
+ return $lei->fail("Unable to handle $input");
+ }
+ } elsif (-f $input) { push @f, $input }
+ elsif (-d _) { push @d, $input }
+ else { return $lei->fail("Unable to handle $input") }
+ }
+ if (@f) { check_input_format($lei, \@f) or return }
+ if (@d) { # TODO: check for MH vs Maildir, here
+ require PublicInbox::MdirReader;
+ }
+ if ($net) {
+ if (my $err = $net->errors) {
+ return $lei->fail($err);
+ }
+ $net->{quiet} = $lei->{opt}->{quiet};
+ require PublicInbox::LeiAuth;
+ $lei->{auth} //= PublicInbox::LeiAuth->new;
+ $lei->{net} //= $net;
+ }
+ $self->{inputs} = $inputs;
+}
+
+1;
^ permalink raw reply related [relevance 38%]
* [PATCH 1/8] lei: support -c <name>=<value> to overrides
2021-03-22 7:53 70% [PATCH 0/8] lei input handling improvements Eric Wong
@ 2021-03-22 7:53 33% ` Eric Wong
2021-03-22 7:53 38% ` [PATCH 3/8] lei: share input code between convert and import Eric Wong
` (2 subsequent siblings)
3 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-22 7:53 UTC (permalink / raw)
To: meta
It's a bit nasty, but seems to mostly work for debugging
IMAP and NNTP commands.
---
lib/PublicInbox/LEI.pm | 109 ++++++++++++++++++++++-----------
lib/PublicInbox/LeiExternal.pm | 2 +-
t/lei.t | 9 +++
3 files changed, 83 insertions(+), 37 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index b6d21af6..9e3bb9b7 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -119,6 +119,8 @@ sub index_opt {
batch_size|batch-size=s skip-docdata)
}
+my @c_opt = qw(c=s@ C=s@ quiet|q);
+
# we generate shell completion + help using %CMD and %OPTDESC,
# see lei__complete() and PublicInbox::LeiHelp
# command => [ positional_args, 1-line description, Getopt::Long option spec ]
@@ -129,82 +131,80 @@ our %CMD = ( # sorted in order of importance/use:
sort|s=s reverse|r offset=i remote! local! external! pretty
include|I=s@ exclude=s@ only=s@ jobs|j=s globoff|g augment|a
import-remote! import-before! lock=s@ rsyncable
- alert=s@ mua=s no-torsocks torsocks=s verbose|v+ quiet|q C=s@),
+ alert=s@ mua=s no-torsocks torsocks=s verbose|v+), @c_opt,
PublicInbox::LeiQuery::curl_opt(), opt_dash('limit|n=i', '[0-9]+') ],
'show' => [ 'MID|OID', 'show a given object (Message-ID or object ID)',
- qw(type=s solve! format|f=s dedupe|d=s threads|t remote local! C=s@),
- pass_through('git show') ],
+ qw(type=s solve! format|f=s dedupe|d=s threads|t remote local!
+ verbose|v+), @c_opt, pass_through('git show') ],
'add-external' => [ 'LOCATION',
'add/set priority of a publicinbox|extindex for extra matches',
- qw(boost=i c=s@ mirror=s no-torsocks torsocks=s inbox-version=i),
- qw(quiet|q verbose|v+ C=s@),
- index_opt(), PublicInbox::LeiQuery::curl_opt() ],
+ qw(boost=i mirror=s no-torsocks torsocks=s inbox-version=i
+ verbose|v+), @c_opt, index_opt(),
+ PublicInbox::LeiQuery::curl_opt() ],
'ls-external' => [ '[FILTER]', 'list publicinbox|extindex locations',
- qw(format|f=s z|0 globoff|g invert-match|v local remote C=s@) ],
+ qw(format|f=s z|0 globoff|g invert-match|v local remote), @c_opt ],
'forget-external' => [ 'LOCATION...|--prune',
'exclude further results from a publicinbox|extindex',
- qw(prune quiet|q C=s@) ],
+ qw(prune), @c_opt ],
'ls-query' => [ '[FILTER...]', 'list saved search queries',
- qw(name-only format|f=s C=s@) ],
-'rm-query' => [ 'QUERY_NAME', 'remove a saved search', qw(C=s@) ],
-'mv-query' => [ qw(OLD_NAME NEW_NAME), 'rename a saved search', qw(C=s@) ],
+ qw(name-only format|f=s), @c_opt ],
+'rm-query' => [ 'QUERY_NAME', 'remove a saved search', @c_opt ],
+'mv-query' => [ qw(OLD_NAME NEW_NAME), 'rename a saved search', @c_opt ],
'plonk' => [ '--threads|--from=IDENT',
'exclude mail matching From: or threads from non-Message-ID searches',
- qw(stdin| threads|t from|f=s mid=s oid=s C=s@) ],
+ qw(stdin| threads|t from|f=s mid=s oid=s), @c_opt ],
'mark' => [ 'MESSAGE_FLAGS...',
'set/unset keywords on message(s) from stdin',
- qw(stdin| oid=s exact by-mid|mid:s C=s@) ],
+ qw(stdin| oid=s exact by-mid|mid:s), @c_opt ],
'forget' => [ '[--stdin|--oid=OID|--by-mid=MID]',
"exclude message(s) on stdin from `q' search results",
- qw(stdin| oid=s exact by-mid|mid:s quiet|q C=s@) ],
+ qw(stdin| oid=s exact by-mid|mid:s), @c_opt ],
'purge-mailsource' => [ 'LOCATION|--all',
'remove imported messages from IMAP, Maildirs, and MH',
- qw(exact! all jobs:i indexed C=s@) ],
+ qw(exact! all jobs:i indexed), @c_opt ],
# code repos are used for `show' to solve blobs from patch mails
'add-coderepo' => [ 'DIRNAME', 'add or set priority of a git code repo',
- qw(boost=i C=s@) ],
+ qw(boost=i), @c_opt ],
'ls-coderepo' => [ '[FILTER_TERMS...]',
- 'list known code repos', qw(format|f=s z C=s@) ],
+ 'list known code repos', qw(format|f=s z), @c_opt ],
'forget-coderepo' => [ 'DIRNAME',
'stop using repo to solve blobs from patches',
- qw(prune C=s@) ],
+ qw(prune), @c_opt ],
'add-watch' => [ 'LOCATION', 'watch for new messages and flag changes',
qw(import! kw|keywords|flags! interval=s recursive|r
- exclude=s include=s C=s@) ],
+ exclude=s include=s), @c_opt ],
'ls-watch' => [ '[FILTER...]', 'list active watches with numbers and status',
- qw(format|f=s z C=s@) ],
-'pause-watch' => [ '[WATCH_NUMBER_OR_FILTER]', qw(all local remote C=s@) ],
-'resume-watch' => [ '[WATCH_NUMBER_OR_FILTER]', qw(all local remote C=s@) ],
+ qw(format|f=s z), @c_opt ],
+'pause-watch' => [ '[WATCH_NUMBER_OR_FILTER]', qw(all local remote), @c_opt ],
+'resume-watch' => [ '[WATCH_NUMBER_OR_FILTER]', qw(all local remote), @c_opt ],
'forget-watch' => [ '{WATCH_NUMBER|--prune}', 'stop and forget a watch',
- qw(prune C=s@) ],
+ qw(prune), @c_opt ],
'import' => [ 'LOCATION...|--stdin',
'one-time import/update from URL or filesystem',
qw(stdin| offset=i recursive|r exclude=s include|I=s
- lock=s@ in-format|F=s kw|keywords|flags! C=s@),
- ],
+ lock=s@ in-format|F=s kw|keywords|flags! verbose|v+), @c_opt ],
'convert' => [ 'LOCATION...|--stdin',
'one-time conversion from URL or filesystem to another format',
- qw(stdin| in-format|F=s out-format|f=s output|mfolder|o=s quiet|q
- lock=s@ kw|keywords|flags! C=s@),
- ],
+ qw(stdin| in-format|F=s out-format|f=s output|mfolder|o=s
+ lock=s@ kw|keywords|flags!), @c_opt ],
'p2q' => [ 'FILE|COMMIT_OID|--stdin',
"use a patch to generate a query for `lei q --stdin'",
- qw(stdin| want|w=s@ uri debug) ],
+ qw(stdin| want|w=s@ uri debug), @c_opt ],
'config' => [ '[...]', sub {
'git-config(1) wrapper for '._config_path($_[0]);
}, qw(config-file|system|global|file|f=s), # for conflict detection
- qw(C=s@), pass_through('git config') ],
+ qw(c=s@ C=s@), pass_through('git config') ],
'init' => [ '[DIRNAME]', sub {
"initialize storage, default: ".store_path($_[0]);
- }, qw(quiet|q C=s@) ],
+ }, @c_opt ],
'daemon-kill' => [ '[-SIGNAL]', 'signal the lei-daemon',
# "-C DIR" conflicts with -CHLD, here, and chdir makes no sense, here
opt_dash('signal|s=s', '[0-9]+|(?:[A-Z][A-Z0-9]+)') ],
@@ -216,7 +216,7 @@ our %CMD = ( # sorted in order of importance/use:
'reorder-local-store-and-break-history' => [ '[REFNAME]',
'rewrite git history in an attempt to improve compression',
- qw(gc! C=s@) ],
+ qw(gc!), @c_opt ],
# internal commands are prefixed with '_'
'_complete' => [ '[...]', 'internal shell completion helper',
@@ -235,6 +235,7 @@ my $ls_format = [ 'OUT|plain|json|null', 'listing output format' ];
# we use \x{a0} (non-breaking SP) to avoid wrapping in PublicInbox::LeiHelp
my %OPTDESC = (
'help|h' => 'show this built-in help',
+'c=s@' => [ 'NAME=VALUE', 'set config option' ],
'C=s@' => [ 'DIR', 'chdir to specify to directory' ],
'quiet|q' => 'be quiet',
'lock=s@' => [ 'METHOD|dotlock|fcntl|flock|none',
@@ -472,7 +473,8 @@ sub lei_atfork_child {
unless ($self->{oneshot}) {
close($_) for @io;
}
- } else {
+ } else { # worker, Net::NNTP (Net::Cmd) uses STDERR directly
+ open STDERR, '+>&='.fileno($self->{2}) or warn "open $!";
delete $self->{0};
}
delete @$self{qw(cnv)};
@@ -586,14 +588,47 @@ sub optparse ($$$) {
$err ? fail($self, "usage: lei $cmd $proto\nE: $err") : 1;
}
+sub _tmp_cfg { # for lei -c <name>=<value> ...
+ my ($self) = @_;
+ my $cfg = _lei_cfg($self, 1);
+ require File::Temp;
+ my $ft = File::Temp->new(TEMPLATE => 'lei_cfg-XXXX', TMPDIR => 1);
+ my $tmp = { '-f' => $ft->filename, -tmp => $ft };
+ $ft->autoflush(1);
+ print $ft <<EOM or return fail($self, "$tmp->{-f}: $!");
+[include]
+ path = $cfg->{-f}
+EOM
+ $tmp = $self->{cfg} = bless { %$cfg, %$tmp }, ref($cfg);
+ for (@{$self->{opt}->{c}}) {
+ /\A([^=\.]+\.[^=]+)(?:=(.*))?\z/ or return fail($self, <<EOM);
+`-c $_' is not of the form -c <name>=<value>'
+EOM
+ my $name = $1;
+ my $value = $2 // 1;
+ _config($self, '--add', $name, $value);
+ if (defined(my $v = $tmp->{$name})) {
+ if (ref($v) eq 'ARRAY') {
+ push @$v, $value;
+ } else {
+ $tmp->{$name} = [ $v, $value ];
+ }
+ } else {
+ $tmp->{$name} = $value;
+ }
+ }
+}
+
sub dispatch {
my ($self, $cmd, @argv) = @_;
local $current_lei = $self; # for __WARN__
dump_and_clear_log("from previous run\n");
return _help($self, 'no command given') unless defined($cmd);
- while ($cmd eq '-C') { # do not support Getopt bundling for this
- my $d = shift(@argv) // return fail($self, '-C DIRECTORY');
- push @{$self->{opt}->{C}}, $d;
+ # do not support Getopt bundling for this
+ while ($cmd eq '-C' || $cmd eq '-c') {
+ my $v = shift(@argv) // return fail($self, $cmd eq '-C' ?
+ '-C DIRECTORY' : '-c <name>=<value>');
+ push @{$self->{opt}->{substr($cmd, 1, 1)}}, $v;
$cmd = shift(@argv) // return _help($self, 'no command given');
}
my $func = "lei_$cmd";
@@ -605,6 +640,7 @@ sub dispatch {
} : undef);
if ($cb) {
optparse($self, $cmd, \@argv) or return;
+ $self->{opt}->{c} and (_tmp_cfg($self) // return);
if (my $chdir = $self->{opt}->{C}) {
for my $d (@$chdir) {
next if $d eq ''; # same as git(1)
@@ -623,6 +659,7 @@ sub dispatch {
sub _lei_cfg ($;$) {
my ($self, $creat) = @_;
+ return $self->{cfg} if $self->{cfg};
my $f = _config_path($self);
my @st = stat($f);
my $cur_st = @st ? pack('dd', $st[10], $st[7]) : ''; # 10:ctime, 7:size
diff --git a/lib/PublicInbox/LeiExternal.pm b/lib/PublicInbox/LeiExternal.pm
index f4e24c2a..9a555831 100644
--- a/lib/PublicInbox/LeiExternal.pm
+++ b/lib/PublicInbox/LeiExternal.pm
@@ -149,7 +149,7 @@ sub lei_add_external {
my $mirror = $opt->{mirror} // do {
my @fail;
for my $sw ($self->index_opt, $self->curl_opt,
- qw(c no-torsocks torsocks inbox-version)) {
+ qw(no-torsocks torsocks inbox-version)) {
my ($f) = (split(/|/, $sw, 2))[0];
next unless defined $opt->{$f};
$f = length($f) == 1 ? "-$f" : "--$f";
diff --git a/t/lei.t b/t/lei.t
index 2bf4b862..0cf97866 100644
--- a/t/lei.t
+++ b/t/lei.t
@@ -93,6 +93,15 @@ my $test_config = sub {
'config set var with -f fails');
like($lei_err, qr/not supported/, 'not supported noted');
ok(!-f "$home/config/f", 'no file created');
+
+ lei_ok(qw(-c imap.debug config --bool imap.debug));
+ is($lei_out, "true\n", "-c sets w/o value");
+ lei_ok(qw(-c imap.debug=1 config --bool imap.debug));
+ is($lei_out, "true\n", "-c coerces value");
+ lei_ok(qw(-c imap.debug=tr00 config imap.debug));
+ is($lei_out, "tr00\n", "-c string value passed as-is");
+ lei_ok(qw(-c imap.debug=a -c imap.debug=b config --get-all imap.debug));
+ is($lei_out, "a\nb\n", '-c and --get-all work together');
};
my $test_completion = sub {
^ permalink raw reply related [relevance 33%]
* [PATCH 0/8] lei input handling improvements
@ 2021-03-22 7:53 70% Eric Wong
2021-03-22 7:53 33% ` [PATCH 1/8] lei: support -c <name>=<value> to overrides Eric Wong
` (3 more replies)
0 siblings, 4 replies; 200+ results
From: Eric Wong @ 2021-03-22 7:53 UTC (permalink / raw)
To: meta
lei <convert|import> share a bit more code, now; and being
able to set "-c imap.debug" on the command-line should make
future work easier.
All this should set us up nicely for implementing "lei mark"
to add/remove keywords and labels.
Eric Wong (8):
lei: support -c <name>=<value> to overrides
net_reader: escape nasty chars from Net::NNTP->message
lei: share input code between convert and import
lei: simplify workers_start and callers
mbox_reader: add ->reads method to avoid nonsensical formats
lei_input: common filehandle reader for eml + mbox
lei_input: drop "From " line on single "eml" (message/rfc822)
lei import: ignore Status headers in "eml" messages
MANIFEST | 1 +
lib/PublicInbox/InboxWritable.pm | 2 +-
lib/PublicInbox/LEI.pm | 137 ++++++++++++++++++-------------
lib/PublicInbox/LeiConvert.pm | 94 ++++-----------------
lib/PublicInbox/LeiExternal.pm | 2 +-
lib/PublicInbox/LeiImport.pm | 107 +++++-------------------
lib/PublicInbox/LeiInput.pm | 106 ++++++++++++++++++++++++
lib/PublicInbox/LeiP2q.pm | 4 +-
lib/PublicInbox/MboxReader.pm | 5 ++
lib/PublicInbox/NetReader.pm | 10 ++-
t/lei-import.t | 37 +++++++--
t/lei.t | 9 ++
12 files changed, 278 insertions(+), 236 deletions(-)
create mode 100644 lib/PublicInbox/LeiInput.pm
^ permalink raw reply [relevance 70%]
* [PATCH] lei: simplify lazy-loading
@ 2021-03-21 11:24 59% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-21 11:24 UTC (permalink / raw)
To: meta
This makes it slightly easier to implement future commands,
since there'll be a couple more relatively self-contained
ones.
---
lib/PublicInbox/LEI.pm | 22 ++++++----------------
lib/PublicInbox/LeiConvert.pm | 6 +++---
lib/PublicInbox/LeiImport.pm | 6 +++---
lib/PublicInbox/LeiP2q.pm | 6 +++---
4 files changed, 15 insertions(+), 25 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index bf97a680..b6d21af6 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -598,7 +598,12 @@ sub dispatch {
}
my $func = "lei_$cmd";
$func =~ tr/-/_/;
- if (my $cb = __PACKAGE__->can($func)) {
+ my $cb = __PACKAGE__->can($func) // ($CMD{$cmd} ? do {
+ my $mod = "PublicInbox::Lei\u$cmd";
+ ($INC{"PublicInbox/Lei\u$cmd.pm"} //
+ eval("require $mod")) ? $mod->can($func) : undef;
+ } : undef);
+ if ($cb) {
optparse($self, $cmd, \@argv) or return;
if (my $chdir = $self->{opt}->{C}) {
for my $d (@$chdir) {
@@ -685,21 +690,6 @@ sub lei_config {
x_it($self, $?) if $?;
}
-sub lei_import {
- require PublicInbox::LeiImport;
- PublicInbox::LeiImport->call(@_);
-}
-
-sub lei_convert {
- require PublicInbox::LeiConvert;
- PublicInbox::LeiConvert->call(@_);
-}
-
-sub lei_p2q {
- require PublicInbox::LeiP2q;
- PublicInbox::LeiP2q->call(@_);
-}
-
sub lei_init {
my ($self, $dir) = @_;
my $cfg = _lei_cfg($self, 1);
diff --git a/lib/PublicInbox/LeiConvert.pm b/lib/PublicInbox/LeiConvert.pm
index fcc67f0b..8d3b221a 100644
--- a/lib/PublicInbox/LeiConvert.pm
+++ b/lib/PublicInbox/LeiConvert.pm
@@ -77,11 +77,11 @@ sub do_convert { # via wq_do
delete $self->{wcb}; # commit
}
-sub call { # the main "lei convert" method
- my ($cls, $lei, @inputs) = @_;
+sub lei_convert { # the main "lei convert" method
+ my ($lei, @inputs) = @_;
my $opt = $lei->{opt};
$opt->{kw} //= 1;
- my $self = $lei->{cnv} = bless {}, $cls;
+ my $self = $lei->{cnv} = bless {}, __PACKAGE__;
my $in_fmt = $opt->{'in-format'};
my (@f, @d);
$opt->{dedupe} //= 'none';
diff --git a/lib/PublicInbox/LeiImport.pm b/lib/PublicInbox/LeiImport.pm
index ae24a1fa..0e2a96e8 100644
--- a/lib/PublicInbox/LeiImport.pm
+++ b/lib/PublicInbox/LeiImport.pm
@@ -63,13 +63,13 @@ sub import_start {
while ($op && $op->{sock}) { $op->event_step }
}
-sub call { # the main "lei import" method
- my ($cls, $lei, @inputs) = @_;
+sub lei_import { # the main "lei import" method
+ my ($lei, @inputs) = @_;
my $sto = $lei->_lei_store(1);
$sto->write_prepare($lei);
my ($net, @f, @d);
$lei->{opt}->{kw} //= 1;
- my $self = $lei->{imp} = bless { inputs => \@inputs }, $cls;
+ my $self = $lei->{imp} = bless { inputs => \@inputs }, __PACKAGE__;
if ($lei->{opt}->{stdin}) {
@inputs and return $lei->fail("--stdin and @inputs do not mix");
$lei->check_input_format or return;
diff --git a/lib/PublicInbox/LeiP2q.pm b/lib/PublicInbox/LeiP2q.pm
index c5718603..302d7864 100644
--- a/lib/PublicInbox/LeiP2q.pm
+++ b/lib/PublicInbox/LeiP2q.pm
@@ -174,9 +174,9 @@ sub do_p2q { # via wq_do
$lei->out(@q, "\n");
}
-sub call { # the "lei patch-to-query" entry point
- my ($cls, $lei, $input) = @_;
- my $self = $lei->{p2q} = bless {}, $cls;
+sub lei_p2q { # the "lei patch-to-query" entry point
+ my ($lei, $input) = @_;
+ my $self = $lei->{p2q} = bless {}, __PACKAGE__;
if ($lei->{opt}->{stdin}) {
$self->{0} = delete $lei->{0}; # guard from lei_atfork_child
} else {
^ permalink raw reply related [relevance 59%]
* [PATCH 0/3] lei import fix, other fixes
@ 2021-03-21 9:50 71% Eric Wong
2021-03-21 9:50 36% ` [PATCH 1/3] lei import: vivify external-only messages Eric Wong
` (2 more replies)
0 siblings, 3 replies; 200+ results
From: Eric Wong @ 2021-03-21 9:50 UTC (permalink / raw)
To: meta
Things are finally coming together...
Eric Wong (3):
lei import: vivify external-only messages
lei q: fix warning on remote imports
lei: fix some warnings in tests
lib/PublicInbox/ContentHash.pm | 15 ++++++++---
lib/PublicInbox/Import.pm | 14 ++++++++++-
lib/PublicInbox/LEI.pm | 8 +++---
lib/PublicInbox/LeiDedupe.pm | 9 ++-----
lib/PublicInbox/LeiExternal.pm | 2 +-
lib/PublicInbox/LeiImport.pm | 22 ++++++++++------
lib/PublicInbox/LeiP2q.pm | 2 +-
lib/PublicInbox/LeiSearch.pm | 5 +++-
lib/PublicInbox/LeiStore.pm | 46 +++++++++++++++++++++++++++++-----
lib/PublicInbox/LeiXSearch.pm | 6 ++++-
lib/PublicInbox/MboxLock.pm | 8 +++---
lib/PublicInbox/Over.pm | 2 +-
lib/PublicInbox/SearchIdx.pm | 12 +++++++--
lib/PublicInbox/TestCommon.pm | 9 +++++++
t/lei-q-kw.t | 44 ++++++++++++++++++++++++++++++++
t/lei-q-remote-import.t | 3 ++-
16 files changed, 167 insertions(+), 40 deletions(-)
^ permalink raw reply [relevance 71%]
* [PATCH 2/3] lei q: fix warning on remote imports
2021-03-21 9:50 71% [PATCH 0/3] lei import fix, other fixes Eric Wong
2021-03-21 9:50 36% ` [PATCH 1/3] lei import: vivify external-only messages Eric Wong
@ 2021-03-21 9:50 57% ` Eric Wong
2021-03-21 9:50 52% ` [PATCH 3/3] lei: fix some warnings in tests Eric Wong
2 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-21 9:50 UTC (permalink / raw)
To: meta
This will let us tie keywords from remote externals
to those which only exist in local externals.
---
lib/PublicInbox/ContentHash.pm | 15 ++++++++++++---
lib/PublicInbox/LeiDedupe.pm | 9 ++-------
lib/PublicInbox/LeiXSearch.pm | 6 +++++-
t/lei-q-remote-import.t | 3 ++-
4 files changed, 21 insertions(+), 12 deletions(-)
diff --git a/lib/PublicInbox/ContentHash.pm b/lib/PublicInbox/ContentHash.pm
index 4dbe7b50..112b1ea6 100644
--- a/lib/PublicInbox/ContentHash.pm
+++ b/lib/PublicInbox/ContentHash.pm
@@ -8,9 +8,9 @@
# See L<public-inbox-v2-format(5)> manpage for more details.
package PublicInbox::ContentHash;
use strict;
-use warnings;
-use base qw/Exporter/;
-our @EXPORT_OK = qw/content_hash content_digest/;
+use v5.10.1;
+use parent qw(Exporter);
+our @EXPORT_OK = qw(content_hash content_digest git_sha);
use PublicInbox::MID qw(mids references);
use PublicInbox::MsgIter;
@@ -94,4 +94,13 @@ sub content_hash ($) {
content_digest($_[0])->digest;
}
+sub git_sha ($$) {
+ my ($n, $eml) = @_;
+ my $dig = Digest::SHA->new($n);
+ my $buf = $eml->as_string;
+ $dig->add('blob '.length($buf)."\0");
+ $dig->add($buf);
+ $dig;
+}
+
1;
diff --git a/lib/PublicInbox/LeiDedupe.pm b/lib/PublicInbox/LeiDedupe.pm
index 5fec9384..a62b3a7c 100644
--- a/lib/PublicInbox/LeiDedupe.pm
+++ b/lib/PublicInbox/LeiDedupe.pm
@@ -3,7 +3,7 @@
package PublicInbox::LeiDedupe;
use strict;
use v5.10.1;
-use PublicInbox::ContentHash qw(content_hash);
+use PublicInbox::ContentHash qw(content_hash git_sha);
use Digest::SHA ();
# n.b. mutt sets most of these headers not sure about Bytes
@@ -18,12 +18,7 @@ sub _regen_oid ($) {
push @stash, [ $k, \@v ];
$eml->header_set($k); # restore below
}
- my $dig = Digest::SHA->new(1); # XXX SHA256 later
- my $buf = $eml->as_string;
- $dig->add('blob '.length($buf)."\0");
- $dig->add($buf);
- undef $buf;
-
+ my $dig = git_sha(1, $eml);
for my $kv (@stash) { # restore stashed headers
my ($k, @v) = @$kv;
$eml->header_set($k, @v);
diff --git a/lib/PublicInbox/LeiXSearch.pm b/lib/PublicInbox/LeiXSearch.pm
index 17171a7f..b6aaf3e1 100644
--- a/lib/PublicInbox/LeiXSearch.pm
+++ b/lib/PublicInbox/LeiXSearch.pm
@@ -18,6 +18,7 @@ use PublicInbox::MID qw(mids);
use PublicInbox::Smsg;
use PublicInbox::Eml;
use Fcntl qw(SEEK_SET F_SETFL O_APPEND O_RDWR);
+use PublicInbox::ContentHash qw(git_sha);
sub new {
my ($class) = @_;
@@ -207,10 +208,13 @@ sub query_mset { # non-parallel for non-"--threads" users
sub each_remote_eml { # callback for MboxReader->mboxrd
my ($eml, $self, $lei, $each_smsg) = @_;
- if ($self->{import_sto} && !$lei->{ale}->xoids_for($eml, 1)) {
+ my $xoids = $lei->{ale}->xoids_for($eml, 1);
+ if ($self->{import_sto} && !$xoids) {
$self->{import_sto}->ipc_do('add_eml', $eml);
}
my $smsg = bless {}, 'PublicInbox::Smsg';
+ $smsg->{blob} = $xoids ? (keys(%$xoids))[0]
+ : git_sha(1, $eml)->hexdigest;
$smsg->populate($eml);
$smsg->parse_references($eml, mids($eml));
$smsg->{$_} //= '' for qw(from to cc ds subject references mid);
diff --git a/t/lei-q-remote-import.t b/t/lei-q-remote-import.t
index 25e461ac..93828a24 100644
--- a/t/lei-q-remote-import.t
+++ b/t/lei-q-remote-import.t
@@ -65,8 +65,9 @@ test_lei({ tmpdir => $tmpdir }, sub {
$im->add(eml_load('t/utf8.eml')) or BAIL_OUT '->add';
};
lei_ok(qw(add-external -q), $ibx->{inboxdir});
- lei_ok(qw(q -o), "mboxrd:$o", '--only', $url,
+ lei_ok(qw(q -q -o), "mboxrd:$o", '--only', $url,
'm:testmessage@example.com');
+ is($lei_err, '', 'no warnings or errors');
ok(-s $o, 'got result from remote external');
my $exp = eml_load('t/utf8.eml');
is_deeply($slurp_emls->($o), [$exp], 'got expected result');
^ permalink raw reply related [relevance 57%]
* [PATCH 3/3] lei: fix some warnings in tests
2021-03-21 9:50 71% [PATCH 0/3] lei import fix, other fixes Eric Wong
2021-03-21 9:50 36% ` [PATCH 1/3] lei import: vivify external-only messages Eric Wong
2021-03-21 9:50 57% ` [PATCH 2/3] lei q: fix warning on remote imports Eric Wong
@ 2021-03-21 9:50 52% ` Eric Wong
2 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-21 9:50 UTC (permalink / raw)
To: meta
And then test the contents of $lei_err to ensure it doesn't
happen again.
We'll also make MboxLock emit nicer warnings without the line
number, since the line number is irrelevant to the user fixing
an mbox lock contention problem.
Finally, we'll also allow showing loud warnings via
TEST_LEI_ERR_LOUD=1
---
lib/PublicInbox/LEI.pm | 8 ++++----
lib/PublicInbox/LeiExternal.pm | 2 +-
lib/PublicInbox/LeiP2q.pm | 2 +-
lib/PublicInbox/MboxLock.pm | 8 ++++----
lib/PublicInbox/TestCommon.pm | 9 +++++++++
5 files changed, 19 insertions(+), 10 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 72a0e52c..bf97a680 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -757,7 +757,7 @@ sub lei__complete {
my ($proto, undef, @spec) = @$info;
my $cur = pop @argv;
my $re = defined($cur) ? qr/\A\Q$cur\E/ : qr/./;
- if (substr($cur // '-', 0, 1) eq '-') { # --switches
+ if (substr(my $_cur = $cur // '-', 0, 1) eq '-') { # --switches
# gross special case since the only git-config options
# Consider moving to a table if we need more special cases
# we use Getopt::Long for are the ones we reject, so these
@@ -781,7 +781,7 @@ sub lei__complete {
}
map {
my $x = length > 1 ? "--$_" : "-$_";
- $x eq $cur ? () : $x;
+ $x eq $_cur ? () : $x;
} grep(!/_/, split(/\|/, $_, -1)) # help|h
} grep { $OPTDESC{"$_\t$cmd"} || $OPTDESC{$_} } @spec);
} elsif ($cmd eq 'config' && !@argv && !$CONFIG_KEYS{$cur}) {
@@ -796,13 +796,13 @@ sub lei__complete {
my @v = ref($v) ? split(/\|/, $v->[0]) : ();
# get rid of ALL CAPS placeholder (e.g "OUT")
# (TODO: completion for external paths)
- shift(@v) if uc($v[0]) eq $v[0];
+ shift(@v) if scalar(@v) && uc($v[0]) eq $v[0];
@v;
} grep(/\A(?:[\w-]+\|)*$opt\b.*?(?:\t$cmd)?\z/, keys %OPTDESC);
}
$cmd =~ tr/-/_/;
if (my $sub = $self->can("_complete_$cmd")) {
- puts $self, $sub->($self, @argv, $cur);
+ puts $self, $sub->($self, @argv, $cur ? ($cur) : ());
}
# TODO: URLs, pathnames, OIDs, MIDs, etc... See optparse() for
# proto parsing.
diff --git a/lib/PublicInbox/LeiExternal.pm b/lib/PublicInbox/LeiExternal.pm
index b5dd85e1..f4e24c2a 100644
--- a/lib/PublicInbox/LeiExternal.pm
+++ b/lib/PublicInbox/LeiExternal.pm
@@ -222,7 +222,7 @@ sub _complete_url_common ($) {
# Maybe there's a better way to go about this in
# contrib/completion/lei-completion.bash
my $re = '';
- my $cur = pop @$argv;
+ my $cur = pop(@$argv) // '';
if (@$argv) {
my @x = @$argv;
if ($cur eq ':' && @x) {
diff --git a/lib/PublicInbox/LeiP2q.pm b/lib/PublicInbox/LeiP2q.pm
index e7ddc852..c5718603 100644
--- a/lib/PublicInbox/LeiP2q.pm
+++ b/lib/PublicInbox/LeiP2q.pm
@@ -144,7 +144,7 @@ sub do_p2q { # via wq_do
my $end = ($pfx =~ s/([0-9\*]+)\z//) ? $1 : '';
my $x = delete($lei->{qterms}->{$pfx}) or next;
my $star = $end =~ tr/*//d ? '*' : '';
- my $min_len = ($end // 0) + 0;
+ my $min_len = ($end || 0) + 0;
# no wildcards for bool_pfx_external
$star = '' if $pfx =~ /\A(dfpre|dfpost|mid)\z/;
diff --git a/lib/PublicInbox/MboxLock.pm b/lib/PublicInbox/MboxLock.pm
index 4e2a2d9a..bea0e325 100644
--- a/lib/PublicInbox/MboxLock.pm
+++ b/lib/PublicInbox/MboxLock.pm
@@ -43,13 +43,13 @@ EOF
}
select(undef, undef, undef, $self->{delay});
} while (now < $end);
- croak "fcntl lock $self->{f}: $!";
+ die "fcntl lock timeout $self->{f}: $!\n";
}
sub acq_dotlock {
my ($self) = @_;
my $dot_lock = "$self->{f}.lock";
- my ($pfx, $base) = ($self->{f} =~ m!(\A.*?/)([^/]+)\z!);
+ my ($pfx, $base) = ($self->{f} =~ m!(\A.*?/)?([^/]+)\z!);
$pfx //= '';
my $pid = $$;
my $end = now + $self->{timeout};
@@ -68,7 +68,7 @@ sub acq_dotlock {
croak "open $tmp (for $dot_lock): $!" if !$!{EXIST};
}
} while (now < $end);
- croak "dotlock $dot_lock";
+ die "dotlock timeout $dot_lock\n";
}
sub acq_flock {
@@ -80,7 +80,7 @@ sub acq_flock {
return if flock($self->{fh}, $op);
select(undef, undef, undef, $self->{delay});
} while (now < $end);
- croak "flock $self->{f}: $!";
+ die "flock timeout $self->{f}: $!\n";
}
sub acq {
diff --git a/lib/PublicInbox/TestCommon.pm b/lib/PublicInbox/TestCommon.pm
index 0d15514e..e67e94ea 100644
--- a/lib/PublicInbox/TestCommon.pm
+++ b/lib/PublicInbox/TestCommon.pm
@@ -457,6 +457,15 @@ sub lei (@) {
my $res = run_script(['lei', @$cmd], $env, $xopt // $lei_opt);
$err_skip and
$lei_err = join('', grep(!/$err_skip/, split(/^/m, $lei_err)));
+ if ($lei_err ne '') {
+ if ($lei_err =~ /Use of uninitialized/ ||
+ $lei_err =~ m!\bArgument .*? isn't numeric in !) {
+ fail "lei_err=$lei_err";
+ } else {
+ state $loud = $ENV{TEST_LEI_ERR_LOUD};
+ diag "lei_err=$lei_err" if $loud;
+ }
+ }
$res;
};
^ permalink raw reply related [relevance 52%]
* [PATCH 1/3] lei import: vivify external-only messages
2021-03-21 9:50 71% [PATCH 0/3] lei import fix, other fixes Eric Wong
@ 2021-03-21 9:50 36% ` Eric Wong
2021-03-21 9:50 57% ` [PATCH 2/3] lei q: fix warning on remote imports Eric Wong
2021-03-21 9:50 52% ` [PATCH 3/3] lei: fix some warnings in tests Eric Wong
2 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-21 9:50 UTC (permalink / raw)
To: meta
Keyword storage for external-only messages was preventing
messages from being explicitly imported. Teach lei_store
to vivify keyword-only entries into fully-indexed messages
on import.
---
lib/PublicInbox/Import.pm | 14 ++++++++++-
lib/PublicInbox/LeiImport.pm | 22 +++++++++++------
lib/PublicInbox/LeiSearch.pm | 5 +++-
lib/PublicInbox/LeiStore.pm | 46 +++++++++++++++++++++++++++++++-----
lib/PublicInbox/Over.pm | 2 +-
lib/PublicInbox/SearchIdx.pm | 12 ++++++++--
t/lei-q-kw.t | 44 ++++++++++++++++++++++++++++++++++
7 files changed, 127 insertions(+), 18 deletions(-)
diff --git a/lib/PublicInbox/Import.pm b/lib/PublicInbox/Import.pm
index b8fa5c21..34738279 100644
--- a/lib/PublicInbox/Import.pm
+++ b/lib/PublicInbox/Import.pm
@@ -413,7 +413,19 @@ sub add {
$smsg->{blob} = $self->get_mark(":$blob");
$smsg->set_bytes($raw_email, $n);
if (my $oidx = delete $smsg->{-oidx}) { # used by LeiStore
- return if $oidx->blob_exists($smsg->{blob});
+ my @docids = $oidx->blob_exists($smsg->{blob});
+ my @vivify_xvmd;
+ for my $id (@docids) {
+ if (my $cur = $oidx->get_art($id)) {
+ # already imported if bytes > 0
+ return if $cur->{bytes} > 0;
+ push @vivify_xvmd, $id;
+ } else {
+ warn "W: $smsg->{blob} ",
+ "#$id gone (bug?)\n";
+ }
+ }
+ $smsg->{-vivify_xvmd} = \@vivify_xvmd;
}
}
my $ref = $self->{ref};
diff --git a/lib/PublicInbox/LeiImport.pm b/lib/PublicInbox/LeiImport.pm
index 137c22fc..ae24a1fa 100644
--- a/lib/PublicInbox/LeiImport.pm
+++ b/lib/PublicInbox/LeiImport.pm
@@ -10,9 +10,14 @@ use PublicInbox::Eml;
use PublicInbox::PktOp qw(pkt_do);
sub _import_eml { # MboxReader callback
- my ($eml, $sto, $set_kw) = @_;
- $sto->ipc_do('set_eml', $eml, $set_kw ?
- { kw => PublicInbox::MboxReader::mbox_keywords($eml) } : ());
+ my ($eml, $lei, $mbox_keywords) = @_;
+ my $vmd;
+ if ($mbox_keywords) {
+ my $kw = $mbox_keywords->($eml);
+ $vmd = { kw => $kw } if scalar(@$kw);
+ }
+ my $xoids = $lei->{ale}->xoids_for($eml);
+ $lei->{sto}->ipc_do('set_eml', $eml, $vmd, $xoids);
}
sub import_done_wait { # dwaitpid callback
@@ -41,6 +46,7 @@ sub net_merge_complete { # callback used by LeiAuth
sub import_start {
my ($lei) = @_;
my $self = $lei->{imp};
+ $lei->ale;
my $j = $lei->{opt}->{jobs} // scalar(@{$self->{inputs}}) || 1;
if (my $net = $lei->{net}) {
# $j = $net->net_concurrency($j); TODO
@@ -130,7 +136,8 @@ sub ipc_atfork_child {
sub _import_fh {
my ($lei, $fh, $input, $ifmt) = @_;
- my $set_kw = $lei->{opt}->{kw};
+ my $kw = $lei->{opt}->{kw} ?
+ PublicInbox::MboxReader->can('mbox_keywords') : undef;
eval {
if ($ifmt eq 'eml') {
my $buf = do { local $/; <$fh> } //
@@ -138,11 +145,11 @@ sub _import_fh {
error reading $input: $!
my $eml = PublicInbox::Eml->new(\$buf);
- _import_eml($eml, $lei->{sto}, $set_kw);
+ _import_eml($eml, $lei, $kw);
} else { # some mbox (->can already checked in call);
my $cb = PublicInbox::MboxReader->can($ifmt) //
die "BUG: bad fmt=$ifmt";
- $cb->(undef, $fh, \&_import_eml, $lei->{sto}, $set_kw);
+ $cb->(undef, $fh, \&_import_eml, $lei, $kw);
}
};
$lei->child_error(1 << 8, "$input: $@") if $@;
@@ -193,7 +200,8 @@ EOM
sub import_stdin {
my ($self) = @_;
my $lei = $self->{lei};
- _import_fh($lei, delete $self->{0}, '<stdin>', $lei->{opt}->{'in-format'});
+ my $in = delete $self->{0};
+ _import_fh($lei, $in, '<stdin>', $lei->{opt}->{'in-format'});
}
no warnings 'once'; # the following works even when LeiAuth is lazy-loaded
diff --git a/lib/PublicInbox/LeiSearch.pm b/lib/PublicInbox/LeiSearch.pm
index 360a37e5..bbb00661 100644
--- a/lib/PublicInbox/LeiSearch.pm
+++ b/lib/PublicInbox/LeiSearch.pm
@@ -63,7 +63,10 @@ sub _cmp_1st { # git->cat_async callback
}
}
-sub xoids_for { # returns { OID => docid } mapping for $eml matches
+# returns { OID => num } mapping for $eml matches
+# The `num' hash value only makes sense from LeiSearch itself
+# and is nonsense from the PublicInbox::LeiALE subclass
+sub xoids_for {
my ($self, $eml, $min) = @_;
my ($chash, $mids) = content_key($eml);
my @overs = ($self->over // $self->overs_all);
diff --git a/lib/PublicInbox/LeiStore.pm b/lib/PublicInbox/LeiStore.pm
index c66d3dc2..b390b318 100644
--- a/lib/PublicInbox/LeiStore.pm
+++ b/lib/PublicInbox/LeiStore.pm
@@ -161,7 +161,7 @@ sub remove_eml_vmd {
}
sub add_eml {
- my ($self, $eml, $vmd) = @_;
+ my ($self, $eml, $vmd, $xoids) = @_;
my $im = $self->importer; # may create new epoch
my $eidx = eidx_init($self); # writes ALL.git/objects/info/alternates
my $oidx = $eidx->{oidx}; # PublicInbox::Import::add checks this
@@ -169,7 +169,40 @@ sub add_eml {
$im->add($eml, undef, $smsg) or return; # duplicate returns undef
local $self->{current_info} = $smsg->{blob};
- if (my @docids = _docids_for($self, $eml)) {
+ my $vivify_xvmd = delete($smsg->{-vivify_xvmd}) // []; # exact matches
+ if ($xoids) { # fuzzy matches from externals in ale->xoids_for
+ delete $xoids->{$smsg->{blob}}; # added later
+ if (scalar keys %$xoids) {
+ my %docids = map { $_ => 1 } @$vivify_xvmd;
+ for my $oid (keys %$xoids) {
+ my @id = $oidx->blob_exists($oid);
+ @docids{@id} = @id;
+ }
+ @$vivify_xvmd = sort { $a <=> $b } keys(%docids);
+ }
+ }
+ if (@$vivify_xvmd) {
+ $xoids //= {};
+ $xoids->{$smsg->{blob}} = 1;
+ for my $docid (@$vivify_xvmd) {
+ my $cur = $oidx->get_art($docid);
+ my $idx = $eidx->idx_shard($docid);
+ if (!$cur || $cur->{bytes} == 0) { # really vivifying
+ $smsg->{num} = $docid;
+ $oidx->add_overview($eml, $smsg);
+ $smsg->{-merge_vmd} = 1;
+ $idx->index_eml($eml, $smsg);
+ } else { # lse fuzzy hit off ale
+ $idx->ipc_do('add_eidx_info', $docid, '.', $eml);
+ }
+ for my $oid (keys %$xoids) {
+ $oidx->add_xref3($docid, -1, $oid, '.');
+ }
+ $idx->ipc_do('add_vmd', $docid, $vmd) if $vmd;
+ }
+ $vivify_xvmd;
+ } elsif (my @docids = _docids_for($self, $eml)) {
+ # fuzzy match from within lei/store
for my $docid (@docids) {
my $idx = $eidx->idx_shard($docid);
$oidx->add_xref3($docid, -1, $smsg->{blob}, '.');
@@ -178,20 +211,21 @@ sub add_eml {
$idx->ipc_do('add_vmd', $docid, $vmd) if $vmd;
}
\@docids;
- } else {
+ } else { # totally new message
$smsg->{num} = $oidx->adj_counter('eidx_docid', '+');
$oidx->add_overview($eml, $smsg);
$oidx->add_xref3($smsg->{num}, -1, $smsg->{blob}, '.');
my $idx = $eidx->idx_shard($smsg->{num});
$idx->index_eml($eml, $smsg);
- $idx->ipc_do('add_vmd', $smsg->{num}, $vmd ) if $vmd;
+ $idx->ipc_do('add_vmd', $smsg->{num}, $vmd) if $vmd;
$smsg;
}
}
sub set_eml {
- my ($self, $eml, $vmd) = @_;
- add_eml($self, $eml, $vmd) // set_eml_vmd($self, $eml, $vmd);
+ my ($self, $eml, $vmd, $xoids) = @_;
+ add_eml($self, $eml, $vmd, $xoids) //
+ set_eml_vmd($self, $eml, $vmd);
}
# set or update keywords for external message, called via ipc_do
diff --git a/lib/PublicInbox/Over.pm b/lib/PublicInbox/Over.pm
index 587e0516..0e191c47 100644
--- a/lib/PublicInbox/Over.pm
+++ b/lib/PublicInbox/Over.pm
@@ -353,7 +353,7 @@ sub blob_exists {
my ($self, $oidhex) = @_;
if (wantarray) {
my $sth = $self->dbh->prepare_cached(<<'', undef, 1);
-SELECT docid FROM xref3 WHERE oidbin = ?
+SELECT docid FROM xref3 WHERE oidbin = ? ORDER BY docid ASC
$sth->bind_param(1, pack('H*', $oidhex), SQL_BLOB);
$sth->execute;
diff --git a/lib/PublicInbox/SearchIdx.pm b/lib/PublicInbox/SearchIdx.pm
index 3237aadc..3f933121 100644
--- a/lib/PublicInbox/SearchIdx.pm
+++ b/lib/PublicInbox/SearchIdx.pm
@@ -11,6 +11,7 @@ use strict;
use v5.10.1;
use parent qw(PublicInbox::Search PublicInbox::Lock Exporter);
use PublicInbox::Eml;
+use PublicInbox::Search qw(xap_terms);
use PublicInbox::InboxWritable;
use PublicInbox::MID qw(mids_for_index mids);
use PublicInbox::MsgIter;
@@ -34,6 +35,7 @@ use constant DEBUG => !!$ENV{DEBUG};
my $xapianlevels = qr/\A(?:full|medium)\z/;
my $hex = '[a-f0-9]';
my $OID = $hex .'{40,}';
+my @VMD_MAP = (kw => 'K', label => 'L');
our $INDEXLEVELS = qr/\A(?:full|medium|basic)\z/;
sub new {
@@ -428,7 +430,15 @@ sub eml2doc ($$$;$) {
sub add_xapian ($$$$) {
my ($self, $eml, $smsg, $mids) = @_;
begin_txn_lazy($self);
+ my $merge_vmd = delete $smsg->{-merge_vmd};
my $doc = eml2doc($self, $eml, $smsg, $mids);
+ if (my $old = $merge_vmd ? _get_doc($self, $smsg->{num}) : undef) {
+ my @x = @VMD_MAP;
+ while (my ($field, $pfx) = splice(@x, 0, 2)) {
+ my $vals = xap_terms($pfx, $old);
+ $doc->add_boolean_term($pfx.$_) for keys %$vals;
+ }
+ }
$self->{xdb}->replace_document($smsg->{num}, $doc);
}
@@ -531,8 +541,6 @@ sub remove_eidx_info {
$self->{xdb}->replace_document($docid, $doc);
}
-my @VMD_MAP = (kw => 'K', label => 'L');
-
sub set_vmd {
my ($self, $docid, $vmd) = @_;
begin_txn_lazy($self);
diff --git a/t/lei-q-kw.t b/t/lei-q-kw.t
index b5e22e9b..4db27363 100644
--- a/t/lei-q-kw.t
+++ b/t/lei-q-kw.t
@@ -161,5 +161,49 @@ like($s, qr/^Status: O\nX-Status: AF\n/ms,
lei_ok(qw(q --pretty), "m:$m", @inc);
like($lei_out, qr/^ "kw": \["answered", "flagged"\],\n/sm,
'--pretty JSON output shows kw: on one line');
+
+# ensure import on previously external-only message works
+lei_ok('q', "m:$m");
+is_deeply(json_utf8->decode($lei_out), [ undef ],
+ 'to-be-imported message non-existent');
+lei_ok(qw(import -F eml t/x-unknown-alpine.eml));
+is($lei_err, '', 'no errors importing previous external-only message');
+lei_ok('q', "m:$m");
+$res = json_utf8->decode($lei_out);
+is($res->[1], undef, 'got one result');
+is_deeply($res->[0]->{kw}, [ qw(answered flagged) ], 'kw preserved on exact');
+
+# ensure fuzzy match import works, too
+$m = 'multipart@example.com';
+$o = "$ENV{HOME}/fuzz";
+lei_ok('q', '-o', $o, "m:$m", @inc);
+@fn = glob("$o/cur/*");
+scalar(@fn) == 1 or BAIL_OUT "wrote multiple or zero files: ".explain(\@fn);
+rename($fn[0], "$fn[0]S") or BAIL_OUT "rename $!";
+lei_ok('q', '-o', $o, "m:$m");
+is_deeply([glob("$o/cur/*")], [], 'clobbered output results');
+my $eml = eml_load('t/plack-2-txt-bodies.eml');
+$eml->header_set('List-Id', '<list.example.com>');
+my $in = $eml->as_string;
+lei_ok([qw(import -F eml --stdin)], undef, { 0 => \$in, %$lei_opt });
+is($lei_err, '', 'no errors from import');
+lei_ok(qw(q -f mboxrd), "m:$m");
+open $fh, '<', \$lei_out or BAIL_OUT $!;
+my @res;
+PublicInbox::MboxReader->mboxrd($fh, sub { push @res, shift });
+is($res[0]->header('Status'), 'RO', 'seen kw set');
+$res[0]->header_set('Status');
+is_deeply(\@res, [ $eml ], 'imported message matches w/ List-Id');
+
+$eml->header_set('List-Id', '<another.example.com>');
+$in = $eml->as_string;
+lei_ok([qw(import -F eml --stdin)], undef, { 0 => \$in, %$lei_opt });
+is($lei_err, '', 'no errors from 2nd import');
+lei_ok(qw(q -f mboxrd), "m:$m", 'l:another.example.com');
+my @another;
+open $fh, '<', \$lei_out or BAIL_OUT $!;
+PublicInbox::MboxReader->mboxrd($fh, sub { push @another, shift });
+is($another[0]->header('Status'), 'RO', 'seen kw set');
+
}); # test_lei
done_testing;
^ permalink raw reply related [relevance 36%]
* [PATCH] lei q: trim JSON output
@ 2021-03-20 12:40 63% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-20 12:40 UTC (permalink / raw)
To: meta
Stop showing `docid' since it's not useful with shards.
`bytes' and `lines' are probably noise, but maybe could be
visible in some "fuller" view.
---
lib/PublicInbox/LeiOverview.pm | 8 ++++++--
lib/PublicInbox/LeiXSearch.pm | 3 ++-
t/lei-import.t | 2 +-
3 files changed, 9 insertions(+), 4 deletions(-)
diff --git a/lib/PublicInbox/LeiOverview.pm b/lib/PublicInbox/LeiOverview.pm
index 521bca50..1ce2a098 100644
--- a/lib/PublicInbox/LeiOverview.pm
+++ b/lib/PublicInbox/LeiOverview.pm
@@ -136,7 +136,10 @@ sub ovv_end {
sub _unbless_smsg {
my ($smsg, $mitem) = @_;
- delete @$smsg{qw(lines bytes num tid)};
+ # TODO: make configurable
+ # num/tid are nonsensical with multi-inbox search,
+ # lines/bytes are not generally useful
+ delete @$smsg{qw(num tid lines bytes)};
$smsg->{rt} = _iso8601(delete $smsg->{ts}); # JMAP receivedAt
$smsg->{dt} = _iso8601(delete $smsg->{ds}); # JMAP UTCDate
$smsg->{pct} = get_pct($mitem) if $mitem;
@@ -151,7 +154,8 @@ sub _unbless_smsg {
$smsg->{substr($f, 0, 1)} = pairs($v);
}
$smsg->{'s'} = delete $smsg->{subject};
- scalar { %$smsg }; # unbless
+ my $kw = delete($smsg->{kw});
+ scalar { %$smsg, ($kw && scalar(@$kw) ? (kw => $kw) : ()) }; # unbless
}
sub ovv_atexit_child {
diff --git a/lib/PublicInbox/LeiXSearch.pm b/lib/PublicInbox/LeiXSearch.pm
index 57717b87..17171a7f 100644
--- a/lib/PublicInbox/LeiXSearch.pm
+++ b/lib/PublicInbox/LeiXSearch.pm
@@ -70,6 +70,8 @@ sub mitem_kw ($$;$) {
my ($smsg, $mitem, $flagged) = @_;
my $kw = xap_terms('K', $mitem->get_document);
$kw->{flagged} = 1 if $flagged;
+ # we keep the empty array here to prevent expensive work in
+ # ->xsmsg_vmd, _unbless_smsg will clobber it iff it's empty
$smsg->{kw} = [ sort keys %$kw ];
}
@@ -85,7 +87,6 @@ sub smsg_for {
my $smsg = $ibx->over->get_art($num);
return if $smsg->{bytes} == 0;
mitem_kw($smsg, $mitem) if $ibx->can('msg_keywords');
- $smsg->{docid} = $docid;
$smsg;
}
diff --git a/t/lei-import.t b/t/lei-import.t
index edb0cd20..e0b517f4 100644
--- a/t/lei-import.t
+++ b/t/lei-import.t
@@ -48,7 +48,7 @@ lei_ok([qw(import --no-kw -F eml -)], undef, $opt,
lei(qw(q m:v@y));
$res = json_utf8->decode($lei_out);
is($res->[1], undef, 'only one result');
-is_deeply($res->[0]->{kw}, [], 'no keywords set');
+is($res->[0]->{kw}, undef, 'no keywords set');
# see t/lei_to_mail.t for "import -F mbox*"
});
^ permalink raw reply related [relevance 63%]
* [PATCH 5/5] lei: tie ALE lifetime to config file
2021-03-20 10:04 67% [PATCH 0/5] lei: preserve keywords across queries Eric Wong
` (2 preceding siblings ...)
2021-03-20 10:04 64% ` [PATCH 3/5] lei q: put keywords on one line in --pretty output Eric Wong
@ 2021-03-20 10:04 59% ` Eric Wong
3 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-20 10:04 UTC (permalink / raw)
To: meta
This should make a future change to "lei import" work more
nicely, since we'll be needing ALE to vivify external-only
messages upon explicit "lei import".
---
lib/PublicInbox/LEI.pm | 3 +--
lib/PublicInbox/LeiALE.pm | 19 ++++++++++++++++---
lib/PublicInbox/LeiExternal.pm | 6 ------
lib/PublicInbox/LeiQuery.pm | 4 ----
t/lei_xsearch.t | 2 +-
5 files changed, 18 insertions(+), 16 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 0da26a32..72a0e52c 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -108,8 +108,7 @@ sub ale {
my ($self) = @_;
$self->{ale} //= do {
require PublicInbox::LeiALE;
- PublicInbox::LeiALE->new(cache_dir($self).
- '/all_locals_ever.git');
+ $self->_lei_cfg(1)->{ale} //= PublicInbox::LeiALE->new($self);
};
}
diff --git a/lib/PublicInbox/LeiALE.pm b/lib/PublicInbox/LeiALE.pm
index bdb50a1a..45748435 100644
--- a/lib/PublicInbox/LeiALE.pm
+++ b/lib/PublicInbox/LeiALE.pm
@@ -11,16 +11,29 @@ use v5.10.1;
use parent qw(PublicInbox::LeiSearch PublicInbox::Lock);
use PublicInbox::Git;
use PublicInbox::Import;
+use PublicInbox::LeiXSearch;
use Fcntl qw(SEEK_SET);
-sub new {
- my ($cls, $d) = @_;
+sub _new {
+ my ($d) = @_;
PublicInbox::Import::init_bare($d, 'ale');
bless {
git => PublicInbox::Git->new($d),
lock_path => "$d/lei_ale.state", # dual-duty lock + state
ibxish => [], # Inbox and ExtSearch (and LeiSearch) objects
- }, $cls;
+ }, __PACKAGE__
+}
+
+sub new {
+ my ($self, $lei) = @_;
+ ref($self) or $self = _new($lei->cache_dir . '/all_locals_ever.git');
+ my $lxs = PublicInbox::LeiXSearch->new;
+ $lxs->prepare_external($lei->_lei_store(1)->search);
+ for my $loc ($lei->externals_each) { # locals only
+ $lxs->prepare_external($loc) if -d $loc;
+ }
+ $self->refresh_externals($lxs);
+ $self;
}
sub over {} # undef for xoids_for
diff --git a/lib/PublicInbox/LeiExternal.pm b/lib/PublicInbox/LeiExternal.pm
index aa09be9e..b5dd85e1 100644
--- a/lib/PublicInbox/LeiExternal.pm
+++ b/lib/PublicInbox/LeiExternal.pm
@@ -139,12 +139,6 @@ sub add_external_finish {
my $key = "external.$location.boost";
my $cur_boost = $cfg->{$key};
return if defined($cur_boost) && $cur_boost == $new_boost; # idempotent
- if (-d $location) {
- require PublicInbox::LeiXSearch;
- my $lxs = PublicInbox::LeiXSearch->new;
- $lxs->prepare_external($location);
- $self->ale->refresh_externals($lxs);
- }
$self->lei_config($key, $new_boost);
}
diff --git a/lib/PublicInbox/LeiQuery.pm b/lib/PublicInbox/LeiQuery.pm
index 007e35fc..148e8524 100644
--- a/lib/PublicInbox/LeiQuery.pm
+++ b/lib/PublicInbox/LeiQuery.pm
@@ -57,10 +57,6 @@ sub lei_q {
}
if ($opt->{'local'} //= scalar(@only) ? 0 : 1) {
$lxs->prepare_external($lse);
- } else {
- my $tmp = PublicInbox::LeiXSearch->new;
- $tmp->prepare_external($lse);
- $self->ale->refresh_externals($tmp);
}
if (@only) {
for my $loc (@only) {
diff --git a/t/lei_xsearch.t b/t/lei_xsearch.t
index 68211d18..e56b2820 100644
--- a/t/lei_xsearch.t
+++ b/t/lei_xsearch.t
@@ -90,7 +90,7 @@ is($lxs->over, undef, '->over fails');
my $mitem = ($mset->items)[0];
my $smsg = $lxs->smsg_for($mitem) or BAIL_OUT 'smsg_for broken';
- my $ale = PublicInbox::LeiALE->new("$home/ale");
+ my $ale = PublicInbox::LeiALE::_new("$home/ale");
$ale->refresh_externals($lxs);
my $exp = [ $smsg->{blob}, 'blob', -s 't/utf8.eml' ];
is_deeply([ $ale->git->check($smsg->{blob}) ], $exp, 'ale->git->check');
^ permalink raw reply related [relevance 59%]
* [PATCH 0/5] lei: preserve keywords across queries
@ 2021-03-20 10:04 67% Eric Wong
2021-03-20 10:04 33% ` [PATCH 1/5] lei: All Local Externals: bare git dir for alternates Eric Wong
` (3 more replies)
0 siblings, 4 replies; 200+ results
From: Eric Wong @ 2021-03-20 10:04 UTC (permalink / raw)
To: meta
2/5 is the major milestone to me, it took me a long while to
figure out and arrive at.
PATCH 1/5 made things click, and
<https://public-inbox.org/meta/20210320023800.1809-1-e@80x24.org/>
("lei_store: initialize IPC lock properly") was also needed :x
There'll still need to be some followup to support import,
and also to deal with public-inbox-edit/purge in externals.
Inotify + EVFILT_VNODE support should make things sweeter, too.
Eric Wong (5):
lei: All Local Externals: bare git dir for alternates
lei q: support vmd for external-only messages
lei q: put keywords on one line in --pretty output
lei_to_mail: match mutt order of status headers
lei: tie ALE lifetime to config file
MANIFEST | 1 +
lib/PublicInbox/LEI.pm | 15 +++++
lib/PublicInbox/LeiALE.pm | 111 +++++++++++++++++++++++++++++++++
lib/PublicInbox/LeiOverview.pm | 12 +++-
lib/PublicInbox/LeiQuery.pm | 1 +
lib/PublicInbox/LeiSearch.pm | 37 +++++++----
lib/PublicInbox/LeiStore.pm | 82 ++++++++++++++----------
lib/PublicInbox/LeiToMail.pm | 38 ++++++-----
lib/PublicInbox/LeiXSearch.pm | 44 +++----------
lib/PublicInbox/Lock.pm | 2 +-
lib/PublicInbox/Over.pm | 22 ++++++-
lib/PublicInbox/OverIdx.pm | 10 ---
lib/PublicInbox/SearchIdx.pm | 3 +
t/eml.t | 2 +
t/lei-convert.t | 3 +-
t/lei-externals.t | 3 +-
t/lei-q-kw.t | 59 ++++++++++++++++--
t/lei-q-remote-import.t | 4 +-
t/lei-q-thread.t | 7 ++-
t/lei_to_mail.t | 2 +-
t/lei_xsearch.t | 22 ++++++-
21 files changed, 355 insertions(+), 125 deletions(-)
create mode 100644 lib/PublicInbox/LeiALE.pm
^ permalink raw reply [relevance 67%]
* [PATCH 3/5] lei q: put keywords on one line in --pretty output
2021-03-20 10:04 67% [PATCH 0/5] lei: preserve keywords across queries Eric Wong
2021-03-20 10:04 33% ` [PATCH 1/5] lei: All Local Externals: bare git dir for alternates Eric Wong
2021-03-20 10:04 25% ` [PATCH 2/5] lei q: support vmd for external-only messages Eric Wong
@ 2021-03-20 10:04 64% ` Eric Wong
2021-03-20 10:04 59% ` [PATCH 5/5] lei: tie ALE lifetime to config file Eric Wong
3 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-20 10:04 UTC (permalink / raw)
To: meta
Don't waste precious terminal space when there are only a small
number of possible keywords supported/reserved for JMAP. In the
future, we may implement more sophisticated wrapping for labels,
but it we'll cross tha bridge when we come to it.
---
lib/PublicInbox/LeiOverview.pm | 5 ++++-
t/lei-q-kw.t | 7 +++++--
2 files changed, 9 insertions(+), 3 deletions(-)
diff --git a/lib/PublicInbox/LeiOverview.pm b/lib/PublicInbox/LeiOverview.pm
index 48237f8a..521bca50 100644
--- a/lib/PublicInbox/LeiOverview.pm
+++ b/lib/PublicInbox/LeiOverview.pm
@@ -176,7 +176,10 @@ sub _json_pretty {
$pair =~ s/(null|"),"/$1, "/g;
$pair;
} @$v) . ']';
- } else { # references
+ } elsif ($k eq 'kw') { # keywords are short, one-line
+ $v = $json->encode($v);
+ $v =~ s/","/", "/g;
+ } else { # refs, labels, ...
$v = '[' . join($sep, map {
substr($json->encode([$_]), 1, -1);
} @$v) . ']';
diff --git a/t/lei-q-kw.t b/t/lei-q-kw.t
index e7e14221..de2c775a 100644
--- a/t/lei-q-kw.t
+++ b/t/lei-q-kw.t
@@ -144,7 +144,7 @@ lei_ok(qw(q -o), "mboxrd:$o", "m:$m", @inc);
# emulate MUA marking mboxrd message as unread
open my $fh, '<', $o or BAIL_OUT;
my $s = do { local $/; <$fh> };
-$s =~ s/^Status: OR\n/Status: O\nX-Status: A\n/sm or
+$s =~ s/^Status: OR\n/Status: O\nX-Status: AF\n/sm or
fail "failed to clear R flag in $s";
open $fh, '>', $o or BAIL_OUT;
print $fh $s or BAIL_OUT;
@@ -156,7 +156,10 @@ lei_ok(qw(q -o), "mboxrd:$o", "m:$m", @inc);
open $fh, '<', $o or BAIL_OUT;
$s = do { local $/; <$fh> };
like($s, qr/^Status: O\n/ms, 'seen keyword gone in mbox');
-like($s, qr/^X-Status: A\n/ms, 'answered flag set');
+like($s, qr/^X-Status: AF\n/ms, 'answered + flagged set');
+lei_ok(qw(q --pretty), "m:$m", @inc);
+like($lei_out, qr/^ "kw": \["answered", "flagged"\],\n/sm,
+ '--pretty JSON output shows kw: on one line');
}); # test_lei
done_testing;
^ permalink raw reply related [relevance 64%]
* [PATCH 1/5] lei: All Local Externals: bare git dir for alternates
2021-03-20 10:04 67% [PATCH 0/5] lei: preserve keywords across queries Eric Wong
@ 2021-03-20 10:04 33% ` Eric Wong
2021-03-20 10:04 25% ` [PATCH 2/5] lei q: support vmd for external-only messages Eric Wong
` (2 subsequent siblings)
3 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-20 10:04 UTC (permalink / raw)
To: meta
This will be used for keyword (and label) storage for externals.
We'll be using this to ensure we don't redundantly auto-import
messages into lei/store if they're already in a local external
(they can still be imported explicitly via "lei import").
---
MANIFEST | 1 +
lib/PublicInbox/LEI.pm | 16 ++++++
lib/PublicInbox/LeiALE.pm | 98 ++++++++++++++++++++++++++++++++++
lib/PublicInbox/LeiExternal.pm | 6 +++
lib/PublicInbox/LeiOverview.pm | 3 +-
lib/PublicInbox/LeiQuery.pm | 5 ++
lib/PublicInbox/LeiStore.pm | 5 +-
lib/PublicInbox/LeiToMail.pm | 10 ++--
lib/PublicInbox/LeiXSearch.pm | 27 +---------
lib/PublicInbox/Lock.pm | 2 +-
t/lei-externals.t | 3 +-
t/lei_xsearch.t | 22 +++++++-
12 files changed, 158 insertions(+), 40 deletions(-)
create mode 100644 lib/PublicInbox/LeiALE.pm
diff --git a/MANIFEST b/MANIFEST
index 775de5cd..b6b4a3ab 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -179,6 +179,7 @@ lib/PublicInbox/InputPipe.pm
lib/PublicInbox/Isearch.pm
lib/PublicInbox/KQNotify.pm
lib/PublicInbox/LEI.pm
+lib/PublicInbox/LeiALE.pm
lib/PublicInbox/LeiAuth.pm
lib/PublicInbox/LeiConvert.pm
lib/PublicInbox/LeiCurl.pm
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index d20ba744..0da26a32 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -97,6 +97,22 @@ sub _config_path ($) {
.'/lei/config');
}
+sub cache_dir ($) {
+ my ($self) = @_;
+ rel2abs($self, ($self->{env}->{XDG_CACHE_HOME} //
+ ($self->{env}->{HOME} // '/nonexistent').'/.cache')
+ .'/lei');
+}
+
+sub ale {
+ my ($self) = @_;
+ $self->{ale} //= do {
+ require PublicInbox::LeiALE;
+ PublicInbox::LeiALE->new(cache_dir($self).
+ '/all_locals_ever.git');
+ };
+}
+
sub index_opt {
# TODO: drop underscore variants everywhere, they're undocumented
qw(fsync|sync! jobs|j=i indexlevel|L=s compact
diff --git a/lib/PublicInbox/LeiALE.pm b/lib/PublicInbox/LeiALE.pm
new file mode 100644
index 00000000..bdb50a1a
--- /dev/null
+++ b/lib/PublicInbox/LeiALE.pm
@@ -0,0 +1,98 @@
+# Copyright (C) 2021 all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+
+# All Locals Ever: track lei/store + externals ever used as
+# long as they're on an accessible FS. Includes "lei q" --include
+# and --only targets that haven't been through "lei add-external".
+# Typically: ~/.cache/lei/all_locals_ever.git
+package PublicInbox::LeiALE;
+use strict;
+use v5.10.1;
+use parent qw(PublicInbox::LeiSearch PublicInbox::Lock);
+use PublicInbox::Git;
+use PublicInbox::Import;
+use Fcntl qw(SEEK_SET);
+
+sub new {
+ my ($cls, $d) = @_;
+ PublicInbox::Import::init_bare($d, 'ale');
+ bless {
+ git => PublicInbox::Git->new($d),
+ lock_path => "$d/lei_ale.state", # dual-duty lock + state
+ ibxish => [], # Inbox and ExtSearch (and LeiSearch) objects
+ }, $cls;
+}
+
+sub over {} # undef for xoids_for
+
+sub overs_all { # for xoids_for (called only in lei workers?)
+ my ($self) = @_;
+ my $pid = $$;
+ if (($self->{owner_pid} // $pid) != $pid) {
+ delete($_->{over}) for @{$self->{ibxish}};
+ }
+ $self->{owner_pid} = $pid;
+ grep(defined, map { $_->over } @{$self->{ibxish}});
+}
+
+sub refresh_externals {
+ my ($self, $lxs) = @_;
+ $self->git->cleanup;
+ my $lk = $self->lock_for_scope;
+ my $cur_lxs = ref($lxs)->new;
+ my $orig = do {
+ local $/;
+ readline($self->{lockfh}) //
+ die "readline($self->{lock_path}): $!";
+ };
+ my $new = '';
+ my $old = '';
+ my $gone = 0;
+ my %seen_ibxish; # $dir => any-defined value
+ for my $dir (split(/\n/, $orig)) {
+ if (-d $dir && -r _ && $cur_lxs->prepare_external($dir)) {
+ $seen_ibxish{$dir} //= length($old .= "$dir\n");
+ } else {
+ ++$gone;
+ }
+ }
+ my @ibxish = $cur_lxs->locals;
+ for my $x ($lxs->locals) {
+ my $d = File::Spec->canonpath($x->{inboxdir} // $x->{topdir});
+ $seen_ibxish{$d} //= do {
+ $new .= "$d\n";
+ push @ibxish, $x;
+ };
+ }
+ if ($new ne '' || $gone) {
+ $self->{lockfh}->autoflush(1);
+ if ($gone) {
+ seek($self->{lockfh}, 0, SEEK_SET) or die "seek: $!";
+ truncate($self->{lockfh}, 0) or die "truncate: $!";
+ } else {
+ $old = '';
+ }
+ print { $self->{lockfh} } $old, $new or die "print: $!";
+ }
+ $new = $old = '';
+ my $f = $self->git->{git_dir}.'/objects/info/alternates';
+ if (open my $fh, '<', $f) {
+ local $/;
+ $old = <$fh> // die "readline($f): $!";
+ }
+ for my $x (@ibxish) {
+ $new .= File::Spec->canonpath($x->git->{git_dir})."/objects\n";
+ }
+ $self->{ibxish} = \@ibxish;
+ return if $old eq $new;
+
+ # this needs to be atomic since child processes may start
+ # git-cat-file at any time
+ my $tmp = "$f.$$.tmp";
+ open my $fh, '>', $tmp or die "open($tmp): $!";
+ print $fh $new or die "print($tmp): $!";
+ close $fh or die "close($tmp): $!";
+ rename($tmp, $f) or die "rename($tmp, $f): $!";
+}
+
+1;
diff --git a/lib/PublicInbox/LeiExternal.pm b/lib/PublicInbox/LeiExternal.pm
index b5dd85e1..aa09be9e 100644
--- a/lib/PublicInbox/LeiExternal.pm
+++ b/lib/PublicInbox/LeiExternal.pm
@@ -139,6 +139,12 @@ sub add_external_finish {
my $key = "external.$location.boost";
my $cur_boost = $cfg->{$key};
return if defined($cur_boost) && $cur_boost == $new_boost; # idempotent
+ if (-d $location) {
+ require PublicInbox::LeiXSearch;
+ my $lxs = PublicInbox::LeiXSearch->new;
+ $lxs->prepare_external($location);
+ $self->ale->refresh_externals($lxs);
+ }
$self->lei_config($key, $new_boost);
}
diff --git a/lib/PublicInbox/LeiOverview.pm b/lib/PublicInbox/LeiOverview.pm
index f6348162..1036f465 100644
--- a/lib/PublicInbox/LeiOverview.pm
+++ b/lib/PublicInbox/LeiOverview.pm
@@ -209,11 +209,10 @@ sub ovv_each_smsg_cb { # runs in wq worker usually
$wcb->(undef, $smsg, $eml);
};
} elsif ($l2m && $l2m->{-wq_s1}) {
- my $git_dir = $ibxish->git->{git_dir};
sub {
my ($smsg, $mitem) = @_;
$smsg->{pct} = get_pct($mitem) if $mitem;
- $l2m->wq_io_do('write_mail', [], $git_dir, $smsg);
+ $l2m->wq_io_do('write_mail', [], $smsg);
}
} elsif ($self->{fmt} =~ /\A(concat)?json\z/ && $lei->{opt}->{pretty}) {
my $EOR = ($1//'') eq 'concat' ? "\n}" : "\n},";
diff --git a/lib/PublicInbox/LeiQuery.pm b/lib/PublicInbox/LeiQuery.pm
index 532668ae..007e35fc 100644
--- a/lib/PublicInbox/LeiQuery.pm
+++ b/lib/PublicInbox/LeiQuery.pm
@@ -57,6 +57,10 @@ sub lei_q {
}
if ($opt->{'local'} //= scalar(@only) ? 0 : 1) {
$lxs->prepare_external($lse);
+ } else {
+ my $tmp = PublicInbox::LeiXSearch->new;
+ $tmp->prepare_external($lse);
+ $self->ale->refresh_externals($tmp);
}
if (@only) {
for my $loc (@only) {
@@ -90,6 +94,7 @@ sub lei_q {
unless ($lxs->locals || $lxs->remotes) {
return $self->fail('no local or remote inboxes to search');
}
+ $self->ale->refresh_externals($lxs);
my ($xj, $mj) = split(/,/, $opt->{jobs} // '');
if (defined($xj) && $xj ne '' && $xj !~ /\A[1-9][0-9]*\z/) {
return $self->fail("`$xj' search jobs must be >= 1");
diff --git a/lib/PublicInbox/LeiStore.pm b/lib/PublicInbox/LeiStore.pm
index 26f975c3..c1abc288 100644
--- a/lib/PublicInbox/LeiStore.pm
+++ b/lib/PublicInbox/LeiStore.pm
@@ -251,10 +251,11 @@ sub refresh_local_externals {
for my $loc (@loc) { # locals only
$lxs->prepare_external($loc) if -d $loc;
}
+ $self->{lei}->ale->refresh_externals($lxs);
+ $lxs->{git} = $self->{lei}->ale->git;
$self->{lxs_all_local} = $lxs;
$self->{cur_cfg} = $cfg;
}
- ($lxs->{git_tmp} //= $lxs->git_tmp)->{git_dir};
}
sub write_prepare {
@@ -268,7 +269,7 @@ sub write_prepare {
$self->ipc_worker_spawn('lei_store', $lei->oldset,
{ lei => $lei });
}
- $lei->{all_ext_git_dir} = $self->ipc_do('refresh_local_externals');
+ my $wait = $self->ipc_do('refresh_local_externals');
$lei->{sto} = $self;
}
diff --git a/lib/PublicInbox/LeiToMail.pm b/lib/PublicInbox/LeiToMail.pm
index 6f386b10..7e821646 100644
--- a/lib/PublicInbox/LeiToMail.pm
+++ b/lib/PublicInbox/LeiToMail.pm
@@ -11,7 +11,6 @@ use PublicInbox::Lock;
use PublicInbox::ProcessPipe;
use PublicInbox::Spawn qw(which spawn popen_rd);
use PublicInbox::LeiDedupe;
-use PublicInbox::Git;
use PublicInbox::GitAsyncCat;
use PublicInbox::PktOp qw(pkt_do);
use Symbol qw(gensym);
@@ -642,18 +641,15 @@ sub poke_dst {
}
sub write_mail { # via ->wq_io_do
- my ($self, $git_dir, $smsg) = @_;
- my $git = $self->{"$$\0$git_dir"} //= PublicInbox::Git->new($git_dir);
- git_async_cat($git, $smsg->{blob}, \&git_to_mail,
+ my ($self, $smsg) = @_;
+ git_async_cat($self->{lei}->{ale}->git, $smsg->{blob}, \&git_to_mail,
[$self->{wcb}, $smsg]);
}
sub wq_atexit_child {
my ($self) = @_;
delete $self->{wcb};
- for my $git (delete @$self{grep(/\A$$\0/, keys %$self)}) {
- $git->async_wait_all;
- }
+ $self->{lei}->{ale}->git->async_wait_all;
$SIG{__WARN__} = 'DEFAULT';
}
diff --git a/lib/PublicInbox/LeiXSearch.pm b/lib/PublicInbox/LeiXSearch.pm
index d95a218e..1266b3b3 100644
--- a/lib/PublicInbox/LeiXSearch.pm
+++ b/lib/PublicInbox/LeiXSearch.pm
@@ -297,27 +297,7 @@ sub query_remote_mboxrd {
$lei->{ovv}->ovv_atexit_child($lei);
}
-# called by LeiOverview::each_smsg_cb
-sub git { $_[0]->{git_tmp} // die 'BUG: caller did not set {git_tmp}' }
-
-sub git_tmp ($) {
- my ($self) = @_;
- my (%seen, @dirs);
- my $tmp = File::Temp->newdir("lei_xsearch_git.$$-XXXX", TMPDIR => 1);
- for my $ibxish (locals($self)) {
- my $d = File::Spec->canonpath($ibxish->git->{git_dir});
- $seen{$d} //= push @dirs, "$d/objects\n"
- }
- my $git_dir = $tmp->dirname;
- PublicInbox::Import::init_bare($git_dir);
- my $f = "$git_dir/objects/info/alternates";
- open my $alt, '>', $f or die "open($f): $!";
- print $alt @dirs or die "print $f: $!";
- close $alt or die "close $f: $!";
- my $git = PublicInbox::Git->new($git_dir);
- $git->{-tmp} = $tmp;
- $git;
-}
+sub git { $_[0]->{git} // die 'BUG: git uninitialized' }
sub xsearch_done_wait { # dwaitpid callback
my ($arg, $pid) = @_;
@@ -460,11 +440,6 @@ sub do_query {
# 1031: F_SETPIPE_SZ
fcntl($lei->{startq}, 1031, 4096) if $^O eq 'linux';
}
- if (!$lei->{opt}->{threads} && locals($self)) { # for query_mset
- # lei->{git_tmp} is set for wq_wait_old so we don't
- # delete until all lei2mail + lei_xsearch workers are reaped
- $lei->{git_tmp} = $self->{git_tmp} = git_tmp($self);
- }
$self->wq_workers_start('lei_xsearch', undef,
$lei->oldset, { lei => $lei });
my $op = delete $lei->{pkt_op_c};
diff --git a/lib/PublicInbox/Lock.pm b/lib/PublicInbox/Lock.pm
index 76c3ffb2..0ee2a8bd 100644
--- a/lib/PublicInbox/Lock.pm
+++ b/lib/PublicInbox/Lock.pm
@@ -16,7 +16,7 @@ sub lock_acquire {
my $lock_path = $self->{lock_path};
croak 'already locked '.($lock_path // '(undef)') if $self->{lockfh};
return unless defined($lock_path);
- sysopen(my $lockfh, $lock_path, O_WRONLY|O_CREAT) or
+ sysopen(my $lockfh, $lock_path, O_RDWR|O_CREAT) or
croak "failed to open $lock_path: $!\n";
flock($lockfh, LOCK_EX) or croak "lock $lock_path failed: $!\n";
$self->{lockfh} = $lockfh;
diff --git a/t/lei-externals.t b/t/lei-externals.t
index 1d2a9a16..2045691f 100644
--- a/t/lei-externals.t
+++ b/t/lei-externals.t
@@ -236,7 +236,8 @@ test_lei(sub {
is(scalar(@s), 2, "2 results in mbox$sfx");
lei_ok('q', '-a', '-o', "mboxcl2:$f", 's:nonexistent');
- is(grep(!/^#/, $lei_err), 0, "no errors on no results ($sfx)");
+ is(grep(!/^#/, $lei_err), 0, "no errors on no results ($sfx)")
+ or diag $lei_err;
my @s2 = grep(/^Subject:/, $cat->());
is_deeply(\@s2, \@s,
diff --git a/t/lei_xsearch.t b/t/lei_xsearch.t
index f626c790..68211d18 100644
--- a/t/lei_xsearch.t
+++ b/t/lei_xsearch.t
@@ -10,6 +10,7 @@ require_mods(qw(DBD::SQLite Search::Xapian));
require PublicInbox::ExtSearchIdx;
require_git 2.6;
require_ok 'PublicInbox::LeiXSearch';
+require_ok 'PublicInbox::LeiALE';
my ($home, $for_destroy) = tmpdir();
my @ibx;
for my $V (1..2) {
@@ -75,7 +76,8 @@ is($lxs->over, undef, '->over fails');
my $v2ibx = create_inbox 'v2full', version => 2, sub {
$_[0]->add(eml_load('t/plack-qp.eml'));
};
- my $v1ibx = create_inbox 'v1medium', indexlevel => 'medium', sub {
+ my $v1ibx = create_inbox 'v1medium', indexlevel => 'medium',
+ tmpdir => "$home/v1tmp", sub {
$_[0]->add(eml_load('t/utf8.eml'));
};
$lxs->prepare_external($v1ibx);
@@ -85,6 +87,24 @@ is($lxs->over, undef, '->over fails');
}
my $mset = $lxs->mset('m:testmessage@example.com');
is($mset->size, 1, 'got m: match on medium+full XSearch mix');
+ my $mitem = ($mset->items)[0];
+ my $smsg = $lxs->smsg_for($mitem) or BAIL_OUT 'smsg_for broken';
+
+ my $ale = PublicInbox::LeiALE->new("$home/ale");
+ $ale->refresh_externals($lxs);
+ my $exp = [ $smsg->{blob}, 'blob', -s 't/utf8.eml' ];
+ is_deeply([ $ale->git->check($smsg->{blob}) ], $exp, 'ale->git->check');
+
+ $lxs = PublicInbox::LeiXSearch->new;
+ $lxs->prepare_external($v2ibx);
+ $ale->refresh_externals($lxs);
+ is_deeply([ $ale->git->check($smsg->{blob}) ], $exp,
+ 'ale->git->check remembered inactive external');
+
+ rename("$home/v1tmp", "$home/v1moved") or BAIL_OUT "rename: $!";
+ $ale->refresh_externals($lxs);
+ is($ale->git->check($smsg->{blob}), undef,
+ 'missing after directory gone');
}
done_testing;
^ permalink raw reply related [relevance 33%]
* [PATCH 2/5] lei q: support vmd for external-only messages
2021-03-20 10:04 67% [PATCH 0/5] lei: preserve keywords across queries Eric Wong
2021-03-20 10:04 33% ` [PATCH 1/5] lei: All Local Externals: bare git dir for alternates Eric Wong
@ 2021-03-20 10:04 25% ` Eric Wong
2021-03-20 10:04 64% ` [PATCH 3/5] lei q: put keywords on one line in --pretty output Eric Wong
2021-03-20 10:04 59% ` [PATCH 5/5] lei: tie ALE lifetime to config file Eric Wong
3 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-20 10:04 UTC (permalink / raw)
To: meta
"lei q" now preserves changes per-message keywords across
invocations when it's --output (Maildir or mbox) is reused
(with or without --augment).
In the future, these changes will be monitored via inotify,
EVFILT_VNODE or IMAP IDLE, too.
Unfortunately, this currently prevents "lei import" from ever
importing a message that's in an external. That will be fixed
in a future change.
---
lib/PublicInbox/LeiOverview.pm | 4 ++
lib/PublicInbox/LeiSearch.pm | 37 ++++++++++-----
lib/PublicInbox/LeiStore.pm | 83 ++++++++++++++++++++--------------
lib/PublicInbox/LeiToMail.pm | 16 ++++++-
lib/PublicInbox/LeiXSearch.pm | 17 +++----
lib/PublicInbox/Over.pm | 22 ++++++++-
lib/PublicInbox/OverIdx.pm | 10 ----
lib/PublicInbox/SearchIdx.pm | 3 ++
t/eml.t | 2 +
t/lei-convert.t | 3 +-
t/lei-q-kw.t | 48 +++++++++++++++++++-
t/lei-q-thread.t | 7 +--
12 files changed, 177 insertions(+), 75 deletions(-)
diff --git a/lib/PublicInbox/LeiOverview.pm b/lib/PublicInbox/LeiOverview.pm
index 1036f465..48237f8a 100644
--- a/lib/PublicInbox/LeiOverview.pm
+++ b/lib/PublicInbox/LeiOverview.pm
@@ -216,9 +216,11 @@ sub ovv_each_smsg_cb { # runs in wq worker usually
}
} elsif ($self->{fmt} =~ /\A(concat)?json\z/ && $lei->{opt}->{pretty}) {
my $EOR = ($1//'') eq 'concat' ? "\n}" : "\n},";
+ my $lse = $lei->{sto}->search;
sub { # DIY prettiness :P
my ($smsg, $mitem) = @_;
return if $dedupe->is_smsg_dup($smsg);
+ $lse->xsmsg_vmd($smsg);
$smsg = _unbless_smsg($smsg, $mitem);
$buf .= "{\n";
$buf .= join(",\n", map {
@@ -238,9 +240,11 @@ sub ovv_each_smsg_cb { # runs in wq worker usually
}
} elsif ($json) {
my $ORS = $self->{fmt} eq 'json' ? ",\n" : "\n"; # JSONL
+ my $lse = $lei->{sto}->search;
sub {
my ($smsg, $mitem) = @_;
return if $dedupe->is_smsg_dup($smsg);
+ $lse->xsmsg_vmd($smsg);
$buf .= $json->encode(_unbless_smsg(@_)) . $ORS;
return if length($buf) < 65536;
my $lk = $self->lock_for_scope;
diff --git a/lib/PublicInbox/LeiSearch.pm b/lib/PublicInbox/LeiSearch.pm
index 2e3f10fd..360a37e5 100644
--- a/lib/PublicInbox/LeiSearch.pm
+++ b/lib/PublicInbox/LeiSearch.pm
@@ -27,6 +27,20 @@ sub msg_keywords {
wantarray ? sort(keys(%$kw)) : $kw;
}
+sub xsmsg_vmd {
+ my ($self, $smsg) = @_;
+ return if $smsg->{kw};
+ my $xdb = $self->xdb; # set {nshard};
+ my %kw;
+ $kw{flagged} = 1 if delete($smsg->{lei_q_tt_flagged});
+ my @num = $self->over->blob_exists($smsg->{blob});
+ for my $num (@num) { # there should only be one...
+ my $kw = xap_terms('K', $xdb, num2docid($self, $num));
+ %kw = (%kw, %$kw);
+ }
+ $smsg->{kw} = [ sort keys %kw ] if scalar(keys(%kw));
+}
+
# when a message has no Message-IDs at all, this is needed for
# unsent Draft messages, at least
sub content_key ($) {
@@ -43,41 +57,42 @@ sub content_key ($) {
}
sub _cmp_1st { # git->cat_async callback
- my ($bref, $oid, $type, $size, $cmp) = @_; # cmp: [chash, found, smsg]
- if (content_hash(PublicInbox::Eml->new($bref)) eq $cmp->[0]) {
+ my ($bref, $oid, $type, $size, $cmp) = @_; # cmp: [chash, xoids, smsg]
+ if ($bref && content_hash(PublicInbox::Eml->new($bref)) eq $cmp->[0]) {
$cmp->[1]->{$oid} = $cmp->[2]->{num};
}
}
-sub xids_for { # returns { OID => docid } mapping for $eml matches
+sub xoids_for { # returns { OID => docid } mapping for $eml matches
my ($self, $eml, $min) = @_;
my ($chash, $mids) = content_key($eml);
my @overs = ($self->over // $self->overs_all);
my $git = $self->git;
- my $found = {};
+ my $xoids = {};
for my $mid (@$mids) {
for my $o (@overs) {
my ($id, $prev);
while (my $cur = $o->next_by_mid($mid, \$id, \$prev)) {
- next if $found->{$cur->{blob}};
+ next if $cur->{bytes} == 0 ||
+ $xoids->{$cur->{blob}};
$git->cat_async($cur->{blob}, \&_cmp_1st,
- [ $chash, $found, $cur ]);
- if ($min && scalar(keys %$found) >= $min) {
+ [ $chash, $xoids, $cur ]);
+ if ($min && scalar(keys %$xoids) >= $min) {
$git->cat_async_wait;
- return $found;
+ return $xoids;
}
}
}
}
$git->cat_async_wait;
- scalar(keys %$found) ? $found : undef;
+ scalar(keys %$xoids) ? $xoids : undef;
}
# returns true if $eml is indexed by lei/store and keywords don't match
sub kw_changed {
my ($self, $eml, $new_kw_sorted) = @_;
- my $found = xids_for($self, $eml, 1) // return;
- my ($num) = values %$found;
+ my $xoids = xoids_for($self, $eml, 1) // return;
+ my ($num) = values %$xoids;
my @cur_kw = msg_keywords($self, $num);
join("\0", @$new_kw_sorted) eq join("\0", @cur_kw) ? 0 : 1;
}
diff --git a/lib/PublicInbox/LeiStore.pm b/lib/PublicInbox/LeiStore.pm
index c1abc288..c66d3dc2 100644
--- a/lib/PublicInbox/LeiStore.pm
+++ b/lib/PublicInbox/LeiStore.pm
@@ -114,6 +114,7 @@ sub _docids_for ($$) {
for my $mid (@$mids) {
my ($id, $prev);
while (my $cur = $oidx->next_by_mid($mid, \$id, \$prev)) {
+ next if $cur->{bytes} == 0; # external-only message
my $oid = $cur->{blob};
my $docid = $cur->{num};
my $bref = $im ? $im->cat_blob($oid) : undef;
@@ -163,7 +164,7 @@ sub add_eml {
my ($self, $eml, $vmd) = @_;
my $im = $self->importer; # may create new epoch
my $eidx = eidx_init($self); # writes ALL.git/objects/info/alternates
- my $oidx = $eidx->{oidx};
+ my $oidx = $eidx->{oidx}; # PublicInbox::Import::add checks this
my $smsg = bless { -oidx => $oidx }, 'PublicInbox::Smsg';
$im->add($eml, undef, $smsg) or return; # duplicate returns undef
@@ -193,22 +194,54 @@ sub set_eml {
add_eml($self, $eml, $vmd) // set_eml_vmd($self, $eml, $vmd);
}
-sub add_eml_maybe {
- my ($self, $eml) = @_;
- my $lxs = $self->{lxs_all_local} // die 'BUG: no {lxs_all_local}';
- return if $lxs->xids_for($eml, 1);
- add_eml($self, $eml);
-}
-
# set or update keywords for external message, called via ipc_do
-sub set_xkw {
- my ($self, $eml, $kw) = @_;
- my $lxs = $self->{lxs_all_local} // die 'BUG: no {lxs_all_local}';
- if ($lxs->xids_for($eml, 1)) { # is it in a local external?
- # TODO: index keywords only
- } else {
- set_eml($self, $eml, { kw => $kw });
+sub set_xvmd {
+ my ($self, $xoids, $eml, $vmd) = @_;
+
+ my $eidx = eidx_init($self);
+ my $oidx = $eidx->{oidx};
+
+ # see if we can just update existing docs
+ for my $oid (keys %$xoids) {
+ my @docids = $oidx->blob_exists($oid) or next;
+ scalar(@docids) > 1 and
+ warn "W: $oid indexed as multiple docids: @docids\n";
+ for my $docid (@docids) {
+ my $idx = $eidx->idx_shard($docid);
+ $idx->ipc_do('set_vmd', $docid, $vmd);
+ }
+ delete $xoids->{$oid}; # all done with this oid
}
+ return unless scalar(keys(%$xoids));
+
+ # see if it was indexed, but with different OID(s)
+ if (my @docids = _docids_for($self, $eml)) {
+ for my $docid (@docids) {
+ for my $oid (keys %$xoids) {
+ $oidx->add_xref3($docid, -1, $oid, '.');
+ }
+ my $idx = $eidx->idx_shard($docid);
+ $idx->ipc_do('set_vmd', $docid, $vmd);
+ }
+ return;
+ }
+ # totally unseen
+ my $smsg = bless { blob => '' }, 'PublicInbox::Smsg';
+ $smsg->{num} = $oidx->adj_counter('eidx_docid', '+');
+ # save space for an externals-only message
+ my $hdr = $eml->header_obj;
+ $smsg->populate($hdr); # sets lines == 0
+ $smsg->{bytes} = 0;
+ delete @$smsg{qw(From Subject)};
+ $smsg->{to} = $smsg->{cc} = $smsg->{from} = '';
+ $oidx->add_overview($hdr, $smsg); # subject+references for threading
+ $smsg->{subject} = '';
+ for my $oid (keys %$xoids) {
+ $oidx->add_xref3($smsg->{num}, -1, $oid, '.');
+ }
+ my $idx = $eidx->idx_shard($smsg->{num});
+ $idx->index_eml(PublicInbox::Eml->new("\n\n"), $smsg);
+ $idx->ipc_do('add_vmd', $smsg->{num}, $vmd);
}
sub checkpoint {
@@ -240,28 +273,9 @@ sub ipc_atfork_child {
$self->SUPER::ipc_atfork_child;
}
-sub refresh_local_externals {
- my ($self) = @_;
- my $cfg = $self->{lei}->_lei_cfg or return;
- my $cur_cfg = $self->{cur_cfg} // -1;
- my $lxs = $self->{lxs_all_local};
- if ($cfg != $cur_cfg || !$lxs) {
- $lxs = PublicInbox::LeiXSearch->new;
- my @loc = $self->{lei}->externals_each;
- for my $loc (@loc) { # locals only
- $lxs->prepare_external($loc) if -d $loc;
- }
- $self->{lei}->ale->refresh_externals($lxs);
- $lxs->{git} = $self->{lei}->ale->git;
- $self->{lxs_all_local} = $lxs;
- $self->{cur_cfg} = $cfg;
- }
-}
-
sub write_prepare {
my ($self, $lei) = @_;
unless ($self->{-ipc_req}) {
- require PublicInbox::LeiXSearch;
$self->ipc_lock_init($lei->store_path . '/ipc.lock');
# Mail we import into lei are private, so headers filtered out
# by -mda for public mail are not appropriate
@@ -269,7 +283,6 @@ sub write_prepare {
$self->ipc_worker_spawn('lei_store', $lei->oldset,
{ lei => $lei });
}
- my $wait = $self->ipc_do('refresh_local_externals');
$lei->{sto} = $self;
}
diff --git a/lib/PublicInbox/LeiToMail.pm b/lib/PublicInbox/LeiToMail.pm
index 7e821646..3e6cf00c 100644
--- a/lib/PublicInbox/LeiToMail.pm
+++ b/lib/PublicInbox/LeiToMail.pm
@@ -11,6 +11,7 @@ use PublicInbox::Lock;
use PublicInbox::ProcessPipe;
use PublicInbox::Spawn qw(which spawn popen_rd);
use PublicInbox::LeiDedupe;
+use PublicInbox::Git;
use PublicInbox::GitAsyncCat;
use PublicInbox::PktOp qw(pkt_do);
use Symbol qw(gensym);
@@ -260,10 +261,12 @@ sub _mbox_write_cb ($$) {
my $atomic_append = !defined($ovv->{lock_path});
my $dedupe = $lei->{dedupe};
$dedupe->prepare_dedupe;
+ my $lse = $lei->{sto} ? $lei->{sto}->search : undef;
sub { # for git_to_mail
my ($buf, $smsg, $eml) = @_;
$eml //= PublicInbox::Eml->new($buf);
return if $dedupe->is_dup($eml, $smsg->{blob});
+ $lse->xsmsg_vmd($smsg) if $lse;
$buf = $eml2mbox->($eml, $smsg);
return atomic_append($lei, $buf) if $atomic_append;
my $lk = $ovv->lock_for_scope;
@@ -275,10 +278,15 @@ sub update_kw_maybe ($$$$) {
my ($lei, $lse, $eml, $kw) = @_;
return unless $lse;
my $x = $lse->kw_changed($eml, $kw);
+ my $vmd = { kw => $kw };
if ($x) {
- $lei->{sto}->ipc_do('set_eml', $eml, { kw => $kw });
+ $lei->{sto}->ipc_do('set_eml', $eml, $vmd);
} elsif (!defined($x)) {
- $lei->{sto}->ipc_do('set_xkw', $eml, $kw);
+ if (my $xoids = $lei->{ale}->xoids_for($eml)) {
+ $lei->{sto}->ipc_do('set_xvmd', $xoids, $eml, $vmd);
+ } else {
+ $lei->{sto}->ipc_do('set_eml', $eml, $vmd);
+ }
}
}
@@ -342,10 +350,12 @@ sub _maildir_write_cb ($$) {
my $dedupe = $lei->{dedupe};
$dedupe->prepare_dedupe if $dedupe;
my $dst = $lei->{ovv}->{dst};
+ my $lse = $lei->{sto} ? $lei->{sto}->search : undef;
sub { # for git_to_mail
my ($buf, $smsg, $eml) = @_;
$dst // return $lei->fail; # dst may be undef-ed in last run
$buf //= \($eml->as_string);
+ $lse->xsmsg_vmd($smsg) if $lse;
return _buf2maildir($dst, $buf, $smsg) if !$dedupe;
$eml //= PublicInbox::Eml->new($$buf); # copy buf
return if $dedupe->is_dup($eml, $smsg->{blob});
@@ -361,6 +371,7 @@ sub _imap_write_cb ($$) {
my $imap_append = $lei->{net}->can('imap_append');
my $mic = $lei->{net}->mic_get($self->{uri});
my $folder = $self->{uri}->mailbox;
+ my $lse = $lei->{sto} ? $lei->{sto}->search : undef;
sub { # for git_to_mail
my ($bref, $smsg, $eml) = @_;
$mic // return $lei->fail; # dst may be undef-ed in last run
@@ -368,6 +379,7 @@ sub _imap_write_cb ($$) {
$eml //= PublicInbox::Eml->new($$bref); # copy bref
return if $dedupe->is_dup($eml, $smsg->{blob});
}
+ $lse->xsmsg_vmd($smsg) if $lse;
eval { $imap_append->($mic, $folder, $bref, $smsg, $eml) };
if (my $err = $@) {
undef $mic;
diff --git a/lib/PublicInbox/LeiXSearch.pm b/lib/PublicInbox/LeiXSearch.pm
index 1266b3b3..57717b87 100644
--- a/lib/PublicInbox/LeiXSearch.pm
+++ b/lib/PublicInbox/LeiXSearch.pm
@@ -83,6 +83,7 @@ sub smsg_for {
my $num = int(($docid - 1) / $nshard) + 1;
my $ibx = $self->{shard2ibx}->[$shard];
my $smsg = $ibx->over->get_art($num);
+ return if $smsg->{bytes} == 0;
mitem_kw($smsg, $mitem) if $ibx->can('msg_keywords');
$smsg->{docid} = $docid;
$smsg;
@@ -97,11 +98,6 @@ sub recent {
sub over {}
-sub overs_all { # for xids_for
- my ($self) = @_;
- grep(defined, map { $_->over } locals($self))
-}
-
sub _mset_more ($$) {
my ($mset, $mo) = @_;
my $size = $mset->size;
@@ -153,7 +149,7 @@ sub query_thread_mset { # for --threads
my $mset;
my $each_smsg = $lei->{ovv}->ovv_each_smsg_cb($lei, $ibxish);
my $can_kw = !!$ibxish->can('msg_keywords');
- my $fl = $lei->{opt}->{threads} > 1 ? [ 'flagged' ] : undef;
+ my $fl = $lei->{opt}->{threads} > 1 ? 1 : undef;
do {
$mset = $srch->mset($mo->{qstr}, $mo);
mset_progress($lei, $desc, $mset->size,
@@ -165,13 +161,14 @@ sub query_thread_mset { # for --threads
while ($over->expand_thread($ctx)) {
for my $n (@{$ctx->{xids}}) {
my $smsg = $over->get_art($n) or next;
- wait_startq($lei);
my $mitem = delete $n2item{$smsg->{num}};
+ next if $smsg->{bytes} == 0;
+ wait_startq($lei); # wait for keyword updates
if ($mitem) {
if ($can_kw) {
mitem_kw($smsg, $mitem, $fl);
} elsif ($fl) {
- $smsg->{kw} = $fl;
+ $smsg->{lei_q_tt_flagged} = 1;
}
}
$each_smsg->($smsg, $mitem);
@@ -209,8 +206,8 @@ sub query_mset { # non-parallel for non-"--threads" users
sub each_remote_eml { # callback for MboxReader->mboxrd
my ($eml, $self, $lei, $each_smsg) = @_;
- if (my $sto = $self->{import_sto}) {
- $sto->ipc_do('add_eml_maybe', $eml);
+ if ($self->{import_sto} && !$lei->{ale}->xoids_for($eml, 1)) {
+ $self->{import_sto}->ipc_do('add_eml', $eml);
}
my $smsg = bless {}, 'PublicInbox::Smsg';
$smsg->populate($eml);
diff --git a/lib/PublicInbox/Over.pm b/lib/PublicInbox/Over.pm
index 06ea439d..587e0516 100644
--- a/lib/PublicInbox/Over.pm
+++ b/lib/PublicInbox/Over.pm
@@ -7,7 +7,7 @@
package PublicInbox::Over;
use strict;
use v5.10.1;
-use DBI;
+use DBI qw(:sql_types); # SQL_BLOB
use DBD::SQLite;
use PublicInbox::Smsg;
use Compress::Zlib qw(uncompress);
@@ -349,4 +349,24 @@ sub check_inodes {
}
}
+sub blob_exists {
+ my ($self, $oidhex) = @_;
+ if (wantarray) {
+ my $sth = $self->dbh->prepare_cached(<<'', undef, 1);
+SELECT docid FROM xref3 WHERE oidbin = ?
+
+ $sth->bind_param(1, pack('H*', $oidhex), SQL_BLOB);
+ $sth->execute;
+ my $tmp = $sth->fetchall_arrayref;
+ map { $_->[0] } @$tmp;
+ } else {
+ my $sth = $self->dbh->prepare_cached(<<'', undef, 1);
+SELECT COUNT(*) FROM xref3 WHERE oidbin = ?
+
+ $sth->bind_param(1, pack('H*', $oidhex), SQL_BLOB);
+ $sth->execute;
+ $sth->fetchrow_array;
+ }
+}
+
1;
diff --git a/lib/PublicInbox/OverIdx.pm b/lib/PublicInbox/OverIdx.pm
index 9013ae23..e1cd31b9 100644
--- a/lib/PublicInbox/OverIdx.pm
+++ b/lib/PublicInbox/OverIdx.pm
@@ -668,14 +668,4 @@ DELETE FROM eidxq WHERE docid = ?
}
-sub blob_exists {
- my ($self, $oidhex) = @_;
- my $sth = $self->dbh->prepare_cached(<<'', undef, 1);
-SELECT COUNT(*) FROM xref3 WHERE oidbin = ?
-
- $sth->bind_param(1, pack('H*', $oidhex), SQL_BLOB);
- $sth->execute;
- $sth->fetchrow_array;
-}
-
1;
diff --git a/lib/PublicInbox/SearchIdx.pm b/lib/PublicInbox/SearchIdx.pm
index e2a1a678..3237aadc 100644
--- a/lib/PublicInbox/SearchIdx.pm
+++ b/lib/PublicInbox/SearchIdx.pm
@@ -494,7 +494,10 @@ sub add_eidx_info {
begin_txn_lazy($self);
my $doc = _get_doc($self, $docid) or return;
term_generator($self)->set_document($doc);
+
+ # '.' is special for lei_store
$doc->add_boolean_term('O'.$eidx_key) if $eidx_key ne '.';
+
index_list_id($self, $doc, $eml);
$self->{xdb}->replace_document($docid, $doc);
}
diff --git a/t/eml.t b/t/eml.t
index ebd45c13..0cf48f22 100644
--- a/t/eml.t
+++ b/t/eml.t
@@ -26,6 +26,8 @@ sub mime_load ($) {
is($str, "hi\n", '->new modified body like Email::Simple');
is($eml->body, "hi\n", '->body works');
is($eml->as_string, "a: b\n\nhi\n", '->as_string');
+ my $empty = PublicInbox::Eml->new("\n\n");
+ is($empty->as_string, "\n\n", 'empty message');
}
for my $cls (@classes) {
diff --git a/t/lei-convert.t b/t/lei-convert.t
index 186cfb13..e147715d 100644
--- a/t/lei-convert.t
+++ b/t/lei-convert.t
@@ -60,7 +60,8 @@ test_lei({ tmpdir => $tmpdir }, sub {
PublicInbox::MdirReader::maildir_each_eml("$d/md", sub {
push @md, $_[2];
});
- is(scalar(@md), scalar(@mboxrd), 'got expected emails in Maildir');
+ is(scalar(@md), scalar(@mboxrd), 'got expected emails in Maildir') or
+ diag $lei_err;
@md = sort { ${$a->{bdy}} cmp ${$b->{bdy}} } @md;
@mboxrd = sort { ${$a->{bdy}} cmp ${$b->{bdy}} } @mboxrd;
my @rd_nostatus = map {
diff --git a/t/lei-q-kw.t b/t/lei-q-kw.t
index 917a2c53..e7e14221 100644
--- a/t/lei-q-kw.t
+++ b/t/lei-q-kw.t
@@ -112,7 +112,51 @@ for my $sfx ('', '.gz') {
lei_ok(qw(q -o), "mboxrd:/dev/stdout", qw(m:qp@example.com)) or
diag $lei_err;
like($lei_out, qr/^Status: OR\n/sm, 'Status set by previous augment');
-}
+} # /mbox + mbox.gz tests
-});
+my ($ro_home, $cfg_path) = setup_public_inboxes;
+
+# import keywords-only for external messages:
+$o = "$ENV{HOME}/kwdir";
+my $m = 'alpine.DEB.2.20.1608131214070.4924@example';
+my @inc = ('-I', "$ro_home/t1");
+lei_ok(qw(q -o), $o, "m:$m", @inc);
+
+# emulate MUA marking a Maildir message as read:
+@fn = glob("$o/cur/*");
+scalar(@fn) == 1 or BAIL_OUT "wrote multiple or zero files: ".explain(\@fn);
+rename($fn[0], "$fn[0]S") or BAIL_OUT "rename $!";
+
+lei_ok(qw(q -o), $o, 'bogus', \'clobber output dir to import keywords');
+@fn = glob("$o/cur/*");
+is_deeply(\@fn, [], 'output dir actually clobbered');
+lei_ok('q', "m:$m", @inc);
+my $res = json_utf8->decode($lei_out);
+is_deeply($res->[0]->{kw}, ['seen'], 'seen flag set for external message')
+ or diag explain($res);
+lei_ok('q', "m:$m", '--no-external');
+is_deeply($res = json_utf8->decode($lei_out), [ undef ],
+ 'external message not imported') or diag explain($res);
+
+$o = "$ENV{HOME}/kwmboxrd";
+lei_ok(qw(q -o), "mboxrd:$o", "m:$m", @inc);
+
+# emulate MUA marking mboxrd message as unread
+open my $fh, '<', $o or BAIL_OUT;
+my $s = do { local $/; <$fh> };
+$s =~ s/^Status: OR\n/Status: O\nX-Status: A\n/sm or
+ fail "failed to clear R flag in $s";
+open $fh, '>', $o or BAIL_OUT;
+print $fh $s or BAIL_OUT;
+close $fh or BAIL_OUT;
+
+lei_ok(qw(q -o), "mboxrd:$o", 'm:bogus', @inc,
+ \'clobber mbox to import keywords');
+lei_ok(qw(q -o), "mboxrd:$o", "m:$m", @inc);
+open $fh, '<', $o or BAIL_OUT;
+$s = do { local $/; <$fh> };
+like($s, qr/^Status: O\n/ms, 'seen keyword gone in mbox');
+like($s, qr/^X-Status: A\n/ms, 'answered flag set');
+
+}); # test_lei
done_testing;
diff --git a/t/lei-q-thread.t b/t/lei-q-thread.t
index e24fb2cb..c999d12b 100644
--- a/t/lei-q-thread.t
+++ b/t/lei-q-thread.t
@@ -43,10 +43,11 @@ test_lei(sub {
'flagged set in direct hit');
lei_ok qw(q -tt m:testmessage@example.com --only), "$ro_home/t2";
$res = json_utf8->decode($lei_out);
- is_deeply($res->[0]->{kw}, [ 'flagged' ],
- 'flagged set on external with -tt');
+ is_deeply($res->[0]->{kw}, [ qw(flagged seen) ],
+ 'flagged set on external with -tt') or diag explain($res);
lei_ok qw(q -t m:testmessage@example.com --only), "$ro_home/t2";
$res = json_utf8->decode($lei_out);
- ok(!exists($res->[0]->{kw}), 'flagged not set on external with 1 -t');
+ is_deeply($res->[0]->{kw}, [ 'seen' ],
+ 'flagged not set on external with 1 -t') or diag explain($res);
});
done_testing;
^ permalink raw reply related [relevance 25%]
* [PATCH] lei q: -I/--include overrides --no-(external|local|remote)
@ 2021-03-19 22:38 59% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-19 22:38 UTC (permalink / raw)
To: meta
Assume that anybody using -I/--include for external locations
will want to override --no-$FOO if they're explicitly including
a location.
With some effort, we could make it order-dependent (e.g.
"-I $LOCATION --no-$FOO" and "--no-$FOO -I $LOCATION"
behave differently). However that's not straightforward
when using Getopt::Long to parse command-line options into
a hashref.
I'm also not sure if order-dependent switches are a desirable
UI/UX quality.
---
lib/PublicInbox/LeiQuery.pm | 7 +++++--
t/lei-externals.t | 15 +++++++++++++--
2 files changed, 18 insertions(+), 4 deletions(-)
diff --git a/lib/PublicInbox/LeiQuery.pm b/lib/PublicInbox/LeiQuery.pm
index 623b92cd..532668ae 100644
--- a/lib/PublicInbox/LeiQuery.pm
+++ b/lib/PublicInbox/LeiQuery.pm
@@ -64,9 +64,12 @@ sub lei_q {
$lxs->prepare_external($_) for @loc;
}
} else {
+ my (@ilocals, @iremotes);
for my $loc (@{$opt->{include} // []}) {
my @loc = $self->get_externals($loc) or return;
$lxs->prepare_external($_) for @loc;
+ @ilocals = @{$lxs->{locals} // []};
+ @iremotes = @{$lxs->{remotes} // []};
}
# --external is enabled by default, but allow --no-external
if ($opt->{external} //= 1) {
@@ -78,9 +81,9 @@ sub lei_q {
my $ne = $self->externals_each(\&prep_ext, $lxs, \%x);
$opt->{remote} //= !($lxs->locals - $opt->{'local'});
if ($opt->{'local'}) {
- delete($lxs->{remotes}) if !$opt->{remote};
+ $lxs->{remotes} = \@iremotes if !$opt->{remote};
} else {
- delete($lxs->{locals});
+ $lxs->{locals} = \@ilocals;
}
}
}
diff --git a/t/lei-externals.t b/t/lei-externals.t
index 1695ff0b..1d2a9a16 100644
--- a/t/lei-externals.t
+++ b/t/lei-externals.t
@@ -127,8 +127,10 @@ test_lei(sub {
lei_ok qw(_complete lei forget-external), \'complete for externals';
my %comp = map { $_ => 1 } split(/\s+/, $lei_out);
ok($comp{'https://example.com/ibx/'}, 'forget external completion');
+ my @dirs;
$cfg->each_inbox(sub {
my ($ibx) = @_;
+ push @dirs, $ibx->{inboxdir};
ok($comp{$ibx->{inboxdir}}, "local $ibx->{name} completion");
});
for my $u (qw(h http https https: https:/ https:// https://e
@@ -157,7 +159,8 @@ test_lei(sub {
lei_ok('ls-external');
unlike($lei_out, qr!https://example\.com/ibx/!s,
'removed canonical URL');
-SKIP: {
+
+ # do some queries
ok(!lei(qw(q s:prefix -o maildir:/dev/null)), 'bad maildir');
like($lei_err, qr!/dev/null exists and is not a directory!,
'error shown');
@@ -249,6 +252,15 @@ SKIP: {
is($? >> 8, 1, 'proper exit code');
like($lei_err, qr/no local or remote.+? to search/, 'no inbox');
+ for my $no (['--no-local'], ['--no-external'],
+ [qw(--no-local --no-external)]) {
+ lei_ok(qw(q mid:testmessage@example.com), @$no,
+ '-I', $dirs[0], \"-I and @$no combine");
+ $res = json_utf8->decode($lei_out);
+ is($res->[0]->{'m'}, 'testmessage@example.com',
+ "-I \$DIR got results regardless of @$no");
+ }
+
{
opendir my $dh, '.' or BAIL_OUT "opendir(.) $!";
my $od = PublicInbox::OnDestroy->new($$, sub {
@@ -278,6 +290,5 @@ SKIP: {
$url = $e{$k} if $url eq '1';
$test_external_remote->($url, $k);
}
- }; # /SKIP
}); # test_lei
done_testing;
^ permalink raw reply related [relevance 59%]
* [PATCH] t/lei-externals: add diagnostic for warning
@ 2021-03-19 12:41 71% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-19 12:41 UTC (permalink / raw)
To: meta
Not sure where it's coming from, but I saw it fail, once
(and we should be doing more "or diag ..." anyways to improve
diagnostics).
---
t/lei-externals.t | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/t/lei-externals.t b/t/lei-externals.t
index 2a92d101..915f25ad 100644
--- a/t/lei-externals.t
+++ b/t/lei-externals.t
@@ -229,7 +229,8 @@ SKIP: {
is(scalar(@s), 2, "2 results in mbox$sfx");
lei_ok('q', '-a', '-o', "mboxcl2:$f", 's:nonexistent');
- is(grep(!/^#/, $lei_err), 0, "no errors on no results ($sfx)");
+ is(grep(!/^#/, $lei_err), 0, "no errors on no results ($sfx)")
+ or diag $lei_err;
my @s2 = grep(/^Subject:/, $cat->());
is_deeply(\@s2, \@s,
^ permalink raw reply related [relevance 71%]
* [PATCH 1/2] lei: disallow "\n" in local externals paths
@ 2021-03-19 12:35 53% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-19 12:35 UTC (permalink / raw)
To: meta
git 2.11 and earlier could not handle git directories with
newlines in them, nor does libgit2 support them.
Followup-to: d87dd0e679587043 ("config: reject `\n' in `inboxdir'")
---
lib/PublicInbox/LeiExternal.pm | 5 +++++
lib/PublicInbox/LeiXSearch.pm | 2 ++
t/lei-externals.t | 8 ++++++--
t/lei-mirror.t | 5 +++++
t/lei.t | 12 ++++++++++++
5 files changed, 30 insertions(+), 2 deletions(-)
diff --git a/lib/PublicInbox/LeiExternal.pm b/lib/PublicInbox/LeiExternal.pm
index 47791d4e..b5dd85e1 100644
--- a/lib/PublicInbox/LeiExternal.pm
+++ b/lib/PublicInbox/LeiExternal.pm
@@ -170,9 +170,14 @@ sub lei_add_external {
$self->fail(<<""); # TODO: did you mean "update-external?"
--mirror destination `$location' already exists
+ } elsif (-d $location) {
+ index($location, "\n") >= 0 and
+ return $self->fail("`\\n' not allowed in `$location'");
}
if ($location !~ m!\Ahttps?://! && !-d $location) {
$mirror // return $self->fail("$location not a directory");
+ index($location, "\n") >= 0 and
+ return $self->fail("`\\n' not allowed in `$location'");
$mirror = ext_canonicalize($mirror);
require PublicInbox::LeiMirror;
PublicInbox::LeiMirror->start($self, $mirror => $location);
diff --git a/lib/PublicInbox/LeiXSearch.pm b/lib/PublicInbox/LeiXSearch.pm
index 22c8026c..d95a218e 100644
--- a/lib/PublicInbox/LeiXSearch.pm
+++ b/lib/PublicInbox/LeiXSearch.pm
@@ -502,8 +502,10 @@ sub prepare_external {
return add_uri($self, URI->new($loc));
} elsif (-f "$loc/ei.lock") {
require PublicInbox::ExtSearch;
+ die "`\\n' not allowed in `$loc'\n" if index($loc, "\n") >= 0;
$loc = PublicInbox::ExtSearch->new($loc);
} elsif (-f "$loc/inbox.lock" || -d "$loc/public-inbox") {
+ die "`\\n' not allowed in `$loc'\n" if index($loc, "\n") >= 0;
require PublicInbox::Inbox; # v2, v1
$loc = bless { inboxdir => $loc }, 'PublicInbox::Inbox';
} else {
diff --git a/t/lei-externals.t b/t/lei-externals.t
index 2a92d101..1695ff0b 100644
--- a/t/lei-externals.t
+++ b/t/lei-externals.t
@@ -78,8 +78,12 @@ test_lei(sub {
ok(!-e $config_file && !-e $store_dir,
'nothing created by ls-external');
- ok(!lei('add-external', "$home/nonexistent",
- "fails on non-existent dir"));
+ ok(!lei('add-external', "$home/nonexistent"),
+ "fails on non-existent dir");
+ like($lei_err, qr/not a directory/, 'noted non-existence');
+ mkdir "$home/new\nline" or BAIL_OUT "mkdir: $!";
+ ok(!lei('add-external', "$home/new\nline"), "fails on newline");
+ like($lei_err, qr/`\\n' not allowed/, 'newline noted in error');
lei_ok('ls-external', \'ls-external works after add failure');
is($lei_out.$lei_err, '', 'ls-external still has no output');
my $cfg = PublicInbox::Config->new($cfg_path);
diff --git a/t/lei-mirror.t b/t/lei-mirror.t
index 1d113e3e..9769f31b 100644
--- a/t/lei-mirror.t
+++ b/t/lei-mirror.t
@@ -29,8 +29,13 @@ test_lei({ tmpdir => $tmpdir }, sub {
ok(!lei('add-external', $t2, '--mirror', "$http/t2/"),
'--mirror fails if reused') or diag "$lei_err.$lei_out = $?";
+ ok(!lei('add-external', "$home/t2\nnewline", '--mirror', "$http/t2/"),
+ '--mirror fails on newline');
+ like($lei_err, qr/`\\n' not allowed/, 'newline noted in error');
+
lei_ok('ls-external');
like($lei_out, qr!\Q$t2\E!, 'still in ls-externals');
+ unlike($lei_out, qr!\Qnewline\E!, 'newline entry not added');
ok(!lei('add-external', "$t2-fail", '-Lmedium'), '--mirror v2');
ok(!-d "$t2-fail", 'destination not created on failure');
diff --git a/t/lei.t b/t/lei.t
index 74a775ca..2bf4b862 100644
--- a/t/lei.t
+++ b/t/lei.t
@@ -133,6 +133,18 @@ my $test_fail = sub {
is($? >> 8, 1, 'chdir at end fails to /dev/null');
lei('-C', '/dev/null', 'q', 'whatever');
is($? >> 8, 1, 'chdir at beginning fails to /dev/null');
+
+ for my $lk (qw(ei inbox)) {
+ my $d = "$home/newline\n$lk";
+ mkdir $d;
+ open my $fh, '>', "$d/$lk.lock" or BAIL_OUT "open $d/$lk.lock";
+ for my $fl (qw(-I --only)) {
+ ok(!lei('q', $fl, $d, 'whatever'),
+ "newline $lk.lock fails with q $fl");
+ like($lei_err, qr/`\\n' not allowed/,
+ "error noted with q $fl");
+ }
+ }
SKIP: {
skip 'no curl', 3 unless which('curl');
lei(qw(q --only http://127.0.0.1:99999/bogus/ t:m));
^ permalink raw reply related [relevance 53%]
* [PATCH] lei: reuse LeiStore object on config changes
@ 2021-03-15 9:32 71% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-15 9:32 UTC (permalink / raw)
To: meta
Unless leistore.dir changes, the same LeiStore object
is should remain reusable and accessible to any clients
This seems to fix problems with t/lei-q-remote-import.t
occasionally getting stuck
---
lib/PublicInbox/LEI.pm | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 59a3338c..31d5b838 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -606,8 +606,10 @@ sub _lei_cfg ($;$) {
my $f = _config_path($self);
my @st = stat($f);
my $cur_st = @st ? pack('dd', $st[10], $st[7]) : ''; # 10:ctime, 7:size
+ my ($sto, $sto_dir);
if (my $cfg = $PATH2CFG{$f}) { # reuse existing object in common case
return ($self->{cfg} = $cfg) if $cur_st eq $cfg->{-st};
+ ($sto, $sto_dir) = @$cfg{qw(-lei_store leistore.dir)};
}
if (!@st) {
unless ($creat) {
@@ -625,6 +627,10 @@ sub _lei_cfg ($;$) {
bless $cfg, 'PublicInbox::Config';
$cfg->{-st} = $cur_st;
$cfg->{'-f'} = $f;
+ if ($sto && File::Spec->canonpath($sto_dir) eq
+ File::Spec->canonpath($cfg->{'leistore.dir'})) {
+ $cfg->{-lei_store} = $sto;
+ }
$self->{cfg} = $PATH2CFG{$f} = $cfg;
}
^ permalink raw reply related [relevance 71%]
* [PATCH] lei q: do not import unnecessarily from externals
@ 2021-03-14 11:12 40% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-14 11:12 UTC (permalink / raw)
To: meta
We only want to auto import messages that are exclusively in
remote externals. Messages in local externals are not
auto-imported to save space and reduce wear on storage device.
---
lib/PublicInbox/LeiSearch.pm | 37 ++++++++++++++++---------
lib/PublicInbox/LeiStore.pm | 52 +++++++++++++++++++++++++++++++----
lib/PublicInbox/LeiToMail.pm | 2 +-
lib/PublicInbox/LeiXSearch.pm | 10 ++++++-
t/lei-q-remote-import.t | 45 +++++++++++++++++++++++++++++-
5 files changed, 124 insertions(+), 22 deletions(-)
diff --git a/lib/PublicInbox/LeiSearch.pm b/lib/PublicInbox/LeiSearch.pm
index ceb3624b..2e3f10fd 100644
--- a/lib/PublicInbox/LeiSearch.pm
+++ b/lib/PublicInbox/LeiSearch.pm
@@ -44,29 +44,40 @@ sub content_key ($) {
sub _cmp_1st { # git->cat_async callback
my ($bref, $oid, $type, $size, $cmp) = @_; # cmp: [chash, found, smsg]
- return if defined($cmp->[1]->[0]); # $found->[0]
if (content_hash(PublicInbox::Eml->new($bref)) eq $cmp->[0]) {
- push @{$cmp->[1]}, $cmp->[2]->{num};
+ $cmp->[1]->{$oid} = $cmp->[2]->{num};
}
}
-# returns true if $eml is indexed by lei/store and keywords don't match
-sub kw_changed {
- my ($self, $eml, $new_kw_sorted) = @_;
+sub xids_for { # returns { OID => docid } mapping for $eml matches
+ my ($self, $eml, $min) = @_;
my ($chash, $mids) = content_key($eml);
- my $over = $self->over;
+ my @overs = ($self->over // $self->overs_all);
my $git = $self->git;
- my $found = [];
+ my $found = {};
for my $mid (@$mids) {
- my ($id, $prev);
- while (my $cur = $over->next_by_mid($mid, \$id, \$prev)) {
- $git->cat_async($cur->{blob}, \&_cmp_1st,
- [ $chash, $found, $cur ]);
- last if scalar(@$found);
+ for my $o (@overs) {
+ my ($id, $prev);
+ while (my $cur = $o->next_by_mid($mid, \$id, \$prev)) {
+ next if $found->{$cur->{blob}};
+ $git->cat_async($cur->{blob}, \&_cmp_1st,
+ [ $chash, $found, $cur ]);
+ if ($min && scalar(keys %$found) >= $min) {
+ $git->cat_async_wait;
+ return $found;
+ }
+ }
}
}
$git->cat_async_wait;
- my $num = $found->[0] // return;
+ scalar(keys %$found) ? $found : undef;
+}
+
+# returns true if $eml is indexed by lei/store and keywords don't match
+sub kw_changed {
+ my ($self, $eml, $new_kw_sorted) = @_;
+ my $found = xids_for($self, $eml, 1) // return;
+ my ($num) = values %$found;
my @cur_kw = msg_keywords($self, $num);
join("\0", @$new_kw_sorted) eq join("\0", @cur_kw) ? 0 : 1;
}
diff --git a/lib/PublicInbox/LeiStore.pm b/lib/PublicInbox/LeiStore.pm
index 6ace2ad1..aaee5874 100644
--- a/lib/PublicInbox/LeiStore.pm
+++ b/lib/PublicInbox/LeiStore.pm
@@ -213,6 +213,24 @@ sub set_eml {
add_eml($self, $eml, @kw) // set_eml_keywords($self, $eml, @kw);
}
+sub add_eml_maybe {
+ my ($self, $eml) = @_;
+ my $lxs = $self->{lxs_all_local} // die 'BUG: no {lxs_all_local}';
+ return if $lxs->xids_for($eml, 1);
+ add_eml($self, $eml);
+}
+
+# set or update keywords for external message, called via ipc_do
+sub set_xkw {
+ my ($self, $eml, $kw) = @_;
+ my $lxs = $self->{lxs_all_local} // die 'BUG: no {lxs_all_local}';
+ if ($lxs->xids_for($eml, 1)) { # is it in a local external?
+ # TODO: index keywords only
+ } else {
+ set_eml($self, $eml, @$kw);
+ }
+}
+
sub checkpoint {
my ($self, $wait) = @_;
if (my $im = $self->{im}) {
@@ -237,18 +255,40 @@ sub done {
sub ipc_atfork_child {
my ($self) = @_;
- my $lei = delete $self->{lei};
+ my $lei = $self->{lei};
$lei->lei_atfork_child(1) if $lei;
$self->SUPER::ipc_atfork_child;
}
+sub refresh_local_externals {
+ my ($self) = @_;
+ my $cfg = $self->{lei}->_lei_cfg or return;
+ my $cur_cfg = $self->{cur_cfg} // -1;
+ my $lxs = $self->{lxs_all_local};
+ if ($cfg != $cur_cfg || !$lxs) {
+ $lxs = PublicInbox::LeiXSearch->new;
+ my @loc = $self->{lei}->externals_each;
+ for my $loc (@loc) { # locals only
+ $lxs->prepare_external($loc) if -d $loc;
+ }
+ $self->{lxs_all_local} = $lxs;
+ $self->{cur_cfg} = $cfg;
+ }
+ ($lxs->{git_tmp} //= $lxs->git_tmp)->{git_dir};
+}
+
sub write_prepare {
my ($self, $lei) = @_;
- $self->ipc_lock_init;
- # Mail we import into lei are private, so headers filtered out
- # by -mda for public mail are not appropriate
- local @PublicInbox::MDA::BAD_HEADERS = ();
- $self->ipc_worker_spawn('lei_store', $lei->oldset, { lei => $lei });
+ unless ($self->{-ipc_req}) {
+ require PublicInbox::LeiXSearch;
+ $self->ipc_lock_init;
+ # Mail we import into lei are private, so headers filtered out
+ # by -mda for public mail are not appropriate
+ local @PublicInbox::MDA::BAD_HEADERS = ();
+ $self->ipc_worker_spawn('lei_store', $lei->oldset,
+ { lei => $lei });
+ }
+ $lei->{all_ext_git_dir} = $self->ipc_do('refresh_local_externals');
$lei->{sto} = $self;
}
diff --git a/lib/PublicInbox/LeiToMail.pm b/lib/PublicInbox/LeiToMail.pm
index 13764d79..587804bb 100644
--- a/lib/PublicInbox/LeiToMail.pm
+++ b/lib/PublicInbox/LeiToMail.pm
@@ -279,7 +279,7 @@ sub update_kw_maybe ($$$$) {
if ($x) {
$lei->{sto}->ipc_do('set_eml', $eml, @$kw);
} elsif (!defined($x)) {
- # TODO: xkw
+ $lei->{sto}->ipc_do('set_xkw', $eml, $kw);
}
}
diff --git a/lib/PublicInbox/LeiXSearch.pm b/lib/PublicInbox/LeiXSearch.pm
index f2c8c02e..22c8026c 100644
--- a/lib/PublicInbox/LeiXSearch.pm
+++ b/lib/PublicInbox/LeiXSearch.pm
@@ -97,6 +97,11 @@ sub recent {
sub over {}
+sub overs_all { # for xids_for
+ my ($self) = @_;
+ grep(defined, map { $_->over } locals($self))
+}
+
sub _mset_more ($$) {
my ($mset, $mo) = @_;
my $size = $mset->size;
@@ -204,7 +209,9 @@ sub query_mset { # non-parallel for non-"--threads" users
sub each_remote_eml { # callback for MboxReader->mboxrd
my ($eml, $self, $lei, $each_smsg) = @_;
- $lei->{sto}->ipc_do('add_eml', $eml) if $lei->{opt}->{'import-remote'};
+ if (my $sto = $self->{import_sto}) {
+ $sto->ipc_do('add_eml_maybe', $eml);
+ }
my $smsg = bless {}, 'PublicInbox::Smsg';
$smsg->populate($eml);
$smsg->parse_references($eml, mids($eml));
@@ -249,6 +256,7 @@ sub query_remote_mboxrd {
my $curl = PublicInbox::LeiCurl->new($lei, $self->{curl}) or return;
push @$curl, '-s', '-d', '';
my $each_smsg = $lei->{ovv}->ovv_each_smsg_cb($lei);
+ $self->{import_sto} = $lei->{sto} if $lei->{opt}->{'import-remote'};
for my $uri (@$uris) {
$lei->{-current_url} = $uri->as_string;
$lei->{-nr_remote_eml} = 0;
diff --git a/t/lei-q-remote-import.t b/t/lei-q-remote-import.t
index 4088b6ad..8b82579c 100644
--- a/t/lei-q-remote-import.t
+++ b/t/lei-q-remote-import.t
@@ -5,6 +5,7 @@ use strict; use v5.10.1; use PublicInbox::TestCommon;
require_git 2.6;
require_mods(qw(json DBD::SQLite Search::Xapian));
use PublicInbox::MboxReader;
+use PublicInbox::InboxWritable;
my ($ro_home, $cfg_path) = setup_public_inboxes;
my $sock = tcp_server;
my ($tmpdir, $for_destroy) = tmpdir;
@@ -36,7 +37,8 @@ test_lei({ tmpdir => $tmpdir }, sub {
is_deeply($slurp_emls->($o), $exp1, 'got results after remote search');
unlink $o or BAIL_OUT $!;
lei_ok(@cmd);
- ok(-f $o && -s _, 'output exists after import but is not empty');
+ ok(-f $o && -s _, 'output exists after import but is not empty') or
+ diag $lei_err;
is_deeply($slurp_emls->($o), $exp1, 'got results w/o remote search');
unlink $o or BAIL_OUT $!;
@@ -58,5 +60,46 @@ test_lei({ tmpdir => $tmpdir }, sub {
unlink "$o.lock" or BAIL_OUT $!;
lei_ok(@cmd, '--lock=dotlock,timeout=0.000001',
\'succeeds after lock removal');
+
+ # XXX memoize this external creation
+ my $inboxdir = "$ENV{HOME}/tmp_git";
+ my $ibx = PublicInbox::InboxWritable->new({
+ name => 'tmp',
+ -primary_address => 'lei@example.com',
+ inboxdir => $inboxdir,
+ indexlevel => 'medium',
+ }, { nproc => 1 });
+ my $im = $ibx->importer(0);
+ $im->add(eml_load('t/utf8.eml')) or BAIL_OUT '->add';
+ $im->done;
+
+ run_script(['-index', $inboxdir], undef) or BAIL_OUT '-init';
+ lei_ok(qw(add-external -q), $inboxdir);
+ lei_ok(qw(q -o), "mboxrd:$o", '--only', $url,
+ 'm:testmessage@example.com');
+ ok(-s $o, 'got result from remote external');
+ my $exp = eml_load('t/utf8.eml');
+ is_deeply($slurp_emls->($o), [$exp], 'got expected result');
+ lei_ok(qw(q --no-external -o), "mboxrd:/dev/stdout",
+ 'm:testmessage@example.com');
+ is($lei_out, '', 'message not imported when in local external');
+
+ open $fh, '>', $o or BAIL_OUT;
+ print $fh <<'EOF' or BAIL_OUT;
+From a@z Mon Sep 17 00:00:00 2001
+From: nobody@localhost
+Date: Sat, 13 Mar 2021 18:23:01 +0600
+Message-ID: <never-before-seen@example.com>
+Status: RO
+
+whatever
+EOF
+ close $fh or BAIL_OUT;
+ lei_ok(qw(q -o), "mboxrd:$o", 'm:testmessage@example.com');
+ is_deeply($slurp_emls->($o), [$exp],
+ 'got expected result after clobber') or diag $lei_err;
+ lei_ok(qw(q -o mboxrd:/dev/stdout m:never-before-seen@example.com));
+ like($lei_out, qr/seen\@example\.com>\nStatus: OR\n\nwhatever/sm,
+ '--import-before imported totally unseen message');
});
done_testing;
^ permalink raw reply related [relevance 40%]
* [PATCH 1/3] lei: add help + completion for --no-external
2021-03-12 10:39 71% [PATCH 0/3] lei CLI option updates Eric Wong
@ 2021-03-12 10:39 71% ` Eric Wong
2021-03-12 10:39 51% ` [PATCH 2/3] lei: rearrange OPT_DESC and drop some TBD switches Eric Wong
2021-03-12 10:39 62% ` [PATCH 3/3] lei q: mbox*: disable changing parallelism, add --rsyncable Eric Wong
2 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-12 10:39 UTC (permalink / raw)
To: meta
I just needed it.
---
lib/PublicInbox/LEI.pm | 1 +
1 file changed, 1 insertion(+)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 50c0a885..ddc27361 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -228,6 +228,7 @@ my %OPTDESC = (
'globoff|g' => "do not match locations using '*?' wildcards ".
"and\xa0'[]'\x{a0}ranges",
'verbose|v+' => 'be more verbose',
+'external!' => 'do not use externals',
'solve!' => 'do not attempt to reconstruct blobs from emails',
'torsocks=s' => ['VAL|auto|no|yes',
'whether or not to wrap git and curl commands with torsocks'],
^ permalink raw reply related [relevance 71%]
* [PATCH 0/3] lei CLI option updates
@ 2021-03-12 10:39 71% Eric Wong
2021-03-12 10:39 71% ` [PATCH 1/3] lei: add help + completion for --no-external Eric Wong
` (2 more replies)
0 siblings, 3 replies; 200+ results
From: Eric Wong @ 2021-03-12 10:39 UTC (permalink / raw)
To: meta
Mentally, I still use "--no-externals" (plural),
but I'm not sure if that's necessary to support with
completion, now...
Eric Wong (3):
lei: add help + completion for --no-external
lei: rearrange OPT_DESC and drop some TBD switches
lei q: mbox*: disable changing parallelism, add --rsyncable
lib/PublicInbox/LEI.pm | 41 +++++++++++++++---------------------
lib/PublicInbox/LeiHelp.pm | 2 +-
lib/PublicInbox/LeiToMail.pm | 6 +++---
3 files changed, 21 insertions(+), 28 deletions(-)
^ permalink raw reply [relevance 71%]
* [PATCH 3/3] lei q: mbox*: disable changing parallelism, add --rsyncable
2021-03-12 10:39 71% [PATCH 0/3] lei CLI option updates Eric Wong
2021-03-12 10:39 71% ` [PATCH 1/3] lei: add help + completion for --no-external Eric Wong
2021-03-12 10:39 51% ` [PATCH 2/3] lei: rearrange OPT_DESC and drop some TBD switches Eric Wong
@ 2021-03-12 10:39 62% ` Eric Wong
2 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-12 10:39 UTC (permalink / raw)
To: meta
Unfortunately, being mairix-compatible with --threads means we
can't change thread-count of gzip, bzip2, or xz when writing to
compressed mbox with a --threads= parameter. It's probably not
worth changing, anyways, so another switch or additional value
for --jobs= won't be added.
While we're in the area, add --rsyncable support since
most installations of gzip support it nowadays.
Fixes: 5beb4a5f6585acd ("lei: replace --thread with --threads")
---
lib/PublicInbox/LEI.pm | 2 +-
lib/PublicInbox/LeiToMail.pm | 6 +++---
2 files changed, 4 insertions(+), 4 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 9bf60ad4..59a3338c 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -113,7 +113,7 @@ our %CMD = ( # sorted in order of importance/use:
qw(save-as=s output|mfolder|o=s format|f=s dedupe|d=s threads|t+
sort|s=s reverse|r offset=i remote! local! external! pretty
include|I=s@ exclude=s@ only=s@ jobs|j=s globoff|g augment|a
- import-remote! import-before! lock=s@
+ import-remote! import-before! lock=s@ rsyncable
alert=s@ mua=s no-torsocks torsocks=s verbose|v+ quiet|q C=s@),
PublicInbox::LeiQuery::curl_opt(), opt_dash('limit|n=i', '[0-9]+') ],
diff --git a/lib/PublicInbox/LeiToMail.pm b/lib/PublicInbox/LeiToMail.pm
index 13b4f672..13764d79 100644
--- a/lib/PublicInbox/LeiToMail.pm
+++ b/lib/PublicInbox/LeiToMail.pm
@@ -170,9 +170,9 @@ sub reap_compress { # dwaitpid callback
# { foo => '' } means "--foo" is passed to the command-line,
# otherwise { foo => '--bar' } passes "--bar"
our %zsfx2cmd = (
- gz => [ qw(GZIP pigz gzip), { rsyncable => '', threads => '-p' } ],
+ gz => [ qw(GZIP pigz gzip), { rsyncable => '' } ],
bz2 => [ 'bzip2', {} ],
- xz => [ 'xz', { threads => '-T' } ],
+ xz => [ 'xz', {} ],
# XXX does anybody care for these? I prefer zstd on entire FSes,
# so it's probably not necessary on a per-file basis
# zst => [ 'zstd', { -default => [ qw(-q) ], # it's noisy by default
@@ -203,7 +203,7 @@ sub zsfx2cmd ($$$) {
my $switch = $cmd_opt->{rsyncable} // next;
push @cmd, '--'.($switch || $bool);
}
- for my $key (qw(threads)) { # support compression level?
+ for my $key (qw(rsyncable)) { # support compression level?
my $switch = $cmd_opt->{$key} // next;
my $val = $lei->{opt}->{$key} // next;
push @cmd, $switch, $val;
^ permalink raw reply related [relevance 62%]
* [PATCH 2/3] lei: rearrange OPT_DESC and drop some TBD switches
2021-03-12 10:39 71% [PATCH 0/3] lei CLI option updates Eric Wong
2021-03-12 10:39 71% ` [PATCH 1/3] lei: add help + completion for --no-external Eric Wong
@ 2021-03-12 10:39 51% ` Eric Wong
2021-03-12 10:39 62% ` [PATCH 3/3] lei q: mbox*: disable changing parallelism, add --rsyncable Eric Wong
2 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-12 10:39 UTC (permalink / raw)
To: meta
It'll be easier for us to have the option-spec in front of the
command instead of the other way around. The option-spec in
front makes it easier to sort and keep track of potentially
confusing/ambiguous use of command-line switches between
different commands.
We'll also update some of the proposed switches while we're
at it.
---
lib/PublicInbox/LEI.pm | 38 +++++++++++++++-----------------------
lib/PublicInbox/LeiHelp.pm | 2 +-
2 files changed, 16 insertions(+), 24 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index ddc27361..9bf60ad4 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -133,7 +133,7 @@ our %CMD = ( # sorted in order of importance/use:
qw(prune quiet|q C=s@) ],
'ls-query' => [ '[FILTER...]', 'list saved search queries',
- qw(name-only format|f=s z C=s@) ],
+ qw(name-only format|f=s C=s@) ],
'rm-query' => [ 'QUERY_NAME', 'remove a saved search', qw(C=s@) ],
'mv-query' => [ qw(OLD_NAME NEW_NAME), 'rename a saved search', qw(C=s@) ],
@@ -240,8 +240,7 @@ my %OPTDESC = (
'dedupe|d=s' => ['STRATEGY|content|oid|mid|none',
'deduplication strategy'],
-'show threads|t' => 'display entire thread a message belongs to',
-'q threads|t+' =>
+'threads|t+' =>
'return all messages in the same threads as the actual match(es)',
'want|w=s@' => [ 'PREFIX|dfpost|dfn', # common ones in help...
@@ -260,17 +259,11 @@ my %OPTDESC = (
'mua=s' => [ 'CMD',
"MUA to run on --output Maildir or mbox (e.g.\xa0`mutt\xa0-f\xa0%f')" ],
-'show format|f=s' => [ 'OUT|plain|raw|html|mboxrd|mboxcl2|mboxcl',
- 'message/object output format' ],
-'mark format|f=s' => $stdin_formats,
-'forget format|f=s' => $stdin_formats,
-
-'add-external inbox-version=i' => [ 'NUM|1|2',
+'inbox-version=i' => [ 'NUM|1|2',
'force a public-inbox version with --mirror'],
-'add-external mirror=s' => [ 'URL', 'mirror a public-inbox'],
+'mirror=s' => [ 'URL', 'mirror a public-inbox'],
# public-inbox-index options
-'add-external jobs|j=i' => 'set parallelism when indexing after --mirror',
'fsync!' => 'speed up indexing after --mirror, risk index corruption',
'compact' => 'run compact index after mirroring',
'indexlevel|L=s' => [ 'LEVEL|full|medium|basic',
@@ -284,23 +277,22 @@ my %OPTDESC = (
'skip-docdata' =>
'drop compatibility w/ public-inbox <1.6 to save ~1.5% space',
-'q format|f=s' => [
+'format|f=s q' => [
'OUT|maildir|mboxrd|mboxcl2|mboxcl|mboxo|html|json|jsonl|concatjson',
'specify output format, default depends on --output'],
-'q exclude=s@' => [ 'LOCATION',
+'exclude=s@ q' => [ 'LOCATION',
'exclude specified external(s) from search' ],
-'q include|I=s@' => [ 'LOCATION',
+'include|I=s@ q' => [ 'LOCATION',
'include specified external(s) in search' ],
-'q only=s@' => [ 'LOCATION',
+'only=s@ q' => [ 'LOCATION',
'only use specified external(s) for search' ],
-
-'q jobs=s' => [ '[SEARCH_JOBS][,WRITER_JOBS]',
+'jobs=s q' => [ '[SEARCH_JOBS][,WRITER_JOBS]',
'control number of search and writer jobs' ],
+'jobs|j=i add-external' => 'set parallelism when indexing after --mirror',
-'import format|f=s' => $stdin_formats,
-
-'ls-query format|f=s' => $ls_format,
-'ls-external format|f=s' => $ls_format,
+'in-format|F=s' => $stdin_formats,
+'format|f=s ls-query' => $ls_format,
+'format|f=s ls-external' => $ls_format,
'limit|n=i@' => ['NUM', 'limit on number of matches (default: 10000)' ],
'offset=i' => ['OFF', 'search result offset (default: 0)'],
@@ -770,7 +762,7 @@ sub lei__complete {
my $x = length > 1 ? "--$_" : "-$_";
$x eq $cur ? () : $x;
} grep(!/_/, split(/\|/, $_, -1)) # help|h
- } grep { $OPTDESC{"$cmd\t$_"} || $OPTDESC{$_} } @spec);
+ } grep { $OPTDESC{"$_\t$cmd"} || $OPTDESC{$_} } @spec);
} elsif ($cmd eq 'config' && !@argv && !$CONFIG_KEYS{$cur}) {
puts $self, grep(/$re/, keys %CONFIG_KEYS);
}
@@ -785,7 +777,7 @@ sub lei__complete {
# (TODO: completion for external paths)
shift(@v) if uc($v[0]) eq $v[0];
@v;
- } grep(/\A(?:$cmd\t|)(?:[\w-]+\|)*$opt\b/, keys %OPTDESC);
+ } grep(/\A(?:[\w-]+\|)*$opt\b.*?(?:\t$cmd)?\z/, keys %OPTDESC);
}
$cmd =~ tr/-/_/;
if (my $sub = $self->can("_complete_$cmd")) {
diff --git a/lib/PublicInbox/LeiHelp.pm b/lib/PublicInbox/LeiHelp.pm
index a654e1c2..be31c2a8 100644
--- a/lib/PublicInbox/LeiHelp.pm
+++ b/lib/PublicInbox/LeiHelp.pm
@@ -20,7 +20,7 @@ sub call {
my @opt_desc;
my $lpad = 2;
for my $sw (grep { !ref } @info) { # ("prio=s", "z", $GLP_PASS)
- my $desc = $OPTDESC->{"$cmd\t$sw"} // $OPTDESC->{$sw} // next;
+ my $desc = $OPTDESC->{"$sw\t$cmd"} // $OPTDESC->{$sw} // next;
my $arg_vals = '';
($arg_vals, $desc) = @$desc if ref($desc) eq 'ARRAY';
^ permalink raw reply related [relevance 51%]
* Re: final "null" in "lei q" JSON output
2021-03-11 22:43 71% final "null" in "lei q" JSON output Eric Wong
@ 2021-03-12 4:22 71% ` Kyle Meyer
0 siblings, 0 replies; 200+ results
From: Kyle Meyer @ 2021-03-12 4:22 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
Eric Wong writes:
> The standard JSON (not line-delimited) output is an array with
> a "null" element as the last element.
>
> I don't really like it, but it seems necessary if we're
> doing parallel writes to stdout and JSON doesn't allow
> trailing commas.
I dunno, it doesn't really bother me, and it seems like a fine solution
to the problem.
> So I don't think that part is considered part of the stable
> output.
I think that's fair enough.
> Maybe it could be tweaked to show stats, or something else,
> I don't know.
Nothing too useful comes to mind, but if someone does think of
something, substituting it for the trailing null makes sense to me.
Until then, my vote would be to leave the helpful creature be :)
^ permalink raw reply [relevance 71%]
* final "null" in "lei q" JSON output
@ 2021-03-11 22:43 71% Eric Wong
2021-03-12 4:22 71% ` Kyle Meyer
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2021-03-11 22:43 UTC (permalink / raw)
To: meta
The standard JSON (not line-delimited) output is an array with
a "null" element as the last element.
I don't really like it, but it seems necessary if we're
doing parallel writes to stdout and JSON doesn't allow
trailing commas.
So I don't think that part is considered part of the stable
output.
Maybe it could be tweaked to show stats, or something else,
I don't know.
Thoughts?
^ permalink raw reply [relevance 71%]
* [PATCH 4/5] lei import: skip trashed Maildir messages
2021-03-10 13:23 67% ` [PATCH 3/5] lei import: simplify Maildir handling Eric Wong
@ 2021-03-10 13:23 68% ` Eric Wong
1 sibling, 0 replies; 200+ results
From: Eric Wong @ 2021-03-10 13:23 UTC (permalink / raw)
To: meta
This matches IMAP behavior in NetReader in skipping \\Deleted
messages. Since lei may be used for personal, non-public mail;
Draft messages are NOT skipped by "lei import".
---
lib/PublicInbox/MdirReader.pm | 1 +
t/lei-import-maildir.t | 7 +++++++
2 files changed, 8 insertions(+)
diff --git a/lib/PublicInbox/MdirReader.pm b/lib/PublicInbox/MdirReader.pm
index 44724af1..06806e80 100644
--- a/lib/PublicInbox/MdirReader.pm
+++ b/lib/PublicInbox/MdirReader.pm
@@ -57,6 +57,7 @@ sub maildir_each_eml ($$;@) {
opendir my $dh, $pfx or return;
while (defined(my $bn = readdir($dh))) {
my $fl = maildir_basename_flags($bn) // next;
+ next if index($fl, 'T') >= 0;
my $f = $pfx.$bn;
my $eml = eml_from_path($f) or next;
my @kw = sort(map { $c2kw{$_} // () } split(//, $fl));
diff --git a/t/lei-import-maildir.t b/t/lei-import-maildir.t
index a3796491..bd89677a 100644
--- a/t/lei-import-maildir.t
+++ b/t/lei-import-maildir.t
@@ -29,5 +29,12 @@ test_lei(sub {
like($res->[0]->{'s'}, qr/use boolean/, 'got expected result');
is_deeply($res->[0]->{kw}, ['answered', 'seen'], 'keywords set');
is($res->[1], undef, 'only got one result');
+
+ symlink(abs_path('t/utf8.eml'), "$md/cur/u:2,ST") or
+ BAIL_OUT "symlink $md $!";
+ lei_ok('import', "maildir:$md", \'import Maildir w/ trashed message');
+ lei_ok(qw(q -d none m:testmessage@example.com));
+ $res = json_utf8->decode($lei_out);
+ is_deeply($res, [ undef ], 'trashed message not imported');
});
done_testing;
^ permalink raw reply related [relevance 68%]
* [PATCH 3/5] lei import: simplify Maildir handling
@ 2021-03-10 13:23 67% ` Eric Wong
2021-03-10 13:23 68% ` [PATCH 4/5] lei import: skip trashed Maildir messages Eric Wong
1 sibling, 0 replies; 200+ results
From: Eric Wong @ 2021-03-10 13:23 UTC (permalink / raw)
To: meta
Having a one-off Maildir functionality in LeiStore doesn't seem
worth the maintenance burden, especially given an upcoming
change to skip trashed messages.
I expect this will hurt performance slightly with extra IPC
overhead for the socket copy, but "lei import" may eventually
become rare or at least not hit messages redundantly.
---
lib/PublicInbox/LeiImport.pm | 8 ++++----
lib/PublicInbox/LeiStore.pm | 6 ------
2 files changed, 4 insertions(+), 10 deletions(-)
diff --git a/lib/PublicInbox/LeiImport.pm b/lib/PublicInbox/LeiImport.pm
index 23cecd53..815788b3 100644
--- a/lib/PublicInbox/LeiImport.pm
+++ b/lib/PublicInbox/LeiImport.pm
@@ -147,9 +147,9 @@ error reading $input: $!
$lei->child_error(1 << 8, "$input: $@") if $@;
}
-sub _import_maildir { # maildir_each_file cb
- my ($f, $sto, $set_kw) = @_;
- $sto->ipc_do('set_eml_from_maildir', $f, $set_kw);
+sub _import_maildir { # maildir_each_eml cb
+ my ($f, $kw, $eml, $sto, $set_kw) = @_;
+ $sto->ipc_do('set_eml', $eml, $set_kw ? @$kw : ());
}
sub _import_net { # imap_each, nntp_each cb
@@ -181,7 +181,7 @@ sub import_path_url {
return $lei->fail(<<EOM) if $ifmt && $ifmt ne 'maildir';
$input appears to a be a maildir, not $ifmt
EOM
- PublicInbox::MdirReader::maildir_each_file($input,
+ PublicInbox::MdirReader::maildir_each_eml($input,
\&_import_maildir,
$lei->{sto}, $lei->{opt}->{kw});
} else {
diff --git a/lib/PublicInbox/LeiStore.pm b/lib/PublicInbox/LeiStore.pm
index 92c29100..6ace2ad1 100644
--- a/lib/PublicInbox/LeiStore.pm
+++ b/lib/PublicInbox/LeiStore.pm
@@ -213,12 +213,6 @@ sub set_eml {
add_eml($self, $eml, @kw) // set_eml_keywords($self, $eml, @kw);
}
-sub set_eml_from_maildir {
- my ($self, $f, $set_kw) = @_;
- my $eml = eml_from_path($f) or return;
- set_eml($self, $eml, $set_kw ? maildir_keywords($f) : ());
-}
-
sub checkpoint {
my ($self, $wait) = @_;
if (my $im = $self->{im}) {
^ permalink raw reply related [relevance 67%]
* Re: release timelines (-extindex, JMAP, lei)
2021-03-08 21:33 71% ` Konstantin Ryabitsev
@ 2021-03-08 22:16 71% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-08 22:16 UTC (permalink / raw)
To: Konstantin Ryabitsev; +Cc: meta
Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> On Fri, Mar 05, 2021 at 10:20:19PM +0000, Eric Wong wrote:
> > * lei saved searches can probably done quick for 1.7,
> > but without full keywords support for externals...
>
> I was wondering if lei should be part of the same suite as public-inbox proper
> as opposed to a standalone (or interdependent) client tool. Unless I'm
> mistaken, someone running a public-inbox origin or mirror server wouldn't
> necessarily be an active lei user. Decoupling them from each-other would allow
> different release cadence, no?
I've considered it, yes, but there's a lot of shared code and
writing release notes/planning/packaging for one project is
challenging enough. As one might imagine, the non-coding
aspects of the project are the least interesting to me so I try
to avoid it as much as possible.
And I'm planning on sharing more code based on the work I've
done so far around IPC, too.
> > In any case, I need to take a few days away to clear my head.
>
> Please take care!
>
> Thanks for your effort on this project.
You're welcome. Should be back to coding in a day or so,
still dealing with leaks of the non-memory variety :x
^ permalink raw reply [relevance 71%]
* Re: release timelines (-extindex, JMAP, lei)
2021-03-05 22:20 64% release timelines (-extindex, JMAP, lei) Eric Wong
2021-03-06 18:31 71% ` Kyle Meyer
@ 2021-03-08 21:33 71% ` Konstantin Ryabitsev
2021-03-08 22:16 71% ` Eric Wong
1 sibling, 1 reply; 200+ results
From: Konstantin Ryabitsev @ 2021-03-08 21:33 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
On Fri, Mar 05, 2021 at 10:20:19PM +0000, Eric Wong wrote:
> So I think the -extindex stuff might be stable and suitable for
> general consumption. The HTML WWW UI around -extindex has some
> rough edges but nothing that would take too much effort to fix.
\o/
> But I'm deeply worried about unleashing a new on-disk format
> that's insufficient and being stuck supporting it forever
> (as I am with v1 inboxes)...
>
> * JMAP is going to be a more effort, but I think our current
> on-disk data model is OK or at least extensible enough for it.
> I might delay JMAP until 1.8.
>
> JMAP will be significantly less effort than inventing something
> new (and one-off) with GraphQL or REST. And it would be less
> effort for client authors, as well; since client code can be
> used for non-public-inbox servers, too.
I don't really have any specific opinion on this matter. I don't really know
of any other provider outside of Fastmail that uses JMAP, but it *is* a
published IETF standard around RFC2822 messages, so it makes sense to use it
for this purpose.
> * lei saved searches can probably done quick for 1.7,
> but without full keywords support for externals...
I was wondering if lei should be part of the same suite as public-inbox proper
as opposed to a standalone (or interdependent) client tool. Unless I'm
mistaken, someone running a public-inbox origin or mirror server wouldn't
necessarily be an active lei user. Decoupling them from each-other would allow
different release cadence, no?
> In any case, I need to take a few days away to clear my head.
Please take care!
Thanks for your effort on this project.
Best,
-K
^ permalink raw reply [relevance 71%]
* [PATCH] lei q: remove angle brackets around Message-IDs
2021-03-06 18:26 71% ` Kyle Meyer
@ 2021-03-08 8:08 53% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-08 8:08 UTC (permalink / raw)
To: Kyle Meyer; +Cc: meta
Kyle Meyer <kyle@kyleam.com> wrote:
> Eric Wong writes:
>
> > I'm thinking these shouldn't include angle brackets:
> >
> > "m": "<20210228122528.18552-2-e@80x24.org>",
> > "refs": ["<20210228122528.18552-1-e@80x24.org>"],
> >
> > Using angle brackets on the command-line requires quoting to
> > disambiguate against redirects, so it's a pain. Leaving the
> > brackets in still works because of how Xapian's query parser
> > works, not because of anything we do on our end.
>
> I think it'd be nice to drop the brackets from a noise perspective too.
Yes, we don't include them in name email address pairs, either.
> Also, does m: work with brackets? Trying it out with a recent message
> ID:
>
> $ lei q -q -I https://public-inbox.org/meta/ -f ldjson \
> '20210304203352.pd5mcg5pw4u2epzl@pengutronix.de'
> {"blob":"87304c8a8cae8ce400443b56309427aeee601505",...}
>
> $ lei q -q -I https://public-inbox.org/meta/ -f ldjson \
> m:'20210304203352.pd5mcg5pw4u2epzl@pengutronix.de'
> {"blob":"87304c8a8cae8ce400443b56309427aeee601505",...}
>
> $ lei q -q -I https://public-inbox.org/meta/ -f ldjson \
> '<20210304203352.pd5mcg5pw4u2epzl@pengutronix.de>'
> {"blob":"87304c8a8cae8ce400443b56309427aeee601505",...}
>
> $ lei q -q -I https://public-inbox.org/meta/ -f ldjson \
> m:'<20210304203352.pd5mcg5pw4u2epzl@pengutronix.de>'
> # no results
Odd, I'm not sure about that one... It's probably something
the Xapian query parser is doing internally and nothing on
our end...
> > Since the actual headers are "Message-ID" and "References", (and
> > not "m" or "refs"), I think it's clear that we don't have to
> > match the raw mail contents exactly. We RFC 2047 decode
> > "f|t|c|s" fields anyways instead of showing the raw values,
> > so more precedence for leaving out <>.
>
> Fwiw I don't think leaving out the brackets would be a source of
> confusion.
Agreed. And I'm now wondering if we should start indexing
"References:" to be a searchable header with the "refs:" prefix
(but also wary about increasing disk space usage as a result...)
In any case, this denoises the output a bit:
------------8<----------
Subject: [PATCH] lei q: remove angle brackets around Message-IDs
They're unnecessary visual noise, and angle brackets don't
always work as intended when going through Xapian's query
parser.
Since we already use "m:" and "refs:" instead of the actual
header names, it should be obvious we're at liberty to
abbreviate such things
Link: https://public-inbox.org/meta/20210304184348.GA19350@dcvr/
---
lib/PublicInbox/LeiOverview.pm | 5 ++---
t/lei-externals.t | 2 +-
t/lei-q-thread.t | 8 ++++----
xt/net_writer-imap.t | 2 +-
4 files changed, 8 insertions(+), 9 deletions(-)
diff --git a/lib/PublicInbox/LeiOverview.pm b/lib/PublicInbox/LeiOverview.pm
index 4db1d8c8..01556273 100644
--- a/lib/PublicInbox/LeiOverview.pm
+++ b/lib/PublicInbox/LeiOverview.pm
@@ -141,17 +141,16 @@ sub _unbless_smsg {
$smsg->{dt} = _iso8601(delete $smsg->{ds}); # JMAP UTCDate
$smsg->{pct} = get_pct($mitem) if $mitem;
if (my $r = delete $smsg->{references}) {
- $smsg->{refs} = [ map { "<$_>" } ($r =~ m/$MID_EXTRACT/go) ];
+ $smsg->{refs} = [ map { $_ } ($r =~ m/$MID_EXTRACT/go) ];
}
if (my $m = delete($smsg->{mid})) {
- $smsg->{'m'} = "<$m>";
+ $smsg->{'m'} = $m;
}
for my $f (qw(from to cc)) {
my $v = delete $smsg->{$f} or next;
$smsg->{substr($f, 0, 1)} = pairs($v);
}
$smsg->{'s'} = delete $smsg->{subject};
- # can we be bothered to parse From/To/Cc into arrays?
scalar { %$smsg }; # unbless
}
diff --git a/t/lei-externals.t b/t/lei-externals.t
index 29667640..2a92d101 100644
--- a/t/lei-externals.t
+++ b/t/lei-externals.t
@@ -25,7 +25,7 @@ SKIP: {
lei_ok(@cmd, \"query $url");
is($lei_err, '', "no errors on $url");
my $res = json_utf8->decode($lei_out);
- is($res->[0]->{'m'}, "<$mid>", "got expected mid from $url") or
+ is($res->[0]->{'m'}, $mid, "got expected mid from $url") or
skip 'further remote tests', 1;
lei_ok(@cmd, 'd:..20101002', \'no results, no error');
is($lei_err, '', 'no output on 404, matching local FS behavior');
diff --git a/t/lei-q-thread.t b/t/lei-q-thread.t
index 28c639f5..e24fb2cb 100644
--- a/t/lei-q-thread.t
+++ b/t/lei-q-thread.t
@@ -27,9 +27,9 @@ test_lei(sub {
is(scalar(@$res), 3, 'got 2 results');
pop @$res;
my %m = map { $_->{'m'} => $_ } @$res;
- is_deeply($m{'<testmessage@example.com>'}->{kw}, ['seen'],
+ is_deeply($m{'testmessage@example.com'}->{kw}, ['seen'],
'flag set in direct hit');
- 'TODO' or is_deeply($m{'<a-reply@miss>'}->{kw}, ['draft'],
+ 'TODO' or is_deeply($m{'a-reply@miss'}->{kw}, ['draft'],
'flag set in thread hit');
lei_ok qw(q -t -t m:testmessage@example.com);
@@ -37,9 +37,9 @@ test_lei(sub {
is(scalar(@$res), 3, 'got 2 results with -t -t');
pop @$res;
%m = map { $_->{'m'} => $_ } @$res;
- is_deeply($m{'<testmessage@example.com>'}->{kw}, ['flagged', 'seen'],
+ is_deeply($m{'testmessage@example.com'}->{kw}, ['flagged', 'seen'],
'flagged set in direct hit');
- 'TODO' or is_deeply($m{'<testmessage@example.com>'}->{kw}, ['draft'],
+ 'TODO' or is_deeply($m{'testmessage@example.com'}->{kw}, ['draft'],
'flagged set in direct hit');
lei_ok qw(q -tt m:testmessage@example.com --only), "$ro_home/t2";
$res = json_utf8->decode($lei_out);
diff --git a/xt/net_writer-imap.t b/xt/net_writer-imap.t
index c24fa993..3631d932 100644
--- a/xt/net_writer-imap.t
+++ b/xt/net_writer-imap.t
@@ -168,7 +168,7 @@ test_lei(sub {
is_deeply($empty, [], 'clobbered folder');
lei_ok qw(q -o /dev/stdout m:testmessage@example.com --no-external);
$res = json_utf8->decode($lei_out)->[0];
- is_deeply([@$res{qw(m kw)}], ['<testmessage@example.com>', ['seen']],
+ is_deeply([@$res{qw(m kw)}], ['testmessage@example.com', ['seen']],
'kw set');
});
^ permalink raw reply related [relevance 53%]
* Re: release timelines (-extindex, JMAP, lei)
2021-03-06 18:31 71% ` Kyle Meyer
@ 2021-03-08 2:54 71% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-08 2:54 UTC (permalink / raw)
To: Kyle Meyer; +Cc: meta
Kyle Meyer <kyle@kyleam.com> wrote:
> Eric Wong writes:
>
> > But I'm deeply worried about unleashing a new on-disk format
> > that's insufficient and being stuck supporting it forever
> > (as I am with v1 inboxes)...
> [...]
> > * lei has a bunch of rough edges and I'm not comfortable declaring
> > it as supported, especially when there's a risk of data loss to
> > users.
>
> What are your thoughts on marking all things lei with a big
> experimental / subject-to-change / may-eat-your-data warning?
Yes, definitely experimental and may-eat-your-data.
I hope the subject-to-change bit can be minimized, though I'll
drop the '<>' in "lei q" JSON before release.
I'm also considering diverging from mairix in making --augment
the default behavior in "lei q". That would make things
significantly safer, and --no-augment would be required to
clobber existing outputs.
^ permalink raw reply [relevance 71%]
* Re: release timelines (-extindex, JMAP, lei)
2021-03-05 22:20 64% release timelines (-extindex, JMAP, lei) Eric Wong
@ 2021-03-06 18:31 71% ` Kyle Meyer
2021-03-08 2:54 71% ` Eric Wong
2021-03-08 21:33 71% ` Konstantin Ryabitsev
1 sibling, 1 reply; 200+ results
From: Kyle Meyer @ 2021-03-06 18:31 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
Eric Wong writes:
> But I'm deeply worried about unleashing a new on-disk format
> that's insufficient and being stuck supporting it forever
> (as I am with v1 inboxes)...
[...]
> * lei has a bunch of rough edges and I'm not comfortable declaring
> it as supported, especially when there's a risk of data loss to
> users.
What are your thoughts on marking all things lei with a big
experimental / subject-to-change / may-eat-your-data warning?
^ permalink raw reply [relevance 71%]
* Re: angle brackets in "m:" and "refs:" in "lei q" JSON
2021-03-04 18:43 71% angle brackets in "m:" and "refs:" in "lei q" JSON Eric Wong
@ 2021-03-06 18:26 71% ` Kyle Meyer
2021-03-08 8:08 53% ` [PATCH] lei q: remove angle brackets around Message-IDs Eric Wong
0 siblings, 1 reply; 200+ results
From: Kyle Meyer @ 2021-03-06 18:26 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
Eric Wong writes:
> I'm thinking these shouldn't include angle brackets:
>
> "m": "<20210228122528.18552-2-e@80x24.org>",
> "refs": ["<20210228122528.18552-1-e@80x24.org>"],
>
> Using angle brackets on the command-line requires quoting to
> disambiguate against redirects, so it's a pain. Leaving the
> brackets in still works because of how Xapian's query parser
> works, not because of anything we do on our end.
I think it'd be nice to drop the brackets from a noise perspective too.
Also, does m: work with brackets? Trying it out with a recent message
ID:
$ lei q -q -I https://public-inbox.org/meta/ -f ldjson \
'20210304203352.pd5mcg5pw4u2epzl@pengutronix.de'
{"blob":"87304c8a8cae8ce400443b56309427aeee601505",...}
$ lei q -q -I https://public-inbox.org/meta/ -f ldjson \
m:'20210304203352.pd5mcg5pw4u2epzl@pengutronix.de'
{"blob":"87304c8a8cae8ce400443b56309427aeee601505",...}
$ lei q -q -I https://public-inbox.org/meta/ -f ldjson \
'<20210304203352.pd5mcg5pw4u2epzl@pengutronix.de>'
{"blob":"87304c8a8cae8ce400443b56309427aeee601505",...}
$ lei q -q -I https://public-inbox.org/meta/ -f ldjson \
m:'<20210304203352.pd5mcg5pw4u2epzl@pengutronix.de>'
# no results
> Since the actual headers are "Message-ID" and "References", (and
> not "m" or "refs"), I think it's clear that we don't have to
> match the raw mail contents exactly. We RFC 2047 decode
> "f|t|c|s" fields anyways instead of showing the raw values,
> so more precedence for leaving out <>.
Fwiw I don't think leaving out the brackets would be a source of
confusion.
^ permalink raw reply [relevance 71%]
* release timelines (-extindex, JMAP, lei)
@ 2021-03-05 22:20 64% Eric Wong
2021-03-06 18:31 71% ` Kyle Meyer
2021-03-08 21:33 71% ` Konstantin Ryabitsev
0 siblings, 2 replies; 200+ results
From: Eric Wong @ 2021-03-05 22:20 UTC (permalink / raw)
To: meta
So I think the -extindex stuff might be stable and suitable for
general consumption. The HTML WWW UI around -extindex has some
rough edges but nothing that would take too much effort to fix.
But I'm deeply worried about unleashing a new on-disk format
that's insufficient and being stuck supporting it forever
(as I am with v1 inboxes)...
* JMAP is going to be a more effort, but I think our current
on-disk data model is OK or at least extensible enough for it.
I might delay JMAP until 1.8.
JMAP will be significantly less effort than inventing something
new (and one-off) with GraphQL or REST. And it would be less
effort for client authors, as well; since client code can be
used for non-public-inbox servers, too.
I'm excited that it allows vendor extensions, which means our
git-show|solver /$INBOX_URL/$OBJECT_ID/s/ blob-reconstruction
endpoint can be exposed via JMAP.
* lei saved searches can probably done quick for 1.7,
but without full keywords support for externals...
* Dealing with per-message keywords and externals in lei
is going to be tough and delayed, I think:
https://public-inbox.org/meta/20210224204950.GA2076@dcvr/
("lei: per-message keywords and externals")
It may involve spinnning up a Python daemon to use
custom PostingSource since Search::Xapian doesn't support it;
and I don't expect anybody outside of FreeBSD to use SWIG Xapian.pm.
* lei has a bunch of rough edges and I'm not comfortable declaring
it as supported, especially when there's a risk of data loss to
users.
* .mailmap + Xapian synonym support would be nice in lei and
HTTP/IMAP/JMAP endpoints
In any case, I need to take a few days away to clear my head.
^ permalink raw reply [relevance 64%]
* [PATCH] lei q: one -t shouldn't set `flagged' on external mail
@ 2021-03-05 4:03 66% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-05 4:03 UTC (permalink / raw)
To: meta
We only want to set `flagged' if a user requests it via
a two '-t' switches.
Fixes: 232f8e376fe2856c ("lei q: -tt marks direct hits as flagged")
---
lib/PublicInbox/LeiXSearch.pm | 6 +++---
t/lei-q-thread.t | 8 ++++++--
2 files changed, 9 insertions(+), 5 deletions(-)
diff --git a/lib/PublicInbox/LeiXSearch.pm b/lib/PublicInbox/LeiXSearch.pm
index 3270b420..f2c8c02e 100644
--- a/lib/PublicInbox/LeiXSearch.pm
+++ b/lib/PublicInbox/LeiXSearch.pm
@@ -148,7 +148,7 @@ sub query_thread_mset { # for --threads
my $mset;
my $each_smsg = $lei->{ovv}->ovv_each_smsg_cb($lei, $ibxish);
my $can_kw = !!$ibxish->can('msg_keywords');
- my $fl = $lei->{opt}->{threads} > 1;
+ my $fl = $lei->{opt}->{threads} > 1 ? [ 'flagged' ] : undef;
do {
$mset = $srch->mset($mo->{qstr}, $mo);
mset_progress($lei, $desc, $mset->size,
@@ -165,8 +165,8 @@ sub query_thread_mset { # for --threads
if ($mitem) {
if ($can_kw) {
mitem_kw($smsg, $mitem, $fl);
- } else {
- $smsg->{kw} = [ 'flagged' ];
+ } elsif ($fl) {
+ $smsg->{kw} = $fl;
}
}
$each_smsg->($smsg, $mitem);
diff --git a/t/lei-q-thread.t b/t/lei-q-thread.t
index 0ddf47a6..28c639f5 100644
--- a/t/lei-q-thread.t
+++ b/t/lei-q-thread.t
@@ -41,8 +41,12 @@ test_lei(sub {
'flagged set in direct hit');
'TODO' or is_deeply($m{'<testmessage@example.com>'}->{kw}, ['draft'],
'flagged set in direct hit');
- lei_ok qw(q -t -t m:testmessage@example.com --only), "$ro_home/t2";
+ lei_ok qw(q -tt m:testmessage@example.com --only), "$ro_home/t2";
$res = json_utf8->decode($lei_out);
- is_deeply($res->[0]->{kw}, [ 'flagged' ], 'flagged set on external');
+ is_deeply($res->[0]->{kw}, [ 'flagged' ],
+ 'flagged set on external with -tt');
+ lei_ok qw(q -t m:testmessage@example.com --only), "$ro_home/t2";
+ $res = json_utf8->decode($lei_out);
+ ok(!exists($res->[0]->{kw}), 'flagged not set on external with 1 -t');
});
done_testing;
^ permalink raw reply related [relevance 66%]
* "lei q" vs mairix notes...
@ 2021-03-05 2:22 63% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-05 2:22 UTC (permalink / raw)
To: meta
I'm not sure if this should be in the lei-q(1) manpage or
another manpage, probably another. There ought to be a similar
doc for notmuch and any other existing mail things I'm not
familiar with.
This is intended to be a neutral document to help and set
expectations for mairix users should they attempt to use lei.
It is NOT intended as advocacy document.
mairix and "lei q" share some similarities around common search
prefixes ("f:", "s:", "nq:") but there are several differences
users familiar with mairix should be aware of.
- lei (Xapian) uses ".." for date and size ranges, mairix uses "-".
This is due to how the Xapian query parser works.
- lei uses git(1) for date and time parsing; mairix has its own
syntax documented in mairix(1).
- lei does not support MH, yet
- lei currently requires mail to be imported into git ("lei import");
mairix indexes mail in IMAP, Maildir, MH, mbox directly
lei may attempt to index mail outside of git if there's interest:
https://public-inbox.org/meta/20210303035359.GA14438@dcvr/
- mairix can use symlinks and/or hardlinks to speed up writing
results when using Maildirs; lei must always extract messages
from git, which will always be slower.
- mairix has different rules around substring matches, negation,
combining, etc. than Xapian <https://xapian.org/docs/queryparser.html>
- lei doesn't yet support config file entries for output
(but will support saved searches)
- --raw-output and --excerpt-output in mairix aren't yet
supported, but the default JSON output in "lei q" may be
similar
- lei indexes positional data by default (and currently lacks a
configuration knob in the CLI), so indices use significantly
more space.
- lei is still in its infancy and far from complete
Again, this is intended to be a neutral document and not
advocacy. Help appreciated with corrections and addendums.
^ permalink raw reply [relevance 63%]
* [PATCH] lei q: fix --import-before default and FIFO output
@ 2021-03-05 1:38 51% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-05 1:38 UTC (permalink / raw)
To: meta
commit 6c551bffd75afb41d9b5e4774068abe7e06ed0e7
("lei q: --import-augment for mbox and mbox.gz") added a check to
in _pre_augment_mbox for the option being a ref() to distinguish
between default values and user-supplied values (which are
non-ref SCALARs from Getopt::Long).
However, LeiQuery failed to use a SCALAR ref as the default
value, making the check in _pre_augment_mbox useless. We
now update LeiQuery to use \1 instead of 1 as the default
value so "lei q -f mboxrd ..." to stdout works once again.
Unfortunately, testing with redirects pointed to regular
files didn't trigger the code paths being updated. Testing
with a FIFO revealed further bugs in the FIFO handling code
which are also fixed in this commit.
We'll also update the $lei->out error message to be
less-specific about "stdout" and use the term "output", instead,
since LeiToMail replaces stdout for all mbox outputs.
---
lib/PublicInbox/LEI.pm | 2 +-
lib/PublicInbox/LeiQuery.pm | 2 +-
lib/PublicInbox/LeiToMail.pm | 8 ++++++--
t/lei-q-kw.t | 25 +++++++++++++++++++------
4 files changed, 27 insertions(+), 10 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 50276a50..50c0a885 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -431,7 +431,7 @@ sub out ($;@) {
my $self = shift;
return if print { $self->{1} // return } @_; # likely
return note_sigpipe($self, 1) if $! == EPIPE;
- my $err = "error writing to stdout: $!";
+ my $err = "error writing to output: $!";
delete $self->{1};
fail($self, $err);
}
diff --git a/lib/PublicInbox/LeiQuery.pm b/lib/PublicInbox/LeiQuery.pm
index 493a8382..623b92cd 100644
--- a/lib/PublicInbox/LeiQuery.pm
+++ b/lib/PublicInbox/LeiQuery.pm
@@ -52,7 +52,7 @@ sub lei_q {
my $sto = $self->_lei_store(1);
my $lse = $sto->search;
if (($opt->{'import-remote'} //= 1) |
- ($opt->{'import-before'} //= 1)) {
+ (($opt->{'import-before'} //= \1) ? 1 : 0)) {
$sto->write_prepare($self);
}
if ($opt->{'local'} //= scalar(@only) ? 0 : 1) {
diff --git a/lib/PublicInbox/LeiToMail.pm b/lib/PublicInbox/LeiToMail.pm
index 1e2060fe..13b4f672 100644
--- a/lib/PublicInbox/LeiToMail.pm
+++ b/lib/PublicInbox/LeiToMail.pm
@@ -223,8 +223,6 @@ sub _post_augment_mbox { # open a compressor process
$dup->{$_} = $lei->{$_} for qw(2 sock);
tie *$pp, 'PublicInbox::ProcessPipe', $pid, $w, \&reap_compress, $dup;
$lei->{1} = $pp;
- die 'BUG: unexpected {ovv}->{lock_path}' if $lei->{ovv}->{lock_path};
- $lei->{ovv}->ovv_out_lk_init;
}
sub decompress_src ($$$) {
@@ -495,6 +493,7 @@ sub _pre_augment_mbox {
my $out = $lei->{1};
if ($dst ne '/dev/stdout') {
if (-p $dst) {
+ $out = undef;
open $out, '>', $dst or die "open($dst): $!";
} elsif (-f _ || !-e _) {
require PublicInbox::MboxLock;
@@ -521,6 +520,11 @@ sub _pre_augment_mbox {
if (($self->{zsfx}) = ($dst =~ /\.($zsfx_allow)\z/)) {
pipe(my ($r, $w)) or die "pipe: $!";
$lei->{zpipe} = [ $r, $w ];
+ $lei->{ovv}->{lock_path} and
+ die 'BUG: unexpected {ovv}->{lock_path}';
+ $lei->{ovv}->ovv_out_lk_init;
+ } elsif (!$self->{seekable} && !$lei->{ovv}->{lock_path}) {
+ $lei->{ovv}->ovv_out_lk_init;
}
$lei->{1} = $out;
undef;
diff --git a/t/lei-q-kw.t b/t/lei-q-kw.t
index 9daeb5b1..917a2c53 100644
--- a/t/lei-q-kw.t
+++ b/t/lei-q-kw.t
@@ -7,7 +7,15 @@ use Fcntl qw(SEEK_SET O_RDONLY O_NONBLOCK);
use IO::Uncompress::Gunzip qw(gunzip);
use IO::Compress::Gzip qw(gzip);
use PublicInbox::MboxReader;
+use PublicInbox::LeiToMail;
use PublicInbox::Spawn qw(popen_rd);
+my $exp = {
+ '<qp@example.com>' => eml_load('t/plack-qp.eml'),
+ '<testmessage@example.com>' => eml_load('t/utf8.eml'),
+};
+$exp->{'<qp@example.com>'}->header_set('Status', 'OR');
+$exp->{'<testmessage@example.com>'}->header_set('Status', 'O');
+
test_lei(sub {
lei_ok(qw(import -F eml t/plack-qp.eml));
my $o = "$ENV{HOME}/dst";
@@ -42,6 +50,17 @@ SKIP: {
ok(!lei(qw(q --import-before bogus -o), "mboxrd:$o"),
'--import-before fails on non-seekable output');
is(do { local $/; <$cat> }, '', 'no output on FIFO');
+ close $cat;
+ $cat = popen_rd(['cat', $o]);
+ lei_ok(qw(q m:qp@example.com -o), "mboxrd:$o");
+ my $buf = do { local $/; <$cat> };
+ open my $fh, '<', \$buf or BAIL_OUT $!;
+ PublicInbox::MboxReader->mboxrd($fh, sub {
+ my ($eml) = @_;
+ $eml->header_set('Status', 'OR');
+ is_deeply($eml, $exp->{'<qp@example.com>'},
+ 'FIFO output works as expected');
+ });
};
lei_ok qw(import -F eml t/utf8.eml), \'for augment test';
@@ -66,12 +85,6 @@ my $write_file = sub {
}
};
-my $exp = {
- '<qp@example.com>' => eml_load('t/plack-qp.eml'),
- '<testmessage@example.com>' => eml_load('t/utf8.eml'),
-};
-$exp->{'<qp@example.com>'}->header_set('Status', 'OR');
-$exp->{'<testmessage@example.com>'}->header_set('Status', 'O');
for my $sfx ('', '.gz') {
$o = "$ENV{HOME}/dst.mboxrd$sfx";
lei_ok(qw(q -o), "mboxrd:$o", qw(m:qp@example.com));
^ permalink raw reply related [relevance 51%]
* angle brackets in "m:" and "refs:" in "lei q" JSON
@ 2021-03-04 18:43 71% Eric Wong
2021-03-06 18:26 71% ` Kyle Meyer
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2021-03-04 18:43 UTC (permalink / raw)
To: meta
I'm thinking these shouldn't include angle brackets:
"m": "<20210228122528.18552-2-e@80x24.org>",
"refs": ["<20210228122528.18552-1-e@80x24.org>"],
Using angle brackets on the command-line requires quoting to
disambiguate against redirects, so it's a pain. Leaving the
brackets in still works because of how Xapian's query parser
works, not because of anything we do on our end.
Since the actual headers are "Message-ID" and "References", (and
not "m" or "refs"), I think it's clear that we don't have to
match the raw mail contents exactly. We RFC 2047 decode
"f|t|c|s" fields anyways instead of showing the raw values,
so more precedence for leaving out <>.
^ permalink raw reply [relevance 71%]
* [PATCH 0/6] lei q --import-augment => --import-before; mbox + IMAP
@ 2021-03-04 9:03 71% Eric Wong
2021-03-04 9:03 43% ` [PATCH 1/6] lei q: support --import-augment for IMAP Eric Wong
` (3 more replies)
0 siblings, 4 replies; 200+ results
From: Eric Wong @ 2021-03-04 9:03 UTC (permalink / raw)
To: meta
mbox support was the trickiest, and necessitated
PATCH 2/6 and 3/6 in addition to
https://public-inbox.org/meta/20210304012039.26900-1-e@80x24.org/
("ds: import croak properly")
6/6 completes the renaming.
Eric Wong (6):
lei q: support --import-augment for IMAP
lei: dclose: do not EPOLL_CTL_DEL w/o event_init
lei_xsearch: cleanup {pkt_op_p} on exceptions
lei q: --import-augment for mbox and mbox.gz
t/lei_to_mail: no need to cat in FIFO test
lei q: s/import-augment/import-before/g
lib/PublicInbox/LEI.pm | 4 +-
lib/PublicInbox/LeiQuery.pm | 2 +-
lib/PublicInbox/LeiToMail.pm | 115 +++++++++++++++++++++++-----------
lib/PublicInbox/LeiXSearch.pm | 6 ++
lib/PublicInbox/NetReader.pm | 9 ++-
lib/PublicInbox/NetWriter.pm | 41 ++++++++++--
t/lei-q-kw.t | 80 +++++++++++++++++++++--
t/lei_to_mail.t | 7 ++-
xt/net_writer-imap.t | 36 +++++++++--
9 files changed, 241 insertions(+), 59 deletions(-)
^ permalink raw reply [relevance 71%]
* [PATCH 2/6] lei: dclose: do not EPOLL_CTL_DEL w/o event_init
2021-03-04 9:03 71% [PATCH 0/6] lei q --import-augment => --import-before; mbox + IMAP Eric Wong
2021-03-04 9:03 43% ` [PATCH 1/6] lei q: support --import-augment for IMAP Eric Wong
@ 2021-03-04 9:03 71% ` Eric Wong
2021-03-04 9:03 43% ` [PATCH 4/6] lei q: --import-augment for mbox and mbox.gz Eric Wong
2021-03-04 9:03 47% ` [PATCH 6/6] lei q: s/import-augment/import-before/g Eric Wong
3 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-04 9:03 UTC (permalink / raw)
To: meta
It's possible we'll hit a die() statement which triggers
lei->dclose, but aren't in the event loop, yet.
---
lib/PublicInbox/LEI.pm | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 1e5b04ca..fdd9f8c8 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -973,7 +973,7 @@ sub dclose {
if (my $sto = delete $self->{sto}) {
$sto->ipc_do('done');
}
- $self->close if $self->{sock}; # PublicInbox::DS::close
+ $self->close if $self->{-event_init_done}; # PublicInbox::DS::close
}
# for long-running results
^ permalink raw reply related [relevance 71%]
* [PATCH 6/6] lei q: s/import-augment/import-before/g
2021-03-04 9:03 71% [PATCH 0/6] lei q --import-augment => --import-before; mbox + IMAP Eric Wong
` (2 preceding siblings ...)
2021-03-04 9:03 43% ` [PATCH 4/6] lei q: --import-augment for mbox and mbox.gz Eric Wong
@ 2021-03-04 9:03 47% ` Eric Wong
3 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-04 9:03 UTC (permalink / raw)
To: meta
Since this importing of keywords is active even when --augment
isn't specified, calling it --import-before seems more
appropriate.
In the future, this will likely default to adding unseen emails
to lei/store, not just updating keywords.
Link: https://public-inbox.org/meta/20210303222930.GA18597@dcvr/T/
---
lib/PublicInbox/LEI.pm | 2 +-
lib/PublicInbox/LeiQuery.pm | 2 +-
lib/PublicInbox/LeiToMail.pm | 16 ++++++++--------
t/lei-q-kw.t | 10 +++++-----
4 files changed, 15 insertions(+), 15 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index fdd9f8c8..50276a50 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -113,7 +113,7 @@ our %CMD = ( # sorted in order of importance/use:
qw(save-as=s output|mfolder|o=s format|f=s dedupe|d=s threads|t+
sort|s=s reverse|r offset=i remote! local! external! pretty
include|I=s@ exclude=s@ only=s@ jobs|j=s globoff|g augment|a
- import-remote! import-augment! lock=s@
+ import-remote! import-before! lock=s@
alert=s@ mua=s no-torsocks torsocks=s verbose|v+ quiet|q C=s@),
PublicInbox::LeiQuery::curl_opt(), opt_dash('limit|n=i', '[0-9]+') ],
diff --git a/lib/PublicInbox/LeiQuery.pm b/lib/PublicInbox/LeiQuery.pm
index c630d628..493a8382 100644
--- a/lib/PublicInbox/LeiQuery.pm
+++ b/lib/PublicInbox/LeiQuery.pm
@@ -52,7 +52,7 @@ sub lei_q {
my $sto = $self->_lei_store(1);
my $lse = $sto->search;
if (($opt->{'import-remote'} //= 1) |
- ($opt->{'import-augment'} //= 1)) {
+ ($opt->{'import-before'} //= 1)) {
$sto->write_prepare($self);
}
if ($opt->{'local'} //= scalar(@only) ? 0 : 1) {
diff --git a/lib/PublicInbox/LeiToMail.pm b/lib/PublicInbox/LeiToMail.pm
index 6290f35e..1e2060fe 100644
--- a/lib/PublicInbox/LeiToMail.pm
+++ b/lib/PublicInbox/LeiToMail.pm
@@ -438,7 +438,7 @@ sub _pre_augment_maildir {
sub _do_augment_maildir {
my ($self, $lei) = @_;
my $dst = $lei->{ovv}->{dst};
- my $lse = $lei->{sto}->search if $lei->{opt}->{'import-augment'};
+ my $lse = $lei->{sto}->search if $lei->{opt}->{'import-before'};
my ($mod, $shard) = @{$self->{shard_info} // []};
if ($lei->{opt}->{augment}) {
my $dedupe = $lei->{dedupe};
@@ -470,7 +470,7 @@ sub _imap_augment_or_delete { # PublicInbox::NetReader::imap_each cb
sub _do_augment_imap {
my ($self, $lei) = @_;
my $net = $lei->{net};
- my $lse = $lei->{sto}->search if $lei->{opt}->{'import-augment'};
+ my $lse = $lei->{sto}->search if $lei->{opt}->{'import-before'};
if ($lei->{opt}->{augment}) {
my $dedupe = $lei->{dedupe};
if ($dedupe && $dedupe->prepare_dedupe) {
@@ -511,8 +511,8 @@ sub _pre_augment_mbox {
die "seek($dst): $!\n";
}
if (!$self->{seekable}) {
- my $ia = $lei->{opt}->{'import-augment'};
- die "--import-augment specified but $dst is not seekable\n"
+ my $ia = $lei->{opt}->{'import-before'};
+ die "--import-before specified but $dst is not seekable\n"
if $ia && !ref($ia);
die "--augment specified but $dst is not seekable\n" if
$lei->{opt}->{augment};
@@ -533,7 +533,7 @@ sub _do_augment_mbox {
my $out = $lei->{1};
my ($fmt, $dst) = @{$lei->{ovv}}{qw(fmt dst)};
return unless -s $out;
- unless ($opt->{augment} || $opt->{'import-augment'}) {
+ unless ($opt->{augment} || $opt->{'import-before'}) {
truncate($out, 0) or die "truncate($dst): $!";
return;
}
@@ -544,14 +544,14 @@ sub _do_augment_mbox {
$dedupe = $lei->{dedupe};
$dedupe->prepare_dedupe if $dedupe;
}
- if ($opt->{'import-augment'}) { # the default
+ if ($opt->{'import-before'}) { # the default
my $lse = $lei->{sto}->search;
PublicInbox::MboxReader->$fmt($rd, \&_mbox_augment_kw_maybe,
$lei, $lse, $opt->{augment});
if (!$opt->{augment} and !truncate($out, 0)) {
die "truncate($dst): $!";
}
- } else { # --augment --no-import-augment
+ } else { # --augment --no-import-before
PublicInbox::MboxReader->$fmt($rd, \&_augment, $lei);
}
# maybe some systems don't honor O_APPEND, Perl does this:
@@ -576,7 +576,7 @@ sub do_augment { # slow, runs in wq worker
# fast (spawn compressor or mkdir), runs in same process as pre_augment
sub post_augment {
my ($self, $lei, @args) = @_;
- my $wait = $lei->{opt}->{'import-augment'} ?
+ my $wait = $lei->{opt}->{'import-before'} ?
$lei->{sto}->ipc_do('checkpoint', 1) : 0;
# _post_augment_mbox
my $m = $self->can("_post_augment_$self->{base_type}") or return;
diff --git a/t/lei-q-kw.t b/t/lei-q-kw.t
index babe9749..9daeb5b1 100644
--- a/t/lei-q-kw.t
+++ b/t/lei-q-kw.t
@@ -23,13 +23,13 @@ lei_ok(qw(q -o), "maildir:$o", qw(m:qp@example.com));
@fn = glob("$o/cur/*:2,S");
is(scalar(@fn), 1, "`seen' flag set on Maildir file");
-# ensure --no-import-augment works
+# ensure --no-import-before works
my $n = $fn[0];
$n =~ s/,S\z/,RS/;
rename($fn[0], $n) or BAIL_OUT "rename $!";
-lei_ok(qw(q --no-import-augment -o), "maildir:$o",
+lei_ok(qw(q --no-import-before -o), "maildir:$o",
qw(m:bogus-noresults@example.com));
-ok(!glob("$o/cur/*"), '--no-import-augment cleared destination');
+ok(!glob("$o/cur/*"), '--no-import-before cleared destination');
lei_ok(qw(q -o), "maildir:$o", qw(m:qp@example.com));
@fn = glob("$o/cur/*:2,S");
is(scalar(@fn), 1, "`seen' flag (but not `replied') set on Maildir file");
@@ -39,8 +39,8 @@ SKIP: {
mkfifo($o, 0600) or skip("mkfifo not supported: $!", 1);
# cat(1) since lei() may not execve for FD_CLOEXEC to work
my $cat = popen_rd(['cat', $o]);
- ok(!lei(qw(q --import-augment bogus -o), "mboxrd:$o"),
- '--import-augment fails on non-seekable output');
+ ok(!lei(qw(q --import-before bogus -o), "mboxrd:$o"),
+ '--import-before fails on non-seekable output');
is(do { local $/; <$cat> }, '', 'no output on FIFO');
};
^ permalink raw reply related [relevance 47%]
* [PATCH 4/6] lei q: --import-augment for mbox and mbox.gz
2021-03-04 9:03 71% [PATCH 0/6] lei q --import-augment => --import-before; mbox + IMAP Eric Wong
2021-03-04 9:03 43% ` [PATCH 1/6] lei q: support --import-augment for IMAP Eric Wong
2021-03-04 9:03 71% ` [PATCH 2/6] lei: dclose: do not EPOLL_CTL_DEL w/o event_init Eric Wong
@ 2021-03-04 9:03 43% ` Eric Wong
2021-03-04 9:03 47% ` [PATCH 6/6] lei q: s/import-augment/import-before/g Eric Wong
3 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-04 9:03 UTC (permalink / raw)
To: meta
The trickiest output formats we support due to the possibility
of filesystem FIFOS and pipes for <gzip|xz|bzip2>.
This completes another phase of keyword sync support.
---
lib/PublicInbox/LeiToMail.pm | 65 ++++++++++++++++++++++---------
t/lei-q-kw.t | 74 +++++++++++++++++++++++++++++++++++-
2 files changed, 119 insertions(+), 20 deletions(-)
diff --git a/lib/PublicInbox/LeiToMail.pm b/lib/PublicInbox/LeiToMail.pm
index b3228a59..6290f35e 100644
--- a/lib/PublicInbox/LeiToMail.pm
+++ b/lib/PublicInbox/LeiToMail.pm
@@ -246,6 +246,13 @@ sub _augment { # MboxReader eml_cb
$lei->{dedupe}->is_dup($eml);
}
+sub _mbox_augment_kw_maybe {
+ my ($eml, $lei, $lse, $augment) = @_;
+ my @kw = PublicInbox::LeiStore::mbox_keywords($eml);
+ update_kw_maybe($lei, $lse, $eml, \@kw);
+ _augment($eml, $lei) if $augment;
+}
+
sub _mbox_write_cb ($$) {
my ($self, $lei) = @_;
my $ovv = $lei->{ovv};
@@ -391,7 +398,7 @@ sub new {
"$dst exists and is not a directory\n";
$lei->{ovv}->{dst} = $dst .= '/' if substr($dst, -1) ne '/';
} elsif (substr($fmt, 0, 4) eq 'mbox') {
- require PublicInbox::MboxReader if $lei->{opt}->{augment};
+ require PublicInbox::MboxReader;
(-d $dst || (-e _ && !-w _)) and die
"$dst exists and is not a writable file\n";
$self->can("eml2$fmt") or die "bad mbox format: $fmt\n";
@@ -485,8 +492,8 @@ sub _do_augment_imap {
sub _pre_augment_mbox {
my ($self, $lei) = @_;
my $dst = $lei->{ovv}->{dst};
+ my $out = $lei->{1};
if ($dst ne '/dev/stdout') {
- my $out;
if (-p $dst) {
open $out, '>', $dst or die "open($dst): $!";
} elsif (-f _ || !-e _) {
@@ -495,36 +502,56 @@ sub _pre_augment_mbox {
PublicInbox::MboxLock->defaults;
$self->{mbl} = PublicInbox::MboxLock->acq($dst, 1, $m);
$out = $self->{mbl}->{fh};
- if (!$lei->{opt}->{augment} and !truncate($out, 0)) {
- die "truncate($dst): $!";
- }
}
$lei->{old_1} = $lei->{1}; # keep for spawning MUA
- $lei->{1} = $out;
}
# Perl does SEEK_END even with O_APPEND :<
- $self->{seekable} = seek($lei->{1}, 0, SEEK_SET);
+ $self->{seekable} = seek($out, 0, SEEK_SET);
if (!$self->{seekable} && $! != ESPIPE && $dst ne '/dev/stdout') {
die "seek($dst): $!\n";
}
+ if (!$self->{seekable}) {
+ my $ia = $lei->{opt}->{'import-augment'};
+ die "--import-augment specified but $dst is not seekable\n"
+ if $ia && !ref($ia);
+ die "--augment specified but $dst is not seekable\n" if
+ $lei->{opt}->{augment};
+ }
state $zsfx_allow = join('|', keys %zsfx2cmd);
- ($self->{zsfx}) = ($dst =~ /\.($zsfx_allow)\z/) or return;
- pipe(my ($r, $w)) or die "pipe: $!";
- $lei->{zpipe} = [ $r, $w ];
+ if (($self->{zsfx}) = ($dst =~ /\.($zsfx_allow)\z/)) {
+ pipe(my ($r, $w)) or die "pipe: $!";
+ $lei->{zpipe} = [ $r, $w ];
+ }
+ $lei->{1} = $out;
+ undef;
}
sub _do_augment_mbox {
my ($self, $lei) = @_;
- return if !$lei->{opt}->{augment};
- my $dedupe = $lei->{dedupe};
- my $dst = $lei->{ovv}->{dst};
- die "cannot augment $dst, not seekable\n" if !$self->{seekable};
+ return unless $self->{seekable};
+ my $opt = $lei->{opt};
my $out = $lei->{1};
- if (-s $out && $dedupe && $dedupe->prepare_dedupe) {
- my $zsfx = $self->{zsfx};
- my $rd = $zsfx ? decompress_src($out, $zsfx, $lei) :
- dup_src($out);
- my $fmt = $lei->{ovv}->{fmt};
+ my ($fmt, $dst) = @{$lei->{ovv}}{qw(fmt dst)};
+ return unless -s $out;
+ unless ($opt->{augment} || $opt->{'import-augment'}) {
+ truncate($out, 0) or die "truncate($dst): $!";
+ return;
+ }
+ my $zsfx = $self->{zsfx};
+ my $rd = $zsfx ? decompress_src($out, $zsfx, $lei) : dup_src($out);
+ my $dedupe;
+ if ($opt->{augment}) {
+ $dedupe = $lei->{dedupe};
+ $dedupe->prepare_dedupe if $dedupe;
+ }
+ if ($opt->{'import-augment'}) { # the default
+ my $lse = $lei->{sto}->search;
+ PublicInbox::MboxReader->$fmt($rd, \&_mbox_augment_kw_maybe,
+ $lei, $lse, $opt->{augment});
+ if (!$opt->{augment} and !truncate($out, 0)) {
+ die "truncate($dst): $!";
+ }
+ } else { # --augment --no-import-augment
PublicInbox::MboxReader->$fmt($rd, \&_augment, $lei);
}
# maybe some systems don't honor O_APPEND, Perl does this:
diff --git a/t/lei-q-kw.t b/t/lei-q-kw.t
index 97b2e08f..babe9749 100644
--- a/t/lei-q-kw.t
+++ b/t/lei-q-kw.t
@@ -2,6 +2,12 @@
# Copyright (C) 2020-2021 all contributors <meta@public-inbox.org>
# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
use strict; use v5.10.1; use PublicInbox::TestCommon;
+use POSIX qw(mkfifo);
+use Fcntl qw(SEEK_SET O_RDONLY O_NONBLOCK);
+use IO::Uncompress::Gunzip qw(gunzip);
+use IO::Compress::Gzip qw(gzip);
+use PublicInbox::MboxReader;
+use PublicInbox::Spawn qw(popen_rd);
test_lei(sub {
lei_ok(qw(import -F eml t/plack-qp.eml));
my $o = "$ENV{HOME}/dst";
@@ -28,6 +34,72 @@ lei_ok(qw(q -o), "maildir:$o", qw(m:qp@example.com));
@fn = glob("$o/cur/*:2,S");
is(scalar(@fn), 1, "`seen' flag (but not `replied') set on Maildir file");
-# TODO: other destination types
+SKIP: {
+ $o = "$ENV{HOME}/fifo";
+ mkfifo($o, 0600) or skip("mkfifo not supported: $!", 1);
+ # cat(1) since lei() may not execve for FD_CLOEXEC to work
+ my $cat = popen_rd(['cat', $o]);
+ ok(!lei(qw(q --import-augment bogus -o), "mboxrd:$o"),
+ '--import-augment fails on non-seekable output');
+ is(do { local $/; <$cat> }, '', 'no output on FIFO');
+};
+
+lei_ok qw(import -F eml t/utf8.eml), \'for augment test';
+my $read_file = sub {
+ if ($_[0] =~ /\.gz\z/) {
+ gunzip($_[0] => \(my $buf = ''), MultiStream => 1) or
+ BAIL_OUT 'gunzip';
+ $buf;
+ } else {
+ open my $fh, '+<', $_[0] or BAIL_OUT $!;
+ do { local $/; <$fh> };
+ }
+};
+
+my $write_file = sub {
+ if ($_[0] =~ /\.gz\z/) {
+ gzip(\($_[1]), $_[0]) or BAIL_OUT 'gzip';
+ } else {
+ open my $fh, '>', $_[0] or BAIL_OUT $!;
+ print $fh $_[1] or BAIL_OUT $!;
+ close $fh or BAIL_OUT;
+ }
+};
+
+my $exp = {
+ '<qp@example.com>' => eml_load('t/plack-qp.eml'),
+ '<testmessage@example.com>' => eml_load('t/utf8.eml'),
+};
+$exp->{'<qp@example.com>'}->header_set('Status', 'OR');
+$exp->{'<testmessage@example.com>'}->header_set('Status', 'O');
+for my $sfx ('', '.gz') {
+ $o = "$ENV{HOME}/dst.mboxrd$sfx";
+ lei_ok(qw(q -o), "mboxrd:$o", qw(m:qp@example.com));
+ my $buf = $read_file->($o);
+ $buf =~ s/^Status: [^\n]*\n//sm or BAIL_OUT "no status in $buf";
+ $write_file->($o, $buf);
+ lei_ok(qw(q -o), "mboxrd:$o", qw(rereadandimportkwchange));
+ $buf = $read_file->($o);
+ is($buf, '', 'emptied');
+ lei_ok(qw(q -o), "mboxrd:$o", qw(m:qp@example.com));
+ $buf = $read_file->($o);
+ $buf =~ s/\nStatus: O\n\n/\nStatus: OR\n\n/s or
+ BAIL_OUT "no Status in $buf";
+ $write_file->($o, $buf);
+ lei_ok(qw(q -a -o), "mboxrd:$o", qw(m:testmessage@example.com));
+ $buf = $read_file->($o);
+ open my $fh, '<', \$buf or BAIL_OUT "PerlIO::scalar $!";
+ my %res;
+ PublicInbox::MboxReader->mboxrd($fh, sub {
+ my ($eml) = @_;
+ $res{$eml->header_raw('Message-ID')} = $eml;
+ });
+ is_deeply(\%res, $exp, '--augment worked');
+
+ lei_ok(qw(q -o), "mboxrd:/dev/stdout", qw(m:qp@example.com)) or
+ diag $lei_err;
+ like($lei_out, qr/^Status: OR\n/sm, 'Status set by previous augment');
+}
+
});
done_testing;
^ permalink raw reply related [relevance 43%]
* [PATCH 1/6] lei q: support --import-augment for IMAP
2021-03-04 9:03 71% [PATCH 0/6] lei q --import-augment => --import-before; mbox + IMAP Eric Wong
@ 2021-03-04 9:03 43% ` Eric Wong
2021-03-04 9:03 71% ` [PATCH 2/6] lei: dclose: do not EPOLL_CTL_DEL w/o event_init Eric Wong
` (2 subsequent siblings)
3 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-04 9:03 UTC (permalink / raw)
To: meta
IMAP is similar to Maildir and we can now preserve keyword
updates done on IMAP folders.
---
lib/PublicInbox/LeiToMail.pm | 48 ++++++++++++++++++++++--------------
lib/PublicInbox/NetReader.pm | 9 +++++--
lib/PublicInbox/NetWriter.pm | 41 ++++++++++++++++++++++++++----
xt/net_writer-imap.t | 36 ++++++++++++++++++++++++---
4 files changed, 105 insertions(+), 29 deletions(-)
diff --git a/lib/PublicInbox/LeiToMail.pm b/lib/PublicInbox/LeiToMail.pm
index 3420b06e..b3228a59 100644
--- a/lib/PublicInbox/LeiToMail.pm
+++ b/lib/PublicInbox/LeiToMail.pm
@@ -267,6 +267,17 @@ sub _mbox_write_cb ($$) {
}
}
+sub update_kw_maybe ($$$$) {
+ my ($lei, $lse, $eml, $kw) = @_;
+ return unless $lse;
+ my $x = $lse->kw_changed($eml, $kw);
+ if ($x) {
+ $lei->{sto}->ipc_do('set_eml', $eml, @$kw);
+ } elsif (!defined($x)) {
+ # TODO: xkw
+ }
+}
+
sub _augment_or_unlink { # maildir_each_eml cb
my ($f, $kw, $eml, $lei, $lse, $mod, $shard, $unlink) = @_;
if ($mod) {
@@ -276,14 +287,7 @@ sub _augment_or_unlink { # maildir_each_eml cb
$1 : sha256_hex($f);
my $recno = hex(substr($hex, 0, 8));
return if ($recno % $mod) != $shard;
- if ($lse) {
- my $x = $lse->kw_changed($eml, $kw);
- if ($x) {
- $lei->{sto}->ipc_do('set_eml', $eml, @$kw);
- } elsif (!defined($x)) {
- # TODO: xkw
- }
- }
+ update_kw_maybe($lei, $lse, $eml, $kw);
}
$unlink ? unlink($f) : _augment($eml, $lei);
}
@@ -446,26 +450,32 @@ sub _do_augment_maildir {
}
}
-sub _post_augment_maildir {
- my ($self, $lei) = @_;
- $lei->{opt}->{'import-augment'} or return;
- my $wait = $lei->{sto}->ipc_do('checkpoint', 1);
-}
-
-sub _augment_imap { # PublicInbox::NetReader::imap_each cb
- my ($url, $uid, $kw, $eml, $lei) = @_;
- _augment($eml, $lei);
+sub _imap_augment_or_delete { # PublicInbox::NetReader::imap_each cb
+ my ($url, $uid, $kw, $eml, $lei, $lse, $delete_mic) = @_;
+ update_kw_maybe($lei, $lse, $eml, $kw);
+ if ($delete_mic) {
+ $lei->{net}->imap_delete_1($url, $uid, $delete_mic);
+ } else {
+ _augment($eml, $lei);
+ }
}
sub _do_augment_imap {
my ($self, $lei) = @_;
my $net = $lei->{net};
+ my $lse = $lei->{sto}->search if $lei->{opt}->{'import-augment'};
if ($lei->{opt}->{augment}) {
my $dedupe = $lei->{dedupe};
if ($dedupe && $dedupe->prepare_dedupe) {
- $net->imap_each($self->{uri}, \&_augment_imap, $lei);
+ $net->imap_each($self->{uri}, \&_imap_augment_or_delete,
+ $lei, $lse);
$dedupe->pause_dedupe;
}
+ } elsif ($lse) {
+ my $delete_mic;
+ $net->imap_each($self->{uri}, \&_imap_augment_or_delete,
+ $lei, $lse, \$delete_mic);
+ $delete_mic->expunge if $delete_mic;
} elsif (!$self->{-wq_worker_nr}) { # undef or 0
# clobber existing IMAP folder
$net->imap_delete_all($self->{uri});
@@ -539,6 +549,8 @@ sub do_augment { # slow, runs in wq worker
# fast (spawn compressor or mkdir), runs in same process as pre_augment
sub post_augment {
my ($self, $lei, @args) = @_;
+ my $wait = $lei->{opt}->{'import-augment'} ?
+ $lei->{sto}->ipc_do('checkpoint', 1) : 0;
# _post_augment_mbox
my $m = $self->can("_post_augment_$self->{base_type}") or return;
$m->($self, $lei, @args);
diff --git a/lib/PublicInbox/NetReader.pm b/lib/PublicInbox/NetReader.pm
index 96d3b2ed..f5f71005 100644
--- a/lib/PublicInbox/NetReader.pm
+++ b/lib/PublicInbox/NetReader.pm
@@ -346,9 +346,14 @@ sub _imap_do_msg ($$$$$) {
$$raw =~ s/\r\n/\n/sg;
my $kw = [];
for my $f (split(/ /, $flags)) {
- my $k = $IMAPflags2kw{$f} // next; # TODO: X-Label?
- push @$kw, $k;
+ if (my $k = $IMAPflags2kw{$f}) {
+ push @$kw, $k;
+ } elsif ($f eq "\\Recent") { # not in JMAP
+ } elsif ($self->{verbose}) {
+ warn "# unknown IMAP flag $f <$uri;uid=$uid>\n";
+ }
}
+ @$kw = sort @$kw; # for all UI/UX purposes
my ($eml_cb, @args) = @{$self->{eml_each}};
$eml_cb->($uri, $uid, $kw, PublicInbox::Eml->new($raw), @args);
}
diff --git a/lib/PublicInbox/NetWriter.pm b/lib/PublicInbox/NetWriter.pm
index e26e9815..49ac02a6 100644
--- a/lib/PublicInbox/NetWriter.pm
+++ b/lib/PublicInbox/NetWriter.pm
@@ -13,27 +13,58 @@ my %IMAPkw2flags;
@IMAPkw2flags{values %PublicInbox::NetReader::IMAPflags2kw} =
keys %PublicInbox::NetReader::IMAPflags2kw;
+sub kw2flags ($) { join(' ', map { $IMAPkw2flags{$_} } @{$_[0]}) }
+
sub imap_append {
my ($mic, $folder, $bref, $smsg, $eml) = @_;
$bref //= \($eml->as_string);
$smsg //= bless {}, 'PublicInbox::Smsg';
bless($smsg, 'PublicInbox::Smsg') if ref($smsg) eq 'HASH';
$smsg->{ts} //= msg_timestamp($eml // PublicInbox::Eml->new($$bref));
- my @f = map { $IMAPkw2flags{$_} } @{$smsg->{kw}};
- $mic->append_string($folder, $$bref, "@f", $smsg->internaldate) or
+ my $f = kw2flags($smsg->{kw});
+ $mic->append_string($folder, $$bref, $f, $smsg->internaldate) or
die "APPEND $folder: $@";
}
+sub mic_for_folder {
+ my ($self, $uri) = @_;
+ if (!ref($uri)) {
+ my $u = PublicInbox::URIimap->new($uri);
+ $_[1] = $uri = $u;
+ }
+ my $mic = $self->mic_get($uri) or die "E: not connected: $@";
+ $mic->select($uri->mailbox) or return;
+ $mic;
+}
+
sub imap_delete_all {
my ($self, $url) = @_;
- my $uri = PublicInbox::URIimap->new($url);
+ my $mic = mic_for_folder($self, my $uri = $url) or return;
my $sec = $self->can('uri_section')->($uri);
local $0 = $uri->mailbox." $sec";
- my $mic = $self->mic_get($uri) or die "E: not connected: $@";
- $mic->select($uri->mailbox) or return; # non-existent
if ($mic->delete_message('1:*')) {
$mic->expunge;
}
}
+sub imap_delete_1 {
+ my ($self, $url, $uid, $delete_mic) = @_;
+ $$delete_mic //= mic_for_folder($self, my $uri = $url) or return;
+ $$delete_mic->delete_message($uid);
+}
+
+sub imap_set_kw {
+ my ($self, $url, $uid, $kw) = @_;
+ my $mic = mic_for_folder($self, my $uri = $url) or return;
+ $mic->set_flag(kw2flags($kw), $uid);
+ $mic; # caller must ->expunge
+}
+
+sub imap_unset_kw {
+ my ($self, $url, $uid, $kw) = @_;
+ my $mic = mic_for_folder($self, my $uri = $url) or return;
+ $mic->unset_flag(kw2flags($kw), $uid);
+ $mic; # caller must ->expunge
+}
+
1;
diff --git a/xt/net_writer-imap.t b/xt/net_writer-imap.t
index da435926..c24fa993 100644
--- a/xt/net_writer-imap.t
+++ b/xt/net_writer-imap.t
@@ -91,7 +91,7 @@ my $smsg = bless { kw => [ 'seen' ] }, 'PublicInbox::Smsg';
$imap_append->($mic, $folder, undef, $smsg, eml_load('t/plack-qp.eml'));
$nwr->{quiet} = 1;
my $imap_slurp_all = sub {
- my ($u, $uid, $kw, $eml, $res) = @_;
+ my ($url, $uid, $kw, $eml, $res) = @_;
push @$res, [ $kw, $eml ];
};
$nwr->imap_each($folder_uri, $imap_slurp_all, my $res = []);
@@ -138,10 +138,38 @@ test_lei(sub {
$nwr->imap_each($folder_uri, $imap_slurp_all, my $empty = []);
is(scalar(@$empty), 0, 'no results w/o augment');
- lei_ok qw(convert -F eml t/msg_iter-order.eml -o), $$folder_uri;
+ my $f = 't/utf8.eml'; # <testmessage@example.com>
+ $exp = eml_load($f);
+ lei_ok qw(convert -F eml -o), $$folder_uri, $f;
+ my (@uid, @res);
+ $nwr->imap_each($folder_uri, sub {
+ my ($u, $uid, $kw, $eml) = @_;
+ push @uid, $uid;
+ push @res, [ $kw, $eml ];
+ });
+ is_deeply(\@res, [ [ [], $exp ] ], 'converted to IMAP destination');
+ is(scalar(@uid), 1, 'got one UID back');
+ lei_ok qw(q -o /dev/stdout m:testmessage@example.com --no-external);
+ is_deeply(json_utf8->decode($lei_out), [undef],
+ 'no results before import');
+
+ lei_ok qw(import -F eml), $f, \'import local copy w/o keywords';
+
+ $nwr->imap_set_kw($folder_uri, $uid[0], [ 'seen' ])->expunge
+ or BAIL_OUT "expunge $@";
+ @res = ();
+ $nwr->imap_each($folder_uri, $imap_slurp_all, \@res);
+ is_deeply(\@res, [ [ ['seen'], $exp ] ], 'seen flag set') or
+ diag explain(\@res);
+
+ lei_ok qw(q s:thisbetternotgiveanyresult -o), $folder_uri->as_string,
+ \'clobber folder but import flag';
$nwr->imap_each($folder_uri, $imap_slurp_all, $empty = []);
- is_deeply($empty, [ [ [], eml_load('t/msg_iter-order.eml') ] ],
- 'converted to IMAP destination');
+ is_deeply($empty, [], 'clobbered folder');
+ lei_ok qw(q -o /dev/stdout m:testmessage@example.com --no-external);
+ $res = json_utf8->decode($lei_out)->[0];
+ is_deeply([@$res{qw(m kw)}], ['<testmessage@example.com>', ['seen']],
+ 'kw set');
});
undef $cleanup; # remove temporary folder
^ permalink raw reply related [relevance 43%]
* Re: RFH: --import-augment naming [was: lei q: import flags when clobbering/augmenting]
2021-03-04 3:31 70% ` Eric Wong
@ 2021-03-04 4:10 71% ` Kyle Meyer
0 siblings, 0 replies; 200+ results
From: Kyle Meyer @ 2021-03-04 4:10 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
Eric Wong writes:
> Oh, I forgot to note it will probably import more than just
> keywords (but maybe it can be tweaked(*)).
>
> The thing I want to protect against is somebody forgetting
> --augment when using "lei q -o imaps://example.com/INBOX ..."
> which would delete mail that hasn't been imported to lei or
> backed-up by another tool.
>
> Causing data loss in the above scenario would be a nightmare,
> even if it's technically user error.
Oy, indeed.
> There's also cases where someone will want to edit a patch in
> the search results mailbox before applying it (e.g. adding
> Acked-by, fixing whitespace, trivial errors, etc...) and
> it might be good to preserve a copy of the edited message.
>
> (*) possible directions:
>
> --import-$FOO=kw-only
> --import-$FOO=not-in-git
> --import-$FOO # same as "not-in-git", this should be the default
> --import-$FOO=none / --no-import-$FOO
Make sense. Your suggested "before" seems like a good choice for $FOO.
^ permalink raw reply [relevance 71%]
* Re: RFH: --import-augment naming [was: lei q: import flags when clobbering/augmenting]
2021-03-04 2:39 71% ` Kyle Meyer
@ 2021-03-04 3:31 70% ` Eric Wong
2021-03-04 4:10 71% ` Kyle Meyer
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2021-03-04 3:31 UTC (permalink / raw)
To: Kyle Meyer; +Cc: meta
Kyle Meyer <kyle@kyleam.com> wrote:
> Eric Wong writes:
>
> > --import-augment is the wrong name for this option,
> > since the import happens even when --augment isn't specified.
> >
> > How about "--import-before" ?
>
> I don't have a good understanding of the internals, but fwiw that sounds
> fine to me. Given your description of "[stash] keyword changes" and
> "import flags", --stash-keywords or --import-keywords came to mind, but
> perhaps those aren't quite accurate.
Oh, I forgot to note it will probably import more than just
keywords (but maybe it can be tweaked(*)).
The thing I want to protect against is somebody forgetting
--augment when using "lei q -o imaps://example.com/INBOX ..."
which would delete mail that hasn't been imported to lei or
backed-up by another tool.
Causing data loss in the above scenario would be a nightmare,
even if it's technically user error.
There's also cases where someone will want to edit a patch in
the search results mailbox before applying it (e.g. adding
Acked-by, fixing whitespace, trivial errors, etc...) and
it might be good to preserve a copy of the edited message.
(*) possible directions:
--import-$FOO=kw-only
--import-$FOO=not-in-git
--import-$FOO # same as "not-in-git", this should be the default
--import-$FOO=none / --no-import-$FOO
^ permalink raw reply [relevance 70%]
* Re: RFH: --import-augment naming [was: lei q: import flags when clobbering/augmenting]
2021-03-03 22:29 71% ` RFH: --import-augment naming [was: lei q: import flags when clobbering/augmenting] Eric Wong
@ 2021-03-04 2:39 71% ` Kyle Meyer
2021-03-04 3:31 70% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Kyle Meyer @ 2021-03-04 2:39 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
Eric Wong writes:
> --import-augment is the wrong name for this option,
> since the import happens even when --augment isn't specified.
>
> How about "--import-before" ?
I don't have a good understanding of the internals, but fwiw that sounds
fine to me. Given your description of "[stash] keyword changes" and
"import flags", --stash-keywords or --import-keywords came to mind, but
perhaps those aren't quite accurate.
^ permalink raw reply [relevance 71%]
* RFH: --import-augment naming [was: lei q: import flags when clobbering/augmenting]
2021-03-03 13:48 33% ` [PATCH 4/4] lei q: import flags when clobbering/augmenting Maildirs Eric Wong
@ 2021-03-03 22:29 71% ` Eric Wong
2021-03-04 2:39 71% ` Kyle Meyer
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2021-03-03 22:29 UTC (permalink / raw)
To: meta
Eric Wong <e@80x24.org> wrote:
> This will eventually be supported for other mail stores,
> but Maildir is the easiest to test and support, here.
>
> This lets us avoid a situation where flag changes get
> lost between search results.
> --- a/lib/PublicInbox/LEI.pm
> +++ b/lib/PublicInbox/LEI.pm
> @@ -113,7 +113,7 @@ our %CMD = ( # sorted in order of importance/use:
> qw(save-as=s output|mfolder|o=s format|f=s dedupe|d=s threads|t+
> sort|s=s reverse|r offset=i remote! local! external! pretty
> include|I=s@ exclude=s@ only=s@ jobs|j=s globoff|g augment|a
> - import-remote! lock=s@
> + import-remote! import-augment! lock=s@
--import-augment is the wrong name for this option,
since the import happens even when --augment isn't specified.
How about "--import-before" ?
^ permalink raw reply [relevance 71%]
* [PATCH 3/4] lei: use maildir_each_eml in more places
@ 2021-03-03 13:48 55% ` Eric Wong
2021-03-03 13:48 33% ` [PATCH 4/4] lei q: import flags when clobbering/augmenting Maildirs Eric Wong
1 sibling, 0 replies; 200+ results
From: Eric Wong @ 2021-03-03 13:48 UTC (permalink / raw)
To: meta
This saves us some code and redundant callsites for
eml_from_path. We'll change maildir_each_eml to include the
filename to facilitate an upcoming change to "lei q" without
--augment
---
lib/PublicInbox/LeiConvert.pm | 3 +--
lib/PublicInbox/LeiToMail.pm | 18 ++++++------------
lib/PublicInbox/MdirReader.pm | 10 ++++++----
t/lei-convert.t | 2 +-
4 files changed, 14 insertions(+), 19 deletions(-)
diff --git a/lib/PublicInbox/LeiConvert.pm b/lib/PublicInbox/LeiConvert.pm
index 4c0bbd88..0c705ba4 100644
--- a/lib/PublicInbox/LeiConvert.pm
+++ b/lib/PublicInbox/LeiConvert.pm
@@ -7,7 +7,6 @@ use strict;
use v5.10.1;
use parent qw(PublicInbox::IPC);
use PublicInbox::Eml;
-use PublicInbox::InboxWritable qw(eml_from_path);
use PublicInbox::LeiStore;
use PublicInbox::LeiOverview;
@@ -24,7 +23,7 @@ sub net_cb { # callback for ->imap_each, ->nntp_each
}
sub mdir_cb {
- my ($kw, $eml, $self) = @_;
+ my ($f, $kw, $eml, $self) = @_;
$self->{wcb}->(undef, { kw => $kw }, $eml);
}
diff --git a/lib/PublicInbox/LeiToMail.pm b/lib/PublicInbox/LeiToMail.pm
index de640657..31b8aba8 100644
--- a/lib/PublicInbox/LeiToMail.pm
+++ b/lib/PublicInbox/LeiToMail.pm
@@ -19,7 +19,6 @@ use IO::Handle; # ->autoflush
use Fcntl qw(SEEK_SET SEEK_END O_CREAT O_EXCL O_WRONLY);
use Errno qw(EEXIST ESPIPE ENOENT EPIPE);
use Digest::SHA qw(sha256_hex);
-my ($maildir_each_file);
# struggles with short-lived repos, Gcf2Client makes little sense with lei;
# but we may use in-process libgit2 in the future.
@@ -268,8 +267,8 @@ sub _mbox_write_cb ($$) {
}
}
-sub _augment_file { # maildir_each_file cb
- my ($f, $lei, $mod, $shard) = @_;
+sub _augment_file { # maildir_each_eml cb
+ my ($f, undef, $eml, $lei, $mod, $shard) = @_;
if ($mod) {
# can't get dirent.d_ino w/ pure Perl, so we extract the OID
# if it looks like one:
@@ -278,7 +277,6 @@ sub _augment_file { # maildir_each_file cb
my $recno = hex(substr($hex, 0, 8));
return if ($recno % $mod) != $shard;
}
- my $eml = PublicInbox::InboxWritable::eml_from_path($f) or return;
_augment($eml, $lei);
}
@@ -375,12 +373,7 @@ sub new {
my $dst = $lei->{ovv}->{dst};
my $self = bless {}, $cls;
if ($fmt eq 'maildir') {
- $maildir_each_file //= do {
- require PublicInbox::MdirReader;
- PublicInbox::MdirReader->can('maildir_each_file');
- };
- $lei->{opt}->{augment} and
- require PublicInbox::InboxWritable; # eml_from_path
+ require PublicInbox::MdirReader;
$self->{base_type} = 'maildir';
-e $dst && !-d _ and die
"$dst exists and is not a directory\n";
@@ -430,12 +423,13 @@ sub _do_augment_maildir {
my $dedupe = $lei->{dedupe};
if ($dedupe && $dedupe->prepare_dedupe) {
my ($mod, $shard) = @{$self->{shard_info} // []};
- $maildir_each_file->($dst, \&_augment_file,
+ PublicInbox::MdirReader::maildir_each_eml($dst,
+ \&_augment_file,
$lei, $mod, $shard);
$dedupe->pause_dedupe;
}
} else { # clobber existing Maildir
- $maildir_each_file->($dst, \&_unlink);
+ PublicInbox::MdirReader::maildir_each_file($dst, \&_unlink);
}
}
diff --git a/lib/PublicInbox/MdirReader.pm b/lib/PublicInbox/MdirReader.pm
index 5fa534f5..44724af1 100644
--- a/lib/PublicInbox/MdirReader.pm
+++ b/lib/PublicInbox/MdirReader.pm
@@ -48,17 +48,19 @@ sub maildir_each_eml ($$;@) {
next if substr($bn, 0, 1) eq '.';
my @f = split(/:/, $bn, -1);
next if scalar(@f) != 1;
- my $eml = eml_from_path($pfx.$bn) or next;
- $cb->([], $eml, @arg);
+ my $f = $pfx.$bn;
+ my $eml = eml_from_path($f) or next;
+ $cb->($f, [], $eml, @arg);
}
}
$pfx = "$dir/cur/";
opendir my $dh, $pfx or return;
while (defined(my $bn = readdir($dh))) {
my $fl = maildir_basename_flags($bn) // next;
- my $eml = eml_from_path($pfx.$bn) or next;
+ my $f = $pfx.$bn;
+ my $eml = eml_from_path($f) or next;
my @kw = sort(map { $c2kw{$_} // () } split(//, $fl));
- $cb->(\@kw, $eml, @arg);
+ $cb->($f, \@kw, $eml, @arg);
}
}
diff --git a/t/lei-convert.t b/t/lei-convert.t
index 20099f65..186cfb13 100644
--- a/t/lei-convert.t
+++ b/t/lei-convert.t
@@ -58,7 +58,7 @@ test_lei({ tmpdir => $tmpdir }, sub {
ok(-d "$d/md", 'Maildir created');
my @md;
PublicInbox::MdirReader::maildir_each_eml("$d/md", sub {
- push @md, $_[1];
+ push @md, $_[2];
});
is(scalar(@md), scalar(@mboxrd), 'got expected emails in Maildir');
@md = sort { ${$a->{bdy}} cmp ${$b->{bdy}} } @md;
^ permalink raw reply related [relevance 55%]
* [PATCH 4/4] lei q: import flags when clobbering/augmenting Maildirs
2021-03-03 13:48 55% ` [PATCH 3/4] lei: use maildir_each_eml in more places Eric Wong
@ 2021-03-03 13:48 33% ` Eric Wong
2021-03-03 22:29 71% ` RFH: --import-augment naming [was: lei q: import flags when clobbering/augmenting] Eric Wong
1 sibling, 1 reply; 200+ results
From: Eric Wong @ 2021-03-03 13:48 UTC (permalink / raw)
To: meta
This will eventually be supported for other mail stores,
but Maildir is the easiest to test and support, here.
This lets us avoid a situation where flag changes get
lost between search results.
---
MANIFEST | 1 +
lib/PublicInbox/ExtSearchIdx.pm | 1 +
lib/PublicInbox/LEI.pm | 2 +-
lib/PublicInbox/LeiQuery.pm | 5 +++-
lib/PublicInbox/LeiSearch.pm | 47 +++++++++++++++++++++++++++++++++
lib/PublicInbox/LeiStore.pm | 27 +++++++++----------
lib/PublicInbox/LeiToMail.pm | 33 ++++++++++++++++++-----
lib/PublicInbox/LeiXSearch.pm | 2 +-
t/lei-q-kw.t | 33 +++++++++++++++++++++++
t/lei.t | 3 ++-
t/lei_store.t | 10 ++++++-
11 files changed, 137 insertions(+), 27 deletions(-)
create mode 100644 t/lei-q-kw.t
diff --git a/MANIFEST b/MANIFEST
index 5044e21c..8c9c86a0 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -375,6 +375,7 @@ t/lei-import-nntp.t
t/lei-import.t
t/lei-mirror.t
t/lei-p2q.t
+t/lei-q-kw.t
t/lei-q-remote-import.t
t/lei-q-thread.t
t/lei.t
diff --git a/lib/PublicInbox/ExtSearchIdx.pm b/lib/PublicInbox/ExtSearchIdx.pm
index d0c9c2f7..a17e7579 100644
--- a/lib/PublicInbox/ExtSearchIdx.pm
+++ b/lib/PublicInbox/ExtSearchIdx.pm
@@ -1128,5 +1128,6 @@ no warnings 'once';
*atfork_child = \&PublicInbox::V2Writable::atfork_child;
*idx_shard = \&PublicInbox::V2Writable::idx_shard;
*reindex_checkpoint = \&PublicInbox::V2Writable::reindex_checkpoint;
+*checkpoint = \&PublicInbox::V2Writable::checkpoint;
1;
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 834e399f..1e5b04ca 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -113,7 +113,7 @@ our %CMD = ( # sorted in order of importance/use:
qw(save-as=s output|mfolder|o=s format|f=s dedupe|d=s threads|t+
sort|s=s reverse|r offset=i remote! local! external! pretty
include|I=s@ exclude=s@ only=s@ jobs|j=s globoff|g augment|a
- import-remote! lock=s@
+ import-remote! import-augment! lock=s@
alert=s@ mua=s no-torsocks torsocks=s verbose|v+ quiet|q C=s@),
PublicInbox::LeiQuery::curl_opt(), opt_dash('limit|n=i', '[0-9]+') ],
diff --git a/lib/PublicInbox/LeiQuery.pm b/lib/PublicInbox/LeiQuery.pm
index b57d1cc5..c630d628 100644
--- a/lib/PublicInbox/LeiQuery.pm
+++ b/lib/PublicInbox/LeiQuery.pm
@@ -51,7 +51,10 @@ sub lei_q {
# we'll allow "--only $LOCATION --local"
my $sto = $self->_lei_store(1);
my $lse = $sto->search;
- $sto->write_prepare($self) if $opt->{'import-remote'} //= 1;
+ if (($opt->{'import-remote'} //= 1) |
+ ($opt->{'import-augment'} //= 1)) {
+ $sto->write_prepare($self);
+ }
if ($opt->{'local'} //= scalar(@only) ? 0 : 1) {
$lxs->prepare_external($lse);
}
diff --git a/lib/PublicInbox/LeiSearch.pm b/lib/PublicInbox/LeiSearch.pm
index 440bacf5..ceb3624b 100644
--- a/lib/PublicInbox/LeiSearch.pm
+++ b/lib/PublicInbox/LeiSearch.pm
@@ -1,11 +1,14 @@
# Copyright (C) 2020-2021 all contributors <meta@public-inbox.org>
# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+# read-only counterpart for PublicInbox::LeiStore
package PublicInbox::LeiSearch;
use strict;
use v5.10.1;
use parent qw(PublicInbox::ExtSearch);
use PublicInbox::Search qw(xap_terms);
+use PublicInbox::ContentHash qw(content_digest content_hash);
+use PublicInbox::MID qw(mids mids_in);
# get combined docid from over.num:
# (not generic Xapian, only works with our sharding scheme)
@@ -24,4 +27,48 @@ sub msg_keywords {
wantarray ? sort(keys(%$kw)) : $kw;
}
+# when a message has no Message-IDs at all, this is needed for
+# unsent Draft messages, at least
+sub content_key ($) {
+ my ($eml) = @_;
+ my $dig = content_digest($eml);
+ my $chash = $dig->clone->digest;
+ my $mids = mids_in($eml,
+ qw(Message-ID X-Alt-Message-ID Resent-Message-ID));
+ unless (@$mids) {
+ $eml->{-lei_fake_mid} = $mids->[0] =
+ PublicInbox::Import::digest2mid($dig, $eml);
+ }
+ ($chash, $mids);
+}
+
+sub _cmp_1st { # git->cat_async callback
+ my ($bref, $oid, $type, $size, $cmp) = @_; # cmp: [chash, found, smsg]
+ return if defined($cmp->[1]->[0]); # $found->[0]
+ if (content_hash(PublicInbox::Eml->new($bref)) eq $cmp->[0]) {
+ push @{$cmp->[1]}, $cmp->[2]->{num};
+ }
+}
+
+# returns true if $eml is indexed by lei/store and keywords don't match
+sub kw_changed {
+ my ($self, $eml, $new_kw_sorted) = @_;
+ my ($chash, $mids) = content_key($eml);
+ my $over = $self->over;
+ my $git = $self->git;
+ my $found = [];
+ for my $mid (@$mids) {
+ my ($id, $prev);
+ while (my $cur = $over->next_by_mid($mid, \$id, \$prev)) {
+ $git->cat_async($cur->{blob}, \&_cmp_1st,
+ [ $chash, $found, $cur ]);
+ last if scalar(@$found);
+ }
+ }
+ $git->cat_async_wait;
+ my $num = $found->[0] // return;
+ my @cur_kw = msg_keywords($self, $num);
+ join("\0", @$new_kw_sorted) eq join("\0", @cur_kw) ? 0 : 1;
+}
+
1;
diff --git a/lib/PublicInbox/LeiStore.pm b/lib/PublicInbox/LeiStore.pm
index 77601828..92c29100 100644
--- a/lib/PublicInbox/LeiStore.pm
+++ b/lib/PublicInbox/LeiStore.pm
@@ -14,8 +14,8 @@ use PublicInbox::ExtSearchIdx;
use PublicInbox::Import;
use PublicInbox::InboxWritable qw(eml_from_path);
use PublicInbox::V2Writable;
-use PublicInbox::ContentHash qw(content_hash content_digest);
-use PublicInbox::MID qw(mids mids_in);
+use PublicInbox::ContentHash qw(content_hash);
+use PublicInbox::MID qw(mids);
use PublicInbox::LeiSearch;
use PublicInbox::MDA;
use List::Util qw(max);
@@ -104,25 +104,13 @@ sub eidx_init {
$eidx;
}
-# when a message has no Message-IDs at all, this is needed for
-# unsent Draft messages, at least
-sub _fake_mid_for ($$) {
- my ($eml, $dig) = @_;
- my $mids = mids_in($eml, qw(X-Alt-Message-ID Resent-Message-ID));
- $eml->{-lei_fake_mid} =
- $mids->[0] // PublicInbox::Import::digest2mid($dig, $eml);
-}
-
sub _docids_for ($$) {
my ($self, $eml) = @_;
my %docids;
- my $dig = content_digest($eml);
- my $chash = $dig->clone->digest;
+ my ($chash, $mids) = PublicInbox::LeiSearch::content_key($eml);
my $eidx = eidx_init($self);
my $oidx = $eidx->{oidx};
my $im = $self->{im};
- my $mids = mids($eml);
- $mids->[0] //= _fake_mid_for($eml, $dig);
for my $mid (@$mids) {
my ($id, $prev);
while (my $cur = $oidx->next_by_mid($mid, \$id, \$prev)) {
@@ -183,6 +171,7 @@ sub mbox_keywords {
sort(keys %kw);
}
+# TODO: move this to MdirReader, maybe...
# cf: https://cr.yp.to/proto/maildir.html
my %c2kw = ('D' => 'draft', F => 'flagged', R => 'answered', S => 'seen');
sub maildir_keywords {
@@ -230,6 +219,14 @@ sub set_eml_from_maildir {
set_eml($self, $eml, $set_kw ? maildir_keywords($f) : ());
}
+sub checkpoint {
+ my ($self, $wait) = @_;
+ if (my $im = $self->{im}) {
+ $wait ? $im->barrier : $im->checkpoint;
+ }
+ $self->{priv_eidx}->checkpoint($wait);
+}
+
sub done {
my ($self) = @_;
my $err = '';
diff --git a/lib/PublicInbox/LeiToMail.pm b/lib/PublicInbox/LeiToMail.pm
index 31b8aba8..3420b06e 100644
--- a/lib/PublicInbox/LeiToMail.pm
+++ b/lib/PublicInbox/LeiToMail.pm
@@ -267,8 +267,8 @@ sub _mbox_write_cb ($$) {
}
}
-sub _augment_file { # maildir_each_eml cb
- my ($f, undef, $eml, $lei, $mod, $shard) = @_;
+sub _augment_or_unlink { # maildir_each_eml cb
+ my ($f, $kw, $eml, $lei, $lse, $mod, $shard, $unlink) = @_;
if ($mod) {
# can't get dirent.d_ino w/ pure Perl, so we extract the OID
# if it looks like one:
@@ -276,8 +276,16 @@ sub _augment_file { # maildir_each_eml cb
$1 : sha256_hex($f);
my $recno = hex(substr($hex, 0, 8));
return if ($recno % $mod) != $shard;
+ if ($lse) {
+ my $x = $lse->kw_changed($eml, $kw);
+ if ($x) {
+ $lei->{sto}->ipc_do('set_eml', $eml, @$kw);
+ } elsif (!defined($x)) {
+ # TODO: xkw
+ }
+ }
}
- _augment($eml, $lei);
+ $unlink ? unlink($f) : _augment($eml, $lei);
}
# maildir_each_file callback, \&CORE::unlink doesn't work with it
@@ -419,20 +427,31 @@ sub _pre_augment_maildir {
sub _do_augment_maildir {
my ($self, $lei) = @_;
my $dst = $lei->{ovv}->{dst};
+ my $lse = $lei->{sto}->search if $lei->{opt}->{'import-augment'};
+ my ($mod, $shard) = @{$self->{shard_info} // []};
if ($lei->{opt}->{augment}) {
my $dedupe = $lei->{dedupe};
if ($dedupe && $dedupe->prepare_dedupe) {
- my ($mod, $shard) = @{$self->{shard_info} // []};
PublicInbox::MdirReader::maildir_each_eml($dst,
- \&_augment_file,
- $lei, $mod, $shard);
+ \&_augment_or_unlink,
+ $lei, $lse, $mod, $shard);
$dedupe->pause_dedupe;
}
- } else { # clobber existing Maildir
+ } elsif ($lse) {
+ PublicInbox::MdirReader::maildir_each_eml($dst,
+ \&_augment_or_unlink,
+ $lei, $lse, $mod, $shard, 1);
+ } else {# clobber existing Maildir
PublicInbox::MdirReader::maildir_each_file($dst, \&_unlink);
}
}
+sub _post_augment_maildir {
+ my ($self, $lei) = @_;
+ $lei->{opt}->{'import-augment'} or return;
+ my $wait = $lei->{sto}->ipc_do('checkpoint', 1);
+}
+
sub _augment_imap { # PublicInbox::NetReader::imap_each cb
my ($url, $uid, $kw, $eml, $lei) = @_;
_augment($eml, $lei);
diff --git a/lib/PublicInbox/LeiXSearch.pm b/lib/PublicInbox/LeiXSearch.pm
index dcc48806..45815180 100644
--- a/lib/PublicInbox/LeiXSearch.pm
+++ b/lib/PublicInbox/LeiXSearch.pm
@@ -204,7 +204,7 @@ sub query_mset { # non-parallel for non-"--threads" users
sub each_remote_eml { # callback for MboxReader->mboxrd
my ($eml, $self, $lei, $each_smsg) = @_;
- $lei->{sto}->ipc_do('add_eml', $eml) if $lei->{sto}; # --import-remote
+ $lei->{sto}->ipc_do('add_eml', $eml) if $lei->{opt}->{'import-remote'};
my $smsg = bless {}, 'PublicInbox::Smsg';
$smsg->populate($eml);
$smsg->parse_references($eml, mids($eml));
diff --git a/t/lei-q-kw.t b/t/lei-q-kw.t
new file mode 100644
index 00000000..97b2e08f
--- /dev/null
+++ b/t/lei-q-kw.t
@@ -0,0 +1,33 @@
+#!perl -w
+# Copyright (C) 2020-2021 all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+use strict; use v5.10.1; use PublicInbox::TestCommon;
+test_lei(sub {
+lei_ok(qw(import -F eml t/plack-qp.eml));
+my $o = "$ENV{HOME}/dst";
+lei_ok(qw(q -o), "maildir:$o", qw(m:qp@example.com));
+my @fn = glob("$o/cur/*:2,");
+scalar(@fn) == 1 or BAIL_OUT "wrote multiple or zero files: ".explain(\@fn);
+rename($fn[0], "$fn[0]S") or BAIL_OUT "rename $!";
+
+lei_ok(qw(q -o), "maildir:$o", qw(m:bogus-noresults@example.com));
+ok(!glob("$o/cur/*"), 'last result cleared after augment-import');
+
+lei_ok(qw(q -o), "maildir:$o", qw(m:qp@example.com));
+@fn = glob("$o/cur/*:2,S");
+is(scalar(@fn), 1, "`seen' flag set on Maildir file");
+
+# ensure --no-import-augment works
+my $n = $fn[0];
+$n =~ s/,S\z/,RS/;
+rename($fn[0], $n) or BAIL_OUT "rename $!";
+lei_ok(qw(q --no-import-augment -o), "maildir:$o",
+ qw(m:bogus-noresults@example.com));
+ok(!glob("$o/cur/*"), '--no-import-augment cleared destination');
+lei_ok(qw(q -o), "maildir:$o", qw(m:qp@example.com));
+@fn = glob("$o/cur/*:2,S");
+is(scalar(@fn), 1, "`seen' flag (but not `replied') set on Maildir file");
+
+# TODO: other destination types
+});
+done_testing;
diff --git a/t/lei.t b/t/lei.t
index ba179b39..74a775ca 100644
--- a/t/lei.t
+++ b/t/lei.t
@@ -138,7 +138,8 @@ SKIP: {
lei(qw(q --only http://127.0.0.1:99999/bogus/ t:m));
is($? >> 8, 3, 'got curl exit for bogus URL');
lei(qw(q --only http://127.0.0.1:99999/bogus/ t:m -o), "$home/junk");
- is($? >> 8, 3, 'got curl exit for bogus URL with Maildir');
+ is($? >> 8, 3, 'got curl exit for bogus URL with Maildir') or
+ diag $lei_err;
is($lei_out, '', 'no output');
}; # /SKIP
};
diff --git a/t/lei_store.t b/t/lei_store.t
index e93fe779..1c3f7841 100644
--- a/t/lei_store.t
+++ b/t/lei_store.t
@@ -124,8 +124,16 @@ SKIP: {
$ids = $sto->ipc_do('set_eml', $eml, qw(seen answered));
is_deeply($ids, [ $no_mid->{num} ], 'docid returned w/o mid w/o ipc');
$wait = $sto->ipc_do('done');
- @kw = $sto->search->msg_keywords($no_mid->{num});
+
+ my $lse = $sto->search;
+ @kw = $lse->msg_keywords($no_mid->{num});
is_deeply(\@kw, [qw(answered seen)], 'set changed kw w/o ipc');
+ is($lse->kw_changed($eml, [qw(answered seen)]), 0,
+ 'kw_changed false when unchanged');
+ is($lse->kw_changed($eml, [qw(answered seen flagged)]), 1,
+ 'kw_changed true when +flagged');
+ is($lse->kw_changed(eml_load('t/plack-qp.eml'), ['seen']), undef,
+ 'kw_changed undef on unknown message');
}
done_testing;
^ permalink raw reply related [relevance 33%]
Results 801-1000 of ~1329 next (older) | prev (newer) | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2021-03-03 13:48 [PATCH 0/4] lei q: avoiding accidental data loss Eric Wong
2021-03-03 13:48 55% ` [PATCH 3/4] lei: use maildir_each_eml in more places Eric Wong
2021-03-03 13:48 33% ` [PATCH 4/4] lei q: import flags when clobbering/augmenting Maildirs Eric Wong
2021-03-03 22:29 71% ` RFH: --import-augment naming [was: lei q: import flags when clobbering/augmenting] Eric Wong
2021-03-04 2:39 71% ` Kyle Meyer
2021-03-04 3:31 70% ` Eric Wong
2021-03-04 4:10 71% ` Kyle Meyer
2021-03-04 9:03 71% [PATCH 0/6] lei q --import-augment => --import-before; mbox + IMAP Eric Wong
2021-03-04 9:03 43% ` [PATCH 1/6] lei q: support --import-augment for IMAP Eric Wong
2021-03-04 9:03 71% ` [PATCH 2/6] lei: dclose: do not EPOLL_CTL_DEL w/o event_init Eric Wong
2021-03-04 9:03 43% ` [PATCH 4/6] lei q: --import-augment for mbox and mbox.gz Eric Wong
2021-03-04 9:03 47% ` [PATCH 6/6] lei q: s/import-augment/import-before/g Eric Wong
2021-03-04 18:43 71% angle brackets in "m:" and "refs:" in "lei q" JSON Eric Wong
2021-03-06 18:26 71% ` Kyle Meyer
2021-03-08 8:08 53% ` [PATCH] lei q: remove angle brackets around Message-IDs Eric Wong
2021-03-05 1:38 51% [PATCH] lei q: fix --import-before default and FIFO output Eric Wong
2021-03-05 2:22 63% "lei q" vs mairix notes Eric Wong
2021-03-05 4:03 66% [PATCH] lei q: one -t shouldn't set `flagged' on external mail Eric Wong
2021-03-05 22:20 64% release timelines (-extindex, JMAP, lei) Eric Wong
2021-03-06 18:31 71% ` Kyle Meyer
2021-03-08 2:54 71% ` Eric Wong
2021-03-08 21:33 71% ` Konstantin Ryabitsev
2021-03-08 22:16 71% ` Eric Wong
2021-03-10 13:23 [PATCH 0/5] no trash, glossary doc Eric Wong
2021-03-10 13:23 67% ` [PATCH 3/5] lei import: simplify Maildir handling Eric Wong
2021-03-10 13:23 68% ` [PATCH 4/5] lei import: skip trashed Maildir messages Eric Wong
2021-03-11 22:43 71% final "null" in "lei q" JSON output Eric Wong
2021-03-12 4:22 71% ` Kyle Meyer
2021-03-12 10:39 71% [PATCH 0/3] lei CLI option updates Eric Wong
2021-03-12 10:39 71% ` [PATCH 1/3] lei: add help + completion for --no-external Eric Wong
2021-03-12 10:39 51% ` [PATCH 2/3] lei: rearrange OPT_DESC and drop some TBD switches Eric Wong
2021-03-12 10:39 62% ` [PATCH 3/3] lei q: mbox*: disable changing parallelism, add --rsyncable Eric Wong
2021-03-14 11:12 40% [PATCH] lei q: do not import unnecessarily from externals Eric Wong
2021-03-15 9:32 71% [PATCH] lei: reuse LeiStore object on config changes Eric Wong
2021-03-19 12:35 [PATCH 0/2] newline rejection for new stuff Eric Wong
2021-03-19 12:35 53% ` [PATCH 1/2] lei: disallow "\n" in local externals paths Eric Wong
2021-03-19 12:41 71% [PATCH] t/lei-externals: add diagnostic for warning Eric Wong
2021-03-19 22:38 59% [PATCH] lei q: -I/--include overrides --no-(external|local|remote) Eric Wong
2021-03-20 10:04 67% [PATCH 0/5] lei: preserve keywords across queries Eric Wong
2021-03-20 10:04 33% ` [PATCH 1/5] lei: All Local Externals: bare git dir for alternates Eric Wong
2021-03-20 10:04 25% ` [PATCH 2/5] lei q: support vmd for external-only messages Eric Wong
2021-03-20 10:04 64% ` [PATCH 3/5] lei q: put keywords on one line in --pretty output Eric Wong
2021-03-20 10:04 59% ` [PATCH 5/5] lei: tie ALE lifetime to config file Eric Wong
2021-03-20 12:40 63% [PATCH] lei q: trim JSON output Eric Wong
2021-03-21 9:50 71% [PATCH 0/3] lei import fix, other fixes Eric Wong
2021-03-21 9:50 36% ` [PATCH 1/3] lei import: vivify external-only messages Eric Wong
2021-03-21 9:50 57% ` [PATCH 2/3] lei q: fix warning on remote imports Eric Wong
2021-03-21 9:50 52% ` [PATCH 3/3] lei: fix some warnings in tests Eric Wong
2021-03-21 11:24 59% [PATCH] lei: simplify lazy-loading Eric Wong
2021-03-22 7:53 70% [PATCH 0/8] lei input handling improvements Eric Wong
2021-03-22 7:53 33% ` [PATCH 1/8] lei: support -c <name>=<value> to overrides Eric Wong
2021-03-22 7:53 38% ` [PATCH 3/8] lei: share input code between convert and import Eric Wong
2021-03-22 7:53 64% ` [PATCH 4/8] lei: simplify workers_start and callers Eric Wong
2021-03-22 7:54 55% ` [PATCH 8/8] lei import: ignore Status headers in "eml" messages Eric Wong
2021-03-23 5:02 71% [PATCH 0/2] lei mark: volatile metadata tagging Eric Wong
2021-03-23 5:02 29% ` [PATCH 1/2] lei mark: command for (un)setting keywords and labels Eric Wong
2021-03-23 5:02 56% ` [PATCH 2/2] lei mark: add support for (bash) completion Eric Wong
2021-03-23 6:51 56% [PATCH] lei: hide *_atfork_child from command-line Eric Wong
2021-03-23 11:48 71% [PATCH 0/5] lei: more input + worker-related stuff Eric Wong
2021-03-23 11:48 70% ` [PATCH 2/5] test_common: check lei/errors.log Eric Wong
2021-03-23 11:48 71% ` [PATCH 3/5] lei: persistent workers (lei_store) run in / Eric Wong
2021-03-23 11:48 47% ` [PATCH 5/5] lei: improve management around short-lived workers Eric Wong
2021-03-24 9:23 70% [PATCH 0/9] lei: various corner case leak fixes Eric Wong
2021-03-24 9:23 71% ` [PATCH 3/9] lei: drop circular reference in lei_store process Eric Wong
2021-03-24 9:23 71% ` [PATCH 4/9] lei: update {3} after -C chdirs Eric Wong
2021-03-24 9:23 61% ` [PATCH 5/9] lei: clean up pkt_op consumer on exception, too Eric Wong
2021-03-24 9:23 58% ` [PATCH 9/9] lei-daemon: do not leak FDs on bogus requests Eric Wong
2021-03-25 4:20 68% [PATCH 00/10] lei testing improvements Eric Wong
2021-03-25 4:20 71% ` [PATCH 02/10] lei: janky $PATH2CFG garbage collection Eric Wong
2021-03-25 4:20 71% ` [PATCH 04/10] lei add-external: do not initialize writable store Eric Wong
2021-03-25 4:20 57% ` [PATCH 07/10] tests: "check-run" uses persistent lei daemon Eric Wong
2021-03-25 4:20 52% ` [PATCH 08/10] lei import: force store, improve test diagnostics Eric Wong
2021-03-25 4:20 56% ` [PATCH 10/10] t/lei: add more diagnostics for failures Eric Wong
2021-03-25 5:22 71% is "lei mark" a good name? Eric Wong
2021-03-26 1:54 71% ` Kyle Meyer
2021-03-26 9:48 68% ` Eric Wong
2021-03-28 2:21 71% ` Eric Wong
2021-03-25 8:32 71% does "lei init" even need to exist? Eric Wong
2021-03-26 1:15 71% ` Kyle Meyer
2021-03-26 4:29 71% [PATCH 0/3] lei labels support Eric Wong
2021-03-26 4:29 71% ` [PATCH 2/3] lei: _lei_store: use default even if unconfigured Eric Wong
2021-03-26 5:01 71% ` [SQUASH 4/3] lei: account for unconfigured leistore.dir Eric Wong
2021-03-26 4:29 30% ` [PATCH 3/3] lei: add some labels support Eric Wong
2021-03-26 10:31 69% ` labels for externals [was: lei labels support] Eric Wong
2021-03-26 9:51 71% [PATCH 0/4] lei minor things Eric Wong
2021-03-26 9:51 68% ` [PATCH 1/4] lei q: skip lei/store->write_prepare for JSON outputs Eric Wong
2021-03-26 9:51 90% ` [PATCH 2/4] lei: do not blindly commit to lei/store on close Eric Wong
2021-03-26 9:51 42% ` [PATCH 3/4] lei: support /dev/fd/[0-2] inputs and outputs in daemon Eric Wong
2021-03-26 9:51 71% ` [PATCH 4/4] lei mark: disallow '!' in labels Eric Wong
2021-03-27 11:45 90% [PATCH 0/4] lei blob (formerly known as "lei show") Eric Wong
2021-03-27 11:45 90% ` [PATCH 2/4] lei help: move "lei help" into LeiHelp.pm Eric Wong
2021-03-27 11:45 38% ` [PATCH 4/4] lei blob: aka "git-show-harder" for blobs Eric Wong
2021-03-27 20:20 71% ` [SQUASH] lei blob: use absolute path Eric Wong
2021-03-27 23:22 69% [PATCH] lei mark: relax label requirements Eric Wong
2021-03-28 9:01 67% [PATCH 00/12] lei blob and some yak-shaving Eric Wong
2021-03-28 9:01 43% ` [PATCH 01/12] lei: simplify PktOp callers Eric Wong
2021-03-28 9:01 58% ` [PATCH 02/12] lei init: split out into separate file Eric Wong
2021-03-28 9:01 71% ` [PATCH 03/12] lei blob: dclose if already failed Eric Wong
2021-03-28 9:01 57% ` [PATCH 04/12] lei blob: support --no-mail switch Eric Wong
2021-03-28 9:01 71% ` [PATCH 05/12] lei blob: fail early if no git dirs Eric Wong
2021-03-28 9:01 67% ` [PATCH 06/12] lei blob: some extra tests Eric Wong
2021-03-28 9:01 71% ` [PATCH 07/12] lei help: show "NAME=VALUE" properly for -c Eric Wong
2021-03-28 9:01 69% ` [PATCH 08/12] lei blob: flesh out help text Eric Wong
2021-03-28 9:01 44% ` [PATCH 10/12] lei blob: add remote external support Eric Wong
2021-03-28 9:01 66% ` [PATCH 11/12] lei: drop coderepo placeholders, submodule TODO Eric Wong
2021-03-28 9:31 71% ` Eric Wong
2021-03-29 3:11 69% [PATCH 0/8] doc: lei manpages, round 4 Kyle Meyer
2021-03-29 3:11 71% ` [PATCH 1/8] doc lei-q: fix typo in -tt description Kyle Meyer
2021-03-29 3:11 71% ` [PATCH 2/8] doc lei: note --stdin shortcut in synopses Kyle Meyer
2021-03-29 3:11 71% ` [PATCH 3/8] doc lei: drop an unnecessary to-do comment Kyle Meyer
2021-03-29 3:11 68% ` [PATCH 4/8] doc lei: don't render most to-do comments Kyle Meyer
2021-03-29 3:11 71% ` [PATCH 5/8] doc lei: update manpages with new options Kyle Meyer
2021-03-29 3:11 30% ` [PATCH 6/8] doc lei: add manpages for new commands Kyle Meyer
2021-03-29 3:25 71% ` Eric Wong
2021-03-29 3:35 71% ` Kyle Meyer
2021-03-29 3:11 90% ` [PATCH 7/8] doc lei overview: note that lei-init is usually unnecessary Kyle Meyer
2021-03-29 3:11 71% ` [PATCH 8/8] doc lei overview: better explain routes into local store Kyle Meyer
2021-03-29 3:18 71% ` [PATCH 0/8] doc: lei manpages, round 4 Eric Wong
2021-03-29 7:08 71% [PATCH 0/3] lei input improvements Eric Wong
2021-03-29 7:08 70% ` [PATCH 2/3] lei: use IO::Uncompress::Gunzip MultiStream Eric Wong
2021-03-29 8:04 [PATCH 0/4] doc: some clarifications and warnings Eric Wong
2021-03-29 8:04 71% ` [PATCH 1/4] doc: lei q: drop NNTP from --output description Eric Wong
2021-03-29 8:04 71% ` [PATCH 2/4] doc: lei q: add warning for --output clobbering Eric Wong
2021-03-29 8:04 71% ` [PATCH 3/4] doc: lei q: clarify default output as stdout Eric Wong
2021-03-29 8:04 67% ` [PATCH 4/4] doc: lei: update description, add warnings Eric Wong
2021-03-29 17:47 71% [PATCH] lei blob: cleanup solver tmpdir on failure Eric Wong
2021-03-30 9:10 70% [PATCH] lei q: avoid redundant default setting for sort with l2m Eric Wong
2021-03-30 9:39 51% [PATCH] lei tag: rename from "lei mark" Eric Wong
2021-03-30 22:29 62% [PATCH] lei: fix IMAP auth failure handling Eric Wong
2021-03-31 0:41 71% [PATCH 0/2] lei doc: mail formats Eric Wong
2021-03-31 0:41 46% ` [PATCH 1/2] doc: add lei-mail-formats(5) manpage Eric Wong
2021-03-31 3:15 71% ` Kyle Meyer
2021-03-31 0:41 66% ` [PATCH 2/2] doc: lei-overview: favor Maildir for mutt examples Eric Wong
2021-03-31 1:53 54% [PATCH] lei blob: "--mail" disables solver, use --include/only Eric Wong
2021-03-31 23:29 62% [PATCH] script/lei: background ourselves on MUA/pager exec Eric Wong
2021-04-01 9:32 71% [PATCH 0/2] lei sucks Eric Wong
2021-04-01 9:32 55% ` [PATCH 2/2] lei sucks: sub-command to aid bug reporting Eric Wong
2021-04-01 10:12 71% [PATCH] lei: maildir: handle "forwarded" keyword as "P" Eric Wong
2021-04-01 12:10 [PATCH 0/5] quieter and less noisy Eric Wong
2021-04-01 12:10 84% ` [PATCH 2/5] lei q: reduce lei/store work for kw changes to stored mail Eric Wong
2021-04-02 9:42 57% [PATCH] lei: fix git-credential handling Eric Wong
2021-04-05 4:17 71% ` Kyle Meyer
2021-04-05 8:36 71% ` [PATCH] script/lei: waitpid for git-credential and pager Eric Wong
2021-04-05 8:59 71% ` [RFC] script/lei: don't setsid on MUA spawn Eric Wong
2021-04-05 21:26 71% ` [PATCH] script/lei: waitpid for git-credential and pager Kyle Meyer
2021-04-03 1:37 71% [PATCH 0/2] lei MUA UX fixes Eric Wong
2021-04-03 1:37 71% ` [PATCH 1/2] lei q: don't show remote progress if MUA is running Eric Wong
2021-04-03 1:37 71% ` [PATCH 2/2] lei: allow progress to non-TTY after MUA spawn Eric Wong
2021-04-03 2:24 71% [PATCH 0/6] lei auth-related fixes Eric Wong
2021-04-03 2:24 57% ` [PATCH 2/6] lei q: ensure wq workers shutdown on IMAP auth failures Eric Wong
2021-04-03 2:24 68% ` [PATCH 3/6] lei tag: fix tagging of IMAP inputs Eric Wong
2021-04-03 2:24 71% ` [PATCH 5/6] net_reader: fix read-only "lei convert" auth failures Eric Wong
2021-04-03 2:24 69% ` [PATCH 6/6] xt/lei-auth-fail: test more failure cases Eric Wong
2021-04-03 10:48 71% [PATCH 0/5] lei/store: fix epoch roll, better errors Eric Wong
2021-04-03 10:48 71% ` [PATCH 3/5] lei tag: note message mismatches on failure Eric Wong
2021-04-03 10:48 55% ` [PATCH 4/5] lei: improve handling of Message-ID-less draft messages Eric Wong
2021-04-03 10:48 43% ` [PATCH 5/5] lei/store: (more) synchronous non-fatal error output Eric Wong
2021-04-05 10:27 [PATCH 0/5] lei_to_mail fixes Eric Wong
2021-04-05 10:27 43% ` [PATCH 3/5] lei: maildir: move shard support to MdirReader Eric Wong
2021-04-05 10:27 60% ` [PATCH 5/5] lei q: fix auth IMAP --output with remote mboxrd Eric Wong
2021-04-12 17:32 68% [PATCH] lei blob: quiet "git rev-parse --git-dir" stderr w/o --cwd Eric Wong
2021-04-13 10:54 90% [PATCH 0/5] "lei q --save" + "lei up" Eric Wong
2021-04-13 10:54 32% ` [PATCH 4/5] lei q: start wiring up saved search Eric Wong
2021-04-13 11:25 71% ` Eric Wong
2021-04-13 19:13 71% ` Eric Wong
2021-04-13 10:54 75% ` [PATCH 5/5] lei: add "lei up" to complement "lei q --save" Eric Wong
2021-04-16 23:10 71% [PATCH 0/9] lei saved search usability improvements Eric Wong
2021-04-16 23:10 53% ` [PATCH 1/9] lei q: --save preserves relative time queries Eric Wong
2021-04-16 23:10 69% ` [PATCH 2/9] lei: expose share_path as a method Eric Wong
2021-04-16 23:10 69% ` [PATCH 3/9] lei: saved searches keyed only by path/URL and format Eric Wong
2021-04-16 23:10 71% ` [PATCH 6/9] lei: fix rel2abs Eric Wong
2021-04-16 23:10 50% ` [PATCH 7/9] lei up: support output destination as arg Eric Wong
2021-04-16 23:10 90% ` [PATCH 8/9] lei q --save: avoid lei.q.format Eric Wong
2021-04-16 23:10 61% ` [PATCH 9/9] lei q --save: clobber config file on repeats Eric Wong
2021-04-17 10:24 62% [PATCH] lei up: fix canonicalization of Maildirs Eric Wong
2021-04-17 19:00 63% ` [PATCH 2/] lei up: further improve Maildir canonicalization Eric Wong
2021-04-17 15:53 71% lei q: --stdin confuses --mua Kyle Meyer
2021-04-17 16:04 71% ` Kyle Meyer
2021-04-17 19:00 60% ` [PATCH] lei q: fix MUA spawn after reading query from stdin Eric Wong
2021-04-17 20:13 70% ` Kyle Meyer
2021-04-18 8:40 36% [PATCH] lei ls-search: command to list saved searches Eric Wong
2021-04-19 8:52 71% [PATCH 0/6] lei saved search improvements Eric Wong
2021-04-19 8:52 68% ` [PATCH 1/6] lei: support unlinked/missing saved searches Eric Wong
2021-04-19 8:52 59% ` [PATCH 2/6] lei q: implement import-before default for --save Eric Wong
2021-04-19 8:52 73% ` [PATCH 5/6] lei_saved_search: split "lei q --save" and "lei up" init paths Eric Wong
2021-04-19 8:52 65% ` [PATCH 6/6] lei q: --save and --augment may be combined Eric Wong
2021-04-19 23:48 71% [PATCH 0/4] "lei up --all=local" support Eric Wong
2021-04-19 23:48 71% ` [PATCH 1/4] lei up: fix help output and ARGV handling Eric Wong
2021-04-19 23:49 65% ` [PATCH 3/4] lei up: more error checking for config loading Eric Wong
2021-04-19 23:49 49% ` [PATCH 4/4] lei up: support --all=local Eric Wong
2021-04-20 7:16 71% [PATCH 0/2] lei {edit,forget}-search Eric Wong
2021-04-20 7:16 56% ` [PATCH 1/2] lei forget-search: new command to forget saved searches Eric Wong
2021-04-20 7:16 64% ` [PATCH 2/2] lei edit-search: command to tweak search parameters Eric Wong
2021-04-20 9:17 51% [PATCH] lei-sigpipe: update and move test from xt => t Eric Wong
2021-04-20 20:33 71% t/lei-daemon.t failure when PERL_INLINE_DIRECTORY is set Konstantin Ryabitsev
2021-04-20 20:38 71% ` Eric Wong
2021-04-20 21:37 68% ` Konstantin Ryabitsev
2021-04-20 22:06 71% ` [PATCH] t/lei-daemon: skip inaccessible socket test as root Eric Wong
2021-04-21 15:03 71% ` Konstantin Ryabitsev
2021-04-21 10:03 46% [PATCH] doc: add lei_design_notes.txt and lei-store-format(5) Eric Wong
2021-04-21 18:36 61% [PATCH] lei: share common *done_wait callbacks Eric Wong
2021-04-21 23:50 50% [PATCH] lei: flesh out `forwarded' kw support for Maildir and IMAP Eric Wong
2021-04-22 9:08 71% [PATCH 0/3] lei import: network sync things Eric Wong
2021-04-22 9:08 70% ` [PATCH 1/3] imap_tracker: prepare for use with lei Eric Wong
2021-04-22 9:08 51% ` [PATCH 2/3] lei import: --incremental default for NNTP and IMAP Eric Wong
2021-04-22 9:08 59% ` [PATCH 3/3] lei import|convert: drop --no-kw aliases Eric Wong
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).