* Re: lei-q-remote-import failures [was: [PATCH] t/index-git-times: support non-master default branch]
@ 2021-10-28 14:03 71% ` Thomas Weißschuh
0 siblings, 1 reply; 200+ results
From: Thomas Weißschuh @ 2021-10-28 14:03 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
On 2021-10-27 23:48+0000, Eric Wong wrote:
> Thomas Weißschuh <thomas@t-8ch.de> wrote:
> > It didn't help, neither did larger sleeps.
>
> Thanks for testing.
>
> > I'll see if I can debug it in the coming days.
>
> OK, maybe the patch below can get you started...
>
> I've also been wondering if something like GIT_TRACE is
> necessary, but it's also more work to support + document.
> And Perl code is easily modified compared to AOT languages.
It turned out that in my $XDG_CONFIG_HOME/lei/all_locals_ever.git/lei_ale.state
there were entries for the repositories in
$PUBLIC_INBOX_SRC/t/home2/{t1,t2}.
I'm not sure how those entries came to be but probably because of some
debugging things I did before.
IMO it would make sense to prevent p-i to read this global state during
unittests.
^ permalink raw reply [relevance 71%]
* Re: [PATCH] test_common: clear XDG_CACHE_HOME before lei tests
@ 2021-10-28 19:22 71% ` Thomas Weißschuh
0 siblings, 0 replies; 200+ results
From: Thomas Weißschuh @ 2021-10-28 19:22 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
On 2021-10-28 19:16+0000, Eric Wong wrote:
> Thomas Weißschuh <thomas@t-8ch.de> wrote:
> > It turned out that in my $XDG_CONFIG_HOME/lei/all_locals_ever.git/lei_ale.state
> > there were entries for the repositories in
> > $PUBLIC_INBOX_SRC/t/home2/{t1,t2}.
> >
> > I'm not sure how those entries came to be but probably because of some
> > debugging things I did before.
>
> The presence of t1 & t2 entries is harmless for normal lei
> operation; but they shouldn't be there...
>
> I wonder, do you have XDG_CACHE_HOME explicitly set in your env?
> If so, that would've caused problems (fixed below).
Yes I have it set explicitly
> > IMO it would make sense to prevent p-i to read this global state during
> > unittests.
>
> Yes, I think this was from our failure to clobber XDG_CACHE_HOME:
That works, thanks!
Tested-by: Thomas Weißschuh <thomas@t-8ch.de>
> ------------8<------------
> Subject: [PATCH] test_common: clear XDG_CACHE_HOME before lei tests
>
> We don't want to read a users'
> $XDG_CACHE_HOME/lei/all_locals_ever.git during tests.
>
> Reported-by: Thomas Weißschuh <thomas@t-8ch.de>
> Link: https://public-inbox.org/meta/f239abac-4aee-4573-a0d6-e533c7a32662@t-8ch.de/
> ---
> lib/PublicInbox/TestCommon.pm | 6 +++++-
> 1 file changed, 5 insertions(+), 1 deletion(-)
>
> diff --git a/lib/PublicInbox/TestCommon.pm b/lib/PublicInbox/TestCommon.pm
> index c3820d3a..052d6e45 100644
> --- a/lib/PublicInbox/TestCommon.pm
> +++ b/lib/PublicInbox/TestCommon.pm
> @@ -556,16 +556,20 @@ SKIP: {
> require_git(2.6, 1) or skip('git 2.6+ required for lei test', 2);
> my $mods = $test_opt->{mods} // [ 'lei' ];
> require_mods(@$mods, 2);
> +
> + # set PERL_INLINE_DIRECTORY before clobbering XDG_CACHE_HOME
> + require PublicInbox::Spawn;
> require PublicInbox::Config;
> require File::Path;
> +
> local %ENV = %ENV;
> delete $ENV{XDG_DATA_HOME};
> delete $ENV{XDG_CONFIG_HOME};
> + delete $ENV{XDG_CACHE_HOME};
> $ENV{GIT_COMMITTER_EMAIL} = 'lei@example.com';
> $ENV{GIT_COMMITTER_NAME} = 'lei user';
> my (undef, $fn, $lineno) = caller(0);
> my $t = "$fn:$lineno";
> - require PublicInbox::Spawn;
> state $lei_daemon = PublicInbox::Spawn->can('send_cmd4') ||
> eval { require Socket::MsgHdr; 1 };
> unless ($lei_daemon) {
^ permalink raw reply [relevance 71%]
* [PATCH 5/5] doc: lei-security: add a note about core dumps
2021-10-30 8:11 71% [PATCH 0/5] lei: fix various SIGPIPE problems Eric Wong
@ 2021-10-30 8:11 71% ` Eric Wong
2021-10-30 8:11 71% ` [PATCH 1/5] lei: do not access {sock} after SIGPIPE Eric Wong
1 sibling, 0 replies; 200+ results
From: Eric Wong @ 2021-10-30 8:11 UTC (permalink / raw)
To: meta
Maybe we can avoid them if we stop having buggy code :P
---
Documentation/lei-security.pod | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/Documentation/lei-security.pod b/Documentation/lei-security.pod
index 8cbd89934568..104bfb48a26c 100644
--- a/Documentation/lei-security.pod
+++ b/Documentation/lei-security.pod
@@ -64,6 +64,12 @@ public-facing L<public-inbox-daemon(8)> processes. They may
reside on shared storage and may be made world-readable to
other users on the local system.
+=head1 CORE DUMPS
+
+In case any process crashes, a core dumps may contain passwords or
+contents of sensitive messages. Please report these so they can be
+fixed (see L</CONTACT>).
+
=head1 NETWORK ACCESS
lei currently uses the L<curl(1)> and L<git(1)> executables in
^ permalink raw reply related [relevance 71%]
* [PATCH 0/5] lei: fix various SIGPIPE problems
@ 2021-10-30 8:11 71% Eric Wong
2021-10-30 8:11 71% ` [PATCH 5/5] doc: lei-security: add a note about core dumps Eric Wong
2021-10-30 8:11 71% ` [PATCH 1/5] lei: do not access {sock} after SIGPIPE Eric Wong
0 siblings, 2 replies; 200+ results
From: Eric Wong @ 2021-10-30 8:11 UTC (permalink / raw)
To: meta
Most worrying was the the bug fixed in 4/5; but at least there
wasn't data loss involved...
While the bug fixed in 4/5 didn't cause data loss, it was
dumping core files and filling up my disk while polluting the
kernel log buffer.
Eric Wong (5):
lei: do not access {sock} after SIGPIPE
lei_to_mail: limit workers for text, reply and v2 outputs
lei_xsearch: quiet error message on SIG{PIPE,TERM}
lei_to_mail: avoid SEGV on worker exit via SIGTERM
doc: lei-security: add a note about core dumps
Documentation/lei-security.pod | 6 ++++++
lib/PublicInbox/LEI.pm | 2 +-
lib/PublicInbox/LeiQuery.pm | 2 +-
lib/PublicInbox/LeiToMail.pm | 10 +++++++++-
lib/PublicInbox/LeiXSearch.pm | 5 ++++-
t/lei-sigpipe.t | 7 +++++--
6 files changed, 26 insertions(+), 6 deletions(-)
^ permalink raw reply [relevance 71%]
* [PATCH 1/5] lei: do not access {sock} after SIGPIPE
2021-10-30 8:11 71% [PATCH 0/5] lei: fix various SIGPIPE problems Eric Wong
2021-10-30 8:11 71% ` [PATCH 5/5] doc: lei-security: add a note about core dumps Eric Wong
@ 2021-10-30 8:11 71% ` Eric Wong
1 sibling, 0 replies; 200+ results
From: Eric Wong @ 2021-10-30 8:11 UTC (permalink / raw)
To: meta
It's possible for this to break out of the event loop if
note_sigpipe fires via PktOp in the same iteration.
---
lib/PublicInbox/LEI.pm | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 96f7c5e315a9..78b49a3bc1af 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -1127,7 +1127,7 @@ sub event_step {
local %ENV = %{$self->{env}};
local $current_lei = $self;
eval {
- my @fds = $recv_cmd->($self->{sock}, my $buf, 4096);
+ my @fds = $recv_cmd->($self->{sock} // return, my $buf, 4096);
if (scalar(@fds) == 1 && !defined($fds[0])) {
return if $! == EAGAIN;
die "recvmsg: $!" if $! != ECONNRESET;
^ permalink raw reply related [relevance 71%]
* [PATCH 0/2] lei: mail-sync docs + compat fix
@ 2021-10-31 9:10 71% Eric Wong
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2021-10-31 9:10 UTC (permalink / raw)
To: meta
Eric Wong (2):
doc: add lei-mail-sync-overview manpage
lei_input: disallow uppercase characters for labels
Documentation/lei-export-kw.pod | 3 ++
Documentation/lei-forget-mail-sync.pod | 5 ++-
Documentation/lei-import.pod | 34 +++++++++++++---
Documentation/lei-index.pod | 5 ++-
Documentation/lei-ls-label.pod | 1 +
Documentation/lei-ls-mail-source.pod | 5 ++-
Documentation/lei-ls-watch.pod | 3 +-
Documentation/lei-mail-formats.pod | 5 ++-
Documentation/lei-mail-sync-overview.pod | 52 ++++++++++++++++++++++++
Documentation/lei-overview.pod | 13 +++++-
Documentation/lei-tag.pod | 21 +++++++++-
Documentation/lei.pod | 4 +-
MANIFEST | 1 +
Makefile.PL | 4 +-
lib/PublicInbox/LeiInput.pm | 2 +-
15 files changed, 137 insertions(+), 21 deletions(-)
create mode 100644 Documentation/lei-mail-sync-overview.pod
^ permalink raw reply [relevance 71%]
* Re: [PATCH 1/2] doc: add lei-mail-sync-overview manpage
@ 2021-10-31 16:32 71% ` Kyle Meyer
0 siblings, 0 replies; 200+ results
From: Kyle Meyer @ 2021-10-31 16:32 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
Eric Wong writes:
> --- a/Documentation/lei-export-kw.pod
> +++ b/Documentation/lei-export-kw.pod
> @@ -12,6 +12,9 @@ lei export-kw MFOLDER [MFOLDER...]
>
> C<lei export-kw> propagates keywords (e.g. C<seen>, C<answered>,
> C<flagged>, etc.) from lei/store to IMAP folders and/or Maildirs.
> +It only works for messages lei knows about (e.g. was used as a
> +C<lei q --output>, or imported via L<lei-import(1)>, or indexed
> +via L<lei-index(1)>.
The closing paren is missing here.
> --- a/Documentation/lei-index.pod
> +++ b/Documentation/lei-index.pod
> @@ -20,6 +20,9 @@ Combined with L<lei-q(1)>, C<lei index> allows Maildir users to
> have similar functionality to L<mairix(1)> by not duplicating
> messages into C<lei/store>.
>
> +Occasional invocations of C<lei-refresh-mail-sync --all=local>
> +is recommended to keep indexed messages retrievable.
"Occasional invocations ... is" -> "Occasional invocations ... are"?
Or perhaps "Occasionally invoking ... is"
> +=head1 TYPICAL WORKFLOW
> +
> + # import mail from a user's IMAP inbox and give it the "inbox" label:
> + lei import +L:inbox imaps://user@example.com/INBOX
> +
> + # dump "inbox" labeled files from the past week to a Maildir
> + lei q L:inbox rt:last.week.. -o /tmp/results
> +
> + # open /tmp/results in your favorite mail agent. If inotify or kevent
> + # works, keyword changes (e.g. marking messages as `seen').
Is this part incomplete (maybe "are synchronized" is missing at the
end)?
^ permalink raw reply [relevance 71%]
* [PATCH] doc: lei-config: fix missing =back
@ 2021-11-01 19:00 71% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-11-01 19:00 UTC (permalink / raw)
To: meta
---
Documentation/lei-config.pod | 2 ++
1 file changed, 2 insertions(+)
diff --git a/Documentation/lei-config.pod b/Documentation/lei-config.pod
index 154295ce..663404fe 100644
--- a/Documentation/lei-config.pod
+++ b/Documentation/lei-config.pod
@@ -95,6 +95,8 @@ will also be parsed for diff coloring. git diff color slots
(C<color.diff.SLOT>) supported are C<new>, C<old>, C<meta>,
C<frag>, C<func>, and C<context>.
+=back
+
=head1 CONTACT
Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org>
^ permalink raw reply related [relevance 71%]
* [PATCH 0/3] lei: more coherent input handling
@ 2021-11-02 18:14 71% Eric Wong
2021-11-02 18:14 71% ` [PATCH 1/3] lei mail-diff: do not default to 'eml' Eric Wong
0 siblings, 2 replies; 200+ results
From: Eric Wong @ 2021-11-02 18:14 UTC (permalink / raw)
To: meta
Hopefully everything makes more sense to new users.
Eric Wong (3):
lei mail-diff: do not default to 'eml'
lei: simplify common LeiInput users with ->wq1_start
lei <rediff|rm|tag>: stdin implies `-F eml'
lib/PublicInbox/LEI.pm | 15 ++++++++++++++-
lib/PublicInbox/LeiExportKw.pm | 7 +------
lib/PublicInbox/LeiForgetSearch.pm | 7 +------
lib/PublicInbox/LeiImport.pm | 7 +------
lib/PublicInbox/LeiLsMailSource.pm | 7 +------
lib/PublicInbox/LeiMailDiff.pm | 8 +-------
lib/PublicInbox/LeiP2q.pm | 7 +------
lib/PublicInbox/LeiRediff.pm | 7 ++-----
lib/PublicInbox/LeiRefreshMailSync.pm | 7 +------
lib/PublicInbox/LeiRm.pm | 7 ++-----
lib/PublicInbox/LeiTag.pm | 11 +++--------
lib/PublicInbox/LeiUp.pm | 5 +----
12 files changed, 29 insertions(+), 66 deletions(-)
^ permalink raw reply [relevance 71%]
* [PATCH 1/3] lei mail-diff: do not default to 'eml'
2021-11-02 18:14 71% [PATCH 0/3] lei: more coherent input handling Eric Wong
@ 2021-11-02 18:14 71% ` Eric Wong
1 sibling, 0 replies; 200+ results
From: Eric Wong @ 2021-11-02 18:14 UTC (permalink / raw)
To: meta
In retrospect, this doesn't make sense, since it needs at least
two messages to diff. So go about "normal" input rules and
require users to specify the format.
---
lib/PublicInbox/LeiMailDiff.pm | 1 -
1 file changed, 1 deletion(-)
diff --git a/lib/PublicInbox/LeiMailDiff.pm b/lib/PublicInbox/LeiMailDiff.pm
index b21a0c36..48ba74cf 100644
--- a/lib/PublicInbox/LeiMailDiff.pm
+++ b/lib/PublicInbox/LeiMailDiff.pm
@@ -76,7 +76,6 @@ sub input_eml_cb { # used by PublicInbox::LeiInput::input_fh
sub lei_mail_diff {
my ($lei, @argv) = @_;
- $lei->{opt}->{'in-format'} //= 'eml' if !grep(/\A[a-z0-9]+:/i, @argv);
my $self = bless {}, __PACKAGE__;
$self->prepare_inputs($lei, \@argv) or return;
my $isatty = -t $lei->{1};
^ permalink raw reply related [relevance 71%]
* Re: [PATCH 2/3] lei: simplify common LeiInput users with ->wq1_start
@ 2021-11-02 18:47 71% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-11-02 18:47 UTC (permalink / raw)
To: meta
Eric Wong <e@80x24.org> wrote:
> This method replaces a common worker of starting workers,
s/common worker/common pattern/ :x
^ permalink raw reply [relevance 71%]
* Re: [PATCH] doc: lei-q: document SEARCH TERMS prefixes
@ 2021-11-03 0:28 71% ` Kyle Meyer
0 siblings, 0 replies; 200+ results
From: Kyle Meyer @ 2021-11-03 0:28 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
Eric Wong writes:
> +=for comment
> +AUTO-GENERATED-SEARCH-TERMS-BEGIN
Looks great, thanks. By the way, this takes care of a to-do in this
file:
=for comment
TODO: Give common prefixes, or at least a description/reference.
^ permalink raw reply [relevance 71%]
* Initial Fedora packaging for lei
@ 2021-11-03 1:28 71% Konstantin Ryabitsev
0 siblings, 0 replies; 200+ results
From: Konstantin Ryabitsev @ 2021-11-03 1:28 UTC (permalink / raw)
To: meta; +Cc: tools
[-- Attachment #1: Type: text/plain, Size: 622 bytes --]
Hi, all:
I did some initial work to package lei for Fedora 34 and 35 (out today). The
lei parts should be ready to use, though I'll continue to work on the server
parts (only needed if you're running httpd/nntpd/imapd daemons).
For now, you'll need to enable my copr repository to use it:
sudo dnf copr enable icon/b4
sudo dnf install lei
(you can also install python3-b4 from there if you're using b4)
I'm still working on some introductory docs, but you can use the following
resource to get yourself going with search-based lei goodness:
https://josefbacik.github.io/kernel/2021/10/18/lei-and-b4.html
-K
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]
^ permalink raw reply [relevance 71%]
* [PATCH 0/4] "lei q" rawstr fallout + fixes
@ 2021-11-10 10:25 71% Eric Wong
2021-11-10 10:28 71% ` [PATCH 4/4] lei q: make HTTP(S) query strings even less ugly Eric Wong
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2021-11-10 10:25 UTC (permalink / raw)
To: meta
The recent rawstr fix caused some errors for existing
--stdin searches, these patches fix them and make some
minor improvements along the way.
Eric Wong (4):
ipc: note failing sub name
lei up: infer rawstr from old searches via trailing "\n"
lei q: disallow "\n" in argv[] elements
lei q: make HTTP(S) query strings even less ugly
lib/PublicInbox/IPC.pm | 2 +-
lib/PublicInbox/LeiQuery.pm | 1 +
lib/PublicInbox/LeiUp.pm | 4 +++-
lib/PublicInbox/LeiXSearch.pm | 2 +-
t/lei.t | 3 +++
5 files changed, 9 insertions(+), 3 deletions(-)
^ permalink raw reply [relevance 71%]
* [PATCH 4/4] lei q: make HTTP(S) query strings even less ugly
2021-11-10 10:25 71% [PATCH 0/4] "lei q" rawstr fallout + fixes Eric Wong
@ 2021-11-10 10:28 71% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-11-10 10:28 UTC (permalink / raw)
To: meta
Following commit 57fed2e4b78ed394 (lei: normalize whitespace in
remote queries, 2021-09-11), leaving the trailing `\n' from
stdin queries to be normalized to ` ' (SP) causes it to appear
as `+' in URLs, which Xapian ignores.
---
lib/PublicInbox/LeiXSearch.pm | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/lib/PublicInbox/LeiXSearch.pm b/lib/PublicInbox/LeiXSearch.pm
index 29df07e0c8a8..2958d3f910b0 100644
--- a/lib/PublicInbox/LeiXSearch.pm
+++ b/lib/PublicInbox/LeiXSearch.pm
@@ -342,7 +342,7 @@ sub query_remote_mboxrd {
local $SIG{TERM} = sub { exit(0) }; # for DESTROY (File::Temp, $reap)
my $lei = $self->{lei};
my $opt = $lei->{opt};
- my $qstr = $lei->{mset_opt}->{qstr};
+ chomp(my $qstr = $lei->{mset_opt}->{qstr});
$qstr =~ s/[ \n\t]+/ /sg; # make URLs less ugly
my @qform = (x => 'm');
push(@qform, t => 1) if $opt->{threads};
^ permalink raw reply related [relevance 71%]
* [PATCH] t/lei-watch: test with with higher sleep
@ 2021-11-10 10:33 71% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-11-10 10:33 UTC (permalink / raw)
To: meta
0.1s may not be enough for a task switch and inotify wakeup,
so try doubling it and see if it fixes test reliability, for
now. A future change may be to implement a watcher/tracer
for inotify -> lei/store events.
Link: https://public-inbox.org/meta/20211104134327.zrf5jijfz7dsvb7l@meerkat.local/
---
t/lei-watch.t | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/t/lei-watch.t b/t/lei-watch.t
index 9158571c..e6066033 100644
--- a/t/lei-watch.t
+++ b/t/lei-watch.t
@@ -52,7 +52,7 @@ test_lei(sub {
my @f = glob("$md/cur/*:2,");
is(scalar(@f), 1, 'got populated maildir with one result');
rename($f[0], "$f[0]S") or xbail "rename $!"; # set (S)een
- tick($have_fast_inotify ? 0.1 : 2.1); # always needed for 1 CPU systems
+ tick($have_fast_inotify ? 0.2 : 2.2); # always needed for 1 CPU systems
lei_ok qw(note-event done); # flushes immediately (instead of 5s)
lei_ok qw(q mid:testmessage@example.com -o), $md2, '-I', "$ro_home/t1";
^ permalink raw reply related [relevance 71%]
* [PATCH] lei forget-search: add help for --prune
@ 2021-11-12 11:08 71% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-11-12 11:08 UTC (permalink / raw)
To: meta
This enables tab-completion, since I'm using --prune quite a bit
and my fingers are about to fall off :<
---
lib/PublicInbox/LEI.pm | 3 +++
1 file changed, 3 insertions(+)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 887025de5ded..192f267ca1dd 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -411,6 +411,9 @@ my %OPTDESC = (
'url ls-mail-source' => 'show full URL of newsgroup or IMAP folder',
'format|f=s ls-external' => $ls_format,
+'prune:s forget-search' =>
+ ['TYPE|local|remote', 'prune all, remote or local folders' ],
+
'limit|n=i@' => ['NUM', 'limit on number of matches (default: 10000)' ],
'offset=i' => ['OFF', 'search result offset (default: 0)'],
^ permalink raw reply related [relevance 71%]
* lei spawns mua before results are written
@ 2021-11-14 20:41 71% Leah Neukirchen
2021-11-15 0:31 71% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Leah Neukirchen @ 2021-11-14 20:41 UTC (permalink / raw)
To: meta
Hi,
I tried building some tooling to use lei with mblaze and stumbled upon
the fact that lei seems to spawn the mua before all mails are written
into the -o maildir; at least, adding a "sleep 1" in front of my
script fixes it. Any way to wait for it? I guess I could just run lei
and my mua scripts afterwards.
Thanks,
--
Leah Neukirchen <leah@vuxu.org> https://leahneukirchen.org/
^ permalink raw reply [relevance 71%]
* Re: lei spawns mua before results are written
2021-11-14 20:41 71% lei spawns mua before results are written Leah Neukirchen
@ 2021-11-15 0:31 71% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-11-15 0:31 UTC (permalink / raw)
To: Leah Neukirchen; +Cc: meta
Leah Neukirchen <leah@vuxu.org> wrote:
> I tried building some tooling to use lei with mblaze and stumbled upon
> the fact that lei seems to spawn the mua before all mails are written
> into the -o maildir; at least, adding a "sleep 1" in front of my
> script fixes it. Any way to wait for it? I guess I could just run lei
> and my mua scripts afterwards.
Yes, it's intentional to allow Maildir and IMAP to allow MUAs to
work progressively while loading similar to how less(1) and
typical web browsers work. It's not really practical to do for
mbox* because of locking and potential incompatible locking
protocols.
You should be able to use --alert=COMMAND to run any arbitrary
command(s) (including an MUA, as an alternative to --mua).
I'm not sure if adding a --no-early-mua switch is needed. My
gut reaction is having yet another switch to document increases
the learning curve and support overhead... *shrug*
^ permalink raw reply [relevance 71%]
* RFC: should lei inject its own "Received:" header?
@ 2021-11-19 20:49 71% Konstantin Ryabitsev
2021-11-21 10:13 71% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Konstantin Ryabitsev @ 2021-11-19 20:49 UTC (permalink / raw)
To: meta
Hello:
I wonder if lei should inject its own "received"-like header on writing to a
maildir/imap target -- to indicate where the copy of the email came from. I
don't think it should use the actual Received: header, as this may cause some
weird SPF/DMARC issues, but perhaps something like:
X-Lei-Received: from https://lore.kernel.org/all/; Fri, 19 Nov 2021 14:32:44 -0500
Just a random thought.
-K
^ permalink raw reply [relevance 71%]
* lei q error after deleting cache (can't find mail_sync.sqlite3)
@ 2021-11-20 16:36 71% ` Johannes Altmanninger
0 siblings, 0 replies; 200+ results
From: Johannes Altmanninger @ 2021-11-20 16:36 UTC (permalink / raw)
To: meta
On Sat, Nov 20, 2021 at 04:30:57PM +0100, Johannes Altmanninger wrote:
> I've been using lei from the 1.7.0 release, which works great.
>
> I had some unrelated problems with one of the created maildirs so I deleted
> said maildir. Now I can't recreate the maildir. I tried deleting all
> persistent data but got an error:
>
> $ rm -rf the-maildir ~/.local/share/lei ~/.cache/{public-inbox,lei}
> $ lei q -I https://lore.kernel.org/git/ -o the-maildir rt:1.week.ago..
> failed to open /home/johannes/.cache/lei/all_locals_ever.git/lei_ale.state: No such file or directory
> at /usr/share/perl5/vendor_perl/PublicInbox/LeiQuery.pm line 117.
>
> A quick look at strace doesn't show access to any other files in $HOME
This is pretty weird: it works in a temporary home, but only for the first time.
rm -rf the-maildir someuniquename; HOME=$PWD/someuniquename lei q -I https://public-inbox.org/meta/ -o the-maildir rt:1.day.ago..
When I re-run this command, I get the same error
There seems to be no mention of someuniquename anywhere on my disk (except
in my shell history).
^ permalink raw reply [relevance 71%]
* Re: RFC: should lei inject its own "Received:" header?
2021-11-19 20:49 71% RFC: should lei inject its own "Received:" header? Konstantin Ryabitsev
@ 2021-11-21 10:13 71% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-11-21 10:13 UTC (permalink / raw)
To: Konstantin Ryabitsev; +Cc: meta
Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> Hello:
>
> I wonder if lei should inject its own "received"-like header on writing to a
> maildir/imap target -- to indicate where the copy of the email came from. I
> don't think it should use the actual Received: header, as this may cause some
> weird SPF/DMARC issues, but perhaps something like:
>
> X-Lei-Received: from https://lore.kernel.org/all/; Fri, 19 Nov 2021 14:32:44 -0500
>
> Just a random thought.
Probably not in the blob itself. Addition of extra headers
means slower and less-effective dedupe (not just for lei, but
also for deduplicating FSes (which operate at block-layer) and
potential FUSE implementation).
Perhaps the mail_sync.sqlite3 stuff could be taught to track the
origin of a blob, though...
^ permalink raw reply [relevance 71%]
* [PATCH] lei: always use 3-arg open perlop
@ 2021-11-22 18:38 71% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-11-22 18:38 UTC (permalink / raw)
To: meta
Future-proofing in case future versions of Perl warn on this, since
2-arg forms of open may be subject to injection vulnerabilities
with non-literal args.
---
lib/PublicInbox/LEI.pm | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 192f267ca1dd..4e0295fa4e8a 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -818,7 +818,8 @@ sub dispatch {
next if $d eq ''; # same as git(1)
chdir $d or return fail($self, "cd $d: $!");
}
- open $self->{3}, '.' or return fail($self, "open . $!");
+ open $self->{3}, '<', '.' or
+ return fail($self, "open . $!");
}
$cb->($self, @argv);
} elsif (grep(/\A-/, $cmd, @argv)) { # --help or -h only
^ permalink raw reply related [relevance 71%]
* Re: [PATCH] t/lei-sigpipe: attempt to improve diagnostics for stuck test
@ 2022-02-22 6:44 71% ` Eric Wong
2022-02-27 4:15 71% ` Julien Moutinho
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2022-02-22 6:44 UTC (permalink / raw)
To: Julien Moutinho; +Cc: Dominique Martinet, meta
Julien Moutinho <julm+public-inbox@sourcephile.fr> wrote:
> This deadlock appears only when the test is run by nix,
> it doesn't when I run it manually from a terminal
> by entering the sandbox:
Oh, btw, is this with the FD_CLOEXEC fixes to nix shell itself?
https://public-inbox.org/meta/YfnoOVOG3TrbqZFs@codewreck.org/
^ permalink raw reply [relevance 71%]
* Re: [PATCH] t/lei-sigpipe: attempt to improve diagnostics for stuck test
2022-02-22 6:44 71% ` Eric Wong
@ 2022-02-27 4:15 71% ` Julien Moutinho
2022-02-27 6:41 71% ` Julien Moutinho
0 siblings, 1 reply; 200+ results
From: Julien Moutinho @ 2022-02-27 4:15 UTC (permalink / raw)
To: Eric Wong; +Cc: Dominique Martinet, meta
Le mar. 22 févr. 2022 06h44 +0000, Eric Wong a écrit :
> Julien Moutinho <julm+public-inbox@sourcephile.fr> wrote:
> > This deadlock appears only when the test is run by nix,
> > it doesn't when I run it manually from a terminal
> > by entering the sandbox:
>
> Oh, btw, is this with the FD_CLOEXEC fixes to nix shell itself?
> https://public-inbox.org/meta/YfnoOVOG3TrbqZFs@codewreck.org/
Yes, I'm using:
$ nix-daemon --version
> nix-daemon (Nix) 2.7.0pre20220127_558c4ee
Which contains the commit from https://github.com/NixOS/nix/pull/5683 :
$ git merge-base --is-ancestor 6e0cbc666b60515b5e201dd28855f5fe1de9a107 558c4ee
$ echo $?
> 0
^ permalink raw reply [relevance 71%]
* Re: [PATCH] t/lei-sigpipe: attempt to improve diagnostics for stuck test
2022-02-27 4:15 71% ` Julien Moutinho
@ 2022-02-27 6:41 71% ` Julien Moutinho
2022-02-27 7:23 71% ` Dominique Martinet
0 siblings, 1 reply; 200+ results
From: Julien Moutinho @ 2022-02-27 6:41 UTC (permalink / raw)
To: Eric Wong; +Cc: Dominique Martinet, meta
Le dim. 27 févr. 2022 05h15 +0100, Julien Moutinho a écrit :
> Yes, I'm using:
> $ nix-daemon --version
> > nix-daemon (Nix) 2.7.0pre20220127_558c4ee
Dominique reminded me that that fix was first intended for
`nix develop`, so I've asked whether it also applies to `nix build`:
https://github.com/NixOS/nix/pull/5683#issuecomment-1053239025
^ permalink raw reply [relevance 71%]
* Re: [PATCH] t/lei-sigpipe: attempt to improve diagnostics for stuck test
2022-02-27 6:41 71% ` Julien Moutinho
@ 2022-02-27 7:23 71% ` Dominique Martinet
2022-02-27 8:04 71% ` Julien Moutinho
0 siblings, 1 reply; 200+ results
From: Dominique Martinet @ 2022-02-27 7:23 UTC (permalink / raw)
To: Julien Moutinho; +Cc: Eric Wong, meta
Hello,
Julien Moutinho wrote on Sun, Feb 27, 2022 at 07:41:44AM +0100:
> Le dim. 27 févr. 2022 05h15 +0100, Julien Moutinho a écrit :
> > Yes, I'm using:
> > $ nix-daemon --version
> > > nix-daemon (Nix) 2.7.0pre20220127_558c4ee
> Dominique reminded me that that fix was first intended for
> `nix develop`, so I've asked whether it also applies to `nix build`:
> https://github.com/NixOS/nix/pull/5683#issuecomment-1053239025
I've replied over there, I don't think leaked fd is a problem.
However Julien's latest traces gave a good enough hint: the lei process
that should die from sigpipe just didn't die...
And surely enough, nix adds SIGPIPE to ignores.
This can be confirmed by looking at SigIgn in /proc/xxx/status of one of
the builder PIDs:
# grep SigIgn /proc/3124202/status
SigIgn: 0000000000001004
Where 0x1000 is the 13th bit:
$ kill -l 13
PIPE
Running tests manually won't inherit masked signals and thus won't
reproduce.
I guess we can now open a nix issue asking to reset signal handlers
after fork, before executing its builder?...
--
Dominique
^ permalink raw reply [relevance 71%]
* Re: [PATCH] t/lei-sigpipe: attempt to improve diagnostics for stuck test
2022-02-27 7:23 71% ` Dominique Martinet
@ 2022-02-27 8:04 71% ` Julien Moutinho
0 siblings, 0 replies; 200+ results
From: Julien Moutinho @ 2022-02-27 8:04 UTC (permalink / raw)
To: Dominique Martinet; +Cc: Eric Wong, meta
Le dim. 27 févr. 2022 16h23 +0900, Dominique Martinet a écrit :
> I guess we can now open a nix issue asking to reset signal handlers
> after fork, before executing its builder?...
Well spotted Dominique, thanks!
Turns out this is actually systemd which ignores SIGPIPE.
I'm able to pass t/lei-sigpipe.t when my `nix build`
connects to a nix-daemon run manually outside systemd.
There is already a 3yo issue on nix's tracker for that:
https://github.com/NixOS/nix/issues/2803
suggesting to add to nix-daemon.service:
> [Service]
> IgnoreSIGPIPE=no
And that indeed also makes t/lei-sigpipe.t succeed.
In the meantime I'll disable t/lei-sigpipe.t
to finally get public-inbox added to Nixpkgs.
Thank you Dominique and Eric!
^ permalink raw reply [relevance 71%]
* Re: [PATCH] t/lei-sigpipe.t: ensure SIGPIPE is not ignored instead of not blocked
@ 2022-03-14 22:14 71% ` Eric Wong
2022-03-15 2:56 71% ` Julien Moutinho
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2022-03-14 22:14 UTC (permalink / raw)
To: Julien Moutinho; +Cc: meta, Dominique Martinet
Julien Moutinho <julm+public-inbox@sourcephile.fr> wrote:
> Ignoring a signal is different than blocking a signal.
Thanks, pushed as commit f1e4e14793d155ea7d6ed7a6858b668e97c7e5d8
I wasn't sure why this patch was necessary at first. Thus I
took take a closer look at systemd behavior to confirm, and
updated the message:
Ignoring a signal is different than blocking a signal, and the
"IgnoreSIGPIPE" option of systemd ignores.
[ew: note systemd behavior]
Acked-by: Eric Wong <e@80x24.org>
I (and any other project maintainer) would appreciate more
complete explanations in the future as to why a patch is needed.
Thanks again.
^ permalink raw reply [relevance 71%]
* Re: [PATCH] t/lei-sigpipe.t: ensure SIGPIPE is not ignored instead of not blocked
2022-03-14 22:14 71% ` Eric Wong
@ 2022-03-15 2:56 71% ` Julien Moutinho
0 siblings, 0 replies; 200+ results
From: Julien Moutinho @ 2022-03-15 2:56 UTC (permalink / raw)
To: Eric Wong; +Cc: meta, Dominique Martinet
Le lun. 14 mars 2022 22h14 +0000, Eric Wong a écrit :
> I (and any other project maintainer) would appreciate more
> complete explanations in the future as to why a patch is needed.
> Thanks again.
Oh sorry, being immersed into the testing I tought
that it was obvious. I should have clarified
that the unblocking fix was not working.
Thanks for the merge!
^ permalink raw reply [relevance 71%]
* [PATCH 0/4] lei: finish wiring up pure-Perl stuff for Linux
@ 2022-04-18 9:50 71% Eric Wong
2022-04-18 9:50 71% ` [PATCH 1/4] lei: clobber recvmsg buffer on errors Eric Wong
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2022-04-18 9:50 UTC (permalink / raw)
To: meta
Some cleanups here, too; but this hopefully makes lei more
accessible to users who don't have space/bandwidth to use
Inline::C.
I thought I sent these out weeks ago, but Real-Life, power outages
and other lei problems got in the way, I guess :x
Eric Wong (4):
lei: clobber recvmsg buffer on errors
syscall: more idiomatic cmsghdr space allocation
lei: wire up pure Perl sendmsg/recvmsg for Linux users
syscall: golf + more idiomatic buffer initialization
lib/PublicInbox/IPC.pm | 4 ++++
lib/PublicInbox/LEI.pm | 4 ++++
lib/PublicInbox/LeiSelfSocket.pm | 1 +
lib/PublicInbox/Syscall.pm | 13 ++++++-------
lib/PublicInbox/TestCommon.pm | 16 ++++++++++++----
t/lei-daemon.t | 6 +++++-
t/lei-externals.t | 1 +
7 files changed, 33 insertions(+), 12 deletions(-)
^ permalink raw reply [relevance 71%]
* [PATCH 1/4] lei: clobber recvmsg buffer on errors
2022-04-18 9:50 71% [PATCH 0/4] lei: finish wiring up pure-Perl stuff for Linux Eric Wong
@ 2022-04-18 9:50 71% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2022-04-18 9:50 UTC (permalink / raw)
To: meta
It will be necessary when we drop the Inline::C requirement
since the pure Perl Linux syscall recvmsg implementation.
This likely would've caused errors for Socket::MsgHdr users
without Inline::C, but I haven't tested it since it's a rare
configuration.
---
lib/PublicInbox/LEI.pm | 1 +
lib/PublicInbox/LeiSelfSocket.pm | 1 +
2 files changed, 2 insertions(+)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index a7ddc21f..9ab91714 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -1149,6 +1149,7 @@ sub event_step {
if (scalar(@fds) == 1 && !defined($fds[0])) {
return if $! == EAGAIN;
die "recvmsg: $!" if $! != ECONNRESET;
+ $buf = '';
@fds = (); # for open loop below:
}
for (@fds) { open my $rfh, '+<&=', $_ }
diff --git a/lib/PublicInbox/LeiSelfSocket.pm b/lib/PublicInbox/LeiSelfSocket.pm
index dd64b6cf..860020cb 100644
--- a/lib/PublicInbox/LeiSelfSocket.pm
+++ b/lib/PublicInbox/LeiSelfSocket.pm
@@ -29,6 +29,7 @@ sub event_step {
if (scalar(@fds) == 1 && !defined($fds[0])) {
return if $!{EAGAIN};
die "recvmsg: $!" unless $!{ECONNRESET};
+ $buf = '';
} else { # just in case open so perl can auto-close them:
for (@fds) { open my $fh, '+<&=', $_ };
}
^ permalink raw reply related [relevance 71%]
* [PATCH] lei: improve diagnosis of errors from children
@ 2022-04-30 21:04 71% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2022-04-30 21:04 UTC (permalink / raw)
To: meta
Not 100% sure what's going on, but maybe this helps.
---
Anybody else see this?
lib/PublicInbox/LEI.pm | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 89aa4119..d81ca296 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -1394,7 +1394,7 @@ sub wq_done_wait { # dwaitpid callback
local $current_lei = $lei;
my $err_type = $lei->{-err_type};
$? and $lei->child_error($?,
- $err_type ? "$err_type errors during $lei->{cmd}" : ());
+ $err_type ? "$err_type errors during $lei->{cmd} \$?=$?" : ());
$lei->dclose;
}
^ permalink raw reply related [relevance 71%]
* [PATCH] lei refresh-mail-sync: filter NNTP(S) from --all
@ 2022-04-30 21:29 71% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2022-04-30 21:29 UTC (permalink / raw)
To: meta
We currently do not support refresh from NNTP since deletes are
rare with public-inbox NNTP servers; but traditional Usenet
servers do delete/expire messages and we should probably support
that at some point.
---
lib/PublicInbox/LeiRefreshMailSync.pm | 2 ++
1 file changed, 2 insertions(+)
diff --git a/lib/PublicInbox/LeiRefreshMailSync.pm b/lib/PublicInbox/LeiRefreshMailSync.pm
index 7821008f..a60a9a5e 100644
--- a/lib/PublicInbox/LeiRefreshMailSync.pm
+++ b/lib/PublicInbox/LeiRefreshMailSync.pm
@@ -73,6 +73,8 @@ lei mail_sync.sqlite3 uninitialized, see lei-import(1)
EOM
if (defined(my $all = $lei->{opt}->{all})) {
$lms->group2folders($lei, $all, \@folders) or return;
+ # TODO: handle NNTP servers which delete messages
+ @folders = grep(!m!\Anntps?://!, @folders);
} else {
$lms->arg2folder($lei, \@folders); # may die
}
^ permalink raw reply related [relevance 71%]
* [PATCH] lei import: add label completions (+L:$LABEL)
@ 2022-05-02 18:10 71% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2022-05-02 18:10 UTC (permalink / raw)
To: meta
This can probably be added for "lei q", too, but we typically
import first. Labels can probably be made persistent on a
per-folder basis in the future.
---
lib/PublicInbox/LeiImport.pm | 2 ++
1 file changed, 2 insertions(+)
diff --git a/lib/PublicInbox/LeiImport.pm b/lib/PublicInbox/LeiImport.pm
index b9865829..2d91e4c4 100644
--- a/lib/PublicInbox/LeiImport.pm
+++ b/lib/PublicInbox/LeiImport.pm
@@ -117,6 +117,8 @@ sub _complete_import {
my ($lei, @argv) = @_;
my ($re, $cur, $match_cb) = $lei->complete_url_prepare(\@argv);
my @k = $lei->url_folder_cache->keys($argv[-1] // undef, 1);
+ my @L = eval { $lei->_lei_store->search->all_terms('L') };
+ push(@k, map { "+L:$_" } @L);
my @m = map { $match_cb->($_) } @k;
my %f = map { $_ => 1 } (@m ? @m : @k);
if (my $lms = $lei->lms) {
^ permalink raw reply related [relevance 71%]
* Re: Trouble running lei
@ 2022-05-03 11:37 71% ` Eric Wong
2022-05-03 12:50 71% ` Filipe Manana
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2022-05-03 11:37 UTC (permalink / raw)
To: Filipe Manana; +Cc: meta
Filipe Manana <fdmanana@kernel.org> wrote:
> Hello,
>
> I tried both 'master' branch and tag v1.8.0, I did the documented build steps:
>
> $ git clone https://public-inbox.org/public-inbox.git/
> $ cd public-inbox
>
> $ perl Makefile.PL
> $ make
> $ echo $? # success, prints 0
>
> $ cd certs
> $ /usr/bin/perl ./create-certs.perl
> $ cd ..
> $ make test
> (...)
Any messages about missing dependencies when running tests?
(e.g. Xapian)
Inline::C or Socket::Msghdr shouldn't be required with 1.8
on most arches...
> All tests successful.
> Files=157, Tests=6785, 491 wallclock secs ( 0.77 usr 0.13 sys + 39.65
> cusr 14.90 csys = 55.45 CPU)
Fwiw, "make check" or "make check-run" can be a lot faster on SMP.
> Result: PASS
>
> $ make symlink-install
>
> $HOME/bin is included in my $PATH, but when I run 'lei', I get an error:
>
> $ lei q -o ~/Mail/overlay -I https://lore.kernel.org/all -t
> 'dfn:fs/btrfs/* AND rt:3.month.ago..'
> Attempt to reload PublicInbox/LeiXSearch.pm aborted.
Odd, "Attempt to reload %s aborted" coming from perl itself means it
already tried and failed before. (man perldiag)
Were there previous errors from other commands?
Perhaps it's already running lei-daemon as an older version?
"lei daemon-kill" should kill it and it'll restart on the next
command, unless something else got wedged.
I've been meaning to make the daemon stuff "worth it"
w.r.t. automatic updates/flag/keyword sync (inotify/IDLE) but
haven't gotten around to it, yet :x
> Compilation failed in require at
> /home/fdmanana/git/hub/public-inbox/lib/PublicInbox/LeiQuery.pm line
> 74.
>
> This is on a Ubuntu 20.04.3 LTS distro.
> I have it working on a Debian SID box without any problems (it was set
> up several months ago).
>
> Any ideas about what's wrong?
Probably daemon-kill will get rid of an old version that's
already running (and verifying that it's actually dead).
^ permalink raw reply [relevance 71%]
* Re: Trouble running lei
2022-05-03 11:37 71% ` Eric Wong
@ 2022-05-03 12:50 71% ` Filipe Manana
2022-05-03 15:24 71% ` Konstantin Ryabitsev
0 siblings, 1 reply; 200+ results
From: Filipe Manana @ 2022-05-03 12:50 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
On Tue, May 3, 2022 at 12:37 PM Eric Wong <e@80x24.org> wrote:
>
> Filipe Manana <fdmanana@kernel.org> wrote:
> > Hello,
> >
> > I tried both 'master' branch and tag v1.8.0, I did the documented build steps:
> >
> > $ git clone https://public-inbox.org/public-inbox.git/
> > $ cd public-inbox
> >
> > $ perl Makefile.PL
> > $ make
> > $ echo $? # success, prints 0
> >
> > $ cd certs
> > $ /usr/bin/perl ./create-certs.perl
> > $ cd ..
> > $ make test
> > (...)
>
> Any messages about missing dependencies when running tests?
> (e.g. Xapian)
>
> Inline::C or Socket::Msghdr shouldn't be required with 1.8
> on most arches...
>
> > All tests successful.
> > Files=157, Tests=6785, 491 wallclock secs ( 0.77 usr 0.13 sys + 39.65
> > cusr 14.90 csys = 55.45 CPU)
>
> Fwiw, "make check" or "make check-run" can be a lot faster on SMP.
Oh, I didn't know about it. Good to know.
>
> > Result: PASS
> >
> > $ make symlink-install
> >
> > $HOME/bin is included in my $PATH, but when I run 'lei', I get an error:
> >
> > $ lei q -o ~/Mail/overlay -I https://lore.kernel.org/all -t
> > 'dfn:fs/btrfs/* AND rt:3.month.ago..'
> > Attempt to reload PublicInbox/LeiXSearch.pm aborted.
>
> Odd, "Attempt to reload %s aborted" coming from perl itself means it
> already tried and failed before. (man perldiag)
>
> Were there previous errors from other commands?
>
> Perhaps it's already running lei-daemon as an older version?
> "lei daemon-kill" should kill it and it'll restart on the next
> command, unless something else got wedged.
Ah, running "lei daemon-kill" fixed it.
I don't know if I did something wrong before, but after running that,
lei is now working fine.
Thanks!
>
> I've been meaning to make the daemon stuff "worth it"
> w.r.t. automatic updates/flag/keyword sync (inotify/IDLE) but
> haven't gotten around to it, yet :x
>
> > Compilation failed in require at
> > /home/fdmanana/git/hub/public-inbox/lib/PublicInbox/LeiQuery.pm line
> > 74.
> >
> > This is on a Ubuntu 20.04.3 LTS distro.
> > I have it working on a Debian SID box without any problems (it was set
> > up several months ago).
> >
> > Any ideas about what's wrong?
>
> Probably daemon-kill will get rid of an old version that's
> already running (and verifying that it's actually dead).
^ permalink raw reply [relevance 71%]
* Re: Trouble running lei
2022-05-03 12:50 71% ` Filipe Manana
@ 2022-05-03 15:24 71% ` Konstantin Ryabitsev
2022-05-03 20:32 71% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Konstantin Ryabitsev @ 2022-05-03 15:24 UTC (permalink / raw)
To: Filipe Manana; +Cc: Eric Wong, meta
On Tue, May 03, 2022 at 01:50:52PM +0100, Filipe Manana wrote:
> > Perhaps it's already running lei-daemon as an older version?
> > "lei daemon-kill" should kill it and it'll restart on the next
> > command, unless something else got wedged.
>
> Ah, running "lei daemon-kill" fixed it.
> I don't know if I did something wrong before, but after running that,
> lei is now working fine.
I think this is actually a common occurrence. Any way lei-daemon can recognize
when there's a version mismatch between itself and the binary talking to it?
Regards,
-K'
^ permalink raw reply [relevance 71%]
* Re: Trouble running lei
2022-05-03 15:24 71% ` Konstantin Ryabitsev
@ 2022-05-03 20:32 71% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2022-05-03 20:32 UTC (permalink / raw)
To: Konstantin Ryabitsev; +Cc: Filipe Manana, meta
Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> On Tue, May 03, 2022 at 01:50:52PM +0100, Filipe Manana wrote:
> > > Perhaps it's already running lei-daemon as an older version?
> > > "lei daemon-kill" should kill it and it'll restart on the next
> > > command, unless something else got wedged.
> >
> > Ah, running "lei daemon-kill" fixed it.
> > I don't know if I did something wrong before, but after running that,
> > lei is now working fine.
Good to know!
> I think this is actually a common occurrence. Any way lei-daemon can recognize
> when there's a version mismatch between itself and the binary talking to it?
Sorta, but even I manage to lose track of which commits cause
protocol or internal API changes since everything is lazy-loaded.
I think the best way would be to have inotify/kqueue watch the /PublicInbox/
directory and auto-restart iff no active commands are running...
^ permalink raw reply [relevance 71%]
* lei missing mails
@ 2022-06-29 16:15 71% Rob Herring
2022-06-29 16:30 71% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Rob Herring @ 2022-06-29 16:15 UTC (permalink / raw)
To: meta
Hi,
I'm using lei with lore where I have 2 queries which overlap. Really,
one is a subset of the other. On those overlapping threads, I'm
finding that sometimes new messages are written to one mailbox and not
the other. (At least sometimes, the messages may be missing from all
mailboxes sometimes too. I'm not certain.) Using --remote-fudge-time
to force refetching seems to get the missing mails. I haven't found
anything strange in timestamps of the missing mails, but otherwise am
not sure how to debug this further. The queries are retrieving full
threads and the missing mails are in the threads, but not direct
matches to the queries. I realize that's not a lot of detail to go on.
Suggestions on debugging this further?
It might be helpful if lei could print out message-ids of messages
written to mailboxes.
Rob
^ permalink raw reply [relevance 71%]
* Re: lei missing mails
2022-06-29 16:15 71% lei missing mails Rob Herring
@ 2022-06-29 16:30 71% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2022-06-29 16:30 UTC (permalink / raw)
To: Rob Herring; +Cc: meta
Rob Herring <robh@kernel.org> wrote:
> Hi,
>
> I'm using lei with lore where I have 2 queries which overlap. Really,
> one is a subset of the other. On those overlapping threads, I'm
> finding that sometimes new messages are written to one mailbox and not
> the other. (At least sometimes, the messages may be missing from all
> mailboxes sometimes too. I'm not certain.) Using --remote-fudge-time
> to force refetching seems to get the missing mails. I haven't found
> anything strange in timestamps of the missing mails, but otherwise am
> not sure how to debug this further. The queries are retrieving full
> threads and the missing mails are in the threads, but not direct
> matches to the queries. I realize that's not a lot of detail to go on.
> Suggestions on debugging this further?
Is this with 1.8 or 1.7?
I forgot to note in the release notes, but there were some
SQLite usage-related fixes which could avoid missing messages.
You'll need "lei daemon-kill" after upgrading to 1.8 to ensure
the new code is running.
What might be interesting is to use the URLs lei prints and
comparing the results w/o lei.
I'll have to double-check if overlapping affects things, but it
shouldn't; since the dedupe logic is per-output.
Is this exclusively with HTTPS endpoints and writing to Maildirs
(or something else?)
> It might be helpful if lei could print out message-ids of messages
> written to mailboxes.
That could get very noisy, especially as mailboxes are written
in parallel.
Thanks.
^ permalink raw reply [relevance 71%]
* Re: lei missing mails
@ 2022-06-29 17:27 71% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2022-06-29 17:27 UTC (permalink / raw)
To: Rob Herring; +Cc: meta
Rob Herring <robh@kernel.org> wrote:
> On Wed, Jun 29, 2022 at 10:30 AM Eric Wong <e@80x24.org> wrote:
> >
> > Rob Herring <robh@kernel.org> wrote:
> > > Hi,
> > >
> > > I'm using lei with lore where I have 2 queries which overlap. Really,
> > > one is a subset of the other. On those overlapping threads, I'm
> > > finding that sometimes new messages are written to one mailbox and not
> > > the other. (At least sometimes, the messages may be missing from all
> > > mailboxes sometimes too. I'm not certain.) Using --remote-fudge-time
> > > to force refetching seems to get the missing mails. I haven't found
> > > anything strange in timestamps of the missing mails, but otherwise am
> > > not sure how to debug this further. The queries are retrieving full
> > > threads and the missing mails are in the threads, but not direct
> > > matches to the queries. I realize that's not a lot of detail to go on.
> > > Suggestions on debugging this further?
> >
> > Is this with 1.8 or 1.7?
>
> Commit 68b53c888911 actually. So post 1.8.
OK, thanks for that info.
> > I forgot to note in the release notes, but there were some
> > SQLite usage-related fixes which could avoid missing messages.
> >
> > You'll need "lei daemon-kill" after upgrading to 1.8 to ensure
> > the new code is running.
>
> It's possible I haven't done that since updating though I do vaguely
> recall seeing something about needing to do that. Is there any way to
> tell before I restart it?
Not really, but it's pretty cheap to restart (assuming there's no
long-running jobs).
> > What might be interesting is to use the URLs lei prints and
> > comparing the results w/o lei.
> >
> > I'll have to double-check if overlapping affects things, but it
> > shouldn't; since the dedupe logic is per-output.
> >
> > Is this exclusively with HTTPS endpoints and writing to Maildirs
> > (or something else?)
>
> Yes. It's querying lore and writing to a maildir. Here's one of the queries:
>
> [lei]
> q = (dfn:drivers OR dfn:arch OR dfn:Documentation/* OR
> dfn:include OR dfn:scripts) AND \
> f:robh@kernel.org AND rt:6.month.ago..
> [lei "q"]
> include = https://lore.kernel.org/all/
> external = 1
> local = 1
> remote = 1
> threads = 1
> dedupe = mid
> output = maildir:/home/rob/Mail/my-patches
Fwiw, dedupe based on mid could be vulnerable to spoofing, which
is why `content' is the default. But yes, in the past, I've
noticed some messages to meta@public-inbox.org not showing up,
though not recently (I guess lack of activity here is a culprit :x)
I also just noticed an inotify-related bug deadlocking the whole
lei-deamon while looking into this :<
> > > It might be helpful if lei could print out message-ids of messages
> > > written to mailboxes.
> >
> > That could get very noisy, especially as mailboxes are written
> > in parallel.
>
> Verbose mode already is. Maybe specifying what info you want to be
> verbose would help. The network side is mostly uninteresting in this
> case for example.
Yes, I've been struggling with the verbosity, too; and many
other things :<
> Is there any tool to list new messages in a maildir? I could do that
> before and after. I've done the clearing the new flag in mutt between
> runs, but that's not really ideal.
I suppose `ls'. There are likely other tools more suited for Maildirs
but I'm not familiar with them off the top of my head.
Maybe lei could grow yet another command.
^ permalink raw reply [relevance 71%]
* Re: lei missing mails
@ 2022-06-30 8:55 71% ` Eric Wong
2022-07-07 9:48 71% ` Eric Wong
0 siblings, 2 replies; 200+ results
From: Eric Wong @ 2022-06-30 8:55 UTC (permalink / raw)
To: Rob Herring; +Cc: meta
Rob Herring <robh@kernel.org> wrote:
> On Wed, Jun 29, 2022 at 11:27 AM Eric Wong <e@80x24.org> wrote:
> > Rob Herring <robh@kernel.org> wrote:
> > > On Wed, Jun 29, 2022 at 10:30 AM Eric Wong <e@80x24.org> wrote:
> > > > Rob Herring <robh@kernel.org> wrote:
> > > > > Hi,
> > > > >
> > > > > I'm using lei with lore where I have 2 queries which overlap. Really,
> > > > > one is a subset of the other. On those overlapping threads, I'm
> > > > > finding that sometimes new messages are written to one mailbox and not
> > > > > the other. (At least sometimes, the messages may be missing from all
> > > > > mailboxes sometimes too. I'm not certain.) Using --remote-fudge-time
> > > > > to force refetching seems to get the missing mails. I haven't found
> > > > > anything strange in timestamps of the missing mails, but otherwise am
> > > > > not sure how to debug this further. The queries are retrieving full
> > > > > threads and the missing mails are in the threads, but not direct
> > > > > matches to the queries. I realize that's not a lot of detail to go on.
> > > > > Suggestions on debugging this further?
> > > >
> > > > Is this with 1.8 or 1.7?
> > >
> > > Commit 68b53c888911 actually. So post 1.8.
> >
> > OK, thanks for that info.
> >
> > > > I forgot to note in the release notes, but there were some
> > > > SQLite usage-related fixes which could avoid missing messages.
> > > >
> > > > You'll need "lei daemon-kill" after upgrading to 1.8 to ensure
> > > > the new code is running.
> > >
> > > It's possible I haven't done that since updating though I do vaguely
> > > recall seeing something about needing to do that. Is there any way to
> > > tell before I restart it?
> >
> > Not really, but it's pretty cheap to restart (assuming there's no
> > long-running jobs).
>
> I've restarted and just hit this again.
Ugh, sorry to hear that :<
> > > > What might be interesting is to use the URLs lei prints and
> > > > comparing the results w/o lei.
>
> $ lei up --all
> # updating /home/rob/Mail/from-me
> # updating /home/rob/Mail/missing-cc
> # updating /home/rob/Mail/my-patches
> # updating /home/rob/Mail/pci
> # https://lore.kernel.org/all/ limiting to 2022-06-27 12:42 -0600 and newer
> # https://lore.kernel.org/all/ limiting to 2022-06-27 9:50 -0600 and newer
> # https://lore.kernel.org/all/ limiting to 2022-06-27 12:42 -0600 and newer
> # /usr/bin/curl -Sf -s -d ''
> https://lore.kernel.org/all/?x=m&t=1&q=(dt%3A20220529211430..+AND+(f%3Arobh%40kernel.org+OR+f%3Arobh%2Bdt%40kernel.org))+AND+dt%3A20220627184226..
> # /home/rob/.local/share/lei/store 144/144
> # /home/rob/.local/share/lei/store 3/3
> # /usr/bin/curl -Sf -s -d ''
> https://lore.kernel.org/all/?x=m&t=1&q=((dfn%3Adrivers+OR+dfn%3Aarch+OR+dfn%3ADocumentation%2F*+OR+dfn%3Ainclude+OR+dfn%3Ascripts)+AND+f%3Arobh%40kernel.org+AND+rt%3A1640812470..)+AND+dt%3A20220627155025..
> # /usr/bin/curl -Sf -s -d ''
> https://lore.kernel.org/all/?x=m&t=1&q=(l%3Alinux-pci+dfn%3Adrivers%2Fpci%2Fcontroller+dt%3A20220529211430..)+AND+dt%3A20220627184226..
> # /home/rob/.local/share/lei/store 0/0
> # /home/rob/.local/share/lei/store 362/362
> # 0 written to /home/rob/Mail/missing-cc/ (0 matches)
> # https://lore.kernel.org/all/ 72/72
> # https://lore.kernel.org/all/ 4/4
> # https://lore.kernel.org/all/ 131/?
> # https://lore.kernel.org/all/ 184/?
> # https://lore.kernel.org/all/ 412/?
> # https://lore.kernel.org/all/ 603/?
> # https://lore.kernel.org/all/ 853/?
> # https://lore.kernel.org/all/ 1069/?
> # https://lore.kernel.org/all/ 1442/?
> # https://lore.kernel.org/all/ 1443/1443
> # 1 written to /home/rob/Mail/pci/ (75 matches)
> # 2 written to /home/rob/Mail/my-patches/ (148 matches)
> # 7 written to /home/rob/Mail/from-me/ (1805 matches)
>
>
> What I expected was 3 messages written to 'my-patches'.
>
> I think the problem is just simply that the new message missing
> doesn't match the query, but is a reply to a match. So with a date
> after the original match in the thread won't pick up anything. The 2nd
> URL above indeed only has 2 results. I guess I just have to fetch a
> wider window like a month every time? What's needed is a get any new
> messages in existing threads. I don't suppose there's an efficient way
> to do that?
No, I don't think so. I think this is a separate issue in lei...
"t=1" in the remote query expands threads in a time-agnostic
way, so I don't think that's the problem (though I may be wrong...).
I'll have to check more closely this week (still stuck with POP3
user account/storage issues :<)
> > > >
> > > > I'll have to double-check if overlapping affects things, but it
> > > > shouldn't; since the dedupe logic is per-output.
> > > >
> > > > Is this exclusively with HTTPS endpoints and writing to Maildirs
> > > > (or something else?)
> > >
> > > Yes. It's querying lore and writing to a maildir. Here's one of the queries:
> > >
> > > [lei]
> > > q = (dfn:drivers OR dfn:arch OR dfn:Documentation/* OR
> > > dfn:include OR dfn:scripts) AND \
> > > f:robh@kernel.org AND rt:6.month.ago..
> > > [lei "q"]
> > > include = https://lore.kernel.org/all/
> > > external = 1
> > > local = 1
> > > remote = 1
> > > threads = 1
> > > dedupe = mid
> > > output = maildir:/home/rob/Mail/my-patches
> >
> > Fwiw, dedupe based on mid could be vulnerable to spoofing, which
> > is why `content' is the default. But yes, in the past, I've
> > noticed some messages to meta@public-inbox.org not showing up,
> > though not recently (I guess lack of activity here is a culprit :x)
>
> Does 'content' ignore trailers that mailman lists like to add? I think
> I switched because of that.
No, unfortunately not. Hopefully the admins can be convinced to
get rid of trailers (I'm happy vger did so a few years back).
But I'd rather deal with duplicates than miss messages (there
have been legitimate messages in the past which reused msgids,
unfortunately).
^ permalink raw reply [relevance 71%]
* [PATCH 0/2] lei: minor diagnostic improvement
@ 2022-07-07 9:40 71% Eric Wong
2022-07-07 9:40 71% ` [PATCH 1/2] lei_xsearch: simplify lei/store import check Eric Wong
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2022-07-07 9:40 UTC (permalink / raw)
To: meta
Still trying to consistently reproduce the source of missing
messages. It may be isolated to HTTP(S) remotes, or not, but
noting the number of seen vs. written messages ought to be a
reasonable start.
patch 1/2 is just a tiny simplification I noticed along the way
Eric Wong (2):
lei_xsearch: simplify lei/store import check
lei: track seen messages to note duplicates
lib/PublicInbox/LeiConvert.pm | 8 +++++---
lib/PublicInbox/LeiToMail.pm | 13 ++++++++++---
lib/PublicInbox/LeiXSearch.pm | 24 ++++++++++++++----------
3 files changed, 29 insertions(+), 16 deletions(-)
^ permalink raw reply [relevance 71%]
* [PATCH 1/2] lei_xsearch: simplify lei/store import check
2022-07-07 9:40 71% [PATCH 0/2] lei: minor diagnostic improvement Eric Wong
@ 2022-07-07 9:40 71% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2022-07-07 9:40 UTC (permalink / raw)
To: meta
There's no need to check for two fields when one will suffice.
---
lib/PublicInbox/LeiXSearch.pm | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/lib/PublicInbox/LeiXSearch.pm b/lib/PublicInbox/LeiXSearch.pm
index 2958d3f9..41e79856 100644
--- a/lib/PublicInbox/LeiXSearch.pm
+++ b/lib/PublicInbox/LeiXSearch.pm
@@ -285,7 +285,7 @@ sub each_remote_eml { # callback for MboxReader->mboxrd
my ($res, $kw) = $self->{import_sto}->wq_do('add_eml', $eml);
if (ref($res) eq ref($smsg)) { # totally new message
$smsg = $res;
- $self->{-imported} = 1;
+ $self->{-sto_imported} = 1;
}
$smsg->{kw} = $kw; # short-circuit xsmsg_vmd
}
@@ -376,7 +376,7 @@ sub query_remote_mboxrd {
$fh = IO::Uncompress::Gunzip->new($fh, MultiStream => 1);
PublicInbox::MboxReader->mboxrd($fh, \&each_remote_eml, $self,
$lei, $each_smsg);
- if ($self->{import_sto} && delete($self->{-imported})) {
+ if (delete($self->{-sto_imported})) {
my $wait = $self->{import_sto}->wq_do('done');
}
$reap_curl->join;
^ permalink raw reply related [relevance 71%]
* Re: lei missing mails
2022-06-30 8:55 71% ` Eric Wong
@ 2022-07-07 9:48 71% ` Eric Wong
2022-07-11 21:17 71% ` Rob Herring
1 sibling, 1 reply; 200+ results
From: Eric Wong @ 2022-07-07 9:48 UTC (permalink / raw)
To: Rob Herring; +Cc: meta
Eric Wong <e@80x24.org> wrote:
> Rob Herring <robh@kernel.org> wrote:
> > I've restarted and just hit this again.
>
> Ugh, sorry to hear that :<
Btw, is this only with HTTP(S) endpoints? (not local mirrors?)
I wonder if it's something wrong on the HTTP end (server or
client), but did more digging the other day and couldn't
reproduce the problem...
I just posted a very minor diagnostic change which might help
(forgot to Cc you :x)
https://public-inbox.org/meta/20220707094030.1185793-1-e@80x24.org/
As usual, daemon-kill is necessary after upgrading.
^ permalink raw reply [relevance 71%]
* Re: lei missing mails
2022-07-07 9:48 71% ` Eric Wong
@ 2022-07-11 21:17 71% ` Rob Herring
0 siblings, 0 replies; 200+ results
From: Rob Herring @ 2022-07-11 21:17 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
On Thu, Jul 7, 2022 at 3:48 AM Eric Wong <e@80x24.org> wrote:
>
> Eric Wong <e@80x24.org> wrote:
> > Rob Herring <robh@kernel.org> wrote:
> > > I've restarted and just hit this again.
> >
> > Ugh, sorry to hear that :<
>
> Btw, is this only with HTTP(S) endpoints? (not local mirrors?)
Yes. Only in that https is the only case I've tried. Don't know about
local mirrors because I don't know how to set that up...
Rob
^ permalink raw reply [relevance 71%]
* Re: lei missing mails
@ 2022-07-18 23:41 71% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2022-07-18 23:41 UTC (permalink / raw)
To: Rob Herring; +Cc: meta
Rob Herring <robh@kernel.org> wrote:
> On Thu, Jun 30, 2022 at 2:55 AM Eric Wong <e@80x24.org> wrote:
> >
> > Rob Herring <robh@kernel.org> wrote:
> > > On Wed, Jun 29, 2022 at 11:27 AM Eric Wong <e@80x24.org> wrote:
> > > > Rob Herring <robh@kernel.org> wrote:
> > > > > On Wed, Jun 29, 2022 at 10:30 AM Eric Wong <e@80x24.org> wrote:
> > > > > > Rob Herring <robh@kernel.org> wrote:
> > > > > > > Hi,
> > > > > > >
> > > > > > > I'm using lei with lore where I have 2 queries which overlap. Really,
> > > > > > > one is a subset of the other. On those overlapping threads, I'm
> > > > > > > finding that sometimes new messages are written to one mailbox and not
> > > > > > > the other. (At least sometimes, the messages may be missing from all
> > > > > > > mailboxes sometimes too. I'm not certain.) Using --remote-fudge-time
> > > > > > > to force refetching seems to get the missing mails. I haven't found
> > > > > > > anything strange in timestamps of the missing mails, but otherwise am
> > > > > > > not sure how to debug this further. The queries are retrieving full
> > > > > > > threads and the missing mails are in the threads, but not direct
> > > > > > > matches to the queries. I realize that's not a lot of detail to go on.
> > > > > > > Suggestions on debugging this further?
> > > > > >
> > > > > > Is this with 1.8 or 1.7?
> > > > >
> > > > > Commit 68b53c888911 actually. So post 1.8.
> > > >
> > > > OK, thanks for that info.
> > > >
> > > > > > I forgot to note in the release notes, but there were some
> > > > > > SQLite usage-related fixes which could avoid missing messages.
> > > > > >
> > > > > > You'll need "lei daemon-kill" after upgrading to 1.8 to ensure
> > > > > > the new code is running.
> > > > >
> > > > > It's possible I haven't done that since updating though I do vaguely
> > > > > recall seeing something about needing to do that. Is there any way to
> > > > > tell before I restart it?
> > > >
> > > > Not really, but it's pretty cheap to restart (assuming there's no
> > > > long-running jobs).
> > >
> > > I've restarted and just hit this again.
> >
> > Ugh, sorry to hear that :<
> >
> > > > > > What might be interesting is to use the URLs lei prints and
> > > > > > comparing the results w/o lei.
> > >
> > > $ lei up --all
> > > # updating /home/rob/Mail/from-me
> > > # updating /home/rob/Mail/missing-cc
> > > # updating /home/rob/Mail/my-patches
> > > # updating /home/rob/Mail/pci
> > > # https://lore.kernel.org/all/ limiting to 2022-06-27 12:42 -0600 and newer
> > > # https://lore.kernel.org/all/ limiting to 2022-06-27 9:50 -0600 and newer
> > > # https://lore.kernel.org/all/ limiting to 2022-06-27 12:42 -0600 and newer
> > > # /usr/bin/curl -Sf -s -d ''
> > > https://lore.kernel.org/all/?x=m&t=1&q=(dt%3A20220529211430..+AND+(f%3Arobh%40kernel.org+OR+f%3Arobh%2Bdt%40kernel.org))+AND+dt%3A20220627184226..
> > > # /home/rob/.local/share/lei/store 144/144
> > > # /home/rob/.local/share/lei/store 3/3
> > > # /usr/bin/curl -Sf -s -d ''
> > > https://lore.kernel.org/all/?x=m&t=1&q=((dfn%3Adrivers+OR+dfn%3Aarch+OR+dfn%3ADocumentation%2F*+OR+dfn%3Ainclude+OR+dfn%3Ascripts)+AND+f%3Arobh%40kernel.org+AND+rt%3A1640812470..)+AND+dt%3A20220627155025..
> > > # /usr/bin/curl -Sf -s -d ''
> > > https://lore.kernel.org/all/?x=m&t=1&q=(l%3Alinux-pci+dfn%3Adrivers%2Fpci%2Fcontroller+dt%3A20220529211430..)+AND+dt%3A20220627184226..
> > > # /home/rob/.local/share/lei/store 0/0
> > > # /home/rob/.local/share/lei/store 362/362
> > > # 0 written to /home/rob/Mail/missing-cc/ (0 matches)
> > > # https://lore.kernel.org/all/ 72/72
> > > # https://lore.kernel.org/all/ 4/4
> > > # https://lore.kernel.org/all/ 131/?
> > > # https://lore.kernel.org/all/ 184/?
> > > # https://lore.kernel.org/all/ 412/?
> > > # https://lore.kernel.org/all/ 603/?
> > > # https://lore.kernel.org/all/ 853/?
> > > # https://lore.kernel.org/all/ 1069/?
> > > # https://lore.kernel.org/all/ 1442/?
> > > # https://lore.kernel.org/all/ 1443/1443
> > > # 1 written to /home/rob/Mail/pci/ (75 matches)
> > > # 2 written to /home/rob/Mail/my-patches/ (148 matches)
> > > # 7 written to /home/rob/Mail/from-me/ (1805 matches)
> > >
> > >
> > > What I expected was 3 messages written to 'my-patches'.
> > >
> > > I think the problem is just simply that the new message missing
> > > doesn't match the query, but is a reply to a match. So with a date
> > > after the original match in the thread won't pick up anything. The 2nd
> > > URL above indeed only has 2 results. I guess I just have to fetch a
> > > wider window like a month every time? What's needed is a get any new
> > > messages in existing threads. I don't suppose there's an efficient way
> > > to do that?
> >
> > No, I don't think so. I think this is a separate issue in lei...
> > "t=1" in the remote query expands threads in a time-agnostic
> > way, so I don't think that's the problem (though I may be wrong...).
>
> Based on what the web interface presents, it sure seems like 't=1' is
> independent of the query. The results listed are only those that match
> the query and date range on the match.
Actually, for the HTML results, t=1 is ignored right now...
it's only for mboxrd downloads (via POST) at the moment...
That should be clarified/changed.
> For example, this query returns 3 matches:
>
> https://lore.kernel.org/all/?x=m&t=1&q=((dfn%3Adrivers+OR+dfn%3Aarch+OR+dfn%3ADocumentation%2F*+OR+dfn%3Ainclude+OR+dfn%3Ascripts)+AND+f%3Arobh%40kernel.org+AND+rt%3A1641934905..)+AND+dt%3A20220630203819..
>
> If I change 'dt' to 1 day earlier, I get 1 more match:
>
> https://lore.kernel.org/all/?x=m&t=1&q=((dfn%3Adrivers+OR+dfn%3Aarch+OR+dfn%3ADocumentation%2F*+OR+dfn%3Ainclude+OR+dfn%3Ascripts)+AND+f%3Arobh%40kernel.org+AND+rt%3A1641934905..)+AND+dt%3A20220629203819..
>
> That 4th match has a reply after 6/30, but the 1st query will not get
> the reply. This is all reproducible without lei involved at all.
Right. t=1 only expands threads if they're linked via
References/In-Reply-To or (loosely) via matching Subjects.
> What seems to be needed is a 'thread date' which is the latest time
> for any message in a thread that matches. Or perhaps some way to
> separate the query from what's transferred. IOW, query for X, but only
> send results newer than some date.
Yeah, the thread ID info is stored independently of the search
index documents, though; so searching (Xapian) vs filtering
(SQLite) gets a bit tricky
I'll look into JMAP further before making more changes to the
data models to accomodate this.
Apologies for the delays; been stressed over other things :<
^ permalink raw reply [relevance 71%]
* [PATCH 2/2] lei note-event: inline note_event_arm_done
2022-07-19 22:42 71% [PATCH 0/2] lei inotify/EVFILT_VNODE deadlock fix Eric Wong
@ 2022-07-19 22:42 71% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2022-07-19 22:42 UTC (permalink / raw)
To: meta
This was a single-caller sub since 47d4e53734820b4e
(lei_mail_sync: rely on flock(2), avoid IPC, 2021-09-18)
and unlikely to be used further, so inline it and save
a few KB of memory.
---
lib/PublicInbox/LeiNoteEvent.pm | 10 ++--------
1 file changed, 2 insertions(+), 8 deletions(-)
diff --git a/lib/PublicInbox/LeiNoteEvent.pm b/lib/PublicInbox/LeiNoteEvent.pm
index 93f80116..8581bd9a 100644
--- a/lib/PublicInbox/LeiNoteEvent.pm
+++ b/lib/PublicInbox/LeiNoteEvent.pm
@@ -27,13 +27,6 @@ sub flush_task { # PublicInbox::DS timer callback
for my $lei (values %$todo) { flush_lei($lei) }
}
-# sets a timer to flush
-sub note_event_arm_done ($) {
- my ($lei) = @_;
- PublicInbox::DS::add_uniq_timer('flush_timer', 5, \&flush_task);
- $to_flush->{$lei->{cfg}->{'-f'}} //= $lei;
-}
-
sub eml_event ($$$$) {
my ($self, $eml, $vmd, $state) = @_;
my $sto = $self->{lei}->{sto};
@@ -92,7 +85,8 @@ sub lei_note_event {
$jobs = 4 if $jobs > 4; # same default as V2Writable
my ($op_c, $ops) = $lei->workers_start($wq, $jobs);
$lei->wait_wq_events($op_c, $ops);
- note_event_arm_done($lei);
+ PublicInbox::DS::add_uniq_timer('flush_timer', 5, \&flush_task);
+ $to_flush->{$lei->{cfg}->{'-f'}} //= $lei;
$wq->prepare_nonblock;
$lei->{lne} = $wq;
};
^ permalink raw reply related [relevance 71%]
* [PATCH 0/2] lei inotify/EVFILT_VNODE deadlock fix
@ 2022-07-19 22:42 71% Eric Wong
2022-07-19 22:42 71% ` [PATCH 2/2] lei note-event: inline note_event_arm_done Eric Wong
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2022-07-19 22:42 UTC (permalink / raw)
To: meta
For every bug I fix, I feel dumber for introducing it :<
Eric Wong (2):
lei: avoid deadlock on inotify/EVFILT_VNODE wakeups
lei note-event: inline note_event_arm_done
MANIFEST | 1 +
lib/PublicInbox/IPC.pm | 26 +++++++++++++++--
lib/PublicInbox/LeiNoteEvent.pm | 15 ++++------
lib/PublicInbox/WQBlocked.pm | 49 +++++++++++++++++++++++++++++++++
4 files changed, 79 insertions(+), 12 deletions(-)
create mode 100644 lib/PublicInbox/WQBlocked.pm
^ permalink raw reply [relevance 71%]
* [PATCH 2/4] lei inspect: less scary exception for invalid "docid:" inspect
2022-08-17 9:33 71% [PATCH 0/4] lei reindex, minor tweaks Eric Wong
2022-08-17 9:33 71% ` [PATCH 3/4] lei/store: reduce work when accessing mail_sync.sqlite3 Eric Wong
@ 2022-08-17 9:33 71% ` Eric Wong
2 siblings, 0 replies; 200+ results
From: Eric Wong @ 2022-08-17 9:33 UTC (permalink / raw)
To: meta
It still says "Exception:", but doesn't pointlessly print out
the line number and file of the exception when it's a data/input
problem, and not a code problem on our end.
---
lib/PublicInbox/LeiInspect.pm | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/lib/PublicInbox/LeiInspect.pm b/lib/PublicInbox/LeiInspect.pm
index d7775d4b..d1dca4ef 100644
--- a/lib/PublicInbox/LeiInspect.pm
+++ b/lib/PublicInbox/LeiInspect.pm
@@ -1,4 +1,4 @@
-# Copyright (C) 2021 all contributors <meta@public-inbox.org>
+# Copyright (C) all contributors <meta@public-inbox.org>
# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
# "lei inspect" general purpose inspector for stuff in SQLite and
@@ -235,7 +235,8 @@ sub inspect_argv { # via wq_do
$lei->{1}->autoflush(0);
$lei->out('[') if $multi;
while (defined(my $x = shift @$argv)) {
- inspect1($lei, $x, scalar(@$argv)) or return;
+ eval { inspect1($lei, $x, scalar(@$argv)) or return };
+ warn "E: $@\n" if $@;
}
$lei->out(']') if $multi;
}
^ permalink raw reply related [relevance 71%]
* [PATCH 0/4] lei reindex, minor tweaks
@ 2022-08-17 9:33 71% Eric Wong
2022-08-17 9:33 71% ` [PATCH 3/4] lei/store: reduce work when accessing mail_sync.sqlite3 Eric Wong
` (2 more replies)
0 siblings, 3 replies; 200+ results
From: Eric Wong @ 2022-08-17 9:33 UTC (permalink / raw)
To: meta
Reindex is far from complete, and probably needs a compact, and
some other fixups for old data + rethread support.
But avoiding false positives from base-85 is nice.
Eric Wong (4):
searchidx: fix spelling error in comment
lei inspect: less scary exception for invalid "docid:" inspect
lei/store: reduce work when accessing mail_sync.sqlite3
lei reindex: new command to reindex lei/store
Documentation/lei-reindex.pod | 47 +++++++++++++++++++++++++++++++++
MANIFEST | 2 ++
lib/PublicInbox/LEI.pm | 2 ++
lib/PublicInbox/LeiInspect.pm | 5 ++--
lib/PublicInbox/LeiReindex.pm | 49 +++++++++++++++++++++++++++++++++++
lib/PublicInbox/LeiStore.pm | 38 ++++++++++++++++++++++++---
lib/PublicInbox/SearchIdx.pm | 2 +-
7 files changed, 138 insertions(+), 7 deletions(-)
create mode 100644 Documentation/lei-reindex.pod
create mode 100644 lib/PublicInbox/LeiReindex.pm
^ permalink raw reply [relevance 71%]
* [PATCH 3/4] lei/store: reduce work when accessing mail_sync.sqlite3
2022-08-17 9:33 71% [PATCH 0/4] lei reindex, minor tweaks Eric Wong
@ 2022-08-17 9:33 71% ` Eric Wong
2022-08-17 9:33 71% ` [PATCH 2/4] lei inspect: less scary exception for invalid "docid:" inspect Eric Wong
2 siblings, 0 replies; 200+ results
From: Eric Wong @ 2022-08-17 9:33 UTC (permalink / raw)
To: meta
There's no need to initialize eidx if we already have an open
handle for mail_sync.sqlite3
---
lib/PublicInbox/LeiStore.pm | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/lib/PublicInbox/LeiStore.pm b/lib/PublicInbox/LeiStore.pm
index 66049dfe..d49746cb 100644
--- a/lib/PublicInbox/LeiStore.pm
+++ b/lib/PublicInbox/LeiStore.pm
@@ -255,13 +255,13 @@ sub remove_eml_vmd { # remove just the VMD
sub _lms_rw ($) { # it is important to have eidx processes open before lms
my ($self) = @_;
- my ($eidx, $tl) = eidx_init($self);
- $self->{lms} //= do {
+ $self->{lms} // do {
require PublicInbox::LeiMailSync;
+ my ($eidx, $tl) = eidx_init($self);
my $f = "$self->{priv_eidx}->{topdir}/mail_sync.sqlite3";
my $lms = PublicInbox::LeiMailSync->new($f);
$lms->lms_write_prepare;
- $lms;
+ $self->{lms} = $lms;
};
}
^ permalink raw reply related [relevance 71%]
* [PATCH 0/4] lei reindex-related stuff
@ 2022-08-19 9:07 71% Eric Wong
2022-08-19 9:07 90% ` [PATCH 1/4] lei reindex: account for parallel lei/store users Eric Wong
2022-08-19 9:07 71% ` [PATCH 4/4] lei/store: reindex culls over-indexed messages Eric Wong
0 siblings, 2 replies; 200+ results
From: Eric Wong @ 2022-08-19 9:07 UTC (permalink / raw)
To: meta
1/4 is an important fix
And I'm still unusually stressed out about how to deal with
ancient, pre-release stuff from lei. --rethread may happen
another time, since AFAIK it's not necessary, right now as there
were no threading fixes required since lei...
Eric Wong (4):
lei reindex: account for parallel lei/store users
tests: add some basic "lei reindex" tests
smsg: ->populate falls back to old {ds}/{ts} values
lei/store: reindex culls over-indexed messages
MANIFEST | 1 +
lib/PublicInbox/LeiStore.pm | 14 ++++++++++++--
lib/PublicInbox/Smsg.pm | 6 ++++--
t/lei-index.t | 12 +++++++++++-
t/lei-reindex.t | 12 ++++++++++++
5 files changed, 40 insertions(+), 5 deletions(-)
create mode 100644 t/lei-reindex.t
^ permalink raw reply [relevance 71%]
* [PATCH 4/4] lei/store: reindex culls over-indexed messages
2022-08-19 9:07 71% [PATCH 0/4] lei reindex-related stuff Eric Wong
2022-08-19 9:07 90% ` [PATCH 1/4] lei reindex: account for parallel lei/store users Eric Wong
@ 2022-08-19 9:07 71% ` Eric Wong
1 sibling, 0 replies; 200+ results
From: Eric Wong @ 2022-08-19 9:07 UTC (permalink / raw)
To: meta
I may be the only lei user who has redundantly-indexed messages
needing this, though...
---
lib/PublicInbox/LeiStore.pm | 9 +++++++++
1 file changed, 9 insertions(+)
diff --git a/lib/PublicInbox/LeiStore.pm b/lib/PublicInbox/LeiStore.pm
index 8e710540..57f0e013 100644
--- a/lib/PublicInbox/LeiStore.pm
+++ b/lib/PublicInbox/LeiStore.pm
@@ -344,6 +344,15 @@ sub _reindex_1 { # git->cat_async callback
my $eml = PublicInbox::Eml->new($bref);
$smsg->{-merge_vmd} = 1; # preserve existing keywords
$eidx->idx_shard($smsg->{num})->index_eml($eml, $smsg);
+ } elsif ($type eq 'missing') {
+ # pre-release/buggy lei may've indexed external-only msgs,
+ # try to correct that, here
+ warn("E: missing $hex, culling (ancient lei artifact?)\n");
+ $smsg->{to} = $smsg->{cc} = $smsg->{from} = '';
+ $smsg->{bytes} = 0;
+ $eidx->{oidx}->update_blob($smsg, '');
+ my $eml = PublicInbox::Eml->new("\r\n\r\n");
+ $eidx->idx_shard($smsg->{num})->index_eml($eml, $smsg);
} else {
warn("E: $type $hex\n");
}
^ permalink raw reply related [relevance 71%]
* [PATCH 1/3] Makefile.PL: add lei-reindex manpage
@ 2022-08-30 9:10 71% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2022-08-30 9:10 UTC (permalink / raw)
To: meta
I forgot to add this when I added the new command :x
---
Makefile.PL | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/Makefile.PL b/Makefile.PL
index 67012d3e..ff03b615 100644
--- a/Makefile.PL
+++ b/Makefile.PL
@@ -53,7 +53,8 @@ $v->{-m1} = [ map {
lei-import lei-index lei-init lei-inspect lei-lcat
lei-ls-external lei-ls-label lei-ls-mail-source lei-ls-mail-sync
lei-ls-search lei-ls-watch lei-mail-diff lei-p2q lei-q
- lei-rediff lei-refresh-mail-sync lei-rm lei-rm-watch lei-tag
+ lei-rediff lei-refresh-mail-sync lei-reindex
+ lei-rm lei-rm-watch lei-tag
lei-up)];
$v->{-m5} = [ qw(public-inbox-config public-inbox-v1-format
public-inbox-v2-format public-inbox-extindex-format
^ permalink raw reply related [relevance 71%]
* [PATCH 0/2] lei-related import tweaks
@ 2022-09-02 18:26 71% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2022-09-02 18:26 UTC (permalink / raw)
To: meta
Just some minor nits I noticed in lei-land...
Eric Wong (2):
lei/store: do not write info/refs file
import: pass --quiet to `git gc' if STDERR isn't a tty
lib/PublicInbox/Import.pm | 7 +++++--
t/lei_store.t | 5 ++++-
2 files changed, 9 insertions(+), 3 deletions(-)
^ permalink raw reply [relevance 71%]
* Re: [PATCH] lei: add diagnostics for IMAP writer failures
2022-09-09 18:00 86% ` [PATCH] doc: document --jobs for `lei q' and `lei up' Eric Wong
@ 2022-09-09 20:35 71% ` Ricardo Ribalda
1 sibling, 1 reply; 200+ results
From: Ricardo Ribalda @ 2022-09-09 20:35 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
Hi Eric
On Fri, 9 Sept 2022 at 19:45, Eric Wong <e@80x24.org> wrote:
>
> Ricardo Ribalda <ribalda@chromium.org> wrote:
> > Hi
> >
> > I am getting a lot of those messages when using lei in imap mode.
> >
> > Nonetheless the mail seems to arrive fine to its destination (on the
> > first run, some of the mail was lost, but for small batches of mails
> > it seems to work fine).
> >
> > It is a valid message or just some red-herring.
> >
> > I am using debian testing/
> >
> > Regards!
> >
> > # https://lore.kernel.org/all/ limiting to 2022-09-07 10:03 +0000 and newer
> > # /usr/local/google/home/ribalda/.local/share/lei/store 13/13
> > # /usr/bin/curl -Sf -s -d ''
> > https://lore.kernel.org/all/?x=m&t=1&q=((ribalda)+AND+rt%3A1660039651..)+AND+dt%3A20220907100321..
> > 1285740 lei2mail 6 wq_worker: do_post_auth: Can't call method
> > "uidvalidity" on an undefined value at
> > /usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
>
> <snip>
>
> > 1285735 lei2mail 1 wq_worker: write_mail: Can't use an undefined value
> > as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
> > line 783.
> > # https://lore.kernel.org/all/ 18/18
> > # 14 written to imaps://imap.gmail.com/lei/me (31 matches)
>
> I wonder if it's excessive parallelism for gmail's IMAP.
> I haven't tested IMAP destinations, much...
>
> Can you try the patch at the bottom?
>
> There's also another patch coming to document the `--jobs|-j' CLI
> switch for `lei up' and `lei q', but trying `-j ,1' may help you
> if it's parallelism. Note the comma before `1', it accepts
> `-j $Q,$W' since $Q is the number of query processes and $W is
> the number of LeiToMail writers.
The patch did not seem to have any effect :(, I never get a "IMAP
LastError: " message
On the other hand, the -j worked! I can go up to -j ,15 without any error.
>
> -------8<-------
> From: Eric Wong <e@80x24.org>
> Subject: [PATCH] lei: add diagnostics for IMAP writer failures
>
> This may help diagnose the problem with IMAP destinations
> encountered at:
> https://public-inbox.org/meta/CANiDSCsDfutAUMBLPZbxdyka+_jnhv+4YNYdL9QPRoC=wNUGCQ@mail.gmail.com/
> ---
> lib/PublicInbox/LeiToMail.pm | 4 ++--
> lib/PublicInbox/NetReader.pm | 8 +++++++-
> 2 files changed, 9 insertions(+), 3 deletions(-)
>
> diff --git a/lib/PublicInbox/LeiToMail.pm b/lib/PublicInbox/LeiToMail.pm
> index 2aa3977e..bc00b96a 100644
> --- a/lib/PublicInbox/LeiToMail.pm
> +++ b/lib/PublicInbox/LeiToMail.pm
> @@ -310,8 +310,8 @@ sub _imap_write_cb ($$) {
> my $dedupe = $lei->{dedupe};
> $dedupe->prepare_dedupe if $dedupe;
> my $append = $lei->{net}->can('imap_append');
> - my $uri = $self->{uri};
> - my $mic = $lei->{net}->mic_get($uri);
> + my $uri = $self->{uri} // die 'BUG: no {uri}';
> + my $mic = $lei->{net}->mic_get($uri) // die 'BUG: no $mic';
> my $folder = $uri->mailbox;
> $uri->uidvalidity($mic->uidvalidity($folder));
> my $lse = $lei->{lse}; # may be undef
> diff --git a/lib/PublicInbox/NetReader.pm b/lib/PublicInbox/NetReader.pm
> index c1af03a3..4de2583e 100644
> --- a/lib/PublicInbox/NetReader.pm
> +++ b/lib/PublicInbox/NetReader.pm
> @@ -685,7 +685,13 @@ sub mic_get {
> }
> my $mic = mic_new($self, $mic_arg, $sec, $uri);
> $cached //= {}; # invalid placeholder if no cache enabled
> - $mic && $mic->IsConnected ? ($cached->{$sec} = $mic) : undef;
> + if ($mic && $mic->IsConnected) {
> + $cached->{$sec} = $mic;
> + } else {
> + warn 'IMAP LastError: ',$mic->LastError, "\n" if $mic;
> + warn "IMAP errno: $!\n" if $!;
> + undef;
> + }
> }
>
> sub imap_each {
--
Ricardo Ribalda
^ permalink raw reply [relevance 71%]
* [PATCH] lei: fix --help for --jobs with `up' and `q'
@ 2022-09-10 1:35 71% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2022-09-10 1:35 UTC (permalink / raw)
To: meta
The help needs to match on the short option, too, and that
`lei q' option is (like most options) shared with `lei up'.
---
lib/PublicInbox/LEI.pm | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 8a3a3ab6..f3e80113 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -399,8 +399,10 @@ my %OPTDESC = (
'include specified external(s) in search' ],
'only|O=s@ q' => [ 'LOCATION',
'only use specified external(s) for search' ],
-'jobs=s q' => [ '[SEARCH_JOBS][,WRITER_JOBS]',
- 'control number of search and writer jobs' ],
+'jobs|j=s' => [ 'JOBSPEC',
+ 'control number of query and writer jobs' .
+ "integers delimited by `,', either of which may be omitted"
+ ],
'jobs|j=i add-external' => 'set parallelism when indexing after --mirror',
'in-format|F=s' => $stdin_formats,
^ permalink raw reply related [relevance 71%]
* Re: [PATCH v2] lei: bail out earlier on IMAP writer failures
@ 2022-09-10 19:50 71% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2022-09-10 19:50 UTC (permalink / raw)
To: Ricardo Ribalda; +Cc: meta
Ricardo Ribalda <ribalda@chromium.org> wrote:
> Similar output:
Oh, wait, did you run `lei daemon-kill' after applying the
patch? That's needed to reload the code, I'll try to make it
either auto-reload or warn users in the future.
Thanks.
^ permalink raw reply [relevance 71%]
* Re: [PATCH v2] lei: bail out earlier on IMAP writer failures
@ 2022-09-10 20:19 71% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2022-09-10 20:19 UTC (permalink / raw)
To: Ricardo Ribalda; +Cc: meta
Ricardo Ribalda <ribalda@chromium.org> wrote:
> ribalda@denia:/tmp/public-inbox$ lei up imaps://imap.gmail.com/lei/me
> # https://lore.kernel.org/all/ limiting to 2022-09-08 19:52 +0000 and newer
> # /usr/local/google/home/ribalda/.local/share/lei/store 12/12
> # /usr/bin/curl -Sf -s -d ''
> https://lore.kernel.org/all/?x=m&t=1&q=((ribalda)+AND+rt%3A1660161176..)+AND+dt%3A20220908195223..
> E: imaps://imap.gmail.com/lei/me connection failed.
> E: Consider using `--jobs ,1' to limit IMAP connections
Thanks for confirming things work as intended. I think the
default should be clamped, though... 15 seems a bit high for
smaller IMAP servers *shrug*
> ribalda@denia:/tmp/public-inbox$ lei up imaps://imap.gmail.com/lei/me --jobs ,15
^ permalink raw reply [relevance 71%]
* [PATCH 0/4] fixes noticed while diagnosing t/lei-up.t
@ 2022-09-30 9:21 71% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2022-09-30 9:21 UTC (permalink / raw)
To: meta
I'm still trying to figure out why OverIdx->adj_counter (via
next_tid) in the LeiSavedSearch dedupe check occasionally fails
under CentOS 7.x (but not other systems).
Meanwhile, some improvements noticed along the way.
The underlying problem remains...
Disabling WAL didn't help t/lei-up.t on CentOS 7.x, so the
obvious newish feature we use is unlikely the culprit...
Eric Wong (4):
tests: favor 3 argument `open' with interopolation
t/lei-up: improve diagnostics for this test
lei_to_mail: propagate errors to script/lei
t/altid_v2: improve test style
lib/PublicInbox/LeiToMail.pm | 34 ++++++++++++++++------------
t/altid_v2.t | 10 ++++-----
t/hl_mod.t | 4 ++--
t/lei-up.t | 43 ++++++++++++++++++++++--------------
t/lei_to_mail.t | 10 ++++-----
5 files changed, 59 insertions(+), 42 deletions(-)
^ permalink raw reply [relevance 71%]
* Re: [Need Help] lei add quotes at the search
@ 2022-10-30 5:13 71% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2022-10-30 5:13 UTC (permalink / raw)
To: Hangbin Liu; +Cc: meta
Hangbin Liu <liuhangbin@gmail.com> wrote:
> Hi,
>
> I used to use a search like
>
> lei q -I https://lore.kernel.org/all/ -o ~/Mail/liuhangbin --threads --dedupe=mid '((tc:liuhangbin AND rt:6.month.ago..) NOT (tc:stable@vger.kernel.org OR f:sfr@canb.auug.org.au)'
>
> It works on fc35. But after I update to fc36 with lei-1.9.0-1.fc36. It start to
> add quotes in the search link and make the search never works. e.g.
Are you able to show the curl CLI from fc35?
Which public-inbox/lei version was it?
I'm actually curious fc35 worked at all, since the quoting would've
been broken, I think...
> $ lei q -I https://lore.kernel.org/all/ -o ~/Mail/liuhangbin --threads --dedupe=mid '((tc:liuhangbin AND rt:6.month.ago..) NOT (tc:stable@vger.kernel.org OR f:sfr@canb.auug.org.au)'
> # /home/Liu/.local/share/lei/store 0/0
> # /usr/bin/curl -Sf -s -d '' https://lore.kernel.org/all/?x=m&t=1&q=((tc%3A%22liuhangbin+AND+rt%3A6.month.ago..)+NOT+(tc%3Astable%40vger.kernel.org+OR+f%3Asfr%40canb.auug.org.au)%22
> # 0 written to /home/Liu/Mail/liuhangbin/ (0 matches)
>
> Do you think if this is a bug, or I should update my search.
The %22 in fc36 is because your entire query is treated as one
element in argv and matches expected behavior.
Since '(' and ')' in the shell CLI is special, I suggest either:
a) using --stdin to enter queries containing '(' and ')'
b) quoting (or escaping) only the '(' and ')':
'('tc:liuhangbin AND rt:6.month.ago..')' NOT ...
or
\(tc:liuhangbin AND rt:6.month.ago..\) NOT ...
Which makes your argv something like:
[ "(tc:liuhangbin", "AND", "rt:6.month.ago..)", "NOT", ... ]
^ permalink raw reply [relevance 71%]
* Re: [Need Help] lei add quotes at the search
2022-10-31 7:47 71% ` Hangbin Liu
@ 2022-10-31 7:36 71% ` Hangbin Liu
1 sibling, 0 replies; 200+ results
From: Hangbin Liu @ 2022-10-31 7:36 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
On Sun, Oct 30, 2022 at 11:06:31PM +0000, Eric Wong wrote:
> Hangbin Liu <liuhangbin@gmail.com> wrote:
> > Sorry, I don't have the fc35 environment now.
>
> No worries, I dont think fc35 is really a culprit. Were you
> running a pre-release version of public-inbox or lei before?
Sorry, I forgot. Maybe I installed via `dnf copr enable icon/b4`
because I start using lei after reading blog
https://people.kernel.org/monsieuricon/lore-lei-part-1-getting-started
> > I'm curious about why the quote(%22) is added after "tc", not after "("
>
> It's because Xapian can only handle a phrase after the `tc:' prefix.
> thus: tc:"foo bar" actually parses `tc:' as a prefix for To/Cc;
> while: "tc:foo bar" looks for the phrase "tc:foo bar" anywhere
> in the message, and won't limit to To/Cc headers.
>
> This happens in the query_argv_to_string sub:
>
> https://public-inbox.org/meta/2feb3e13b49d222bc7bd28430a9cf159692a933f/s/?b=lib/PublicInbox/Search.pm#n358
>
> From the CLI: lei q "tc:foo bar" is indistinguishable
> from lei q tc:"foo bar" , so it gets treated as the latter.
Thanks for the explanation.
> > But if I have a long search line. This will breaks too much and hard to edit.
> > e.g. My real previous search is like
> >
> > [lei]
> > q = (tc:liuhangbin OR \
> > (dfn:drivers/net/wireguard/ AND rt:6.month.ago..) OR \
> > (dfn:tools/testing/selftests/net/ AND rt:1.month.ago..) OR \
> > (dfn:drivers/net/team/ AND rt:6.month.ago..) OR \
> > (dfn:net/ipv4/igmp.c AND rt:6.month.ago..) OR \
> > (dfn:net/ipv6/mcast.c AND rt:6.month.ago..)) \
> > NOT (tc:stable@vger.kernel.org OR f:sfr@canb.auug.org.au)
> >
> > If I add "\" on each "(", this will break to a very long config search.
> > I tried to adjust it to
>
> I think that can work if lei.internal.rawstr is set in the
> config to indicate stdin was used (It's auto-set by --stdin).
> I guess it also works if it's the only lei.q config entry
> and the lei.q entry contains "\n"
>
> cf. https://public-inbox.org/meta/20211110102837.41721-1-e@80x24.org/
>
> > [lei]
> > q = (tc:liuhangbin OR \
> > (dfn:drivers/net/wireguard/ AND rt:6.month.ago..) OR \
> > (dfn:tools/testing/selftests/net/ AND rt:1.month.ago..) OR \
> > (dfn:drivers/net/team/ AND rt:6.month.ago..) OR \
> > (dfn:net/ipv4/igmp.c AND rt:6.month.ago..) OR \
> > (dfn:net/ipv6/mcast.c AND rt:6.month.ago..))
> > q = NOT
> > q = (tc:stable@vger.kernel.org
> > q = OR
> > q = f:sfr@canb.auug.org.au)
> >
> > And now it works...
>
> Sorta... at least for remotes it does:
>
> > $ lei up /home/Liu/Mail/gmail/Linux_Kernel
> > # https://lore.kernel.org/all/ limiting to 2022-09-30 17:00 +0800 and newer
> > 60927 lei_xsearch 0 wq_worker: query_one_mset: Exception: Unknown range operation at /usr/share/perl5/vendor_perl/PublicInbox/IPC.pm line 254.
>
> Note that Exception means it's not handling the first part of
> the query when hitting the local Xapian DB. It's not doing the
> approxidate ($X.month.ago) substitution for the local Xapian DB,
> thus you got the "Unknown range operation".
>
> > # /usr/bin/curl -Sf -s -d '' https://lore.kernel.org/all/?x=m&t=1&q=((tc%3Aliuhangbin+OR+(dfn%3Adrivers%2Fnet%2Fwireguard%2F+AND+rt%3A6.month.ago..)+OR+(dfn%3Atools%2Ftesting%2Fselftests%2Fnet%2F+AND+rt%3A1.month.ago..)+OR+(dfn%3Adrivers%2Fnet%2Fteam%2F+AND+rt%3A6.month.ago..)+OR+(dfn%3Anet%2Fipv4%2Figmp.c+AND+rt%3A6.month.ago..)+OR+(dfn%3Anet%2Fipv6%2Fmcast.c+AND+rt%3A1651301673..))+NOT+(tc%3Astable%40vger.kernel.org+OR+f%3Asfr%40canb.auug.org.au))+AND+dt%3A20220930090001..
> > # https://lore.kernel.org/all/ 43/?
>
> Of course, the lack of approxidate parsing there inside lei is
> fine, since the lore.kernel.org instance will do it remotely...
>
> > So I want to know when/why *lei* add the quotes.
>
> lei adds quotes since it can't distinguish if the shell user
> used single or double quotes. Xapian uses double quotes for
> phrase search, and I wanted: lei q "this is a phrase"
> to work naturally, which means: lei q 'this is a phrase'
> (with single quotes) works the same way as with double quotes
> because the difference is handled by the shell and lei never
> sees it.
Thanks for the help.
Hangbin
^ permalink raw reply [relevance 71%]
* Re: [Need Help] lei add quotes at the search
@ 2022-10-31 7:47 71% ` Hangbin Liu
2022-10-31 7:36 71% ` Hangbin Liu
1 sibling, 0 replies; 200+ results
From: Hangbin Liu @ 2022-10-31 7:47 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
On Sun, Oct 30, 2022 at 11:06:31PM +0000, Eric Wong wrote:
> > If I add "\" on each "(", this will break to a very long config search.
> > I tried to adjust it to
>
> I think that can work if lei.internal.rawstr is set in the
> config to indicate stdin was used (It's auto-set by --stdin).
OH, BTW, I will get error
fatal: bad config line 4 in file [..snip..]/lei.saved-search
if adding '\' in the config file.
> I guess it also works if it's the only lei.q config entry
> and the lei.q entry contains "\n"
But with "\n" in the config file. My previous config will works fine.
>
> cf. https://public-inbox.org/meta/20211110102837.41721-1-e@80x24.org/
Thanks
Hangbin
^ permalink raw reply [relevance 71%]
* [PATCH 3/6] doc: txt2pre: linkify "lei COMMAND" form
2022-11-03 0:48 90% ` [PATCH 6/6] txt2pre: linkify lei/store => lei-store-format.html Eric Wong
2022-11-03 0:48 90% ` [PATCH 5/6] doc: lei-import: link to lei-store-format(5) Eric Wong
@ 2022-11-03 0:48 71% ` Eric Wong
2 siblings, 0 replies; 200+ results
From: Eric Wong @ 2022-11-03 0:48 UTC (permalink / raw)
To: meta
While manpages are named `L<lei-COMMAND(1)>', `lei COMMAND'
can be worth linkifying for ease-of-navigation, too.
---
Documentation/txt2pre | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/Documentation/txt2pre b/Documentation/txt2pre
index c8dbd2ba..82573a30 100755
--- a/Documentation/txt2pre
+++ b/Documentation/txt2pre
@@ -9,7 +9,7 @@ use strict;
use warnings;
use PublicInbox::Linkify;
use PublicInbox::Hval qw(ascii_html);
-my %xurls;
+my (%xurls, %lei);
for (qw[lei(1)
lei-add-external(1)
lei-add-watch(1)
@@ -77,6 +77,7 @@ for (qw[lei(1)
my ($n) = (/([\w\-\.]+)/);
$xurls{$_} = "$n.html";
$xurls{$n} = "$n.html";
+ /\Alei-(.+?)\(1\)\z/ and $xurls{"lei $1"} = "$n.html";
}
for (qw[make(1) flock(2) setrlimit(2) vfork(2) tmpfs(5) inotify(7) unix(7)
@@ -161,6 +162,9 @@ if ($str =~ /^NAME\n\s+([^\n]+)/sm) {
if ($title =~ /([\w\.\-]+)/) {
delete $xurls{$1};
}
+ if ($title =~ /\blei-([\w\-]+)\b/) {
+ delete $xurls{"lei $1"};
+ }
}
$title = ascii_html($title);
my $l = PublicInbox::Linkify->new;
^ permalink raw reply related [relevance 71%]
* [PATCH 0/2] fix xt/lei-auth-fail.t
@ 2023-01-29 22:58 71% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2023-01-29 22:58 UTC (permalink / raw)
To: meta
I need a failing mock IMAP server for this, or remember xt/ exists :<
Eric Wong (2):
lei_input: give a hint for upper-case in labels
xt/lei-auth-fail: use valid label name
lib/PublicInbox/LeiInput.pm | 2 ++
xt/lei-auth-fail.t | 7 ++++---
2 files changed, 6 insertions(+), 3 deletions(-)
^ permalink raw reply [relevance 71%]
* Re: FUSE3 vs read-write IMAP for lei
@ 2023-02-20 19:27 71% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2023-02-20 19:27 UTC (permalink / raw)
To: meta
Eric Wong <e@80x24.org> wrote:
> == FUSE3 - Maildir-oriented FS
>
> + will require C compiler since old FUSE XS modules don't do
> FUSE3 (for readdirplus); and going w/o readdirplus is
> unimaginable for Maildir.
>
> + I already have existing (unreleased) AGPL-3 work based on
> FUSE3 + URCU + Perl5 with just-ahead-of-time (JAOT) compilation
Fwiw, I've pushed out a new "fuse3" branch to public-inbox.git:
https://80x24.org/public-inbox.git/80ce906027eeb7b4cc5cc7d3858294927951988a/s/
I think I need some C + URCU in my life to keep my brain working
> * I've seen the light w/ URCU, and can't go back to C without it :P
>
> - likely Linux-only (not sure how good FUSE support will be if
> depending on FUSE3 features)
Well, I got rid of the futex requirement from the original...
> - kernel caches still incur nasty memory overhead w/ Maildir
>
> - readdir(3) userspace API still sucks
^ permalink raw reply [relevance 71%]
* Re: [PATCH] lei q: do not collapse threads with `-tt'
@ 2023-02-26 12:17 71% ` Maxim Mikityanskiy
0 siblings, 1 reply; 200+ results
From: Maxim Mikityanskiy @ 2023-02-26 12:17 UTC (permalink / raw)
To: Eric Wong; +Cc: meta, Kyle Meyer
On Tue, Feb 14, 2023 at 02:42:32AM +0000, Eric Wong wrote:
> Maxim Mikityanskiy <maxtram95@gmail.com> wrote:
> > lei q --no-save -a -o /tmp/lei-test -I 'https://lore.kernel.org/all' \
> > -tt 'a:syzbot AND rt:2023-01-01..2023-01-07'
>
> At first, I thought -a (--augment) was causing it...
>
> Sidenote: you also don't need to quote the query (I forget the exact
> rules, but I tried to keep quotes easier for phrase searches).
>
> > It looks as if the match works correctly, but the -tt option fails to
> > mark most of the matched emails as important, except a few that actually
> > got marked (I couldn't find a pattern here). It's also not consistent,
> > for example, after I removed /tmp/lei-test and restarted the lei q
> > command, I got many more important emails, almost in each thread, but
> > there were still threads without flagged emails.
>
> Yes, now it seems it's the collapsing optimization.
>
> > I'm checking the flags with mutt.
> >
> > Does anyone know what could be the reason for such behavior?
>
> I think the following patch fixes it.
Sorry for taking too long, I finally found a minute to test it, and
unfortunately I didn't see a difference. I queried for:
a:syzbot AND rt:2023-02-01..2023-02-07
and I still saw I lot of threads without a single flag.
I double-checked that the patch was actually applied, killed lei-daemon,
and removed the mailbox directory, but it didn't help.
> (I accidentally sent you a private copy with invalid blobs since
> I had other unpublished changes)
>
> -----8<-------
> Subject: [PATCH] lei q: do not collapse threads with `-tt'
>
> While having Xapian collapse threads is an easy way to reduce
> the amount of deduplication work we need to do when writing
> out threads; we can't rely on it when using `lei q -tt` since
> that needs to flag all hits.
>
> Reported-by: Maxim Mikityanskiy <maxtram95@gmail.com>
> Link: https://public-inbox.org/git/Y+pgBmj0jxR+cVkD@mail.gmail.com/
> ---
> lib/PublicInbox/Search.pm | 5 +++--
> 1 file changed, 3 insertions(+), 2 deletions(-)
>
> diff --git a/lib/PublicInbox/Search.pm b/lib/PublicInbox/Search.pm
> index 2feb3e13..273cc57c 100644
> --- a/lib/PublicInbox/Search.pm
> +++ b/lib/PublicInbox/Search.pm
> @@ -460,8 +460,9 @@ sub _enquire_once { # retry_reopen callback
> $enquire->set_sort_by_relevance_then_value(TS, !$opts->{asc});
> }
>
> - # `mairix -t / --threads' or JMAP collapseThreads
> - if ($opts->{threads} && has_threadid($self)) {
> + # `lei q -t / --threads' or JMAP collapseThreads; but don't collapse
> + # on `-tt' ({threads} > 1) which sets the Flagged|Important keyword
> + if (($opts->{threads} // 0) == 1 && has_threadid($self)) {
> $enquire->set_collapse_key(THREADID);
> }
> $enquire->get_mset($opts->{offset} || 0, $opts->{limit} || 50);
^ permalink raw reply [relevance 71%]
* [PATCH] doc: note "lei q -tt" is broken with HTTP(S) remotes
@ 2023-02-26 17:15 71% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2023-02-26 17:15 UTC (permalink / raw)
To: Maxim Mikityanskiy; +Cc: meta, Kyle Meyer
Eric Wong <e@80x24.org> wrote:
> Getting -tt to work on remote inboxes will take more effort.
> I'm not sure which option is better:
I suppose documenting the current breakage first is important:
--------- 8< --------
Subject: [PATCH] doc: note "lei q -tt" is broken with HTTP(S) remotes
I'm still trying to decide how to handle HTTP(S) remotes
properly...
Link: https://public-inbox.org/meta/20230226170931.M947721@dcvr/
---
Documentation/lei-q.pod | 3 +++
1 file changed, 3 insertions(+)
diff --git a/Documentation/lei-q.pod b/Documentation/lei-q.pod
index d52c5b04..5e9a5658 100644
--- a/Documentation/lei-q.pod
+++ b/Documentation/lei-q.pod
@@ -124,6 +124,9 @@ of the same thread.
TODO: Warning: this flag may become persistent and saved in
lei/store unless an MUA unflags it! (Behavior undecided)
+Caveat: C<-tt> only works on locally-indexed messages at the
+moment, and not on remote (HTTP(S)) endpoints.
+
=item --jobs=QUERY_WORKERS[,WRITE_WORKERS]
=item --jobs=,WRITE_WORKERS
^ permalink raw reply related [relevance 71%]
* [PATCH 4/6] doc: lei import: add hints about nntp.* and imap.* config options
@ 2023-03-09 19:28 71% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2023-03-09 19:28 UTC (permalink / raw)
To: meta
I'm setting up more imports and forgot about them :x
---
Documentation/lei-import.pod | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/Documentation/lei-import.pod b/Documentation/lei-import.pod
index 69ec6497..31d6db13 100644
--- a/Documentation/lei-import.pod
+++ b/Documentation/lei-import.pod
@@ -86,8 +86,13 @@ Default: C<auto>
Use the specified proxy (e.g., C<socks5h://0:9050>).
+Consider L<imap.proxy> and L<nntp.proxy> which can be persistently
+configured on a per-host basis in L<lei-config(1)>.
+
=back
+See L<lei-config(1)> for various C<imap.*> and C<nntp.*> options.
+
=head1 CONTACT
Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org>
@@ -103,4 +108,4 @@ License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>
=head1 SEE ALSO
-L<lei-index(1)>, L<lei-store-format(5)>
+L<lei-config(1)>, L<lei-index(1)>, L<lei-store-format(5)>
^ permalink raw reply related [relevance 71%]
* repeat `lei import' users?
@ 2023-03-23 22:05 71% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2023-03-23 22:05 UTC (permalink / raw)
To: meta
Just wondering if there's lei-mail-sync-overview(7) followers,
yet... It could be a bit less clunky :x
^ permalink raw reply [relevance 71%]
* Issues with `lei` as non-root
@ 2023-03-28 1:00 71% Louis DeLosSantos
2023-03-28 1:32 71% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Louis DeLosSantos @ 2023-03-28 1:00 UTC (permalink / raw)
To: meta
Hello,
I'm experimenting with `lei` as a nice search tool for `lore.kernel.org`
Everything works fine with the caveat that it seems to break if I'm not root.
When using `lei` as non-root we get this error:
```
E: Linux::Inotify2->new: Too many open files at
/usr/share/perl5/vendor_perl/PublicInbox/DirIdle.pm line 40.
connect(/run/user/1000/lei/5.seq.sock): Connection refused (after
attempted daemon start)
```
Any ideas why this may occur? Is `lei` designed to only be ran as root
or is Fedora installing perl in an odd fashion which results in root
needing to be used?
^ permalink raw reply [relevance 71%]
* Re: Issues with `lei` as non-root
2023-03-28 1:00 71% Issues with `lei` as non-root Louis DeLosSantos
@ 2023-03-28 1:32 71% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2023-03-28 1:32 UTC (permalink / raw)
To: Louis DeLosSantos; +Cc: meta
Louis DeLosSantos <louis.delos@gmail.com> wrote:
> Hello,
>
> I'm experimenting with `lei` as a nice search tool for `lore.kernel.org`
>
> Everything works fine with the caveat that it seems to break if I'm not root.
>
> When using `lei` as non-root we get this error:
I've never used lei as root nor has any part of public-inbox
ever been intended to run as root.
> ```
> E: Linux::Inotify2->new: Too many open files at
> /usr/share/perl5/vendor_perl/PublicInbox/DirIdle.pm line 40.
> connect(/run/user/1000/lei/5.seq.sock): Connection refused (after
> attempted daemon start)
> ```
>
> Any ideas why this may occur? Is `lei` designed to only be ran as root
> or is Fedora installing perl in an odd fashion which results in root
> needing to be used?
What's the output of `ulimit -n` and `lsof -p $(lei daemon-pid)`?
(you may need to use `ps -ef |grep lei-daemon` to get the PID
if lei is broken and using too many FDs, though)
`ulimit -n' is the open file limit, typically 1024 or higher.
If `lsof -p $PID` may reveal a bug in lei which leaves too many
files open. lei (especially with inotify on Linux) should use
far less than 1024.
(FreeBSD may end up using far more open files, but that's a
different story)
^ permalink raw reply [relevance 71%]
* Re: Issues with `lei` as non-root
@ 2023-03-28 2:52 71% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2023-03-28 2:52 UTC (permalink / raw)
To: Louis DeLosSantos; +Cc: meta
Louis DeLosSantos <louis.delos@gmail.com> wrote:
<snip>
> Above is 54 open sockets. Which seems fine.
Actually, only 18 (0..17). The `mem' stuff is mmap-ed and
doesn't count against `ulimit -n` (RLIMIT_NOFILE).
> Should daemon be running as root, if I intend to only use lei as user?
Definitely not; the lei-daemon is per-user.
I also forgot, inotify has its own per-user limits; perhaps
you're hitting those?
# show system-wide limits
$ head /proc/sys/fs/inotify/max_*
# show per-user inotify FDs (-nP speeds up lsof by avoiding lookups)
$ lsof -nP -u $USER |grep inotify
^ permalink raw reply [relevance 71%]
* Re: Issues with `lei` as non-root
@ 2023-03-28 3:38 71% ` Eric Wong
2023-03-28 4:08 71% ` Louis DeLosSantos
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2023-03-28 3:38 UTC (permalink / raw)
To: Louis DeLosSantos; +Cc: meta
Louis DeLosSantos <louis.delos@gmail.com> wrote:
> > Definitely not; the lei-daemon is per-user.
>
> Okay, maybe this is the issue to begin with? I installed lei from dnf.
> I'm not sure what launches the daemon, is it launched on first run?
>
> If that is the case, it was probably launched when I restored to `sudo
> lei q ....` command.
> But, if its running as systemd service, I could move it to user service.
You shouldn't need to manage it as a service; it's auto-started
and killing it is harmless in most cases. I'm considering it
have it auto-exit if it stays idle for a long time and there's
no active inotify watches.
lei-daemon doesn't start until any other lei command is invoked;
so it shouldn't be started on installation.
> # show system-wide limits
> ==> /proc/sys/fs/inotify/max_user_instances <==
> 128
<snip>
> tail 367093 louis 4r a_inode 0,14 0
Thats a lot of tail processes....
I wonder if they were spawned by `lei q -v' for emitting curl stderr?
They should be auto-killed.
(or if you have some other reason for running tail on your system).
^ permalink raw reply [relevance 71%]
* Re: Issues with `lei` as non-root
2023-03-28 3:38 71% ` Eric Wong
@ 2023-03-28 4:08 71% ` Louis DeLosSantos
0 siblings, 0 replies; 200+ results
From: Louis DeLosSantos @ 2023-03-28 4:08 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
> Thats a lot of tail processes....
Ugh, sorry to waste your time.
All the tails were from a run away UI program I'm working on.
Once I killed them, `let` runs just fine as non-root.
Thanks for the free tech support, hope I didn't steal your attention
from something valuable :-D.
On Mon, Mar 27, 2023 at 11:38 PM Eric Wong <e@80x24.org> wrote:
>
> Louis DeLosSantos <louis.delos@gmail.com> wrote:
> > > Definitely not; the lei-daemon is per-user.
> >
> > Okay, maybe this is the issue to begin with? I installed lei from dnf.
> > I'm not sure what launches the daemon, is it launched on first run?
> >
> > If that is the case, it was probably launched when I restored to `sudo
> > lei q ....` command.
> > But, if its running as systemd service, I could move it to user service.
>
> You shouldn't need to manage it as a service; it's auto-started
> and killing it is harmless in most cases. I'm considering it
> have it auto-exit if it stays idle for a long time and there's
> no active inotify watches.
>
> lei-daemon doesn't start until any other lei command is invoked;
> so it shouldn't be started on installation.
>
> > # show system-wide limits
>
> > ==> /proc/sys/fs/inotify/max_user_instances <==
> > 128
>
> <snip>
>
> > tail 367093 louis 4r a_inode 0,14 0
>
> Thats a lot of tail processes....
> I wonder if they were spawned by `lei q -v' for emitting curl stderr?
> They should be auto-killed.
> (or if you have some other reason for running tail on your system).
^ permalink raw reply [relevance 71%]
* [PATCH] t/lei-refresh-mail-sync: improve test reliability
@ 2023-03-28 10:53 71% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2023-03-28 10:53 UTC (permalink / raw)
To: meta
Lack of signalfd/EVFILT_SIGNAL means we need to kill a
process repeatedly to ensure it wakes up.
---
t/lei-refresh-mail-sync.t | 8 ++++++--
1 file changed, 6 insertions(+), 2 deletions(-)
diff --git a/t/lei-refresh-mail-sync.t b/t/lei-refresh-mail-sync.t
index 0498a0c4..8ccc68c6 100644
--- a/t/lei-refresh-mail-sync.t
+++ b/t/lei-refresh-mail-sync.t
@@ -137,8 +137,12 @@ SKIP: {
my $ar = PublicInbox::AutoReap->new($pid);
ok(!(lei 'refresh-mail-sync', $url), 'URL fails on dead -imapd');
ok(!(lei 'refresh-mail-sync', '--all'), '--all fails on dead -imapd');
- $ar->kill for qw(avoid sig wake miss-no signalfd or EVFILT_SIG);
- $ar->join('TERM');
+ {
+ local $SIG{CHLD} = sub { $ar->join('TERM'); undef $ar };
+ do {
+ eval { $ar->kill and tick(0.01) }
+ } while (defined($ar));
+ }
my $cmd = $srv->{imapd}->{cmd};
my $s = $srv->{imapd}->{s};
^ permalink raw reply related [relevance 71%]
* [PATCH] t/lei-import-nntp: dump $lei_err on failure
@ 2023-04-29 7:18 71% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2023-04-29 7:18 UTC (permalink / raw)
To: meta
I hit an error on the backwards range import test and can't
reproduce it, perhaps dumping $lei_err can help diagnose it
in the future.
---
t/lei-import-nntp.t | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/t/lei-import-nntp.t b/t/lei-import-nntp.t
index eb1ae312..2c48d973 100644
--- a/t/lei-import-nntp.t
+++ b/t/lei-import-nntp.t
@@ -43,7 +43,8 @@ test_lei({ tmpdir => $tmpdir }, sub {
lei_ok 'ls-mail-sync';
like($lei_out, qr!\A\Q$url\E\n\z!, 'ls-mail-sync output as-expected');
- ok(!lei(qw(import), "$url/12-1"), 'backwards range rejected');
+ ok(!lei(qw(import), "$url/12-1"), 'backwards range rejected') or
+ diag $lei_err;
# new home
local $ENV{HOME} = "$tmpdir/h2";
^ permalink raw reply related [relevance 71%]
* [PATCH] t/lei.t: quiet newline warning on older Perls
@ 2023-06-08 18:26 71% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2023-06-08 18:26 UTC (permalink / raw)
To: meta
Perl < 5.22 warned on newlines in the middle of a string instead
of just the end. Workaround it by disabling all warnings on older
Perls while running File::Path::mkpath.
---
t/lei.t | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/t/lei.t b/t/lei.t
index a80143ef..5d0fa622 100644
--- a/t/lei.t
+++ b/t/lei.t
@@ -149,7 +149,10 @@ my $test_fail = sub {
for my $lk (qw(ei inbox)) {
my $d = "$home/newline\n$lk";
my $all = $lk eq 'ei' ? 'ALL' : 'all';
- File::Path::mkpath("$d/$all.git/objects");
+ { # quiet newline warning on older Perls
+ local $^W = undef if $^V lt v5.22.0;
+ File::Path::mkpath("$d/$all.git/objects");
+ }
open my $fh, '>', "$d/$lk.lock" or BAIL_OUT "open $d/$lk.lock";
for my $fl (qw(-I --only)) {
ok(!lei('q', $fl, $d, 'whatever'),
^ permalink raw reply related [relevance 71%]
* [PATCH] doc: lei q: document v2:$INBOX_DIR output format
@ 2023-06-15 0:08 71% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2023-06-15 0:08 UTC (permalink / raw)
To: meta
This has been supported in every lei release, actually.
---
Documentation/lei-q.pod | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/Documentation/lei-q.pod b/Documentation/lei-q.pod
index 5e9a5658..c0254ba0 100644
--- a/Documentation/lei-q.pod
+++ b/Documentation/lei-q.pod
@@ -50,6 +50,10 @@ A prefix can specify the format of the output: C<maildir>,
C<mboxrd>, C<mboxcl2>, C<mboxcl>, C<mboxo>. For a description of
mail formats, see L<lei-mail-formats(5)>.
+C<v2:/path/to/inbox> may be used to create a new inbox of
+L<public-inbox-v2-format(5)>. The new inbox will not be configured
+in the L<public-inbox-config(5)> file.
+
C<maildir> is the default for an existing directory or non-existing path.
Default: C<-> (stdout)
^ permalink raw reply related [relevance 71%]
* [PATCH] t/lei-mirror: do not bail out on `make help' failure
@ 2023-09-14 12:12 71% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2023-09-14 12:12 UTC (permalink / raw)
To: meta
I'm not sure why, but this test occasionally fails on OpenBSD
and I can't reproduce it on a repeatable basis. In any case,
there's no reason we can't continue the rest of the test if
`make help' fails.
---
t/lei-mirror.t | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/t/lei-mirror.t b/t/lei-mirror.t
index 2400578a..9b5d73ec 100644
--- a/t/lei-mirror.t
+++ b/t/lei-mirror.t
@@ -27,7 +27,8 @@ test_lei({ tmpdir => $tmpdir }, sub {
"mirror of $http/t1/\n", 'description set');
ok(-f "$t1/Makefile", 'convenience Makefile added (v1)');
my $make = which('make');
- xsys_e([$make, 'help'], undef, { -C => $t1, 1 => \(my $help) });
+ is(xsys([$make, 'help'], undef, { -C => $t1, 1 => \(my $help) }), 0,
+ 'make help');
ok(-f "$t1/inbox.config.example", 'inbox.config.example downloaded');
isnt($help, '', 'make help worked');
is((stat(_))[9], $created{v1},
^ permalink raw reply related [relevance 71%]
* Re: [PATCH 03/15] t/lei-p2q: extra diagnostics
@ 2023-09-21 10:23 71% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2023-09-21 10:23 UTC (permalink / raw)
To: meta
Eric Wong <e@80x24.org> wrote:
> I got one mysterious test failure here, once, and can't seem
> to reproduce it...
nearly 2 years later: I've hit it again, but once again cannot
reproduce it...
> +++ b/t/lei-p2q.t
> @@ -7,7 +7,7 @@ require_mods(qw(json DBD::SQLite Search::Xapian));
>
> test_lei(sub {
> ok(!lei(qw(p2q this-better-cause-format-patch-to-fail)),
> - 'p2q fails on bogus arg');
> + 'p2q fails on bogus arg') or diag $lei_err;
> like($lei_err, qr/format-patch.*failed/, 'notes format-patch failure');
$? does get printed (32768, so (>> 8) means 128), as it should be.
So I wonder if there's a place we drop the socket prematurely or
something else is amiss...
On a side note, lei's internal IPC could probably be simplified
a bit w/o sacrificing parallelism.
^ permalink raw reply [relevance 71%]
* [PATCH 0/4] small lei fixes
@ 2023-09-22 21:13 71% Eric Wong
2023-09-22 21:13 90% ` [PATCH 1/4] lei blob|rediff: fix usage of lei->fail Eric Wong
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2023-09-22 21:13 UTC (permalink / raw)
To: meta
Only noticed while working on bigger fixes...
Eric Wong (4):
lei blob|rediff: fix usage of lei->fail
lei: improve ->fail internal API
lei_to_mail: drop awkward duplication of $lei object
lei: use File::Temp for listing saved searches
lib/PublicInbox/LEI.pm | 19 ++++++++++++-------
lib/PublicInbox/LeiBlob.pm | 2 +-
lib/PublicInbox/LeiRediff.pm | 2 +-
lib/PublicInbox/LeiSavedSearch.pm | 20 +++++++++-----------
lib/PublicInbox/LeiToMail.pm | 12 +++++-------
5 files changed, 28 insertions(+), 27 deletions(-)
^ permalink raw reply [relevance 71%]
* [PATCH 3/4] lei: use scalar %SIG assignment
@ 2023-09-24 21:08 71% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2023-09-24 21:08 UTC (permalink / raw)
To: meta
Perl v5.16.3 (and possibly some later versions) complain about
this, but newer (v5.32.1) are fine with it.
Fixes: e281363ba937 ("lei: ensure we run DESTROY|END at daemon exit w/ kqueue")
---
lib/PublicInbox/LEI.pm | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 1ead9bf6..be77fa90 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -1381,9 +1381,9 @@ sub lazy_start {
PublicInbox::DS::sig_setmask($oldset) if @kq_ign;
# exit() may trigger waitpid via various DESTROY, ensure interruptible
- local @SIG{TERM} = sub { exit(POSIX::SIGTERM + 128) };
- local @SIG{INT} = sub { exit(POSIX::SIGINT + 128) };
- local @SIG{QUIT} = sub { exit(POSIX::SIGQUIT + 128) };
+ local $SIG{TERM} = sub { exit(POSIX::SIGTERM + 128) };
+ local $SIG{INT} = sub { exit(POSIX::SIGINT + 128) };
+ local $SIG{QUIT} = sub { exit(POSIX::SIGQUIT + 128) };
PublicInbox::DS::sig_setmask($oldset) if !@kq_ign;
dump_and_clear_log();
exit($exit_code // 0);
^ permalink raw reply related [relevance 71%]
* [PATCH 0/2] lei: support reading inboxes & extindex w/o search
@ 2023-09-30 0:36 71% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2023-09-30 0:36 UTC (permalink / raw)
To: meta
This works on completely unindexed inboxes, even, as long as the
inbox.lock (or ssoma.lock) file exists.
Eric Wong (2):
lei_input: always prefix `maildir:' internally
lei convert: support reading from v1, v2, and extindex
lib/PublicInbox/ExtSearch.pm | 6 +-
lib/PublicInbox/LeiInput.pm | 113 ++++++++++++++++++++++++++---------
t/extsearch.t | 24 ++++++++
t/lei-convert.t | 40 +++++++++++++
4 files changed, 153 insertions(+), 30 deletions(-)
^ permalink raw reply [relevance 71%]
* [PATCH] t/lei-convert: fix uninitialized variable w/o pigz
@ 2023-09-30 16:17 71% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2023-09-30 16:17 UTC (permalink / raw)
To: meta
`backtick` captures return `undef' when a command is missing
Fixes: 5df0446abcca (lei: don't gzip --rsyncable by default for mbox*)
---
t/lei-convert.t | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/t/lei-convert.t b/t/lei-convert.t
index d75110cb..84b57f81 100644
--- a/t/lei-convert.t
+++ b/t/lei-convert.t
@@ -132,7 +132,7 @@ test_lei({ tmpdir => $tmpdir }, sub {
SKIP: {
my $ok;
for my $x (($ENV{GZIP}//''), qw(pigz gzip)) {
- $x && `$x -h 2>&1` =~ /--rsyncable\b/s or next;
+ $x && (`$x -h 2>&1`//'') =~ /--rsyncable\b/s or next;
$ok = $x;
last;
}
^ permalink raw reply related [relevance 71%]
* [PATCH 07/13] lei: correct exit signal
@ 2023-10-01 9:54 71% ` Eric Wong
2023-10-01 9:54 71% ` [PATCH 12/13] lei: ->fail only allows integer exit codes Eric Wong
2023-10-01 9:54 71% ` [PATCH 08/13] lei mail-diff: don't remove temporary subdirectory Eric Wong
2 siblings, 0 replies; 200+ results
From: Eric Wong @ 2023-10-01 9:54 UTC (permalink / raw)
To: meta
The first argument passed to Perl signal handlers is a
signal name (e.g. "TERM") and not an integer that can
be passed to the `exit' perlop. Thus we must look up the
integer value from the POSIX module.
---
lib/PublicInbox/LEI.pm | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 48c5644b..1b14d5e1 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -1310,9 +1310,9 @@ sub lazy_start {
local $quit = do {
my (undef, $eof_p) = PublicInbox::PktOp->pair;
sub {
- $exit_code //= shift;
+ $exit_code //= eval("POSIX::SIG$_[0] + 128") if @_;
eval 'PublicInbox::LeiNoteEvent::flush_task()';
- my $lis = $pil or exit($exit_code);
+ my $lis = $pil or exit($exit_code // 0);
# closing eof_p triggers \&noop wakeup
$listener = $eof_p = $pil = $path = undef;
$lis->close; # DS::close
^ permalink raw reply related [relevance 71%]
* [PATCH 12/13] lei: ->fail only allows integer exit codes
2023-10-01 9:54 71% ` [PATCH 07/13] lei: correct exit signal Eric Wong
@ 2023-10-01 9:54 71% ` Eric Wong
2023-10-01 9:54 71% ` [PATCH 08/13] lei mail-diff: don't remove temporary subdirectory Eric Wong
2 siblings, 0 replies; 200+ results
From: Eric Wong @ 2023-10-01 9:54 UTC (permalink / raw)
To: meta
We can't use floating point numbers nor Inf/-Inf as exit codes;
but we can allow `-1' as shorthand for 255.
---
lib/PublicInbox/LEI.pm | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 1b14d5e1..1899bf38 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -524,7 +524,7 @@ sub sigpipe_handler { # handles SIGPIPE from @WQ_KEYS workers
sub fail ($;@) {
my ($lei, @msg) = @_;
- my $exit_code = looks_like_number($msg[0]) ? shift(@msg) : undef;
+ my $exit_code = ($msg[0]//'') =~ /\A-?[0-9]+\z/ ? shift(@msg) : undef;
local $current_lei = $lei;
$lei->{failed}++;
if (@msg) {
^ permalink raw reply related [relevance 71%]
* [PATCH 08/13] lei mail-diff: don't remove temporary subdirectory
2023-10-01 9:54 71% ` [PATCH 07/13] lei: correct exit signal Eric Wong
2023-10-01 9:54 71% ` [PATCH 12/13] lei: ->fail only allows integer exit codes Eric Wong
@ 2023-10-01 9:54 71% ` Eric Wong
2 siblings, 0 replies; 200+ results
From: Eric Wong @ 2023-10-01 9:54 UTC (permalink / raw)
To: meta
->{curdir} is localized inside MailDiff->dump_eml anyways, so it
was attempting to remove `undef' :x. Since most messages don't
have too many attachments, save some opcodes on our end and just
let File::Temp::Dir->DESTROY handle all the cleanup.
---
lib/PublicInbox/LeiMailDiff.pm | 2 --
1 file changed, 2 deletions(-)
diff --git a/lib/PublicInbox/LeiMailDiff.pm b/lib/PublicInbox/LeiMailDiff.pm
index 5e2e4b0b..af6ecf82 100644
--- a/lib/PublicInbox/LeiMailDiff.pm
+++ b/lib/PublicInbox/LeiMailDiff.pm
@@ -7,7 +7,6 @@ package PublicInbox::LeiMailDiff;
use v5.12;
use parent qw(PublicInbox::IPC PublicInbox::LeiInput PublicInbox::MailDiff);
use PublicInbox::Spawn qw(run_wait);
-use File::Path ();
require PublicInbox::LeiRediff;
sub diff_a ($$) {
@@ -21,7 +20,6 @@ sub diff_a ($$) {
my $rdr = { -C => "$self->{tmp}" };
@$rdr{1, 2} = @$lei{1, 2};
run_wait($cmd, $lei->{env}, $rdr) and $lei->child_error($?);
- File::Path::remove_tree($self->{curdir});
}
sub input_eml_cb { # used by PublicInbox::LeiInput::input_fh
^ permalink raw reply related [relevance 71%]
* [PATCH 10/21] lei: reuse PublicInbox::Config::noop
@ 2023-10-04 3:49 71% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2023-10-04 3:49 UTC (permalink / raw)
To: meta
No need to define our own empty `noop' sub when PublicInbox::Config
already has one and is loaded anyways.
---
lib/PublicInbox/LEI.pm | 14 +++-----------
1 file changed, 3 insertions(+), 11 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index fba4edf3..c9ad46e2 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -1217,8 +1217,6 @@ sub event_step_init {
};
}
-sub noop {}
-
sub oldset { $oldset }
sub dump_and_clear_log {
@@ -1364,15 +1362,9 @@ sub lazy_start {
$lis->close; # DS::close
};
};
- my $sig = {
- CHLD => \&PublicInbox::DS::enqueue_reap,
- QUIT => $quit,
- INT => $quit,
- TERM => $quit,
- HUP => \&noop,
- USR1 => \&noop,
- USR2 => \&noop,
- };
+ my $sig = { CHLD => \&PublicInbox::DS::enqueue_reap };
+ $sig->{$_} = $quit for qw(QUIT INT TERM);
+ $sig->{$_} = \&PublicInbox::Config::noop for qw(HUP USR1 USR2);
# for EVFILT_SIGNAL and signalfd behavioral difference:
my @kq_ign = eval { require PublicInbox::DSKQXS } ? keys(%$sig) : ();
^ permalink raw reply related [relevance 71%]
* [PATCH 2/9] lei: do not issue sto->done if socket is inactive
@ 2023-10-07 21:24 71% ` Eric Wong
1 sibling, 0 replies; 200+ results
From: Eric Wong @ 2023-10-07 21:24 UTC (permalink / raw)
To: meta
This fixes attempts to use an undefined value as an ARRAY reference
in PublicInbox::IPC::wq_io_do
---
lib/PublicInbox/LEI.pm | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index f8bcd43d..f00b2465 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -1296,7 +1296,7 @@ sub can_stay_alive { # PublicInbox::DS::post_loop_do cb
my $lne = delete($cfg->{-lei_note_event});
$lne->wq_close if $lne;
my $sto = delete($cfg->{-lei_store}) // next;
- eval { $sto->wq_io_do('done') };
+ eval { $sto->wq_do('done') if $sto->{-wq_s1} };
warn "E: $@ (dropping store for $cfg->{-f})" if $@;
$sto->wq_close;
}
^ permalink raw reply related [relevance 71%]
* Re: [PATCH 3/9] lei: always use async `done' requests to store
@ 2023-10-08 1:58 71% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2023-10-08 1:58 UTC (permalink / raw)
To: meta
Eric Wong <e@80x24.org> wrote:
> diff --git a/lib/PublicInbox/LeiRemote.pm b/lib/PublicInbox/LeiRemote.pm
> index 54750062..15013baa 100644
> --- a/lib/PublicInbox/LeiRemote.pm
> +++ b/lib/PublicInbox/LeiRemote.pm
> @@ -52,7 +52,7 @@ sub mset {
> $self->{smsg} = [];
> $fh = IO::Uncompress::Gunzip->new($fh, MultiStream => 1);
> PublicInbox::MboxReader->mboxrd($fh, \&_each_mboxrd_eml, $self);
> - my $wait = $self->{lei}->{sto}->wq_do('done');
> + $self->{lei}->sto_done_request;
That's usually not the normal lei/store, but the {tmp_sto} one
from LeiRediff. So it must be synchronous in that case because
we make multiple queries, there.
So I'll revert that hunk.
And ugh, the new lei-store-fail.t test is so nasty; I don't
think a 100ms delay is enough for some systems...
diff --git a/t/lei-store-fail.t b/t/lei-store-fail.t
index e9ad779f..fb0f2b75 100644
--- a/t/lei-store-fail.t
+++ b/t/lei-store-fail.t
@@ -31,7 +31,7 @@ Message-ID: <$_\@t>
will this save?
EOM
}
- tick 0.1; # XXX ugh, this is so hacky
+ tick 0.2; # XXX ugh, this is so hacky
# make sto_done_request fail:
remove_tree("$ENV{HOME}/.local/share/lei/store");
^ permalink raw reply related [relevance 71%]
* [PATCH 2/3] doc: lei-q: drop stale TODO comment (fixed in 1f1b1f0e22f7)
@ 2023-10-16 11:33 71% ` Štěpán Němec
2023-10-16 21:17 71% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Štěpán Němec @ 2023-10-16 11:33 UTC (permalink / raw)
To: meta
Fixes: 1f1b1f0e22f7 ("doc: lei-q: document SEARCH TERMS prefixes")
---
I also wonder about CAVEATS from lei-overview.pod:
IMAP and NNTP client performance is poor on high-latency connections.
It will hopefully be fixed in 2022.
I think this needs an update (or removal, if the issue was indeed
fixed).
Thanks.
Documentation/lei-q.pod | 3 ---
1 file changed, 3 deletions(-)
diff --git a/Documentation/lei-q.pod b/Documentation/lei-q.pod
index 5f5338490b07..4862ce78b709 100644
--- a/Documentation/lei-q.pod
+++ b/Documentation/lei-q.pod
@@ -12,9 +12,6 @@ lei q [OPTIONS] (--stdin|-)
Search for messages across the lei/store and externals.
-=for comment
-TODO: Give common prefixes, or at least a description/reference.
-
=head1 OPTIONS
=for comment
--
2.42.0
^ permalink raw reply related [relevance 71%]
* Re: [PATCH 2/3] doc: lei-q: drop stale TODO comment (fixed in 1f1b1f0e22f7)
2023-10-16 11:33 71% ` [PATCH 2/3] doc: lei-q: drop stale TODO comment (fixed in 1f1b1f0e22f7) Štěpán Němec
@ 2023-10-16 21:17 71% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2023-10-16 21:17 UTC (permalink / raw)
To: Štěpán Němec; +Cc: meta
Štěpán Němec <stepnem@smrk.net> wrote:
> I also wonder about CAVEATS from lei-overview.pod:
>
> IMAP and NNTP client performance is poor on high-latency connections.
> It will hopefully be fixed in 2022.
>
> I think this needs an update (or removal, if the issue was indeed
> fixed).
Unfortunately not :< It's a PITA to deal with some of that stuff
and I'm still using offlineimap (as I've been for ~20 years, now?)
I'll try to take care of it in 2024 at latest...
^ permalink raw reply [relevance 71%]
* [PATCH 0/3] lei: stdin handling improvements
@ 2023-10-17 10:11 71% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2023-10-17 10:11 UTC (permalink / raw)
To: meta
I'm not sure about 3/3 since termios(3) stuff is mostly alien to
me. I'm very slowly expanding my horizons...
Eric Wong (3):
lei: consolidate stdin slurp, fix warnings
input_pipe: improve error handling
input_pipe: handle noncanonical TTY
lib/PublicInbox/InputPipe.pm | 79 ++++++++++++++++++++++++++++-------
lib/PublicInbox/LEI.pm | 13 ++++++
lib/PublicInbox/LeiInspect.pm | 12 +-----
lib/PublicInbox/LeiLcat.pm | 13 +-----
lib/PublicInbox/LeiQuery.pm | 14 ++-----
t/lei.t | 5 +++
6 files changed, 88 insertions(+), 48 deletions(-)
^ permalink raw reply [relevance 71%]
* [PATCH 18/30] t/lei-up: additional diagnostics for match failures
@ 2023-10-17 23:38 71% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2023-10-17 23:38 UTC (permalink / raw)
To: meta
I'm not sure why, but this test just failed for some odd reason
from `make check-run' on my Debian bullseye workstatation.
---
t/lei-up.t | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/t/lei-up.t b/t/lei-up.t
index baed6507..2d3afd82 100644
--- a/t/lei-up.t
+++ b/t/lei-up.t
@@ -18,11 +18,11 @@ test_lei(sub {
gunzip("$home/$x.mbox.gz" => \$uc, MultiStream => 1) or
xbail "gunzip $GunzipError";
ok(index($uc, $qp->body_raw) >= 0,
- "original mail in $x.mbox.gz");
+ "original mail in $x.mbox.gz") or diag $uc;
open my $fh, '<', "$home/$x" or xbail $!;
$uc = do { local $/; <$fh> } // xbail $!;
ok(index($uc, $qp->body_raw) >= 0,
- "original mail in uncompressed $x");
+ "original mail in uncompressed $x") or diag $uc;
}
lei_ok qw(ls-search);
$s = eml_load('t/utf8.eml')->as_string;
^ permalink raw reply related [relevance 71%]
* lei - dfn filters for net/* catching drivers/net/*
@ 2023-11-02 21:16 71% David Wei
2023-11-02 21:27 71% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: David Wei @ 2023-11-02 21:16 UTC (permalink / raw)
To: meta
Hi,
I have a problem with lei dfn filters. Here is my query:
lei q -o ~/Mail/overlay -I https://lore.kernel.org/all -t '(dfn:net/* OR dfn:drivers/net/ethernet/mellanox/mlx5/* OR dfn:drivers/net/ethernet/broadcom/bnxt/*) AND tc:netdev@vger.kernel.org AND rt:2.week.ago..'
I'm seeing patches that touch drivers/net/* whereas I only want to match
net/*.
I tried changing it to dfn:^net/* and dfn:b/net/* but neither is
working,
I also read the Xapian docs: https://xapian.org/docs/queryparser.html
but didn't see anything more than * wildcards.
Could you please advise on how I can limit my query to only net/*?
Thanks!
David
^ permalink raw reply [relevance 71%]
* Re: lei - dfn filters for net/* catching drivers/net/*
2023-11-02 21:16 71% lei - dfn filters for net/* catching drivers/net/* David Wei
@ 2023-11-02 21:27 71% ` Eric Wong
2023-11-03 18:29 71% ` David Wei
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2023-11-02 21:27 UTC (permalink / raw)
To: David Wei; +Cc: meta
David Wei <dw@davidwei.uk> wrote:
> Hi,
>
> I have a problem with lei dfn filters. Here is my query:
>
> lei q -o ~/Mail/overlay -I https://lore.kernel.org/all -t '(dfn:net/* OR dfn:drivers/net/ethernet/mellanox/mlx5/* OR dfn:drivers/net/ethernet/broadcom/bnxt/*) AND tc:netdev@vger.kernel.org AND rt:2.week.ago..'
>
> I'm seeing patches that touch drivers/net/* whereas I only want to match
> net/*.
>
> I tried changing it to dfn:^net/* and dfn:b/net/* but neither is
> working,
Right, ^ is a regexp thing and I don't think Xapian supports anything
like it.
> I also read the Xapian docs: https://xapian.org/docs/queryparser.html
> but didn't see anything more than * wildcards.
>
> Could you please advise on how I can limit my query to only net/*?
I'm not an expert in Xapian's parser, either, but I think `AND NOT'
is appropriate here. So something like:
dfn:net/* AND NOT dfn:drivers/net/*
Would be helpful to know if it works for you.
(having NOT only is very expensive and not allowed via the web interface,
but combining it a positive match should be fine)
^ permalink raw reply [relevance 71%]
* Re: lei - dfn filters for net/* catching drivers/net/*
2023-11-02 21:27 71% ` Eric Wong
@ 2023-11-03 18:29 71% ` David Wei
0 siblings, 0 replies; 200+ results
From: David Wei @ 2023-11-03 18:29 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
On 2023-11-02 14:27, Eric Wong wrote:
> David Wei <dw@davidwei.uk> wrote:
>> Hi,
>>
>> I have a problem with lei dfn filters. Here is my query:
>>
>> lei q -o ~/Mail/overlay -I https://lore.kernel.org/all -t '(dfn:net/* OR dfn:drivers/net/ethernet/mellanox/mlx5/* OR dfn:drivers/net/ethernet/broadcom/bnxt/*) AND tc:netdev@vger.kernel.org AND rt:2.week.ago..'
>>
>> I'm seeing patches that touch drivers/net/* whereas I only want to match
>> net/*.
>>
>> I tried changing it to dfn:^net/* and dfn:b/net/* but neither is
>> working,
>
> Right, ^ is a regexp thing and I don't think Xapian supports anything
> like it.
>
>> I also read the Xapian docs: https://xapian.org/docs/queryparser.html
>> but didn't see anything more than * wildcards.
>>
>> Could you please advise on how I can limit my query to only net/*?
>
> I'm not an expert in Xapian's parser, either, but I think `AND NOT'
> is appropriate here. So something like:
>
> dfn:net/* AND NOT dfn:drivers/net/*
>
> Would be helpful to know if it works for you.
> (having NOT only is very expensive and not allowed via the web interface,
> but combining it a positive match should be fine)
Thank you, using AND NOT does work. However, there are many more file
paths that partially match "net/", and excluding each one by one using
AND NOT is tedious.
I found that using b:b/net/* works very well to match patch diffs in
message bodies. This achieves my intended goal of matching only ^net/*.
^ permalink raw reply [relevance 71%]
* Re: lei interactive TUIs (ncurses/vim/emacs)
@ 2023-11-09 4:14 71% ` Kyle Meyer
0 siblings, 0 replies; 200+ results
From: Kyle Meyer @ 2023-11-09 4:14 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
[ I missed this (just saw it mentioned in a recent message). ]
Eric Wong writes:
> I've also noticed vim has scripting abilities (like Emacs?) and
> notmuch bundles a vim extension we can take inspiration from.
> Perhaps we could bundle vim and Emacs extensions for lei, too...
[...]
> So, any thoughts on this matter?
>
> Anybody willing to maintain an lei TUI for emacs?
As part of piem [1], I've done some work on an Emacs interface for lei,
in particular `lei q'. It's pretty bare-bones (been meaning to get back
to it...), but it's functional for searching and basic viewing.
[1]: https://git.kyleam.com/piem/about/
^ permalink raw reply [relevance 71%]
* Re: [Bug] lei: extra quotes inserted into query with AND/OR
@ 2023-11-12 0:10 71% ` Eric Wong
2023-11-12 8:23 71% ` Henrik Grimler
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2023-11-12 0:10 UTC (permalink / raw)
To: Henrik Grimler; +Cc: meta
Henrik Grimler <henrik@grimler.se> wrote:
> Hi,
>
> I recently found out about lei and installed it through archlinux's
> package manager and am trying out queries. When using AND/OR extra
> quotes are inserted in the curl command which messes it up, for
> example:
>
> $ lei q -I https://lore.kernel.org/all/ -o ~/mail/foo 'dfn:COPYING OR dfn:Makefile'
> # /home/grimler/.local/share/lei/store 0/0
> # /usr/bin/curl -Sf -s -d '' https://lore.kernel.org/all/?x=m&q=dfn%3A%22COPYING+OR+dfn%3AMakefile%22
> # 0 written to /home/grimler/mail/foo/ (0 matches)
>
> where it can be seen that it tries to search for 'dfn:"COPYING OR
> dfn:Makefile"', and no hits are returned since there is no file named
> "COPYING OR dfn:Makefile".
Don't use quotes unless you want a phrase search.
Basically, I wanted the CLI and WWW search to feel the same.
> Is this a known issue, or am I doing something wrong?
I think you're the second user to add unnecessary quotes;
is it learned behavior from another search tool?
In my experience, generic web search engines don't use quotes
outside of phrase search, either...
My primary mail experience is from using mairix, so lei borrows
heavily from it. But IIRC, mairix doesn't support phrase search.
Anyways, thanks for the note and any future comments you provide :>
^ permalink raw reply [relevance 71%]
* Re: [Bug] lei: extra quotes inserted into query with AND/OR
2023-11-12 0:10 71% ` Eric Wong
@ 2023-11-12 8:23 71% ` Henrik Grimler
0 siblings, 1 reply; 200+ results
From: Henrik Grimler @ 2023-11-12 8:23 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
Hi Eric,
On Sun, Nov 12, 2023 at 12:10:50AM +0000, Eric Wong wrote:
> Henrik Grimler <henrik@grimler.se> wrote:
> > Hi,
> >
> > I recently found out about lei and installed it through archlinux's
> > package manager and am trying out queries. When using AND/OR extra
> > quotes are inserted in the curl command which messes it up, for
> > example:
> >
> > $ lei q -I https://lore.kernel.org/all/ -o ~/mail/foo 'dfn:COPYING OR dfn:Makefile'
> > # /home/grimler/.local/share/lei/store 0/0
> > # /usr/bin/curl -Sf -s -d '' https://lore.kernel.org/all/?x=m&q=dfn%3A%22COPYING+OR+dfn%3AMakefile%22
> > # 0 written to /home/grimler/mail/foo/ (0 matches)
> >
> > where it can be seen that it tries to search for 'dfn:"COPYING OR
> > dfn:Makefile"', and no hits are returned since there is no file named
> > "COPYING OR dfn:Makefile".
>
> Don't use quotes unless you want a phrase search.
The quotes are added by lei (or some dependency) when query contains
space. Happens also if I search for a single file:
lei q -I https://lore.kernel.org/all/ -o ~/mail/foo ' dfn:COPYING'
which results in this curl cmd:
/usr/bin/curl -Sf -s -d '' https://lore.kernel.org/all/?x=m&q=+dfn%3A%22COPYING%22
where %22 then is "
Without spaces in the query all is well:
lei q -I https://lore.kernel.org/all/ -o ~/mail/foo 'dfn:COPYING'
which gives expected results
/usr/bin/curl -Sf -s -d '' https://lore.kernel.org/all/?x=m&q=dfn%3ACOPYING
So, maybe there is an issue in some perl dependency on archlinux, any
suggestion where I should start digging?
Best regards,
Henrik Grimler
> Basically, I wanted the CLI and WWW search to feel the same.
>
> > Is this a known issue, or am I doing something wrong?
>
> I think you're the second user to add unnecessary quotes;
> is it learned behavior from another search tool?
>
> In my experience, generic web search engines don't use quotes
> outside of phrase search, either...
>
> My primary mail experience is from using mairix, so lei borrows
> heavily from it. But IIRC, mairix doesn't support phrase search.
>
> Anyways, thanks for the note and any future comments you provide :>
^ permalink raw reply [relevance 71%]
* Re: [Bug] lei: extra quotes inserted into query with AND/OR
@ 2023-11-12 11:59 71% ` Henrik Grimler
2023-11-12 13:24 71% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Henrik Grimler @ 2023-11-12 11:59 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
Hi Eric,
On Sun, Nov 12, 2023 at 09:02:49AM +0000, Eric Wong wrote:
> Henrik Grimler <henrik@grimler.se> wrote:
> > Hi Eric,
> >
> > On Sun, Nov 12, 2023 at 12:10:50AM +0000, Eric Wong wrote:
> > > Henrik Grimler <henrik@grimler.se> wrote:
> > > > Hi,
> > > >
> > > > I recently found out about lei and installed it through archlinux's
> > > > package manager and am trying out queries. When using AND/OR extra
> > > > quotes are inserted in the curl command which messes it up, for
> > > > example:
> > > >
> > > > $ lei q -I https://lore.kernel.org/all/ -o ~/mail/foo 'dfn:COPYING OR dfn:Makefile'
> > > > # /home/grimler/.local/share/lei/store 0/0
> > > > # /usr/bin/curl -Sf -s -d '' https://lore.kernel.org/all/?x=m&q=dfn%3A%22COPYING+OR+dfn%3AMakefile%22
> > > > # 0 written to /home/grimler/mail/foo/ (0 matches)
> > > >
> > > > where it can be seen that it tries to search for 'dfn:"COPYING OR
> > > > dfn:Makefile"', and no hits are returned since there is no file named
> > > > "COPYING OR dfn:Makefile".
> > >
> > > Don't use quotes unless you want a phrase search.
> >
> > The quotes are added by lei (or some dependency) when query contains
> > space. Happens also if I search for a single file:
> > lei q -I https://lore.kernel.org/all/ -o ~/mail/foo ' dfn:COPYING'
> > which results in this curl cmd:
> > /usr/bin/curl -Sf -s -d '' https://lore.kernel.org/all/?x=m&q=+dfn%3A%22COPYING%22
> > where %22 then is "
>
> Right, spaces require quotes in sh and lei inserts quotes when
> it sees spaces assuming it's a phrase search. Most queries
> involving filenames don't have spaces, and your original query
> shouldn't have spaces. It's 3 separate args in @argv of
> `lei_q': [ "dfn:COPYING", "OR", "dfn:Makefile" ]
>
> In other words, no quotes or spaces are needed in your case at all:
>
> $ lei q dfn:COPYING OR dfn:Makefile
> (I've omitted the -I and -o args for brevity)
>
> Your original query only passes 1 arg due to single or double quotes
> handled in the shell (assuming POSIX-like sh or bash):
>
> $ lei q 'dfn:COPYING OR dfn:Makefile' # don't do this
> $ lei q "dfn:COPYING OR dfn:Makefile" # don't do this, either
>
> In both cases the `lei_q' subroutine would only see
> [ "dfn:COPYING OR dfn:Makefile" ] in its @argv.
>
> If you have odd cases where you really need spaces in a single
> token and maybe not phrase search, --stdin can probably get what
> you want more reliably:
>
> $ echo 'dfn:"some filename with spaces" AND something.else' | lei q --stdin
>
> Hope that helps.
Aha, I see, thanks for the explanation! Without the single quotes, and
after escaping parantheses, lei works as expected.
For the record, I read some old posts where query was '' quoted, and
thought it was the way to do it (for example
https://people.kernel.org/monsieuricon/lore-lei-part-1-getting-started
and https://josefbacik.github.io/kernel/2021/10/18/lei-and-b4.html)
Best regards,
Henrik Grimler
^ permalink raw reply [relevance 71%]
* Re: [Bug] lei: extra quotes inserted into query with AND/OR
2023-11-12 11:59 71% ` Henrik Grimler
@ 2023-11-12 13:24 71% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2023-11-12 13:24 UTC (permalink / raw)
To: Henrik Grimler; +Cc: meta
Henrik Grimler <henrik@grimler.se> wrote:
> Aha, I see, thanks for the explanation! Without the single quotes, and
> after escaping parantheses, lei works as expected.
Good to know.
> For the record, I read some old posts where query was '' quoted, and
> thought it was the way to do it (for example
> https://people.kernel.org/monsieuricon/lore-lei-part-1-getting-started
> and https://josefbacik.github.io/kernel/2021/10/18/lei-and-b4.html)
I'm not sure, I'm a little surprised that they do, but it may be
the preprocessing that the hacky approxidate wrapper (for
rt:1.month.ago..) somehow smooths things over a bit...
I'm sleepy now and head hurts from trying to figure out some
-cindex bugs :x, so maybe I missed some things.
I do recommend --stdin for more complex queries, and I just
posted a patch which should let it work better for terminals
(not just pipes/regular files):
https://public-inbox.org/meta/20231112131233.718614-1-e@80x24.org/
^ permalink raw reply [relevance 71%]
* [PATCH 0/4] lei convert: support idempotent v2 outputs
@ 2023-11-15 9:21 71% Eric Wong
2023-11-15 9:21 71% ` [PATCH 1/4] lei: fix idempotent STDERR redirect in workers Eric Wong
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2023-11-15 9:21 UTC (permalink / raw)
To: meta
This may make it easier for public-inbox admins to forcibly
inject missing messages from existing mbox*/maildir/IMAP/NNTP
archives.
1/4 was only needed to get 2/4 working, but 3/4 makes
it unnecessary with our current codebase (though we may
still need 1/4 in the future).
4/4 was noticed while working on 3/4.
Eric Wong (4):
lei: fix idempotent STDERR redirect in workers
lei convert: fix repeat and idempotent v2 output
lei: avoid extra fork for v2 outputs
lei q|up|convert: common finish_output to detect errors
lib/PublicInbox/LEI.pm | 2 +-
lib/PublicInbox/LeiConvert.pm | 9 ++++++---
lib/PublicInbox/LeiOverview.pm | 4 ++--
lib/PublicInbox/LeiToMail.pm | 33 +++++++++++++++++++++------------
lib/PublicInbox/LeiXSearch.pm | 13 +------------
lib/PublicInbox/V2Writable.pm | 1 -
t/lei-convert.t | 31 ++++++++++++++++++++++++++++++-
7 files changed, 61 insertions(+), 32 deletions(-)
^ permalink raw reply [relevance 71%]
* [PATCH 1/4] lei: fix idempotent STDERR redirect in workers
2023-11-15 9:21 71% [PATCH 0/4] lei convert: support idempotent v2 outputs Eric Wong
@ 2023-11-15 9:21 71% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2023-11-15 9:21 UTC (permalink / raw)
To: meta
This is needed to support forking from already-forked lei workers
and $lei->{2} is already STDERR.
Fixes: e015c3742f91 (lei: use autodie where appropriate, 2023-10-17)
---
lib/PublicInbox/LEI.pm | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 460aed40..8d235b37 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -581,7 +581,7 @@ sub _lei_atfork_child {
close($_) for (grep(defined, delete @$self{qw(0 1 2 sock)}));
delete $cfg->{-lei_store};
} else { # worker, Net::NNTP (Net::Cmd) uses STDERR directly
- open STDERR, '+>&', $self->{2};
+ open STDERR, '+>&='.fileno($self->{2}); # idempotent w/ fileno
STDERR->autoflush(1);
$self->{2} = \*STDERR;
POSIX::setpgid(0, $$) // die "setpgid(0, $$): $!";
^ permalink raw reply related [relevance 71%]
* lei up without creating xapian database
@ 2023-12-12 11:17 71% Aneesh Kumar K.V (IBM)
2023-12-12 11:41 71% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Aneesh Kumar K.V (IBM) @ 2023-12-12 11:17 UTC (permalink / raw)
To: meta
Hi,
I currently use notmuch to manage my emails, and I already have a xapian
index created for all the emails on my system. I want to switch to using
lei to download emails from lore.kernel.org based on query patterns, and
I want to index these emails with notmuch. However, I noticed that lei
also creates an index for these emails, which is not necessary for my
workflow. Is there a way to use lei to download emails to maildir from a
public-inbox server without creating the xapian index on the local disk?
-aneesh
^ permalink raw reply [relevance 71%]
* Re: lei up without creating xapian database
2023-12-12 11:17 71% lei up without creating xapian database Aneesh Kumar K.V (IBM)
@ 2023-12-12 11:41 71% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2023-12-12 11:41 UTC (permalink / raw)
To: Aneesh Kumar K.V (IBM); +Cc: meta
"Aneesh Kumar K.V (IBM)" <aneesh.kumar@kernel.org> wrote:
> I currently use notmuch to manage my emails, and I already have a xapian
> index created for all the emails on my system. I want to switch to using
> lei to download emails from lore.kernel.org based on query patterns, and
> I want to index these emails with notmuch. However, I noticed that lei
> also creates an index for these emails, which is not necessary for my
> workflow. Is there a way to use lei to download emails to maildir from a
> public-inbox server without creating the xapian index on the local disk?
Not yet, unfortunately. I have plans to make lei independent of
Xapian (SQLite will still be necessary), but it hasn't been a
priority this year, maybe next year...
The (hopefully) coming lei FUSE3 backend may be helpful for notmuch
users, too, since it'll allow storing compressed/deltafied/deduped
messages in git w/o eating millions of real inodes.
^ permalink raw reply [relevance 71%]
* [PATCH 05/14] lei inspect: drop unneeded strftime import
@ 2023-12-13 0:50 71% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2023-12-13 0:50 UTC (permalink / raw)
To: meta
`lei inspect' uses the `iso8601' sub from LeiOverview.
---
lib/PublicInbox/LeiInspect.pm | 1 -
1 file changed, 1 deletion(-)
diff --git a/lib/PublicInbox/LeiInspect.pm b/lib/PublicInbox/LeiInspect.pm
index 88d7949c..576ab2c7 100644
--- a/lib/PublicInbox/LeiInspect.pm
+++ b/lib/PublicInbox/LeiInspect.pm
@@ -12,7 +12,6 @@ use parent qw(PublicInbox::IPC);
use PublicInbox::Config;
use PublicInbox::MID qw(mids);
use PublicInbox::NetReader qw(imap_uri nntp_uri);
-use POSIX qw(strftime);
use PublicInbox::LeiOverview;
*iso8601 = \&PublicInbox::LeiOverview::iso8601;
^ permalink raw reply related [relevance 71%]
* [PATCH 0/2] lei bugfixes
@ 2023-12-16 11:13 71% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2023-12-16 11:13 UTC (permalink / raw)
To: meta
Eric Wong (2):
lei index: support +L: labels
lei: use ->child_error API properly
lib/PublicInbox/LEI.pm | 2 +-
lib/PublicInbox/LeiExportKw.pm | 4 ++--
lib/PublicInbox/LeiMirror.pm | 2 +-
lib/PublicInbox/LeiToMail.pm | 4 ++--
t/lei-index.t | 3 ++-
5 files changed, 8 insertions(+), 7 deletions(-)
^ permalink raw reply [relevance 71%]
* Re: [PATCH] lei: support reading MH for convert+import+index
@ 2023-12-16 16:15 71% ` Konstantin Ryabitsev
2023-12-16 18:17 71% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Konstantin Ryabitsev @ 2023-12-16 16:15 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
On Sat, Dec 16, 2023 at 01:09:32PM +0000, Eric Wong wrote:
> The MH format is widely-supported and used by various MUAs such
> as mutt and sylpheed, and a MH-like format is used by mlmmj for
> archives, as well. Locking implementations for writes are
> inconsistent, so this commit doesn't support writes, yet.
Nice, so eventually we should be able to specify the following instead of
faking out a maildir?
watch=mh:/var/spool/mlmmj/list.name/archive
> inotify|EVFILT_VNODE watches aren't supported, yet, either.
In the case of mlmmj it's sufficient to watch the
/var/spool/mlmmj/list.name/index file for updates, but I don't know how well
this lends itself to other implementations (I am not at all familiar with MH).
-K
^ permalink raw reply [relevance 71%]
* Re: [PATCH] lei: support reading MH for convert+import+index
2023-12-16 16:15 71% ` Konstantin Ryabitsev
@ 2023-12-16 18:17 71% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2023-12-16 18:17 UTC (permalink / raw)
To: Konstantin Ryabitsev; +Cc: meta
Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> Nice, so eventually we should be able to specify the following instead of
> faking out a maildir?
>
> watch=mh:/var/spool/mlmmj/list.name/archive
Yes, that's the plan.
> > inotify|EVFILT_VNODE watches aren't supported, yet, either.
>
> In the case of mlmmj it's sufficient to watch the
> /var/spool/mlmmj/list.name/index file for updates, but I don't know how well
> this lends itself to other implementations (I am not at all familiar with MH).
Just watching the directory itself is sufficient (like Maildir)
and will report new files. We just have to check /\A[0-9]+\z/
^ permalink raw reply [relevance 71%]
* [PATCH 0/3] lei NNTP + error handling fixes
@ 2024-01-10 11:18 71% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2024-01-10 11:18 UTC (permalink / raw)
To: meta
These apply to public-inbox-watch, too, actually. I also
just noticed 3/3 while testing something after 2/3.
Eric Wong (3):
net_reader: fix NNTP credential use
lei+net_reader: show NNTP message in more failures
lei_to_mail: show supported mbox formats on error
lib/PublicInbox/LeiLsMailSource.pm | 6 +++++-
lib/PublicInbox/LeiToMail.pm | 4 +++-
lib/PublicInbox/NetReader.pm | 24 +++++++++++++++---------
3 files changed, 23 insertions(+), 11 deletions(-)
^ permalink raw reply [relevance 71%]
* [PATCH 0/2] watch: add MH support + lei doc
@ 2024-01-30 6:31 71% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2024-01-30 6:31 UTC (permalink / raw)
To: meta
I'm not sure if I want to deal with supporting MH writes;
but reads should be mostly safe (I hope...)
Eric Wong (2):
watch: support incremental updates from MH
doc/lei-mail-formats: update MH read-only status
Documentation/lei-mail-formats.pod | 16 +++-
Documentation/public-inbox-watch.pod | 16 ++--
MANIFEST | 1 +
lib/PublicInbox/Watch.pm | 92 ++++++++++++++------
t/watch_maildir.t | 2 +-
t/watch_mh.t | 120 +++++++++++++++++++++++++++
6 files changed, 211 insertions(+), 36 deletions(-)
create mode 100644 t/watch_mh.t
^ permalink raw reply [relevance 71%]
* Re: lei up can't fetch new thread messages when searching by mid
@ 2024-02-09 17:35 71% ` Eric Wong
2024-02-12 11:11 71% ` Pratyush Yadav
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2024-02-09 17:35 UTC (permalink / raw)
To: Pratyush Yadav; +Cc: meta
Pratyush Yadav <me@yadavpratyush.com> wrote:
<snip> yeah, known problem, whole thread from last year...
https://public-inbox.org/meta/20230330112951.M493025@dcvr/T/
> Is there a way to properly subscribe to an email thread? I suppose I can
> do query the subject instead but that can also run into some corner
> cases like multiple threads with the same subject, or if someone changes
> the subject when replying to the thread.
The infrastructure exists for it, but hasn't been wired into
lei, yet:
https://public-inbox.org/meta/20230330112951.M493025@dcvr/
I also don't know if lore is running it, yet, but the
80x24.org/lore mirror is still alive... And I've still yet to
make a release. Hopefully coming soon, can't wrap my head
around coderepo <=> inbox mapping parts and also dealing with
extreme anxiety around making releases :<
^ permalink raw reply [relevance 71%]
* Re: lei up can't fetch new thread messages when searching by mid
2024-02-09 17:35 71% ` Eric Wong
@ 2024-02-12 11:11 71% ` Pratyush Yadav
0 siblings, 0 replies; 200+ results
From: Pratyush Yadav @ 2024-02-12 11:11 UTC (permalink / raw)
To: Eric Wong; +Cc: Pratyush Yadav, meta
Hi,
On Fri, Feb 09 2024, Eric Wong wrote:
> Pratyush Yadav <me@yadavpratyush.com> wrote:
>
> <snip> yeah, known problem, whole thread from last year...
>
> https://public-inbox.org/meta/20230330112951.M493025@dcvr/T/
>
>> Is there a way to properly subscribe to an email thread? I suppose I can
>> do query the subject instead but that can also run into some corner
>> cases like multiple threads with the same subject, or if someone changes
>> the subject when replying to the thread.
>
> The infrastructure exists for it, but hasn't been wired into
> lei, yet:
>
> https://public-inbox.org/meta/20230330112951.M493025@dcvr/
>
> I also don't know if lore is running it, yet, but the
> 80x24.org/lore mirror is still alive... And I've still yet to
> make a release. Hopefully coming soon, can't wrap my head
> around coderepo <=> inbox mapping parts and also dealing with
> extreme anxiety around making releases :<
I see. Thanks for the pointer. I will use the subject to query for now
and wait until this comes to lei :-)
--
Regards,
Pratyush Yadav
^ permalink raw reply [relevance 71%]
* Lei exception
@ 2024-03-13 15:35 71% Gonsolo
2024-03-13 19:20 71% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Gonsolo @ 2024-03-13 15:35 UTC (permalink / raw)
To: meta
Hi!
Since a few days I'm getting the following error when running "lei up --all":
10770 lei2mail 1 (nshard=3) 8b214638a3a05e3d9f2345a632a5eed0de7f9ab6:
Exception: Document 22720 not found at
/usr/share/perl5/PublicInbox/LeiSearch.pm line 68.
Is there anything I can do?
Best regards
--
g
^ permalink raw reply [relevance 71%]
* Re: Lei exception
2024-03-13 15:35 71% Lei exception Gonsolo
@ 2024-03-13 19:20 71% ` Eric Wong
[not found] ` <CANL0fFSMQ1YL1a8PEpU39pYQ7d6vmmndughvJVue=SWNYNdqGQ@mail.gmail.com>
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2024-03-13 19:20 UTC (permalink / raw)
To: Gonsolo; +Cc: meta
Gonsolo <gonsolo@gmail.com> wrote:
> Hi!
>
> Since a few days I'm getting the following error when running "lei up --all":
>
> 10770 lei2mail 1 (nshard=3) 8b214638a3a05e3d9f2345a632a5eed0de7f9ab6:
> Exception: Document 22720 not found at
> /usr/share/perl5/PublicInbox/LeiSearch.pm line 68.
>
> Is there anything I can do?
Which version of public-inbox is this? It looks like 1.9 based
on the line number[1].
Any info you can share about the queries you use? (https, local
public-inbox clones).
I've seen it a few times in the past, but IIRC couldn't reliably
reproduce it and haven't seen it in a while. I'm always running
latest <https://80x24.org/public-inbox.git>, though, and that
has numerous reliability improvements over 1.9.
In any case, there's no mail data loss from xsmsg_vmd, only
metadata (missed flags) and I think it could've been a data
synchronization problem that was fixed in public-inbox.git.
[1] Running "lei sucks" should show all relevant version info. The blob OID
for LeiSearch.pm would certainly help confirm that.
^ permalink raw reply [relevance 71%]
* Re: Lei exception
[not found] ` <CANL0fFSMQ1YL1a8PEpU39pYQ7d6vmmndughvJVue=SWNYNdqGQ@mail.gmail.com>
@ 2024-03-14 14:46 71% ` Eric Wong
2024-03-15 17:36 71% ` Gonsolo
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2024-03-14 14:46 UTC (permalink / raw)
To: Gonsolo; +Cc: meta
please keep meta@public-inbox.org in the Cc:
Gonsolo <gonsolo@gmail.com> wrote:
> > Which version of public-inbox is this? It looks like 1.9 based
> > on the line number[1].
>
> Package from Ubuntu/Debian.
>
> lei 1.9.0-1 (Ubuntu)
<snip>
> 936c275178dfc6908577487ce97d3a83c58c5449 PublicInbox/LeiSearch.pm
OK.
> > Any info you can share about the queries you use? (https, local
> > public-inbox clones).
>
> lei edit-search:
<snip>
> [lei "q"]
> include = https://lore.kernel.org/all/
> external = 1
> local = 1
> remote = 1
> threads = 1
OK, yeah. I seem to recall problems with http(s) externals
being racy and made more reliable with the use of OnDestroy
callbacks in public-inbox.git
> > I've seen it a few times in the past, but IIRC couldn't reliably
> > reproduce it and haven't seen it in a while. I'm always running
> > latest <https://80x24.org/public-inbox.git>, though, and that
> > has numerous reliability improvements over 1.9.
>
> Ok. Maybe I'm going to ping the Debian maintainer.
No, please don't ask Debian maintainers to package unreleased
versions. The non-lei feature: codesearch, is not ready to be
supported in a 2.0 release.
The only good news is 2.0 will probably be the last release
involving large data model additions/changes.
^ permalink raw reply [relevance 71%]
* Re: Lei exception
2024-03-14 14:46 71% ` Eric Wong
@ 2024-03-15 17:36 71% ` Gonsolo
0 siblings, 0 replies; 200+ results
From: Gonsolo @ 2024-03-15 17:36 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
> please keep meta@public-inbox.org in the Cc:
Ok.
> No, please don't ask Debian maintainers to package unreleased
> versions. The non-lei feature: codesearch, is not ready to be
> supported in a 2.0 release.
>
> The only good news is 2.0 will probably be the last release
> involving large data model additions/changes.
Ok. Thanks. I'll just wait for 2.0.
--
g
^ permalink raw reply [relevance 71%]
* v1.9.0 : `ls-search' is not an lei command
@ 2024-04-04 19:04 71% Josh Steadmon
2024-04-04 19:55 71% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Josh Steadmon @ 2024-04-04 19:04 UTC (permalink / raw)
To: meta
Hello all,
I recently had to reinstall my work machine, and after doing so I now
get an error when running `lei ls-search`:
`ls-search' is not an lei command
This happens with both version 1.9.0-1+build1 of the Debian "lei"
package, and with version 1.9.0 of the Nix "public-inbox" package.
Unfortunately I'm not sure which version I had installed prior to wiping
my machine.
Any advice is appreciated, thanks!
Josh
^ permalink raw reply [relevance 71%]
* Re: v1.9.0 : `ls-search' is not an lei command
2024-04-04 19:04 71% v1.9.0 : `ls-search' is not an lei command Josh Steadmon
@ 2024-04-04 19:55 71% ` Eric Wong
2024-04-09 19:59 71% ` Josh Steadmon
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2024-04-04 19:55 UTC (permalink / raw)
To: Josh Steadmon; +Cc: meta
Josh Steadmon <steadmon@google.com> wrote:
> Hello all,
>
> I recently had to reinstall my work machine, and after doing so I now
> get an error when running `lei ls-search`:
>
> `ls-search' is not an lei command
>
> This happens with both version 1.9.0-1+build1 of the Debian "lei"
> package, and with version 1.9.0 of the Nix "public-inbox" package.
Are you sure the `lei' in $PATH is pointed to the right
installation?
`lei sucks' should give diagnostic info (or similarly broken if
the lei-daemon is running on an uninstalled version).
`lei daemon-kill' should always work and rerunning any `lei'
should restart it.
Check for a LeiLsSearch.pm on your machine and see if it lines
up with your expected installation location (and there should be
a bunch of Lei*.pm siblings in the same directory).
^ permalink raw reply [relevance 71%]
* Re: v1.9.0 : `ls-search' is not an lei command
2024-04-04 19:55 71% ` Eric Wong
@ 2024-04-09 19:59 71% ` Josh Steadmon
0 siblings, 0 replies; 200+ results
From: Josh Steadmon @ 2024-04-09 19:59 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
On 2024.04.04 19:55, Eric Wong wrote:
> Josh Steadmon <steadmon@google.com> wrote:
> > Hello all,
> >
> > I recently had to reinstall my work machine, and after doing so I now
> > get an error when running `lei ls-search`:
> >
> > `ls-search' is not an lei command
> >
> > This happens with both version 1.9.0-1+build1 of the Debian "lei"
> > package, and with version 1.9.0 of the Nix "public-inbox" package.
>
> Are you sure the `lei' in $PATH is pointed to the right
> installation?
>
> `lei sucks' should give diagnostic info (or similarly broken if
> the lei-daemon is running on an uninstalled version).
Thanks for the pointer to `lei sucks'. It reported that lei was for some
reason trying to load packages from a prior attempt at installing
public-inbox from source, even after deleting the relevant directory.
I'm not sure how that happened, since I installed via apt and AFAICT
there's nothing in my environment that would tell Perl to look in that
location.
In any case, `apt-get purge lei` followed by a reinstall worked after
deleting the old source directory.
Thanks again!
^ permalink raw reply [relevance 71%]
* [PATCH] lei blob: fix attachment extraction for unimported||inflight
@ 2024-04-11 18:58 71% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2024-04-11 18:58 UTC (permalink / raw)
To: meta
Noticed while trying to make other reliability improvements to
lei...
---
lib/PublicInbox/LeiBlob.pm | 11 +++++++----
1 file changed, 7 insertions(+), 4 deletions(-)
diff --git a/lib/PublicInbox/LeiBlob.pm b/lib/PublicInbox/LeiBlob.pm
index 127cc81e..00697097 100644
--- a/lib/PublicInbox/LeiBlob.pm
+++ b/lib/PublicInbox/LeiBlob.pm
@@ -119,14 +119,17 @@ sub lei_blob {
} else {
open $rdr->{2}, '>', '/dev/null' or die "open: $!";
}
- my $cmd = [ 'git', '--git-dir='.$lei->ale->git->{git_dir},
- 'cat-file', 'blob', $blob ];
+ my $cmd = $lei->ale->git->cmd('cat-file', 'blob', $blob);
+ my $cerr;
if (defined $lei->{-attach_idx}) {
my $buf = run_qx($cmd, $lei->{env}, $rdr);
return extract_attach($lei, $blob, \$buf) unless $?;
+ $cerr = $?;
+ } else {
+ $rdr->{1} = $lei->{1}; # write directly to client
+ $cerr = run_wait($cmd, $lei->{env}, $rdr) or return;
}
- $rdr->{1} = $lei->{1};
- my $cerr = run_wait($cmd, $lei->{env}, $rdr) or return;
+ # fall back to unimported ('lei index') and inflight blobs
my $lms = $lei->lms;
my $bref = ($lms ? $lms->local_blob($blob, 1) : undef) // do {
my $sto = $lei->{sto} // $lei->_lei_store;
^ permalink raw reply related [relevance 71%]
* Re: lei-up doesn't output replies to matching thread
@ 2024-04-12 2:07 71% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2024-04-12 2:07 UTC (permalink / raw)
To: Josh Steadmon; +Cc: meta
Josh Steadmon <steadmon@google.com> wrote:
> Hello again,
>
> I'm having trouble where `lei up --all` is not outputting threaded
> replies, despite the fact that the saved search requests them. I noticed
> the problem on this Git thread:
I think this needs the notmuch-style subquery support I mentioned here:
https://public-inbox.org/meta/20230412201743.M20097@dcvr/T/#u
The external process part works nowadays (for new -cindex), but isn't
wired up to old lei and WWW code, yet... And the subquery code will
probably be stealing C++ from notmuch.
Slightly related, I just posted a patch to finish off a per-thread
subscription feature but it requires recent HTTP(S) endpoints if
using HTTP(S):
https://public-inbox.org/meta/20240412020103.2665237-1-e@80x24.org/
(I've been sidetracked badly this year dealing with offline things :<)
^ permalink raw reply [relevance 71%]
* Re: [PATCH] lei q: support --thread-id=$MSGID || -T $MSGID
@ 2024-04-12 8:03 71% ` Štěpán Němec
2024-04-12 9:43 71% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Štěpán Němec @ 2024-04-12 8:03 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
Typo squad alert!
On Fri, 12 Apr 2024 02:01:03 +0000
Eric Wong wrote:
> diff --git a/t/psgi_v2.t b/t/psgi_v2.t
> index 54faae9b..56a6ae8e 100644
> --- a/t/psgi_v2.t
> +++ b/t/psgi_v2.t
> @@ -101,6 +101,19 @@ EOM
> }
> };
>
> +my $test_lei_q_threadid = sub {
> + my ($u) = @_;
> + test_lei(sub {
> + lei_ok qw(q -f text --only), $u, qw(-T t@1 s:unrelated);
> + is $lei_out, '', 'no results on unlrelated thread';
^
s/unlrelated/unrelated/
Thanks,
Štěpán
^ permalink raw reply [relevance 71%]
* Re: [PATCH] lei q: support --thread-id=$MSGID || -T $MSGID
2024-04-12 8:03 71% ` Štěpán Němec
@ 2024-04-12 9:43 71% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2024-04-12 9:43 UTC (permalink / raw)
To: Štěpán Němec; +Cc: meta
Štěpán Němec <stepnem@smrk.net> wrote:
> Eric Wong wrote:
> > + is $lei_out, '', 'no results on unlrelated thread';
> ^
> s/unlrelated/unrelated/
Thanks, squashed:
diff --git a/t/psgi_v2.t b/t/psgi_v2.t
index 56a6ae8e..d5c328f0 100644
--- a/t/psgi_v2.t
+++ b/t/psgi_v2.t
@@ -105,7 +105,7 @@ my $test_lei_q_threadid = sub {
my ($u) = @_;
test_lei(sub {
lei_ok qw(q -f text --only), $u, qw(-T t@1 s:unrelated);
- is $lei_out, '', 'no results on unlrelated thread';
+ is $lei_out, '', 'no results on unrelated thread';
lei_ok qw(q -f text --only), $u, qw(-T t@1 dt:19931002000300..);
my @m = ($lei_out =~ m!^Message-ID: <([^>]+)>\n!gms);
is_deeply \@m, ['t@3'], 'got expected result from -T MSGID';
And pushed as commit 873066744d1b105da4cfafb1c7312ca11b579864
^ permalink raw reply related [relevance 71%]
* [PATCH 0/3] some lei fixes
@ 2024-04-12 18:04 71% Eric Wong
2024-04-12 18:04 71% ` [PATCH 3/3] lei: remove leftover debugging message Eric Wong
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2024-04-12 18:04 UTC (permalink / raw)
To: meta
Some trivial fixes I noticed while (still) working on getting
lei to use checkpoint to improve parallelism.
Eric Wong (3):
lei_remote: solver supports uncommitted blobs
io: avoid redundant waitpid in DESTROY
lei: remove leftover debugging message
lib/PublicInbox/IO.pm | 10 +++++-----
lib/PublicInbox/LEI.pm | 2 --
lib/PublicInbox/LeiRemote.pm | 13 ++++++++++---
3 files changed, 15 insertions(+), 10 deletions(-)
^ permalink raw reply [relevance 71%]
* [PATCH 3/3] lei: remove leftover debugging message
2024-04-12 18:04 71% [PATCH 0/3] some lei fixes Eric Wong
@ 2024-04-12 18:04 71% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2024-04-12 18:04 UTC (permalink / raw)
To: meta
Noticed while working on other things...
Fixes: 299aac294ec3 (lei: do label/keyword parsing in optparse, 2023-10-02)
---
lib/PublicInbox/LEI.pm | 2 --
1 file changed, 2 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 06592358..5b46686a 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -728,8 +728,6 @@ sub optparse ($$$) {
require PublicInbox::LeiInput;
my @err = PublicInbox::LeiInput::vmd_mod_extract($self, $argv);
return $self->fail(join("\n", @err)) if @err;
- } else {
- warn "proto $proto\n" if $cmd =~ /(add-watch|tag|index)/;
}
my $i = 0;
^ permalink raw reply related [relevance 71%]
* [PATCH 0/4] lei parallelism fixes
@ 2024-04-16 20:56 71% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2024-04-16 20:56 UTC (permalink / raw)
To: meta
This series allows `lei reindex' to run in parallel with other
lei commands which write to lei/store.
Eric Wong (4):
v2 + lei/store: always wait for fast-import checkpoint
lei: use ->barrier to commit to lei/store
lei/store: stop shard workers + cat-file on idle
lei: use async barrier for --import-before
lib/PublicInbox/EOFpipe.pm | 7 ++--
lib/PublicInbox/ExtSearchIdx.pm | 1 +
lib/PublicInbox/LEI.pm | 6 ++--
lib/PublicInbox/LeiInput.pm | 2 +-
lib/PublicInbox/LeiRefreshMailSync.pm | 2 +-
lib/PublicInbox/LeiRemote.pm | 4 +--
lib/PublicInbox/LeiStore.pm | 46 ++++++++++++++++-----------
lib/PublicInbox/LeiToMail.pm | 28 ++++++++++++----
lib/PublicInbox/LeiXSearch.pm | 17 ++++++----
lib/PublicInbox/V2Writable.pm | 8 +----
t/lei-store-fail.t | 2 +-
11 files changed, 74 insertions(+), 49 deletions(-)
^ permalink raw reply [relevance 71%]
* Sharing lei searches
@ 2024-04-18 6:24 71% Gonsolo
2024-04-18 10:26 71% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Gonsolo @ 2024-04-18 6:24 UTC (permalink / raw)
To: meta
Hi!
Is there an easy way to share a lei configuration for a different computer?
Right now I'm relying on the following (clumsy) workflow:
1. lei edit-search on computer A, copy and mail to myself
2. Dummy lei q on computer B.
3. lei edit-search on computer B, use email from 1.
4. Do this for all searches in lei ls-search
Is there a better way?
Thanks
--
g
^ permalink raw reply [relevance 71%]
* Re: Sharing lei searches
2024-04-18 6:24 71% Sharing lei searches Gonsolo
@ 2024-04-18 10:26 71% ` Eric Wong
2024-04-18 15:17 71% ` Gonsolo
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2024-04-18 10:26 UTC (permalink / raw)
To: Gonsolo; +Cc: meta
Gonsolo <gonsolo@gmail.com> wrote:
> Hi!
>
> Is there an easy way to share a lei configuration for a different computer?
> Right now I'm relying on the following (clumsy) workflow:
>
> 1. lei edit-search on computer A, copy and mail to myself
> 2. Dummy lei q on computer B.
> 3. lei edit-search on computer B, use email from 1.
> 4. Do this for all searches in lei ls-search
I do the above for some simple searches.
> Is there a better way?
You can also make `lei up --all' run as cronjob on a server
and output to IMAP folders; then use IMAP normally w/o needing
lei on the other machines.
^ permalink raw reply [relevance 71%]
* Re: Sharing lei searches
2024-04-18 10:26 71% ` Eric Wong
@ 2024-04-18 15:17 71% ` Gonsolo
0 siblings, 0 replies; 200+ results
From: Gonsolo @ 2024-04-18 15:17 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
Hi!
> I do the above for some simple searches.
Ok.
> > Is there a better way?
> You can also make `lei up --all' run as cronjob on a server
> and output to IMAP folders; then use IMAP normally w/o needing
> lei on the other machines.
Ok.
I thought about something like lei export-searches/import-searches but
I guess I will never suffer enough to try to implement it.
Thanks for the help!
--
g
^ permalink raw reply [relevance 71%]
* [PATCH 5/6] lei_saved_search: split "lei q --save" and "lei up" init paths
@ 2021-04-19 8:52 73% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-19 8:52 UTC (permalink / raw)
To: meta
They're more different than alike, and having two separate
methods seems less confusing to me.
---
lib/PublicInbox/LeiLsSearch.pm | 2 +-
lib/PublicInbox/LeiSavedSearch.pm | 79 ++++++++++++++++---------------
lib/PublicInbox/LeiUp.pm | 2 +-
3 files changed, 44 insertions(+), 39 deletions(-)
diff --git a/lib/PublicInbox/LeiLsSearch.pm b/lib/PublicInbox/LeiLsSearch.pm
index 2aa457c0..9ac4870f 100644
--- a/lib/PublicInbox/LeiLsSearch.pm
+++ b/lib/PublicInbox/LeiLsSearch.pm
@@ -29,7 +29,7 @@ sub do_ls_search_long {
my @x = sort(grep(/\A\Q$pfx/, PublicInbox::LeiSavedSearch::list($lei)));
while (my $x = shift @x) {
$ORS = '' if !scalar(@x);
- my $lss = PublicInbox::LeiSavedSearch->new($lei, $x) or next;
+ my $lss = PublicInbox::LeiSavedSearch->up($lei, $x) or next;
my $cfg = $lss->{-cfg};
my $ent = {
q => $cfg->get_all('lei.q'),
diff --git a/lib/PublicInbox/LeiSavedSearch.pm b/lib/PublicInbox/LeiSavedSearch.pm
index d3a32d36..948e4954 100644
--- a/lib/PublicInbox/LeiSavedSearch.pm
+++ b/lib/PublicInbox/LeiSavedSearch.pm
@@ -65,52 +65,57 @@ sub list {
} @$out
}
-sub new {
+sub up { # updating existing saved search via "lei up"
my ($cls, $lei, $dst) = @_;
+ my $f;
my $self = bless { ale => $lei->ale }, $cls;
- my $dir;
- if (defined $dst) { # updating existing saved search via "lei up"
- my $f;
- $dir = $dst;
- output2lssdir($self, $lei, \$dir, \$f) or
- return $lei->fail("--save was not used with $dst cwd=".
- $lei->rel2abs('.'));
- $self->{'-f'} = $f;
- } else { # new saved search "lei q --save"
- $dst = $lei->{ovv}->{dst};
- $dir = lss_dir_for($lei, \$dst);
- require File::Path;
- File::Path::make_path($dir); # raises on error
- $self->{-cfg} = {};
- my $f = $self->{'-f'} = "$dir/lei.saved-search";
- open my $fh, '>', $f or return $lei->fail("open $f: $!");
- my $sq_dst = PublicInbox::Config::squote_maybe($dst);
- my $q = $lei->{mset_opt}->{q_raw} // die 'BUG: {q_raw} missing';
- if (ref $q) {
- $q = join("\n", map { "\tq = ".cquote_val($_) } @$q);
- } else {
- $q = "\tq = ".cquote_val($q);
- }
- $dst = "$lei->{ovv}->{fmt}:$dst" if $dst !~ m!\Aimaps?://!i;
- print $fh <<EOM;
+ my $dir = $dst;
+ output2lssdir($self, $lei, \$dir, \$f) or
+ return $lei->fail("--save was not used with $dst cwd=".
+ $lei->rel2abs('.'));
+ $self->{-cfg} = PublicInbox::Config->git_config_dump($f);
+ $self->{-ovf} = "$dir/over.sqlite3";
+ $self->{'-f'} = $f;
+ $self->{lock_path} = "$self->{-f}.flock";
+ $self;
+}
+
+sub new { # new saved search "lei q --save"
+ my ($cls, $lei) = @_;
+ my $self = bless { ale => $lei->ale }, $cls;
+ my $dst = $lei->{ovv}->{dst};
+ my $dir = lss_dir_for($lei, \$dst);
+ require File::Path;
+ File::Path::make_path($dir); # raises on error
+ $self->{-cfg} = {};
+ my $f = $self->{'-f'} = "$dir/lei.saved-search";
+ open my $fh, '>', $f or return $lei->fail("open $f: $!");
+ my $sq_dst = PublicInbox::Config::squote_maybe($dst);
+ my $q = $lei->{mset_opt}->{q_raw} // die 'BUG: {q_raw} missing';
+ if (ref $q) {
+ $q = join("\n", map { "\tq = ".cquote_val($_) } @$q);
+ } else {
+ $q = "\tq = ".cquote_val($q);
+ }
+ $dst = "$lei->{ovv}->{fmt}:$dst" if $dst !~ m!\Aimaps?://!i;
+ print $fh <<EOM;
; to refresh with new results, run: lei up $sq_dst
[lei]
-$q
+ $q
[lei "q"]
output = $dst
EOM
- for my $k (ARRAY_FIELDS) {
- my $ary = $lei->{opt}->{$k} // next;
- for my $x (@$ary) {
- print $fh "\t$k = ".cquote_val($x)."\n";
- }
- }
- for my $k (BOOL_FIELDS) {
- my $val = $lei->{opt}->{$k} // next;
- print $fh "\t$k = ".($val ? 1 : 0)."\n";
+ for my $k (ARRAY_FIELDS) {
+ my $ary = $lei->{opt}->{$k} // next;
+ for my $x (@$ary) {
+ print $fh "\t$k = ".cquote_val($x)."\n";
}
- close($fh) or return $lei->fail("close $f: $!");
}
+ for my $k (BOOL_FIELDS) {
+ my $val = $lei->{opt}->{$k} // next;
+ print $fh "\t$k = ".($val ? 1 : 0)."\n";
+ }
+ close($fh) or return $lei->fail("close $f: $!");
$self->{lock_path} = "$self->{-f}.flock";
$self->{-ovf} = "$dir/over.sqlite3";
$self;
diff --git a/lib/PublicInbox/LeiUp.pm b/lib/PublicInbox/LeiUp.pm
index e4cbc825..23c5c606 100644
--- a/lib/PublicInbox/LeiUp.pm
+++ b/lib/PublicInbox/LeiUp.pm
@@ -11,7 +11,7 @@ use PublicInbox::LeiOverview;
sub lei_up {
my ($lei, $out) = @_;
$lei->{lse} = $lei->_lei_store(1)->search;
- my $lss = PublicInbox::LeiSavedSearch->new($lei, $out) or return;
+ my $lss = PublicInbox::LeiSavedSearch->up($lei, $out) or return;
my $mset_opt = $lei->{mset_opt} = { relevance => -2 };
$mset_opt->{limit} = $lei->{opt}->{limit} // 10000;
my $q = $mset_opt->{q_raw} = $lss->{-cfg}->{'lei.q'} //
^ permalink raw reply related [relevance 73%]
* [PATCH 5/5] lei edit-search: support relocating lei.q.output
@ 2021-05-01 6:21 75% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-05-01 6:21 UTC (permalink / raw)
To: meta
The contents of the old lei.q.output will not be removed,
but will be converted into the new one.
---
lib/PublicInbox/LeiConvert.pm | 5 ++-
lib/PublicInbox/LeiEditSearch.pm | 15 +++++++--
lib/PublicInbox/LeiSavedSearch.pm | 51 +++++++++++++++++++++++++++++++
script/lei | 7 +++++
4 files changed, 74 insertions(+), 4 deletions(-)
diff --git a/lib/PublicInbox/LeiConvert.pm b/lib/PublicInbox/LeiConvert.pm
index cefcaf65..5b27ec2d 100644
--- a/lib/PublicInbox/LeiConvert.pm
+++ b/lib/PublicInbox/LeiConvert.pm
@@ -37,8 +37,11 @@ sub process_inputs { # via wq_do
my ($self) = @_;
local $PublicInbox::DS::in_loop = 0; # force synchronous dwaitpid
$self->SUPER::process_inputs;
- delete $self->{lei}->{1};
+ my $lei = $self->{lei};
+ delete $lei->{1};
delete $self->{wcb}; # commit
+ my $nr = delete($lei->{-nr_write}) // 0;
+ $lei->err("# converted $nr messages") if $lei->{opt}->{verbose};
}
sub lei_convert { # the main "lei convert" method
diff --git a/lib/PublicInbox/LeiEditSearch.pm b/lib/PublicInbox/LeiEditSearch.pm
index fb36fdcd..30ac65bd 100644
--- a/lib/PublicInbox/LeiEditSearch.pm
+++ b/lib/PublicInbox/LeiEditSearch.pm
@@ -13,10 +13,19 @@ sub lei_edit_search {
my $lss = PublicInbox::LeiSavedSearch->up($lei, $out) or return;
my @cmd = (qw(git config --edit -f), $lss->{'-f'});
$lei->qerr("# spawning @cmd");
+ $lss->edit_begin($lei);
if ($lei->{oneshot}) {
- exec(@cmd) or die "exec @cmd: $!\n";
- } else {
- $lei->send_exec_cmd([], \@cmd, {});
+ require PublicInbox::Spawn;
+ waitpid(PublicInbox::Spawn::spawn(\@cmd), 0);
+ # non-fatal, editor could fail after successful write
+ $lei->child_error($?) if $?;
+ $lss->edit_done($lei);
+ } else { # run in script/lei foreground
+ require PublicInbox::PktOp;
+ my ($op_c, $op_p) = PublicInbox::PktOp->pair;
+ # $op_p will EOF when $EDITOR is done
+ $op_c->{ops} = { '' => [$lss->can('edit_done'), $lss, $lei] };
+ $lei->send_exec_cmd([ @$lei{qw(0 1 2)}, $op_p ], \@cmd, {});
}
}
diff --git a/lib/PublicInbox/LeiSavedSearch.pm b/lib/PublicInbox/LeiSavedSearch.pm
index 79125214..8177c98e 100644
--- a/lib/PublicInbox/LeiSavedSearch.pm
+++ b/lib/PublicInbox/LeiSavedSearch.pm
@@ -259,6 +259,57 @@ sub output2lssdir {
undef;
}
+sub edit_begin {
+ my ($self, $lei) = @_;
+ if (ref($self->{-cfg}->{'lei.q.output'})) {
+ delete $self->{-cfg}->{'lei.q.output'}; # invalid
+ $lei->err(<<EOM);
+$self->{-f} has multiple values of lei.q.output
+please remove redundant ones
+EOM
+ }
+ $lei->{-lss_for_edit} = $self;
+}
+
+sub edit_done {
+ my ($self, $lei) = @_;
+ my $cfg = PublicInbox::Config->git_config_dump($self->{'-f'});
+ my $new_out = $cfg->{'lei.q.output'} // '';
+ return $lei->fail(<<EOM) if ref $new_out;
+$self->{-f} has multiple values of lei.q.output
+please edit again
+EOM
+ return $lei->fail(<<EOM) if $new_out eq '';
+$self->{-f} needs lei.q.output
+please edit again
+EOM
+ my $old_out = $self->{-cfg}->{'lei.q.output'} // '';
+ return if $old_out eq $new_out;
+ my $old_path = $old_out;
+ my $new_path = $new_out;
+ s!$LOCAL_PFX!! for ($old_path, $new_path);
+ my $dir_old = lss_dir_for($lei, \$old_path, 1);
+ my $dir_new = lss_dir_for($lei, \$new_path);
+ return if $dir_new eq $dir_old; # no change, likely
+ return $lei->fail(<<EOM) if -e $dir_new;
+lei.q.output changed from `$old_out' to `$new_out'
+However, $dir_new exists
+EOM
+ # start the conversion asynchronously
+ my $old_sq = PublicInbox::Config::squote_maybe($old_out);
+ my $new_sq = PublicInbox::Config::squote_maybe($new_out);
+ $lei->puts("lei.q.output changed from $old_sq to $new_sq");
+ $lei->qerr("# lei convert $old_sq -o $new_sq");
+ my $v = !$lei->{opt}->{quiet};
+ $lei->{opt} = { output => $new_out, verbose => $v };
+ require PublicInbox::LeiConvert;
+ PublicInbox::LeiConvert::lei_convert($lei, $old_out);
+
+ $lei->fail(<<EOM) if -e $dir_old && !rename($dir_old, $dir_new);
+E: rename($dir_old, $dir_new) error: $!
+EOM
+}
+
no warnings 'once';
*nntp_url = \&cloneurl;
*base_url = \&PublicInbox::Inbox::base_url;
diff --git a/script/lei b/script/lei
index 90a93839..bec6b001 100755
--- a/script/lei
+++ b/script/lei
@@ -33,8 +33,15 @@ my $exec_cmd = sub {
push @rdr, shift(@old), $newfh;
}
my $do_exec = sub {
+ my @non_std; # ex. $op_p from lei_edit_search
while (my ($io, $newfh) = splice(@rdr, 0, 2)) {
+ my $old_io = !!$io;
open $io, '+<&', $newfh or die "open +<&=: $!";
+ push @non_std, $io unless $old_io;
+ }
+ if (@non_std) {
+ require Fcntl;
+ fcntl($_, Fcntl::F_SETFD(), 0) for @non_std;
}
my %env = map { split(/=/, $_, 2) } splice(@argv, $argc);
@ENV{keys %env} = values %env;
^ permalink raw reply related [relevance 75%]
* [PATCH 1/7] lei: always pass $lei to LeiAuth->op_merge
@ 2021-10-24 0:20 75% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-10-24 0:20 UTC (permalink / raw)
To: meta
This will make future developments easier.
---
lib/PublicInbox/LeiExportKw.pm | 2 +-
lib/PublicInbox/LeiForgetSearch.pm | 2 +-
lib/PublicInbox/LeiImport.pm | 2 +-
lib/PublicInbox/LeiLsMailSource.pm | 2 +-
lib/PublicInbox/LeiMailDiff.pm | 2 +-
lib/PublicInbox/LeiRefreshMailSync.pm | 2 +-
lib/PublicInbox/LeiTag.pm | 2 +-
lib/PublicInbox/LeiXSearch.pm | 2 +-
8 files changed, 8 insertions(+), 8 deletions(-)
diff --git a/lib/PublicInbox/LeiExportKw.pm b/lib/PublicInbox/LeiExportKw.pm
index 5be9e51f..756d0e9c 100644
--- a/lib/PublicInbox/LeiExportKw.pm
+++ b/lib/PublicInbox/LeiExportKw.pm
@@ -115,7 +115,7 @@ EOM
$self->{nwr}->{-skip_creat} = 1;
}
my $ops = {};
- $lei->{auth}->op_merge($ops, $self) if $lei->{auth};
+ $lei->{auth}->op_merge($ops, $self, $lei) if $lei->{auth};
(my $op_c, $ops) = $lei->workers_start($self, 1, $ops);
$lei->{wq1} = $self;
$lei->{-err_type} = 'non-fatal';
diff --git a/lib/PublicInbox/LeiForgetSearch.pm b/lib/PublicInbox/LeiForgetSearch.pm
index f353fe52..dfeb0293 100644
--- a/lib/PublicInbox/LeiForgetSearch.pm
+++ b/lib/PublicInbox/LeiForgetSearch.pm
@@ -46,7 +46,7 @@ sub lei_forget_search {
$self->prepare_inputs($lei, $self->{o_remote}) or return;
}
my $ops = {};
- $lei->{auth}->op_merge($ops, $self) if $lei->{auth};
+ $lei->{auth}->op_merge($ops, $self, $lei) if $lei->{auth};
(my $op_c, $ops) = $lei->workers_start($self, 1, $ops);
$lei->{wq1} = $self;
net_merge_all_done($self) unless $lei->{auth};
diff --git a/lib/PublicInbox/LeiImport.pm b/lib/PublicInbox/LeiImport.pm
index 2f8fd6c6..d8f39fdf 100644
--- a/lib/PublicInbox/LeiImport.pm
+++ b/lib/PublicInbox/LeiImport.pm
@@ -103,7 +103,7 @@ sub do_import_index ($$@) {
($lei->{opt}->{'new-only'} && (!$net || !$net->{imap_order})) and
warn "# --new-only is only for IMAP\n";
my $ops = {};
- $lei->{auth}->op_merge($ops, $self) if $lei->{auth};
+ $lei->{auth}->op_merge($ops, $self, $lei) if $lei->{auth};
$lei->{-eml_noisy} = 1;
(my $op_c, $ops) = $lei->workers_start($self, $j, $ops);
$lei->{wq1} = $self;
diff --git a/lib/PublicInbox/LeiLsMailSource.pm b/lib/PublicInbox/LeiLsMailSource.pm
index 7c3c0cda..5eb7032d 100644
--- a/lib/PublicInbox/LeiLsMailSource.pm
+++ b/lib/PublicInbox/LeiLsMailSource.pm
@@ -96,7 +96,7 @@ sub lei_ls_mail_source {
}
$lei->start_pager if $isatty;
my $ops = {};
- $lei->{auth}->op_merge($ops, $self);
+ $lei->{auth}->op_merge($ops, $self, $lei);
(my $op_c, $ops) = $lei->workers_start($self, 1, $ops);
$lei->{wq1} = $self;
$lei->{-err_type} = 'non-fatal';
diff --git a/lib/PublicInbox/LeiMailDiff.pm b/lib/PublicInbox/LeiMailDiff.pm
index a29ae225..4f3a4608 100644
--- a/lib/PublicInbox/LeiMailDiff.pm
+++ b/lib/PublicInbox/LeiMailDiff.pm
@@ -83,7 +83,7 @@ sub lei_mail_diff {
$lei->{opt}->{color} //= $isatty;
$lei->start_pager if $isatty;
my $ops = {};
- $lei->{auth}->op_merge($ops, $self) if $lei->{auth};
+ $lei->{auth}->op_merge($ops, $self, $lei) if $lei->{auth};
(my $op_c, $ops) = $lei->workers_start($self, 1, $ops);
$lei->{wq1} = $self;
$lei->{-err_type} = 'non-fatal';
diff --git a/lib/PublicInbox/LeiRefreshMailSync.pm b/lib/PublicInbox/LeiRefreshMailSync.pm
index 0cb9f3dc..f516f572 100644
--- a/lib/PublicInbox/LeiRefreshMailSync.pm
+++ b/lib/PublicInbox/LeiRefreshMailSync.pm
@@ -82,7 +82,7 @@ EOM
$lei->{opt}->{'mail-sync'} = 1; # for prepare_inputs
$self->prepare_inputs($lei, \@folders) or return;
my $ops = {};
- $lei->{auth}->op_merge($ops, $self) if $lei->{auth};
+ $lei->{auth}->op_merge($ops, $self, $lei) if $lei->{auth};
(my $op_c, $ops) = $lei->workers_start($self, 1, $ops);
$lei->{wq1} = $self;
$lei->{-err_type} = 'non-fatal';
diff --git a/lib/PublicInbox/LeiTag.pm b/lib/PublicInbox/LeiTag.pm
index 817b87f8..77654d1a 100644
--- a/lib/PublicInbox/LeiTag.pm
+++ b/lib/PublicInbox/LeiTag.pm
@@ -49,7 +49,7 @@ sub lei_tag { # the "lei tag" method
grep(defined, @$vmd_mod{qw(+kw +L -L -kw)}) or
return $lei->fail('no keywords or labels specified');
my $ops = {};
- $lei->{auth}->op_merge($ops, $self) if $lei->{auth};
+ $lei->{auth}->op_merge($ops, $self, $lei) if $lei->{auth};
(my $op_c, $ops) = $lei->workers_start($self, 1, $ops);
$lei->{wq1} = $self;
$lei->{-err_type} = 'non-fatal';
diff --git a/lib/PublicInbox/LeiXSearch.pm b/lib/PublicInbox/LeiXSearch.pm
index 119070a2..acc36897 100644
--- a/lib/PublicInbox/LeiXSearch.pm
+++ b/lib/PublicInbox/LeiXSearch.pm
@@ -548,7 +548,7 @@ sub do_query {
'child_error' => [ $lei ],
'incr_start_query' => [ $self, $lei ],
};
- $lei->{auth}->op_merge($ops, $l2m) if $l2m && $lei->{auth};
+ $lei->{auth}->op_merge($ops, $l2m, $lei) if $l2m && $lei->{auth};
my $end = $lei->pkt_op_pair;
$lei->{1}->autoflush(1);
$lei->start_pager if delete $lei->{need_pager};
^ permalink raw reply related [relevance 75%]
* [PATCH 5/5] lei: add "lei up" to complement "lei q --save"
2021-04-13 10:54 90% [PATCH 0/5] "lei q --save" + "lei up" Eric Wong
@ 2021-04-13 10:54 75% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-13 10:54 UTC (permalink / raw)
To: meta
The command isn't finalized, yet, but it's intended to update
an existing saved search.
---
MANIFEST | 1 +
lib/PublicInbox/LEI.pm | 2 ++
lib/PublicInbox/LeiQuery.pm | 2 +-
lib/PublicInbox/LeiSavedSearch.pm | 24 ++++++++++------
lib/PublicInbox/LeiToMail.pm | 12 ++++----
lib/PublicInbox/LeiUp.pm | 46 +++++++++++++++++++++++++++++++
t/lei-q-save.t | 17 ++++++++++--
7 files changed, 88 insertions(+), 16 deletions(-)
create mode 100644 lib/PublicInbox/LeiUp.pm
diff --git a/MANIFEST b/MANIFEST
index 20615abc..1b7d16ee 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -208,6 +208,7 @@ lib/PublicInbox/LeiStoreErr.pm
lib/PublicInbox/LeiSucks.pm
lib/PublicInbox/LeiTag.pm
lib/PublicInbox/LeiToMail.pm
+lib/PublicInbox/LeiUp.pm
lib/PublicInbox/LeiXSearch.pm
lib/PublicInbox/Linkify.pm
lib/PublicInbox/Listener.pm
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 7292d0f2..4b87c104 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -135,6 +135,8 @@ our %CMD = ( # sorted in order of importance/use:
sort|s=s reverse|r offset=i pretty jobs|j=s globoff|g augment|a
import-before! lock=s@ rsyncable alert=s@ mua=s verbose|v+), @c_opt,
opt_dash('limit|n=i', '[0-9]+') ],
+'up' => [ 'SEARCH_TERMS...', 'update saved search',
+ qw(jobs|j=s lock=s@ alert=s@ mua=s verbose|v+), @c_opt ],
'blob' => [ 'OID', 'show a git blob, reconstructing from mail if necessary',
qw(git-dir=s@ cwd! verbose|v+ mail! oid-a|A=s path-a|a=s path-b|b=s),
diff --git a/lib/PublicInbox/LeiQuery.pm b/lib/PublicInbox/LeiQuery.pm
index 8bca1020..7456f7f9 100644
--- a/lib/PublicInbox/LeiQuery.pm
+++ b/lib/PublicInbox/LeiQuery.pm
@@ -12,7 +12,7 @@ sub prep_ext { # externals_each callback
$lxs->prepare_external($loc) unless $exclude->{$loc};
}
-sub _start_query {
+sub _start_query { # used by "lei q" and "lei up"
my ($self) = @_;
PublicInbox::LeiOverview->new($self) or return;
my $opt = $self->{opt};
diff --git a/lib/PublicInbox/LeiSavedSearch.pm b/lib/PublicInbox/LeiSavedSearch.pm
index ab9f393b..815008fd 100644
--- a/lib/PublicInbox/LeiSavedSearch.pm
+++ b/lib/PublicInbox/LeiSavedSearch.pm
@@ -17,18 +17,12 @@ use PublicInbox::Hval qw(to_filename);
sub new {
my ($cls, $lei, $dir) = @_;
my $self = bless { ale => $lei->ale, -cfg => {} }, $cls;
- if (defined $dir) { # updating existing saved search
+ if (defined $dir) { # updating existing saved search via "lei up"
my $f = $self->{'-f'} = "$dir/lei.saved-search";
-f $f && -r _ or
return $lei->fail("$f non-existent or unreadable");
$self->{-cfg} = PublicInbox::Config::git_config_dump($f);
- my $q = $lei->{mset_opt}->{q_raw} = $self->{-cfg}->{'lei.q'} //
- return $lei->fail("lei.q unset in $f");
- my $lse = $lei->{lse} // die 'BUG: {lse} missing';
- $lei->{mset_opt}->{qstr} = ref($q) ?
- $lse->query_argv_to_string($lse->git, $q) :
- $lse->query_approxidate($lse->git, $q);
- } else { # new saved search
+ } else { # new saved search "lei q --save"
my $saved_dir = $lei->store_path . '/../saved-searches/';
my (@name) = ($lei->{ovv}->{dst} =~ m{([\w\-\.]+)/*\z});
push @name, to_filename($lei->{mset_opt}->{qstr});
@@ -42,6 +36,20 @@ sub new {
} else {
cfg_set($self, 'lei.q', $q);
}
+ my $fmt = $lei->{opt}->{'format'};
+ cfg_set($self, 'lei.q.format', $fmt) if defined $fmt;
+ cfg_set($self, 'lei.q.output', $lei->{opt}->{output});
+ for my $k (qw(only include exclude)) {
+ my $ary = $lei->{opt}->{$k} // next;
+ for my $x (@$ary) {
+ cfg_set($self, '--add', "lei.q.$k", $x);
+ }
+ }
+ for my $k (qw(external local remote import-remote
+ import-before threads)) {
+ my $val = $lei->{opt}->{$k} // next;
+ cfg_set($self, "lei.q.$k", $val);
+ }
}
bless $self->{-cfg}, 'PublicInbox::Config';
$self->{lock_path} = "$self->{-f}.flock";
diff --git a/lib/PublicInbox/LeiToMail.pm b/lib/PublicInbox/LeiToMail.pm
index bd2b714a..4ebaf8f3 100644
--- a/lib/PublicInbox/LeiToMail.pm
+++ b/lib/PublicInbox/LeiToMail.pm
@@ -349,11 +349,13 @@ sub new {
die "bad mail --format=$fmt\n";
}
$self->{dst} = $dst;
- my $dd_cls = 'PublicInbox::'.
- ($lei->{opt}->{save} ? 'LeiSavedSearch' : 'LeiDedupe');
- eval "require $dd_cls";
- die "$dd_cls: $@" if $@;
- $lei->{dedupe} = $dd_cls->new($lei);
+ $lei->{dedupe} = $lei->{lss} // do {
+ my $dd_cls = 'PublicInbox::'.
+ ($lei->{opt}->{save} ? 'LeiSavedSearch' : 'LeiDedupe');
+ eval "require $dd_cls";
+ die "$dd_cls: $@" if $@;
+ $dd_cls->new($lei);
+ };
$self;
}
diff --git a/lib/PublicInbox/LeiUp.pm b/lib/PublicInbox/LeiUp.pm
new file mode 100644
index 00000000..386a7566
--- /dev/null
+++ b/lib/PublicInbox/LeiUp.pm
@@ -0,0 +1,46 @@
+# Copyright (C) 2021 all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+
+# "lei up" - updates the result of "lei q --save"
+package PublicInbox::LeiUp;
+use strict;
+use v5.10.1;
+use PublicInbox::LeiSavedSearch;
+use PublicInbox::LeiOverview;
+
+sub lei_up {
+ my ($lei, $dir) = @_;
+ $lei->{lse} = $lei->_lei_store(1)->search;
+ my $lss = PublicInbox::LeiSavedSearch->new($lei, $dir) or return;
+ my $mset_opt = $lei->{mset_opt} = { relevance => -2 };
+ $mset_opt->{limit} = $lei->{opt}->{limit} // 10000;
+ my $q = $mset_opt->{q_raw} = $lss->{-cfg}->{'lei.q'} //
+ return $lei->fail("lei.q unset in $lss->{-f}");
+ my $lse = $lei->{lse} // die 'BUG: {lse} missing';
+ if (ref($q)) {
+ $mset_opt->{qstr} = $lse->query_argv_to_string($lse->git, $q);
+ } else {
+ $lse->query_approxidate($lse->git, $mset_opt->{qstr} = $q);
+ }
+ $lei->{opt}->{output} = $lss->{-cfg}->{'lei.q.output'} //
+ return $lei->fail("lei.q.output unset in $lss->{-f}");
+ $lei->{opt}->{'format'} //= $lss->{-cfg}->{'lei.q.format'}; # optional
+
+ my $to_avref = $lss->{-cfg}->can('_array');
+ for my $k (qw(only include exclude)) {
+ my $v = $lss->{-cfg}->{"lei.q.$k"} // next;
+ $lei->{opt}->{$k} = $to_avref->($v);
+ }
+ for my $k (qw(external local remote
+ import-remote import-before threads)) {
+ my $v = $lss->{-cfg}->{"lei.q.$k"} // next;
+ $lei->{opt}->{$k} = $v;
+ }
+ $lei->{lss} = $lss; # for LeiOverview->new
+ my $lxs = $lei->lxs_prepare or return;
+ $lei->ale->refresh_externals($lxs);
+ $lei->{opt}->{save} = 1;
+ $lei->_start_query;
+}
+
+1;
diff --git a/t/lei-q-save.t b/t/lei-q-save.t
index 56f7cb37..a6d579cf 100644
--- a/t/lei-q-save.t
+++ b/t/lei-q-save.t
@@ -2,11 +2,24 @@
# Copyright (C) 2021 all contributors <meta@public-inbox.org>
# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
use strict; use v5.10.1; use PublicInbox::TestCommon;
+my $doc1 = eml_load('t/plack-qp.eml');
+my $doc2 = eml_load('t/utf8.eml');
test_lei(sub {
my $home = $ENV{HOME};
- lei_ok qw(import t/plack-qp.eml);
- lei_ok qw(q --save z:0..), '-o', "$home/md/";
+ lei_ok qw(import -q t/plack-qp.eml);
+ lei_ok qw(q -q --save z:0..), '-o', "$home/md/";
+ my %before = map { $_ => 1 } glob("$home/md/cur/*");
+ is_deeply(eml_load((keys %before)[0]), $doc1, 'doc1 matches');
+
my @s = glob("$home/.local/share/lei/saved-searches/md-*");
is(scalar(@s), 1, 'got one saved search');
+
+ # ensure "lei up" works, since it compliments "lei q --save"
+ lei_ok qw(import t/utf8.eml);
+ lei_ok qw(up), $s[0];
+ my %after = map { $_ => 1 } glob("$home/md/cur/*");
+ is(delete $after{(keys(%before))[0]}, 1, 'original message kept');
+ is(scalar(keys %after), 1, 'one new message added');
+ is_deeply(eml_load((keys %after)[0]), $doc2, 'doc2 matches');
});
done_testing;
^ permalink raw reply related [relevance 75%]
* [PATCH 08/11] doc: lei: describe lei-daemon-kill and upgrades
@ 2021-10-19 9:33 80% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-10-19 9:33 UTC (permalink / raw)
To: meta
While we're at it, start dropping copyright years
since it seems acceptable to not have them:
https://www.linuxfoundation.org/blog/copyright-notices-in-open-source-software-projects/
Copyright years are also a noisy to update every year (maybe,
just maybe, we'll make it to 2022...)
---
Documentation/lei-daemon-kill.pod | 29 ++++++++++++++++++++++++++---
Documentation/lei-overview.pod | 8 +++++++-
2 files changed, 33 insertions(+), 4 deletions(-)
diff --git a/Documentation/lei-daemon-kill.pod b/Documentation/lei-daemon-kill.pod
index 7fb0fb25ad0e..48c237b8d3a6 100644
--- a/Documentation/lei-daemon-kill.pod
+++ b/Documentation/lei-daemon-kill.pod
@@ -8,7 +8,30 @@ lei daemon-kill [-SIGNAL | -s SIGNAL | --signal SIGNAL]
=head1 DESCRIPTION
-Send a signal to the lei-daemon. C<SIGNAL> defaults to C<TERM>.
+Send a signal to the L<lei-daemon(8)>. C<SIGNAL> defaults to C<TERM>.
+
+This command should be run after updating the code of lei.
+
+=head1 SIGNALS
+
+=over 8
+
+=item SIGTERM
+
+Send a graceful termination signal. L<lei-daemon(8)> will exit
+when all currently running lei commands are done. The listen
+socket will be released as soon as the signal is processed
+so another L<lei-daemon(8)> process can take its place.
+
+=item SIGKILL
+
+Kills L<lei-daemon(8)> immediately. Some worker processes may
+remain running after a short while after this takes effect.
+
+=back
+
+=for comment
+SIGQUIT and SIGINT currently do what SIGTERM do, may change...
=head1 CONTACT
@@ -19,10 +42,10 @@ L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/>
=head1 COPYRIGHT
-Copyright 2021 all contributors L<mailto:meta@public-inbox.org>
+Copyright all contributors L<mailto:meta@public-inbox.org>
License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>
=head1 SEE ALSO
-L<lei-daemon-pid(1)>
+L<lei-daemon-pid(1)>, L<lei-daemon(8)>
diff --git a/Documentation/lei-overview.pod b/Documentation/lei-overview.pod
index 40a7b0aadd04..bb2fe50f7cd9 100644
--- a/Documentation/lei-overview.pod
+++ b/Documentation/lei-overview.pod
@@ -131,6 +131,12 @@ C<contrib/completion/>. Contributions adding support for other
shells, as well as improvements to the existing Bash completion, are
welcome.
+=head1 UPGRADING
+
+Since lei runs as a daemon, L<lei-daemon-kill(1)> is required to kill
+the daemon so it can load new code. It will be restarted with the
+next invocation of any lei command.
+
=head1 CONTACT
Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org>
@@ -140,6 +146,6 @@ L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/>
=head1 COPYRIGHT
-Copyright 2021 all contributors L<mailto:meta@public-inbox.org>
+Copyright all contributors L<mailto:meta@public-inbox.org>
License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>
^ permalink raw reply related [relevance 80%]
* [PATCH 11/11] tests|lei: fixes for TEST_RUN_MODE=0 and lei oneshot
@ 2021-02-09 8:09 81% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-02-09 8:09 UTC (permalink / raw)
To: meta
DESTROY callbacks can clobber $?, so we must take care to
preserve it when exiting. We'll also try to make an effort to
ensure better DESTROY ordering and delete as much as possible
before x_it finishes.
We also need to load PublicInbox::Config when setting up
public inboxes.
---
lib/PublicInbox/IPC.pm | 2 ++
lib/PublicInbox/LEI.pm | 7 +++++--
lib/PublicInbox/TestCommon.pm | 3 ++-
t/lei-mirror.t | 2 +-
4 files changed, 10 insertions(+), 4 deletions(-)
diff --git a/lib/PublicInbox/IPC.pm b/lib/PublicInbox/IPC.pm
index 9331233a..efac4c4d 100644
--- a/lib/PublicInbox/IPC.pm
+++ b/lib/PublicInbox/IPC.pm
@@ -412,9 +412,11 @@ sub DESTROY {
my ($self) = @_;
my $ppid = $self->{-wq_ppid};
wq_kill($self) if $ppid && $ppid == $$;
+ my $err = $?;
wq_close($self);
wq_wait_old($self);
ipc_worker_stop($self);
+ $? = $err if $err;
}
sub detect_nproc () {
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 5f265087..dd831c54 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -336,8 +336,9 @@ sub x_it ($$) {
my $wq = delete $self->{$f} or next;
$wq->DESTROY;
}
- # cleanup anything that has tempfiles
- delete @$self{qw(ovv dedupe)};
+ # cleanup anything that has tempfiles or open file handles
+ %PATH2CFG = ();
+ delete @$self{qw(ovv dedupe sto cfg)};
if (my $signum = ($code & 127)) { # usually SIGPIPE (13)
$SIG{PIPE} = 'DEFAULT'; # $SIG{$signum} doesn't work
kill $signum, $$;
@@ -1072,8 +1073,10 @@ sub DESTROY {
my ($self) = @_;
$self->{1}->autoflush(1) if $self->{1};
stop_pager($self);
+ my $err = $?;
my $oneshot_pids = delete $self->{"pid.$self.$$"} or return;
waitpid($_, 0) for keys %$oneshot_pids;
+ $? = $err if $err; # preserve ->fail or ->x_it code
}
1;
diff --git a/lib/PublicInbox/TestCommon.pm b/lib/PublicInbox/TestCommon.pm
index 53f13437..63d45ac3 100644
--- a/lib/PublicInbox/TestCommon.pm
+++ b/lib/PublicInbox/TestCommon.pm
@@ -541,7 +541,6 @@ sub setup_public_inboxes () {
my $end = $lk->lock_for_scope;
return @ret if -f $stamp;
- require PublicInbox::InboxWritable;
local $ENV{PI_CONFIG} = $pi_config;
for my $V (1, 2) {
run_script([qw(-init), "-V$V", "t$V",
@@ -549,6 +548,8 @@ sub setup_public_inboxes () {
"$test_home/t$V", "http://example.com/t$V",
"t$V\@example.com" ]) or BAIL_OUT "init v$V";
}
+ require PublicInbox::Config;
+ require PublicInbox::InboxWritable;
my $cfg = PublicInbox::Config->new;
my $seen = 0;
$cfg->each_inbox(sub {
diff --git a/t/lei-mirror.t b/t/lei-mirror.t
index e3707979..cbe300da 100644
--- a/t/lei-mirror.t
+++ b/t/lei-mirror.t
@@ -27,7 +27,7 @@ test_lei({ tmpdir => $tmpdir }, sub {
like($lei_out, qr!\Q$t2\E!, 't2 added to ls-externals');
ok(!$lei->('add-external', $t2, '--mirror', "$http/t2/"),
- '--mirror fails if reused');
+ '--mirror fails if reused') or diag "$lei_err.$lei_out = $?";
ok($lei->('ls-external'), 'ls-external');
like($lei_out, qr!\Q$t2\E!, 'still in ls-externals');
^ permalink raw reply related [relevance 81%]
* [PATCH 08/21] lei: keep $lei around until workers are reaped
@ 2021-02-01 8:28 82% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-02-01 8:28 UTC (permalink / raw)
To: meta
This prevents SharedKV->DESTROY in lei-daemon from triggering
before DB handles are closed in lei2mail processes. The
{each_smsg_not_done} pipe was not sufficient in this case:
that gets closed at the end of the last git_to_mail callback
invocation.
---
lib/PublicInbox/IPC.pm | 10 +++++-----
lib/PublicInbox/LEI.pm | 2 +-
lib/PublicInbox/LeiXSearch.pm | 4 ++--
3 files changed, 8 insertions(+), 8 deletions(-)
diff --git a/lib/PublicInbox/IPC.pm b/lib/PublicInbox/IPC.pm
index 37f02944..689f32d0 100644
--- a/lib/PublicInbox/IPC.pm
+++ b/lib/PublicInbox/IPC.pm
@@ -137,7 +137,7 @@ sub ipc_worker_spawn {
}
sub ipc_worker_reap { # dwaitpid callback
- my ($self, $pid) = @_;
+ my ($args, $pid) = @_;
return if !$?;
# TERM(15) is our default exit signal, PIPE(13) is likely w/ pager
my $s = $? & 127;
@@ -145,9 +145,9 @@ sub ipc_worker_reap { # dwaitpid callback
}
sub wq_wait_old {
- my ($self) = @_;
+ my ($self, $args) = @_;
my $pids = delete $self->{"-wq_old_pids.$$"} or return;
- dwaitpid($_, \&ipc_worker_reap, $self) for @$pids;
+ dwaitpid($_, \&ipc_worker_reap, [$self, $args]) for @$pids;
}
# for base class, override in sub classes
@@ -164,7 +164,7 @@ sub ipc_atfork_child {
# idempotent, can be called regardless of whether worker is active or not
sub ipc_worker_stop {
- my ($self) = @_;
+ my ($self, $args) = @_;
my ($pid, $ppid) = delete(@$self{qw(-ipc_pid -ipc_ppid)});
my ($w_req, $r_res) = delete(@$self{qw(-ipc_req -ipc_res)});
if (!$w_req && !$r_res) {
@@ -175,7 +175,7 @@ sub ipc_worker_stop {
$w_req = $r_res = undef;
return if $$ != $ppid;
- dwaitpid($pid, \&ipc_worker_reap, $self);
+ dwaitpid($pid, \&ipc_worker_reap, [$self, $args]);
}
# use this if we have multiple readers reading curl or "pigz -dc"
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index c0b90451..4f7ed171 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -860,7 +860,7 @@ sub dclose {
if ($wq->wq_kill) {
$wq->wq_close
} elsif ($wq->wq_kill_old) {
- $wq->wq_wait_old;
+ $wq->wq_wait_old($self);
}
}
close(delete $self->{1}) if $self->{1}; # may reap_compress
diff --git a/lib/PublicInbox/LeiXSearch.pm b/lib/PublicInbox/LeiXSearch.pm
index de82a7da..b4a9b89d 100644
--- a/lib/PublicInbox/LeiXSearch.pm
+++ b/lib/PublicInbox/LeiXSearch.pm
@@ -283,7 +283,7 @@ sub query_done { # EOF callback
my $has_l2m = exists $lei->{l2m};
for my $f (qw(lxs l2m)) {
my $wq = delete $lei->{$f} or next;
- $wq->wq_wait_old;
+ $wq->wq_wait_old($lei);
}
$lei->{ovv}->ovv_end($lei);
if ($has_l2m) { # close() calls LeiToMail reap_compress
@@ -359,7 +359,7 @@ sub sigpipe_handler { # handles SIGPIPE from l2m/lxs workers
my ($lei) = @_;
my $lxs = delete $lei->{lxs};
if ($lxs && $lxs->wq_kill_old) { # is this the daemon?
- $lxs->wq_wait_old;
+ $lxs->wq_wait_old($lei);
}
close(delete $lei->{1}) if $lei->{1};
$lei->x_it(13);
^ permalink raw reply related [relevance 82%]
* [PATCH 3/9] doc lei blob: point to lei-q for shared options
2021-05-17 3:35 90% ` [PATCH 4/9] doc lei: resort lei-tag entries Kyle Meyer
@ 2021-05-17 3:35 82% ` Kyle Meyer
1 sibling, 0 replies; 200+ results
From: Kyle Meyer @ 2021-05-17 3:35 UTC (permalink / raw)
To: meta
lei-blob shares a good number of options with lei-q. Refer to lei-q's
manpage rather than repeating the descriptions.
Also, add lei-q to lei-blob's "see also" section.
Link: https://public-inbox.org/meta/YGFImEcWX1mCJJwv@dcvr/
---
Documentation/lei-blob.pod | 47 ++++++++++----------------------------
1 file changed, 12 insertions(+), 35 deletions(-)
diff --git a/Documentation/lei-blob.pod b/Documentation/lei-blob.pod
index 76918324..36c75d53 100644
--- a/Documentation/lei-blob.pod
+++ b/Documentation/lei-blob.pod
@@ -49,60 +49,37 @@ Provide pre-image pathname as a hint for reconstructing C<OID>.
Provide post-image pathname as a hint for reconstructing C<OID>.
-=for comment
-TODO: The below options are shared with lei-q. Any good approaches to
-not repeating the text?
+=item -v
-=item --[no-]remote
+=item --verbose
-Whether to include results requiring network access. When local
-externals are configured, C<--remote> must be explicitly passed to
-enable reporting of results from remote externals.
+Provide more feedback on stderr.
-=item --no-local
+=back
-Limit operations to those requiring network access.
+The following options are also supported and are described in
+L<lei-q(1)>.
-=item --no-external
+=over
-Don't include results from externals.
+=item --[no-]remote
-=item -I LOCATION
+=item --no-local
-=item --include=LOCATION
+=item --no-external
-Include specified external in search. This option may be given
-multiple times.
+=item -I LOCATION, --include=LOCATION
=item --exclude=LOCATION
-Exclude specified external from search. This option may be given
-multiple times.
-
=item --only=LOCATION
-Use only the specified external for search. This option may be given
-multiple times, in which case the search uses only the specified set.
-
=item --no-import-remote
-Disable the default behavior of memoizing remote messages into the
-local store.
-
-=item -v
-
-=item --verbose
-
-Provide more feedback on stderr.
-
=item --torsocks=auto|no|yes
=item --no-torsocks
-Whether to wrap L<git(1)> and L<curl(1)> commands with torsocks.
-
-Default: C<auto>
-
=back
=head1 CONTACT
@@ -121,4 +98,4 @@ License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>
=head1 SEE ALSO
-L<lei-add-external(1)>
+L<lei-add-external(1)>, L<lei-q(1)>
--
2.31.1
^ permalink raw reply related [relevance 82%]
* [PATCH 4/8] lei: use lei->lms in place of lse->lms in a few places
@ 2021-09-03 8:54 83% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-09-03 8:54 UTC (permalink / raw)
To: meta
We can golf out some code and refcounts this way.
---
lib/PublicInbox/LeiInspect.pm | 3 +--
lib/PublicInbox/LeiPruneMailSync.pm | 26 +++++++++++++-------------
2 files changed, 14 insertions(+), 15 deletions(-)
diff --git a/lib/PublicInbox/LeiInspect.pm b/lib/PublicInbox/LeiInspect.pm
index 2d2ff1a0..2ade17af 100644
--- a/lib/PublicInbox/LeiInspect.pm
+++ b/lib/PublicInbox/LeiInspect.pm
@@ -206,8 +206,7 @@ sub lei_inspect {
sub _complete_inspect {
my ($lei, @argv) = @_;
- my $sto = $lei->_lei_store or return;
- my $lms = $sto->search->lms or return;
+ my $lms = $lei->lms or return;
my $match_cb = $lei->complete_url_prepare(\@argv);
map { $match_cb->($_) } $lms->folders;
}
diff --git a/lib/PublicInbox/LeiPruneMailSync.pm b/lib/PublicInbox/LeiPruneMailSync.pm
index 79f3325d..98239a13 100644
--- a/lib/PublicInbox/LeiPruneMailSync.pm
+++ b/lib/PublicInbox/LeiPruneMailSync.pm
@@ -40,7 +40,7 @@ sub prune_imap { # lms->each_src callback
sub input_path_url { # overrides PublicInbox::LeiInput::input_path_url
my ($self, $input, @args) = @_;
- my $lms = $self->{-lms_ro} //= $self->{lse}->lms;
+ my $lms = $self->{-lms_ro} //= $self->{lei}->lms;
if ($input =~ /\Amaildir:(.+)/i) {
my $mdir = $1;
$lms->each_src($input, \&prune_mdir, $self, $mdir);
@@ -59,24 +59,24 @@ sub lei_prune_mail_sync {
my $sto = $lei->_lei_store or return $lei->fail(<<EOM);
lei/store uninitialized, see lei-import(1)
EOM
- my $lse = $sto->search;
- my $lms = $lse->lms or return $lei->fail(<<EOM);
-lei mail_sync uninitialized, see lei-import(1)
-EOM
- if (defined(my $all = $lei->{opt}->{all})) {
- $lms->group2folders($lei, $all, \@folders) or return;
+ if (my $lms = $lei->lms) {
+ if (defined(my $all = $lei->{opt}->{all})) {
+ $lms->group2folders($lei, $all, \@folders) or return;
+ } else {
+ my $err = $lms->arg2folder($lei, \@folders);
+ $lei->qerr(@{$err->{qerr}}) if $err->{qerr};
+ return $lei->fail($err->{fail}) if $err->{fail};
+ }
} else {
- my $err = $lms->arg2folder($lei, \@folders);
- $lei->qerr(@{$err->{qerr}}) if $err->{qerr};
- return $lei->fail($err->{fail}) if $err->{fail};
+ return $lei->fail(<<EOM);
+lei mail_sync.sqlite3 uninitialized, see lei-import(1)
+EOM
}
- delete $lms->{dbh};
$sto->write_prepare($lei);
- my $self = bless { lse => $lse }, __PACKAGE__;
+ my $self = bless {}, __PACKAGE__;
$lei->{opt}->{'mail-sync'} = 1; # for prepare_inputs
$self->prepare_inputs($lei, \@folders) or return;
my $j = $lei->{opt}->{jobs} || scalar(@{$self->{inputs}}) || 1;
- undef $lms; # for fork
my $ops = {};
$sto->write_prepare($lei);
$lei->{auth}->op_merge($ops, $self) if $lei->{auth};
^ permalink raw reply related [relevance 83%]
* [PATCH] t/lei: skip "lei q" tests on missing dependencies
@ 2021-02-04 2:10 83% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-02-04 2:10 UTC (permalink / raw)
To: meta
... for now. It's probably possible to just use send()
recv() without CMSG_* eventually.
---
t/lei.t | 15 +++++++++++----
1 file changed, 11 insertions(+), 4 deletions(-)
diff --git a/t/lei.t b/t/lei.t
index 01eed1da..a08a6d0d 100644
--- a/t/lei.t
+++ b/t/lei.t
@@ -9,6 +9,9 @@ use PublicInbox::Config;
use File::Path qw(rmtree);
use Fcntl qw(SEEK_SET);
use PublicInbox::Spawn qw(which);
+my $req_sendcmd = 'Socket::MsgHdr or Inline::C missing or unconfigured';
+undef($req_sendcmd) if PublicInbox::Spawn->can('send_cmd4');
+eval { require Socket::MsgHdr; undef $req_sendcmd };
require_git 2.6;
require_mods(qw(json DBD::SQLite Search::Xapian));
my $opt = { 1 => \(my $out = ''), 2 => \(my $err = '') };
@@ -165,6 +168,7 @@ my $test_external_remote = sub {
SKIP: {
my $nr = 5;
skip "$k unset", $nr if !$url;
+ skip $req_sendcmd, $nr if $req_sendcmd;
$curl or skip 'no curl', $nr;
which('torsocks') or skip 'no torsocks', $nr if $url =~ m!\.onion/!;
my $mid = '20140421094015.GA8962@dcvr.yhbt.net';
@@ -245,6 +249,8 @@ my $test_external = sub {
$lei->('ls-external');
unlike($out, qr!https://example\.com/ibx/!s, 'removed canonical URL');
+SKIP: {
+ skip $req_sendcmd, 52 if $req_sendcmd;
ok(!$lei->(qw(q s:prefix -o /dev/null -f maildir)), 'bad maildir');
like($err, qr!/dev/null exists and is not a directory!,
'error shown');
@@ -342,6 +348,7 @@ my $test_external = sub {
$url = $e{$k} if $url eq '1';
$test_external_remote->($url, $k);
}
+ }; # /SKIP
};
my $test_completion = sub {
@@ -372,11 +379,14 @@ my $test_completion = sub {
};
my $test_fail = sub {
+SKIP: {
+ skip $req_sendcmd, 3 if $req_sendcmd;
$lei->(qw(q --only http://127.0.0.1:99999/bogus/ t:m));
is($? >> 8, 3, 'got curl exit for bogus URL');
$lei->(qw(q --only http://127.0.0.1:99999/bogus/ t:m -o), "$home/junk");
is($? >> 8, 3, 'got curl exit for bogus URL with Maildir');
is($out, '', 'no output');
+}; # /SKIP
};
my $test_lei_common = sub {
@@ -397,10 +407,7 @@ if ($ENV{TEST_LEI_ONESHOT}) {
$test_lei_common->();
} else {
SKIP: { # real socket
- eval { require Socket::MsgHdr; 1 } // do {
- require PublicInbox::Spawn;
- PublicInbox::Spawn->can('send_cmd4');
- } // skip 'Socket::MsgHdr or Inline::C missing or unconfigured', 115;
+ skip $req_sendcmd, 115 if $req_sendcmd;
local $ENV{XDG_RUNTIME_DIR} = "$home/xdg_run";
my $sock = "$ENV{XDG_RUNTIME_DIR}/lei/5.seq.sock";
my $err_log = "$ENV{XDG_RUNTIME_DIR}/lei/errors.log";
^ permalink raw reply related [relevance 83%]
* [PATCH 2/5] lei q: reduce lei/store work for kw changes to stored mail
@ 2021-04-01 12:10 84% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-01 12:10 UTC (permalink / raw)
To: meta
We can tweak lse->kw_changed to return docids and reduce IPC
traffic and reduce work the lei/store worker needs to do.
---
lib/PublicInbox/LeiSearch.pm | 9 +++++----
lib/PublicInbox/LeiStore.pm | 8 ++++----
lib/PublicInbox/LeiToMail.pm | 6 +++---
3 files changed, 12 insertions(+), 11 deletions(-)
diff --git a/lib/PublicInbox/LeiSearch.pm b/lib/PublicInbox/LeiSearch.pm
index 07d570ec..69ba8303 100644
--- a/lib/PublicInbox/LeiSearch.pm
+++ b/lib/PublicInbox/LeiSearch.pm
@@ -100,10 +100,11 @@ sub xoids_for {
# returns true if $eml is indexed by lei/store and keywords don't match
sub kw_changed {
- my ($self, $eml, $new_kw_sorted) = @_;
- my $xoids = xoids_for($self, $eml, 1) // return;
- my ($num) = values %$xoids;
- my @cur_kw = msg_keywords($self, $num);
+ my ($self, $eml, $new_kw_sorted, $docids) = @_;
+ my $xoids = xoids_for($self, $eml) // return;
+ $docids //= [];
+ @$docids = sort { $a <=> $b } values %$xoids;
+ my @cur_kw = msg_keywords($self, $docids->[0]);
join("\0", @$new_kw_sorted) eq join("\0", @cur_kw) ? 0 : 1;
}
diff --git a/lib/PublicInbox/LeiStore.pm b/lib/PublicInbox/LeiStore.pm
index b76af4d3..48ab1d76 100644
--- a/lib/PublicInbox/LeiStore.pm
+++ b/lib/PublicInbox/LeiStore.pm
@@ -131,13 +131,13 @@ sub _docids_for ($$) {
}
sub set_eml_vmd {
- my ($self, $eml, $vmd) = @_;
+ my ($self, $eml, $vmd, $docids) = @_;
my $eidx = eidx_init($self);
- my @docids = _docids_for($self, $eml);
- for my $docid (@docids) {
+ $docids //= [ _docids_for($self, $eml) ];
+ for my $docid (@$docids) {
$eidx->idx_shard($docid)->ipc_do('set_vmd', $docid, $vmd);
}
- \@docids;
+ $docids;
}
sub add_eml_vmd {
diff --git a/lib/PublicInbox/LeiToMail.pm b/lib/PublicInbox/LeiToMail.pm
index da633da4..0364d8ef 100644
--- a/lib/PublicInbox/LeiToMail.pm
+++ b/lib/PublicInbox/LeiToMail.pm
@@ -228,10 +228,10 @@ sub _mbox_write_cb ($$) {
sub update_kw_maybe ($$$$) {
my ($lei, $lse, $eml, $kw) = @_;
return unless $lse;
- my $lse_oids = $lse->kw_changed($eml, $kw);
+ my $c = $lse->kw_changed($eml, $kw, my $docids = []);
my $vmd = { kw => $kw };
- if ($lse_oids) { # already in lei/store
- $lei->{sto}->ipc_do('set_eml', $eml, $vmd);
+ if (scalar @$docids) { # already in lei/store
+ $lei->{sto}->ipc_do('set_eml_vmd', undef, $vmd, $docids) if $c;
} elsif (my $xoids = $lei->{ale}->xoids_for($eml)) {
# it's in an external, only set kw, here
$lei->{sto}->ipc_do('set_xvmd', $xoids, $eml, $vmd);
^ permalink raw reply related [relevance 84%]
* [PATCH 05/11] lei import: move check_input_format to lei
@ 2021-02-17 10:07 85% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-02-17 10:07 UTC (permalink / raw)
To: meta
We'll be supporting "lei convert" in a future change; so it
makes sense to share a common internal API for common error
messages.
---
lib/PublicInbox/LEI.pm | 14 ++++++++++++++
lib/PublicInbox/LeiImport.pm | 17 ++---------------
2 files changed, 16 insertions(+), 15 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 12deedd8..1fa9f751 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -391,6 +391,20 @@ sub fail ($$;$) {
undef;
}
+sub check_input_format ($;$) {
+ my ($self, $files) = @_;
+ my $fmt = $self->{opt}->{'format'};
+ if (!$fmt) {
+ my $err = $files ? "regular file(s):\n@$files" : '--stdin';
+ return fail($self, "--format unset for $err");
+ }
+ return 1 if $fmt eq 'eml';
+ # XXX: should this handle {gz,bz2,xz}? that's currently in LeiToMail
+ require PublicInbox::MboxReader;
+ PublicInbox::MboxReader->can($fmt) ||
+ fail($self, "--format=$fmt unrecognized");
+}
+
sub out ($;@) {
my $self = shift;
return if print { $self->{1} // return } @_; # likely
diff --git a/lib/PublicInbox/LeiImport.pm b/lib/PublicInbox/LeiImport.pm
index b25d7e97..32f3a467 100644
--- a/lib/PublicInbox/LeiImport.pm
+++ b/lib/PublicInbox/LeiImport.pm
@@ -29,19 +29,6 @@ sub import_done { # EOF callback for main daemon
$imp->wq_wait_old(\&import_done_wait, $lei);
}
-sub check_fmt ($;$) {
- my ($lei, $f) = @_;
- my $fmt = $lei->{opt}->{'format'};
- if (!$fmt) {
- my $err = $f ? "regular file(s):\n@$f" : '--stdin';
- return $lei->fail("--format unset for $err");
- }
- return 1 if $fmt eq 'eml';
- require PublicInbox::MboxReader;
- PublicInbox::MboxReader->can($fmt) ||
- $lei->fail( "--format=$fmt unrecognized\n");
-}
-
sub do_import {
my ($lei) = @_;
my $ops = {
@@ -82,7 +69,7 @@ sub call { # the main "lei import" method
if ($lei->{opt}->{stdin}) {
@argv and return
$lei->fail("--stdin and locations (@argv) do not mix");
- check_fmt($lei) or return;
+ $lei->check_input_format or return;
$self->{0} = $lei->{0};
} else {
my @f;
@@ -95,7 +82,7 @@ sub call { # the main "lei import" method
$lei->{nrd}->add_url($x);
}
}
- if (@f) { check_fmt($lei, \@f) or return }
+ if (@f) { $lei->check_input_format(\@f) or return }
if ($lei->{nrd} && (my @err = $lei->{nrd}->errors)) {
return $lei->fail(@err);
}
^ permalink raw reply related [relevance 85%]
* [PATCH 8/9] lei blob: run cat_blob on lei/store for pending blobs
@ 2023-10-11 7:20 86% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2023-10-11 7:20 UTC (permalink / raw)
To: meta
This can probably be made asynchronous in the future via
PublicInbox::InputPipe, but it's good enough for testing.
---
lib/PublicInbox/LeiBlob.pm | 16 ++++++++++------
lib/PublicInbox/LeiStore.pm | 5 +++++
2 files changed, 15 insertions(+), 6 deletions(-)
diff --git a/lib/PublicInbox/LeiBlob.pm b/lib/PublicInbox/LeiBlob.pm
index 8df83b1d..d069d4a8 100644
--- a/lib/PublicInbox/LeiBlob.pm
+++ b/lib/PublicInbox/LeiBlob.pm
@@ -9,6 +9,7 @@ use v5.10.1;
use parent qw(PublicInbox::IPC);
use PublicInbox::Spawn qw(run_wait popen_rd which);
use PublicInbox::DS;
+use PublicInbox::Eml;
sub get_git_dir ($$) {
my ($lei, $d) = @_;
@@ -121,18 +122,21 @@ sub lei_blob {
'cat-file', 'blob', $blob ];
if (defined $lei->{-attach_idx}) {
my $fh = popen_rd($cmd, $lei->{env}, $rdr);
- require PublicInbox::Eml;
my $buf = do { local $/; <$fh> };
return extract_attach($lei, $blob, \$buf) if close($fh);
}
$rdr->{1} = $lei->{1};
my $cerr = run_wait($cmd, $lei->{env}, $rdr) or return;
my $lms = $lei->lms;
- if (my $bref = $lms ? $lms->local_blob($blob, 1) : undef) {
- defined($lei->{-attach_idx}) and
- return extract_attach($lei, $blob, $bref);
- return $lei->out($$bref);
- } elsif ($opt->{mail}) {
+ my $bref = ($lms ? $lms->local_blob($blob, 1) : undef) // do {
+ my $sto = $lei->{sto} // $lei->_lei_store;
+ $sto && $sto->{-wq_s1} ? $sto->wq_do('cat_blob', $blob)
+ : undef;
+ };
+ $bref and return $lei->{-attach_idx} ?
+ extract_attach($lei, $blob, $bref) :
+ $lei->out($$bref);
+ if ($opt->{mail}) {
my $eh = $rdr->{2};
seek($eh, 0, 0);
return $lei->child_error($cerr, do { local $/; <$eh> });
diff --git a/lib/PublicInbox/LeiStore.pm b/lib/PublicInbox/LeiStore.pm
index e19ec88e..9c07af14 100644
--- a/lib/PublicInbox/LeiStore.pm
+++ b/lib/PublicInbox/LeiStore.pm
@@ -108,6 +108,11 @@ sub search {
PublicInbox::LeiSearch->new($_[0]->{priv_eidx}->{topdir});
}
+sub cat_blob {
+ my ($self, $oid) = @_;
+ $self->{im} ? $self->{im}->cat_blob($oid) : undef;
+}
+
# follows the stderr file
sub _tail_err {
my ($self) = @_;
^ permalink raw reply related [relevance 86%]
* lei: incorrect quoting on saved searches (was Re: lore+lei: getting started)
[not found] ` <CAL_JsqJBh1O3H2-P07AHzVq0x89BoP_N6P=rT5up6=3QyF_B0Q@mail.gmail.com>
@ 2021-11-08 20:22 86% ` Konstantin Ryabitsev
2021-11-08 20:53 90% ` Rob Herring
2021-11-08 20:49 90% ` Eric Wong
0 siblings, 2 replies; 200+ results
From: Konstantin Ryabitsev @ 2021-11-08 20:22 UTC (permalink / raw)
To: Rob Herring; +Cc: meta
On Mon, Nov 08, 2021 at 01:49:07PM -0600, Rob Herring wrote:
Moving this to meta.
> > lei q -I https://lore.kernel.org/all/ -o ~/Mail/floppy \
> > --threads --dedupe=mid \
> > '(dfn:drivers/block/floppy.c OR dfhh:floppy_* OR s:floppy \
> > OR ((nq:bug OR nq:regression) AND nq:floppy)) \
> > AND rt:1.month.ago..'
>
> I tried a similar one which I had working as a bookmark:
>
> $ lei q -I https://lore.kernel.org/all/ -o ~/Mail/my-patches
> --threads --dedupe=mid '(dfn:drivers OR dfn:arch OR dfn:Documentation
> OR dfn:include OR dfn:scripts) AND f:robh@kernel.org'
> # /home/rob/.local/share/lei/store 0/0
> # /usr/bin/curl -Sf -s -d ''
> https://lore.kernel.org/all/?x=m&t=1&q=(dfn%3A%22drivers+OR+dfn%3Aarch+OR+dfn%3ADocumentation+OR+dfn%3Ainclude+OR+dfn%3Ascripts)+AND+f%3Arobh%40kernel.org%22
> # 0 written to /home/rob/Mail/my-patches/ (0 matches)
It's true, I get the same thing if I omit "AND rt:" at the end.
$ lei q -I https://lore.kernel.org/all/ -o ~/work/temp/lei/robh-patches --threads --dedupe=mid '(dfn:drivers OR dfn:arch OR dfn:Documentation OR dfn:include OR dfn:scripts) AND f:robh@kernel.org'
# /home/user/.local/share/lei/store 0/0
# /usr/bin/curl -Sf -s -d '' https://lore.kernel.org/all/?x=m&t=1&q=(dfn%3A%22drivers+OR+dfn%3Aarch+OR+dfn%3ADocumentation+OR+dfn%3Ainclude+OR+dfn%3Ascripts)+AND+f%3Arobh%40kernel.org%22
# 0 written to /home/user/work/temp/lei/robh-patches/ (0 matches)
$ lei forget-search ~/work/temp/lei/robh-patches
$ lei q -I https://lore.kernel.org/all/ -o ~/work/temp/lei/robh-patches --threads --dedupe=mid '(dfn:drivers OR dfn:arch OR dfn:Documentation OR dfn:include OR dfn:scripts) AND f:robh@kernel.org AND rt:1.month.ago..'
# /usr/bin/curl -Sf -s -d '' https://lore.kernel.org/all/?x=m&t=1&q=(dfn%3Adrivers+OR+dfn%3Aarch+OR+dfn%3ADocumentation+OR+dfn%3Ainclude+OR+dfn%3Ascripts)+AND+f%3Arobh%40kernel.org+AND+rt%3A1633724105..
# /home/user/.local/share/lei/store 13/13
# https://lore.kernel.org/all/ 65/?
# https://lore.kernel.org/all/ 75/75
# 45 written to /home/user/work/temp/lei/robh-patches/ (88 matches)
> It seems there is some problem in quoting. Notice the '%22' that's
> inserted in the url.
Deferring to Eric here.
> Also, the above query is a bit of a work-around as what I really want
> is just all patches from me. I haven't been able to get something to
> work. I've tried things like 'dfn:*' or 'dfn:/' or 'dfn:b/'.
I think 's:patch AND nq:diff' is a good option here.
-K
^ permalink raw reply [relevance 86%]
* [PATCH] doc: document --jobs for `lei q' and `lei up'
@ 2022-09-09 18:00 86% ` Eric Wong
2022-09-09 20:35 71% ` [PATCH] lei: add diagnostics for IMAP writer failures Ricardo Ribalda
1 sibling, 0 replies; 200+ results
From: Eric Wong @ 2022-09-09 18:00 UTC (permalink / raw)
To: meta; +Cc: Ricardo Ribalda
Eric Wong <e@80x24.org> wrote:
> There's also another patch coming to document the `--jobs|-j' CLI
> switch for `lei up' and `lei q', but trying `-j ,1' may help you
> if it's parallelism. Note the comma before `1', it accepts
> `-j $Q,$W' since $Q is the number of query processes and $W is
> the number of LeiToMail writers.
-------8<-----
From: Eric Wong <e@80x24.org>
Subject: [PATCH] doc: document --jobs for `lei q' and `lei up'
These may be helpful for users on slow disks or limited IMAP
connections.
---
Documentation/lei-q.pod | 17 +++++++++++++++++
Documentation/lei-up.pod | 4 +++-
2 files changed, 20 insertions(+), 1 deletion(-)
diff --git a/Documentation/lei-q.pod b/Documentation/lei-q.pod
index 2f0c3bc6..8134223e 100644
--- a/Documentation/lei-q.pod
+++ b/Documentation/lei-q.pod
@@ -124,6 +124,23 @@ of the same thread.
TODO: Warning: this flag may become persistent and saved in
lei/store unless an MUA unflags it! (Behavior undecided)
+=item --jobs=QUERY_WORKERS[,WRITE_WORKERS]
+=item --jobs=,WRITE_WORKERS
+
+=item -j QUERY_WORKERS[,WRITE_WORKERS]
+=item -j ,WRITE_WORKERS
+
+Set the number of query and write worker processes for parallelism.
+
+C<QUERY_WORKERS> defaults to the number of CPUs available, but 4 per
+remote (HTTP/HTTPS) host.
+
+C<WRITE_WORKERS> defaults to the number of CPUs available for Maildir,
+IMAP/IMAPS, and mbox* destinations.
+
+Omitting C<QUERY_WORKERS> but leaving the comma (C<,>) allows
+one to only set C<WRITE_WORKERS>
+
=item --dedupe=STRATEGY
=item -d STRATEGY
diff --git a/Documentation/lei-up.pod b/Documentation/lei-up.pod
index ac644a96..3b7c6f46 100644
--- a/Documentation/lei-up.pod
+++ b/Documentation/lei-up.pod
@@ -64,7 +64,9 @@ specified via C<lei q --only>.
=item --mua=CMD
-C<--lock>, C<--alert>, and C<--mua> are all supported and
+=item --jobs QUERY_WORKERS[,WRITE_WORKERS]
+
+C<--lock>, C<--alert>, C<--mua>, and C<--jobs> are all supported and
documented in L<lei-q(1)>.
C<--mua> is incompatible with C<--all>.
^ permalink raw reply related [relevance 86%]
* [PATCH] lei import: reduce writes to lei/store on IMAP sync
@ 2021-05-31 10:20 86% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-05-31 10:20 UTC (permalink / raw)
To: meta
We don't need to write VMD changes to lei/store if local
keywords are unchanged.
---
lib/PublicInbox/LeiImport.pm | 1 +
lib/PublicInbox/LeiSearch.pm | 8 +++++---
lib/PublicInbox/NetReader.pm | 2 +-
3 files changed, 7 insertions(+), 4 deletions(-)
diff --git a/lib/PublicInbox/LeiImport.pm b/lib/PublicInbox/LeiImport.pm
index f9a46ec5..860a2c98 100644
--- a/lib/PublicInbox/LeiImport.pm
+++ b/lib/PublicInbox/LeiImport.pm
@@ -50,6 +50,7 @@ sub input_net_cb { # imap_each / nntp_each
} elsif ($vmd) { # old message, kw only
my $oid = $self->{-lms_ro}->imap_oid2($uri, $uid) // return;
my @docids = $self->{lse}->over->blob_exists($oid) or return;
+ $self->{lse}->kw_changed(undef, $kw, \@docids) or return;
my $lei = $self->{lei};
$lei->qerr("# $oid => @$kw\n") if $lei->{opt}->{verbose};
$self->{lei}->{sto}->ipc_do('set_eml_vmd', undef,
diff --git a/lib/PublicInbox/LeiSearch.pm b/lib/PublicInbox/LeiSearch.pm
index b09d1e45..d0963e92 100644
--- a/lib/PublicInbox/LeiSearch.pm
+++ b/lib/PublicInbox/LeiSearch.pm
@@ -133,9 +133,11 @@ sub xoids_for {
# returns true if $eml is indexed by lei/store and keywords don't match
sub kw_changed {
my ($self, $eml, $new_kw_sorted, $docids) = @_;
- my $xoids = xoids_for($self, $eml) // return;
- $docids //= [];
- @$docids = sort { $a <=> $b } values %$xoids;
+ if ($eml) {
+ my $xoids = xoids_for($self, $eml) // return;
+ $docids //= [];
+ @$docids = sort { $a <=> $b } values %$xoids;
+ }
my $cur_kw = msg_keywords($self, $docids->[0]);
# RFC 5550 sec 5.9 on the $Forwarded keyword states:
diff --git a/lib/PublicInbox/NetReader.pm b/lib/PublicInbox/NetReader.pm
index b97444fd..39a8f7fc 100644
--- a/lib/PublicInbox/NetReader.pm
+++ b/lib/PublicInbox/NetReader.pm
@@ -409,7 +409,7 @@ sub flags2kw ($$$$) {
warn "# unknown IMAP flag $f <$uri/;UID=$uid>\n";
}
}
- @$kw = sort @$kw; # for all UI/UX purposes
+ @$kw = sort @$kw; # for LeiSearch->kw_changed and UI/UX purposes
$kw;
}
^ permalink raw reply related [relevance 86%]
* Failed 'lei q' blocks 'lei init' from working
@ 2022-03-12 21:04 87% Nícolas F. R. A. Prado
2022-03-13 0:06 90% ` Kyle Meyer
0 siblings, 1 reply; 200+ results
From: Nícolas F. R. A. Prado @ 2022-03-12 21:04 UTC (permalink / raw)
To: meta
Hi,
Right after building lei from source, I get an error running the 'q' command:
[nfraprado@notapiano public-inbox]$ ./lei.sh q -I https://lore.kernel.org/all/ -o ~/.mail/lei/test --threads --dedupe=mid 'dfn:mt8192.dtsi rt:1.month.ago..'
open(/home/nfraprado/.local/share/lei/store/mail_sync.sqlite3): No such file or directory at /home/nfraprado/ext/git/public-inbox/lib/PublicInbox/LeiQuery.pm line 37.
That's somewhat reasonable since I haven't issued an init first. But if I then
try to init:
[nfraprado@notapiano public-inbox]$ ./lei.sh init
# leistore.dir=/home/nfraprado/.local/share/lei/store newly initialized
[nfraprado@notapiano public-inbox]$ ./lei.sh init
# leistore.dir=/home/nfraprado/.local/share/lei/store newly initialized
No store is ever created. If I reboot (or issue the 'e' sysrq to basically
restart userspace) and then run the init command right away, it does work.
So it seems to me that there's some state that is kept after running the q
command which blocks init from working, hence requiring the reboot to clear that
state.
Not quite sure if it's a Perl thing or a bug in public-inbox, but since it's
unusual behavior (and took me some time to find out) I figured I'd report it.
Making 'q' run 'init' if not already done so could also help to work around this
issue.
Thanks,
Nícolas
^ permalink raw reply [relevance 87%]
* [PATCH 2/2] lei blob: support "lei index"-ed mail
@ 2021-05-05 10:46 87% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-05-05 10:46 UTC (permalink / raw)
To: meta
Normal git retrieval don't work for Maildir blobs indexed using
"lei index". Fortunately, this oddness is limited to the
LeiStore class and we can override smsg_eml with a fallback
to read blobs from Maildirs.
---
lib/PublicInbox/LeiSearch.pm | 10 ++++++++++
t/solver_git.t | 10 ++++++++++
2 files changed, 20 insertions(+)
diff --git a/lib/PublicInbox/LeiSearch.pm b/lib/PublicInbox/LeiSearch.pm
index cd28a700..c2b12146 100644
--- a/lib/PublicInbox/LeiSearch.pm
+++ b/lib/PublicInbox/LeiSearch.pm
@@ -144,4 +144,14 @@ sub lms {
-f $f ? PublicInbox::LeiMailSync->new($f) : undef;
}
+# allow SolverGit->resolve_patch to work with "lei index"
+sub smsg_eml {
+ my ($self, $smsg) = @_;
+ PublicInbox::Inbox::smsg_eml($self, $smsg) // do {
+ my $lms = lms($self);
+ my $bref = $lms ? $lms->local_blob($smsg->{blob}, 1) : undef;
+ $bref ? PublicInbox::Eml->new($bref) : undef;
+ };
+}
+
1;
diff --git a/t/solver_git.t b/t/solver_git.t
index e566efb3..44cbbfdb 100644
--- a/t/solver_git.t
+++ b/t/solver_git.t
@@ -27,6 +27,11 @@ my $ibx = create_inbox 'v2', version => 2,
$im->add(eml_load 't/solve/0001-simple-mod.patch') or BAIL_OUT;
$im->add($patch2) or BAIL_OUT;
};
+my $md = "$tmpdir/md";
+File::Path::mkpath([map { $md.$_ } (qw(/ /cur /new /tmp))]);
+symlink(abs_path('t/solve/0001-simple-mod.patch'), "$md/cur/foo:2,") or
+ xbail "symlink: $!";
+
my $v1_0_0_tag = 'cb7c42b1e15577ed2215356a2bf925aef59cdd8d';
my $v1_0_0_tag_short = substr($v1_0_0_tag, 0, 16);
my $expect = '69df7d565d49fbaaeb0a067910f03dc22cd52bd0';
@@ -70,6 +75,11 @@ test_lei({tmpdir => "$tmpdir/rediff"}, sub {
'got more context with -U9');
});
+test_lei({tmpdir => "$tmpdir/index-eml-only"}, sub {
+ lei_ok(qw(index), $md);
+ lei_ok(qw(blob 69df7d5)); # hits LeiSearch->smsg_eml -> lms->local_blob
+});
+
my $git = PublicInbox::Git->new($git_dir);
$ibx->{-repo_objs} = [ $git ];
my $res;
^ permalink raw reply related [relevance 87%]
* Re: lei: incorrect quoting on saved searches (was Re: lore+lei: getting started)
2021-11-08 21:48 90% ` Eric Wong
@ 2021-11-08 22:36 88% ` Konstantin Ryabitsev
2021-11-08 22:57 90% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Konstantin Ryabitsev @ 2021-11-08 22:36 UTC (permalink / raw)
To: Eric Wong; +Cc: Rob Herring, meta, workflows
On Mon, Nov 08, 2021 at 09:48:36PM +0000, Eric Wong wrote:
> > Hmm... I noticed that when I `lei edit-search` the initial query that was
> > causing quoting issues, I get the following:
> >
> > [lei]
> > q = (dfn:drivers OR dfn:arch OR dfn:Documentation OR dfn:include OR dfn:scripts) AND f:robh@kernel.org
> >
> > So, the extra quotes didn't get added to the config file. Running `lei up` on
> > that saved search seems to do the right thing, so the erroneous quotes are
> > only added during the initial `lei q` call.
>
> Right, each entry in lei.q is actually an entry in argv[].
> So the correct query should look something like:
So, to be clear here... the following doesn't work because instead of multiple
query parameters to 'lei q' the single-quoted string becomes a single
parameter?
lei q -I https://lore.kernel.org/all/ -o ~/work/temp/lei/robh-patches \
--threads --dedupe=mid \
'(dfn:drivers OR dfn:arch OR dfn:Documentation OR dfn:include OR dfn:scripts) AND f:robh@kernel.org'
Any way to make this work? I find that it's more easily readable than the
"echo | lei q" version.
For bash users, the following should work as well:
lei q -I https://lore.kernel.org/all/ -o ~/work/temp/lei/robh-patches \
--threads --dedupe=mid <<< \
'(dfn:drivers OR dfn:arch OR dfn:Documentation OR dfn:include OR dfn:scripts) AND f:robh@kernel.org'
Suggestion, can -I accept the URL containing the query, so that the command
becomes:
lei q -o ~/mail/foo --threads --dedupe=mid -I \
https://lore.kernel.org/all/?q=f%3Atorvalds+AND+nq%3Agarbage
This way we pass both the location of the extindex to query AND the parameters
we should use, avoiding shell quoting problems?
-K
^ permalink raw reply [relevance 88%]
* [PATCH] lei: don't exit lei-daemon on ovv_begin failure
@ 2023-10-27 1:14 88% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2023-10-27 1:14 UTC (permalink / raw)
To: meta
When ->ovv_begin is called in LeiXSearch->do_query in the top-level
lei-daemon process, $lei->{pkt_op_p} still exists. We must make
sure we're exiting the correct process since lei->out can call
lei->fail and lei->fail calls lei->x_it.
As to avoiding how I caused ->ovv_begin failures to begin with,
that's for a much bigger change...
---
lib/PublicInbox/LEI.pm | 7 ++++---
1 file changed, 4 insertions(+), 3 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 7bc7b2dc..e060bcbe 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -40,6 +40,7 @@ $GLP_PASS->configure(qw(gnu_getopt no_ignore_case auto_abbrev pass_through));
our (%PATH2CFG, # persistent for socket daemon
$MDIR2CFGPATH, # /path/to/maildir => { /path/to/config => [ ino watches ] }
$OPT, # shared between optparse and opt_dash callback (for Getopt::Long)
+$daemon_pid
);
# TBD: this is a documentation mechanism to show a subcommand
@@ -486,7 +487,7 @@ sub x_it ($$) {
stop_pager($self);
if ($self->{pkt_op_p}) { # worker => lei-daemon
$self->{pkt_op_p}->pkt_do('x_it', $code);
- exit($code >> 8);
+ exit($code >> 8) if $$ != $daemon_pid;
} elsif ($self->{sock}) { # lei->daemon => lei(1) client
send($self->{sock}, "x_it $code", 0);
} elsif ($quit == \&CORE::exit) { # an admin (one-shot) command
@@ -1341,8 +1342,8 @@ sub lazy_start {
my $pid = fork;
return if $pid;
$0 = "lei-daemon $path";
- local %PATH2CFG;
- local $MDIR2CFGPATH;
+ local (%PATH2CFG, $MDIR2CFGPATH);
+ local $daemon_pid = $$;
$listener->blocking(0);
my $exit_code;
my $pil = PublicInbox::Listener->new($listener, \&accept_dispatch);
^ permalink raw reply related [relevance 88%]
* [PATCH 3/3] lei forget-mail-sync: rely on lei/store process
@ 2021-08-17 8:52 88% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-08-17 8:52 UTC (permalink / raw)
To: meta
As implied in commit 6ff03ba2be9247f1
("lei export-kw: do not write directly to mail_sync.sqlite3"),
modifying mail_sync.sqlite3 directly can lead to conflicts
and making everything go through lei/store is easier.
---
lib/PublicInbox/LeiForgetMailSync.pm | 8 ++++----
lib/PublicInbox/LeiStore.pm | 6 ++++++
2 files changed, 10 insertions(+), 4 deletions(-)
diff --git a/lib/PublicInbox/LeiForgetMailSync.pm b/lib/PublicInbox/LeiForgetMailSync.pm
index c74ba25d..940ca1b6 100644
--- a/lib/PublicInbox/LeiForgetMailSync.pm
+++ b/lib/PublicInbox/LeiForgetMailSync.pm
@@ -15,13 +15,13 @@ use PublicInbox::LeiExportKw;
sub lei_forget_mail_sync {
my ($lei, @folders) = @_;
my $lms = $lei->lms or return;
+ my $sto = $lei->_lei_store or return; # may disappear due to race
+ $sto->write_prepare;
my $err = $lms->arg2folder($lei, \@folders);
$lei->qerr(@{$err->{qerr}}) if $err->{qerr};
return $lei->fail($err->{fail}) if $err->{fail};
- delete $lms->{dbh};
- $lms->lms_begin;
- $lms->forget_folder($_) for @folders;
- $lms->lms_commit;
+ $sto->ipc_do('lms_forget_folders', @folders);
+ my $wait = $sto->ipc_do('done');
}
*_complete_forget_mail_sync = \&PublicInbox::LeiExportKw::_complete_export_kw;
diff --git a/lib/PublicInbox/LeiStore.pm b/lib/PublicInbox/LeiStore.pm
index 3f33d114..e8334187 100644
--- a/lib/PublicInbox/LeiStore.pm
+++ b/lib/PublicInbox/LeiStore.pm
@@ -216,6 +216,12 @@ sub lms_mv_src {
_lms_rw($self)->mv_src($folder, $oidbin, $id, $newbn);
}
+sub lms_forget_folders {
+ my ($self, @folders) = @_;
+ my $lms = _lms_rw($self);
+ for my $f (@folders) { $lms->forget_folder($f) }
+}
+
sub set_sync_info {
my ($self, $oidhex, $folder, $id) = @_;
_lms_rw($self)->set_src($oidhex, $folder, $id);
^ permalink raw reply related [relevance 88%]
* Re: lei import on epochs [was: make menuconfig interface for lei / grok-pull]
2021-09-16 23:53 90% ` Luis Chamberlain
@ 2021-09-17 0:22 89% ` Eric Wong
2021-09-17 0:48 90% ` Luis Chamberlain
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2021-09-17 0:22 UTC (permalink / raw)
To: Luis Chamberlain; +Cc: meta
Luis Chamberlain <mcgrof@kernel.org> wrote:
> On Thu, Sep 16, 2021 at 09:09:02PM +0000, Eric Wong wrote:
> > Luis Chamberlain <mcgrof@kernel.org> wrote:
> > > # The next two don't work
> > > lei import import https://lore.kernel.org/linux-fsdevel/git/0.git
> >
> > Btw, extra "import" aside; I never intended to support the above
> > case. However, your post got me thinking we could and probably
> > should... :>
> >
> > (epochs are git storage with a loose limit of ~1G packed data,
> > "slices" are public-inbox-imapd folders capped at 50K to
> > accomodate limitations of existing clients/filesystems)
>
> So... by default all epochs are pulled? If so, yeah geesh.
> I don't want to pull all linux-kernel epochs. Only if doing
> R&D on that list would I need it.
Right now, "lei import" doesn't read nor understand git repos at all.
If you meant add-external, yes, all epochs are pulled when using
"lei add-external --mirror" (and public-inbox-clone). Yes,
there should be a way to do partial mirrors, just haven't
gotten around to it...
"lei import" currently reads IMAP, NNTP, Maildir, mbox*, etc.
It also understands various https?://*/(raw,t.mbox.gz) URLs used by
PublicInbox::WWW, since they're (gzipped) mboxrd.
But none of that is reading from git repos directly...
> Maybe a --query-epochs and then a --use-epoch 0 or whatever
> as well?
Since epochs expose the gittyness of lei, perhaps a limited form
of commitish-like ranges used by git would work:
lei add-external --mirror $URL --epoch=0..9
lei add-external --mirror $URL --epoch=5..
lei add-external --mirror $URL --epoch=~2.. # like HEAD~2..
That only works for contiguous ranges, though; but I'm also
unsure if discontiguous ranges make sense to anyone.
There's other syntaxes (e.g. (sed|Perl)-like), of course,
but I think git-like here would be most natural.
Thoughts?
^ permalink raw reply [relevance 89%]
* [PATCH 0/7] lei sync preparations, "lei inspect"
@ 2021-04-24 9:28 89% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-24 9:28 UTC (permalink / raw)
To: meta
"lei import" now tracks IMAP and Maildir source information in
preparation for propagating keywords back to IMAP|Maildir.
"lei inspect" is a long-overdue debug/diagnostic thing.
Eric Wong (7):
lei_input: drop outdated comment w.r.t. compression
t/lei_to_mail: split "lei import" test $HOME directory
URIimap: support ->uidvalidity and ->iuid
net_reader: imap_each: add UIDVALIDITY to URL arg
doc: lei_design_notes: add a bit on WAL usage
lei_mail_sync: for bidirectional keyword sync
lei import: keep sync info for Maildir and IMAP folders
Documentation/lei_design_notes.txt | 12 ++
MANIFEST | 3 +
lib/PublicInbox/LEI.pm | 16 ++-
lib/PublicInbox/LeiImport.pm | 22 ++-
lib/PublicInbox/LeiInput.pm | 42 +++++-
lib/PublicInbox/LeiInspect.pm | 96 +++++++++++++
lib/PublicInbox/LeiMailSync.pm | 211 +++++++++++++++++++++++++++++
lib/PublicInbox/LeiSearch.pm | 7 +
lib/PublicInbox/LeiStore.pm | 20 ++-
lib/PublicInbox/NetReader.pm | 11 +-
lib/PublicInbox/TestCommon.pm | 2 +
lib/PublicInbox/URIimap.pm | 38 +++++-
t/lei-import-imap.t | 27 +++-
t/lei-import-maildir.t | 21 +++
t/lei_mail_sync.t | 68 ++++++++++
t/lei_to_mail.t | 8 +-
t/net_reader-imap.t | 4 +-
t/uri_imap.t | 32 ++++-
18 files changed, 612 insertions(+), 28 deletions(-)
create mode 100644 lib/PublicInbox/LeiInspect.pm
create mode 100644 lib/PublicInbox/LeiMailSync.pm
create mode 100644 t/lei_mail_sync.t
^ permalink raw reply [relevance 89%]
* [PATCH 2/4] lei: do not blindly commit to lei/store on close
@ 2021-03-26 9:51 90% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-26 9:51 UTC (permalink / raw)
To: meta
It may hide errors/bugs, instead do it explicitly for each
worker that writes to it. For lei_xsearch, it will be better
to close before spawning the MUA for future use since we may
need it again once the user starts changing keywords.
---
lib/PublicInbox/LEI.pm | 3 ---
lib/PublicInbox/LeiXSearch.pm | 1 +
2 files changed, 1 insertion(+), 3 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 6a5c32b3..59715633 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -1030,9 +1030,6 @@ sub dclose {
}
}
close(delete $self->{1}) if $self->{1}; # may reap_compress
- if (my $sto = delete $self->{sto}) {
- $sto->ipc_do('done');
- }
$self->close if $self->{-event_init_done}; # PublicInbox::DS::close
}
diff --git a/lib/PublicInbox/LeiXSearch.pm b/lib/PublicInbox/LeiXSearch.pm
index 6410e0ea..b41daffe 100644
--- a/lib/PublicInbox/LeiXSearch.pm
+++ b/lib/PublicInbox/LeiXSearch.pm
@@ -319,6 +319,7 @@ sub query_done { # EOF callback for main daemon
if (my $lxs = delete $lei->{lxs}) {
$lxs->wq_wait_old(\&xsearch_done_wait, $lei);
}
+ my $wait = $lei->{sto} ? $lei->{sto}->ipc_do('done') : undef;
$lei->{ovv}->ovv_end($lei);
if ($l2m) { # close() calls LeiToMail reap_compress
if (my $out = delete $lei->{old_1}) {
^ permalink raw reply related [relevance 90%]
* [PATCH 0/5] "lei q --save" + "lei up"
@ 2021-04-13 10:54 90% Eric Wong
2021-04-13 10:54 75% ` [PATCH 5/5] lei: add "lei up" to complement "lei q --save" Eric Wong
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2021-04-13 10:54 UTC (permalink / raw)
To: meta
"--save" may become the default when writing to a
pathname or URL. "lei up" will be used to update
the results of "--save".
This only supports local externals at the moment,
remote externals won't be able to avoid excess
traffic easily.
Usability improvements are coming...
Eric Wong (5):
lei_xsearch: use per-external queries when not sorting
lei_dedupe: adjust to prepare for saved searches
lei_query: rearrange internals to capture query early
lei q: start wiring up saved search
lei: add "lei up" to complement "lei q --save"
MANIFEST | 4 +
lib/PublicInbox/LEI.pm | 6 +-
lib/PublicInbox/LeiDedupe.pm | 16 ++--
lib/PublicInbox/LeiQuery.pm | 59 +++++++------
lib/PublicInbox/LeiSavedSearch.pm | 142 ++++++++++++++++++++++++++++++
lib/PublicInbox/LeiToMail.pm | 18 ++--
lib/PublicInbox/LeiUp.pm | 46 ++++++++++
lib/PublicInbox/LeiXSearch.pm | 94 +++++++++++++-------
t/lei-q-save.t | 25 ++++++
t/lei.t | 2 +-
t/lei_dedupe.t | 11 ++-
t/lei_saved_search.t | 10 +++
12 files changed, 356 insertions(+), 77 deletions(-)
create mode 100644 lib/PublicInbox/LeiSavedSearch.pm
create mode 100644 lib/PublicInbox/LeiUp.pm
create mode 100644 t/lei-q-save.t
create mode 100644 t/lei_saved_search.t
^ permalink raw reply [relevance 90%]
* [PATCH 4/9] doc lei: resort lei-tag entries
@ 2021-05-17 3:35 90% ` Kyle Meyer
2021-05-17 3:35 82% ` [PATCH 3/9] doc lei blob: point to lei-q for shared options Kyle Meyer
1 sibling, 0 replies; 200+ results
From: Kyle Meyer @ 2021-05-17 3:35 UTC (permalink / raw)
To: meta
The command was renamed in 54da988cfb049ea2 (lei tag: rename from "lei
mark", 2021-03-30). Relocate its entries in txt2pre and Makefile.PL
to restore alphabetical sorting.
---
Documentation/txt2pre | 2 +-
Makefile.PL | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/Documentation/txt2pre b/Documentation/txt2pre
index 7b9d7853..25d81cf9 100755
--- a/Documentation/txt2pre
+++ b/Documentation/txt2pre
@@ -21,10 +21,10 @@ for (qw[lei(1)
lei-init(1)
lei-ls-external(1)
lei-ls-label(1)
- lei-tag(1)
lei-overview(7)
lei-p2q(1)
lei-q(1)
+ lei-tag(1)
public-inbox.cgi(1)
public-inbox-compact(1)
public-inbox-config(5)
diff --git a/Makefile.PL b/Makefile.PL
index 401c033e..b06b6ab8 100644
--- a/Makefile.PL
+++ b/Makefile.PL
@@ -47,7 +47,7 @@ $v->{-m1} = [ map {
qw(
lei-add-external lei-blob lei-config lei-daemon-kill lei-daemon-pid
lei-forget-external lei-import lei-init lei-ls-external lei-ls-label
- lei-tag lei-p2q lei-q)];
+ lei-p2q lei-q lei-tag)];
$v->{-m5} = [ qw(public-inbox-config public-inbox-v1-format
public-inbox-v2-format public-inbox-extindex-format
lei-mail-formats lei-store-format
--
2.31.1
^ permalink raw reply related [relevance 90%]
* Re: lei: incorrect quoting on saved searches (was Re: lore+lei: getting started)
2021-11-08 20:22 86% ` lei: incorrect quoting on saved searches (was Re: lore+lei: getting started) Konstantin Ryabitsev
2021-11-08 20:53 90% ` Rob Herring
@ 2021-11-08 20:49 90% ` Eric Wong
2021-11-08 21:36 90% ` Konstantin Ryabitsev
1 sibling, 1 reply; 200+ results
From: Eric Wong @ 2021-11-08 20:49 UTC (permalink / raw)
To: Rob Herring; +Cc: Konstantin Ryabitsev, meta, workflows
Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> On Mon, Nov 08, 2021 at 01:49:07PM -0600, Rob Herring wrote:
>
> Moving this to meta.
I don't think workflows should've been dropped, though.
> > > lei q -I https://lore.kernel.org/all/ -o ~/Mail/floppy \
> > > --threads --dedupe=mid \
> > > '(dfn:drivers/block/floppy.c OR dfhh:floppy_* OR s:floppy \
> > > OR ((nq:bug OR nq:regression) AND nq:floppy)) \
> > > AND rt:1.month.ago..'
> >
> > I tried a similar one which I had working as a bookmark:
That's actually treating the entire single-quoted section as
a phrase search for Xapian.
The correct way to use '(', ')', and '*' on the command-line for
Xapian is to shell escape them:
lei q -I https://lore.kernel.org/all/ -o ~/Mail/floppy \
--threads --dedupe=mid \
\( dfn:drivers/block/floppy.c OR dfhh:floppy_\* OR s:floppy \
OR \(\(nq:bug OR nq:regression\) AND nq:floppy\)\) \
AND rt:1.month.ago...
Since shell escaping so many metacharacters is annoying,
stdin is supported (and implicit iff file|pipe):
echo '(dfn:drivers/block/floppy.c OR dfhh:floppy_* OR s:floppy \
OR ((nq:bug OR nq:regression) AND nq:floppy)) \
AND rt:1.month.ago..' | \
lei q -I https://lore.kernel.org/all/ -o ~/Mail/floppy \
--threads --dedupe=mid
^ permalink raw reply [relevance 90%]
* [PATCH 8/9] lei q --save: avoid lei.q.format
@ 2021-04-16 23:10 90% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-16 23:10 UTC (permalink / raw)
To: meta
It is redundant since we stuff everything into the lei.q.output
config key.
---
lib/PublicInbox/LeiSavedSearch.pm | 2 --
lib/PublicInbox/LeiUp.pm | 1 -
2 files changed, 3 deletions(-)
diff --git a/lib/PublicInbox/LeiSavedSearch.pm b/lib/PublicInbox/LeiSavedSearch.pm
index 93b1b23a..a8bf470b 100644
--- a/lib/PublicInbox/LeiSavedSearch.pm
+++ b/lib/PublicInbox/LeiSavedSearch.pm
@@ -51,8 +51,6 @@ sub new {
} else {
cfg_set($self, 'lei.q', $q);
}
- my $fmt = $lei->{opt}->{'format'};
- cfg_set($self, 'lei.q.format', $fmt) if defined $fmt;
$dst = "$lei->{ovv}->{fmt}:$dst" if $dst !~ m!\Aimaps?://!i;
cfg_set($self, 'lei.q.output', $dst);
for my $k (qw(only include exclude)) {
diff --git a/lib/PublicInbox/LeiUp.pm b/lib/PublicInbox/LeiUp.pm
index 7ddb1dd0..9fe4901b 100644
--- a/lib/PublicInbox/LeiUp.pm
+++ b/lib/PublicInbox/LeiUp.pm
@@ -24,7 +24,6 @@ sub lei_up {
}
$lei->{opt}->{output} = $lss->{-cfg}->{'lei.q.output'} //
return $lei->fail("lei.q.output unset in $lss->{-f}");
- $lei->{opt}->{'format'} //= $lss->{-cfg}->{'lei.q.format'}; # optional
my $to_avref = $lss->{-cfg}->can('_array');
for my $k (qw(only include exclude)) {
^ permalink raw reply related [relevance 90%]
* lei-q doc thoughts... [was: doc: start manpages for lei commands]
@ 2021-02-06 9:01 90% ` Eric Wong
2021-02-06 19:57 90% ` Kyle Meyer
2021-02-07 19:58 90% ` lei q --output vs --mfolder [was: [PATCH 1/2] " Eric Wong
1 sibling, 1 reply; 200+ results
From: Eric Wong @ 2021-02-06 9:01 UTC (permalink / raw)
To: Kyle Meyer; +Cc: meta
Kyle Meyer <kyle@kyleam.com> wrote:
> +=item --mua-cmd=COMMAND, --mua=COMMAND
On second thought: is the long "--mua-cmd" even worth having or
supporting given "--mua=" exists? I will likely remove it from
the documentation and filter it out from the help text.
Technically "mua-cmd" is more descriptive since it's a command
with a %f placeholder, but I can't imagine anybody wanting to
type "--mua-cmd" over "--mua".
> +=item -t, --thread
> +
> +Return all messages in the same thread as the actual match(es).
Heh, it turns out mairix uses "--threads" (plural). I never
knew that since I always used "-t". Not sure if it's worth
pluralizing on our end...
^ permalink raw reply [relevance 90%]
* Re: lei-q doc thoughts... [was: doc: start manpages for lei commands]
2021-02-06 9:01 90% ` lei-q doc thoughts... [was: doc: start manpages for lei commands] Eric Wong
@ 2021-02-06 19:57 90% ` Kyle Meyer
2021-02-07 3:33 90% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Kyle Meyer @ 2021-02-06 19:57 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
Eric Wong writes:
> Kyle Meyer <kyle@kyleam.com> wrote:
>> +=item --mua-cmd=COMMAND, --mua=COMMAND
>
> On second thought: is the long "--mua-cmd" even worth having or
> supporting given "--mua=" exists? I will likely remove it from
> the documentation and filter it out from the help text.
>
> Technically "mua-cmd" is more descriptive since it's a command
> with a %f placeholder, but I can't imagine anybody wanting to
> type "--mua-cmd" over "--mua".
No, I can't either. Dropping --mua-cmd makes sense.
>> +=item -t, --thread
>> +
>> +Return all messages in the same thread as the actual match(es).
>
> Heh, it turns out mairix uses "--threads" (plural). I never
> knew that since I always used "-t". Not sure if it's worth
> pluralizing on our end...
I'd vote for following mairix here. I guess most people will use -t,
but it's just one less thing to get tripped up on.
By the way, if you'd like, I'd be happy to do a round (or more) of lei
manpage updates for new additions whenever you think things are in a
good spot for it.
^ permalink raw reply [relevance 90%]
* Re: lei-q doc thoughts... [was: doc: start manpages for lei commands]
2021-02-06 19:57 90% ` Kyle Meyer
@ 2021-02-07 3:33 90% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-02-07 3:33 UTC (permalink / raw)
To: Kyle Meyer; +Cc: meta
Kyle Meyer <kyle@kyleam.com> wrote:
>
> No, I can't either. Dropping --mua-cmd makes sense.
> Eric Wong writes:
> > Heh, it turns out mairix uses "--threads" (plural). I never
> > knew that since I always used "-t". Not sure if it's worth
> > pluralizing on our end...
>
> I'd vote for following mairix here. I guess most people will use -t,
> but it's just one less thing to get tripped up on.
Thanks, I'll queue up necessary patches for those.
> By the way, if you'd like, I'd be happy to do a round (or more) of lei
> manpage updates for new additions whenever you think things are in a
> good spot for it.
Yes, much appreciated, thanks in advance :>
Anything covered by tests should be considered
ready-for-documentation, I think. That can help us flush out
any strange/unexpected behaviors before it's finalized in a
release.
Fwiw, I'm likely to apply+push any doc-only patches as quickly as
I see them.
^ permalink raw reply [relevance 90%]
* lei q --output vs --mfolder [was: [PATCH 1/2] doc: start manpages for lei commands]
2021-02-06 9:01 90% ` lei-q doc thoughts... [was: doc: start manpages for lei commands] Eric Wong
@ 2021-02-07 19:58 90% ` Eric Wong
2021-02-07 20:33 90% ` Kyle Meyer
1 sibling, 1 reply; 200+ results
From: Eric Wong @ 2021-02-07 19:58 UTC (permalink / raw)
To: Kyle Meyer; +Cc: meta
Kyle Meyer <kyle@kyleam.com> wrote:
> +++ b/Documentation/lei-q.pod
> +=item -o PATH, --output=PATH, --mfolder=PATH
> +
> +Destination for results (e.g., C<path/to/Maildir> or - for stdout).
Fwiw, I didn't really like the term "mfolder" but that's what
mairix users are used to... Perhaps:
=item -o MFOLDER, --output=MFOLDER
Or is "mfolder" not that bad and we ditch --output instead?
(mairix uses '-o', so we match it there)
I never really got used to using "folder" to describe mailboxes
or directories. I associate "folder" with something that holds
only a few pages of paper in the physical world; not thousands
or millions of items.
^ permalink raw reply [relevance 90%]
* Re: lei q --output vs --mfolder [was: [PATCH 1/2] doc: start manpages for lei commands]
2021-02-07 19:58 90% ` lei q --output vs --mfolder [was: [PATCH 1/2] " Eric Wong
@ 2021-02-07 20:33 90% ` Kyle Meyer
2021-02-07 20:59 90% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Kyle Meyer @ 2021-02-07 20:33 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
Eric Wong writes:
> Kyle Meyer <kyle@kyleam.com> wrote:
>> +++ b/Documentation/lei-q.pod
>
>> +=item -o PATH, --output=PATH, --mfolder=PATH
>> +
>> +Destination for results (e.g., C<path/to/Maildir> or - for stdout).
>
> Fwiw, I didn't really like the term "mfolder" but that's what
> mairix users are used to... Perhaps:
>
> =item -o MFOLDER, --output=MFOLDER
>
> Or is "mfolder" not that bad and we ditch --output instead?
Yeah, I don't like "mfolder" either. In the --thread{,s} case,
consistency seemed worth it to me because either one sounded okay, and a
one character difference is easy to not notice. But in this case, I
think --output is the better name, so if we're dropping one, I'd vote to
cast out --mfolder.
Using MFOLDER rather than PATH as the metavariable seems a bit confusing
to me because the target isn't a directory for --format values other
than "maildir".
^ permalink raw reply [relevance 90%]
* Re: lei q --output vs --mfolder [was: [PATCH 1/2] doc: start manpages for lei commands]
2021-02-07 20:33 90% ` Kyle Meyer
@ 2021-02-07 20:59 90% ` Eric Wong
2021-02-07 21:47 90% ` Kyle Meyer
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2021-02-07 20:59 UTC (permalink / raw)
To: Kyle Meyer; +Cc: meta
Kyle Meyer <kyle@kyleam.com> wrote:
> Eric Wong writes:
> > Kyle Meyer <kyle@kyleam.com> wrote:
> >> +++ b/Documentation/lei-q.pod
> >
> >> +=item -o PATH, --output=PATH, --mfolder=PATH
> >> +
> >> +Destination for results (e.g., C<path/to/Maildir> or - for stdout).
> >
> > Fwiw, I didn't really like the term "mfolder" but that's what
> > mairix users are used to... Perhaps:
> >
> > =item -o MFOLDER, --output=MFOLDER
> >
> > Or is "mfolder" not that bad and we ditch --output instead?
>
> Yeah, I don't like "mfolder" either. In the --thread{,s} case,
> consistency seemed worth it to me because either one sounded okay, and a
> one character difference is easy to not notice. But in this case, I
> think --output is the better name, so if we're dropping one, I'd vote to
> cast out --mfolder.
Alright, I'll think about dropping ... Right now, it's still
supported, but masked out of --help output but with a "MFOLDER"
placeholder.
Perhaps a note in the man page noting it's mairix analogue is
sufficient?
> Using MFOLDER rather than PATH as the metavariable seems a bit confusing
> to me because the target isn't a directory for --format values other
> than "maildir".
Fwiw, mairix uses "mfolder" for mbox and IMAP destinations, too;
(I've never used IMAP with mairix, but have every intention of
supporting IMAP in lei).
"LOCATION" may also be a suitable placeholder *shrug*
^ permalink raw reply [relevance 90%]
* Re: lei q --output vs --mfolder [was: [PATCH 1/2] doc: start manpages for lei commands]
2021-02-07 20:59 90% ` Eric Wong
@ 2021-02-07 21:47 90% ` Kyle Meyer
2021-02-07 21:55 90% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Kyle Meyer @ 2021-02-07 21:47 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
Eric Wong writes:
> Kyle Meyer <kyle@kyleam.com> wrote:
[...]
>> But in this case, I think --output is the better name, so if we're
>> dropping one, I'd vote to cast out --mfolder.
>
> Alright, I'll think about dropping ... Right now, it's still
> supported, but masked out of --help output but with a "MFOLDER"
> placeholder.
>
> Perhaps a note in the man page noting it's mairix analogue is
> sufficient?
As someone that has only looked into mairix a few times but not used it,
I'd find the pointer helpful, I think.
>> Using MFOLDER rather than PATH as the metavariable seems a bit confusing
>> to me because the target isn't a directory for --format values other
>> than "maildir".
>
> Fwiw, mairix uses "mfolder" for mbox and IMAP destinations, too;
> (I've never used IMAP with mairix, but have every intention of
> supporting IMAP in lei).
>
> "LOCATION" may also be a suitable placeholder *shrug*
Oh, IMAP destinations. So PATH isn't a great choice then. In that
case, switching to MFOLDER or LOCATION as the placeholder sounds fine to
me. I guess MFOLDER would be good at this point for consistency with `q
--help'. I've made a note to do that when updating the lei manpages
(hope to get to that tomorrow or the next day) assuming you haven't
handle it before then.
^ permalink raw reply [relevance 90%]
* Re: lei q --output vs --mfolder [was: [PATCH 1/2] doc: start manpages for lei commands]
2021-02-07 21:47 90% ` Kyle Meyer
@ 2021-02-07 21:55 90% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-02-07 21:55 UTC (permalink / raw)
To: Kyle Meyer; +Cc: meta
Kyle Meyer <kyle@kyleam.com> wrote:
> Eric Wong writes:
> > Fwiw, mairix uses "mfolder" for mbox and IMAP destinations, too;
> > (I've never used IMAP with mairix, but have every intention of
> > supporting IMAP in lei).
> >
> > "LOCATION" may also be a suitable placeholder *shrug*
>
> Oh, IMAP destinations. So PATH isn't a great choice then. In that
> case, switching to MFOLDER or LOCATION as the placeholder sounds fine to
> me. I guess MFOLDER would be good at this point for consistency with `q
> --help'. I've made a note to do that when updating the lei manpages
> (hope to get to that tomorrow or the next day) assuming you haven't
> handle it before then.
Right, I started using LOCATION for external inboxes instead of
"URL_OR_PATH" in --help; so MFOLDER may help disambiguate it for
non-external destinations.
Thanks again for tackling the doc stuff; I'll leave the .pod
untouched for now to avoid conflicts. I have an aversion to all
mark{up,down} languages, thus it's most natural for me to
document stuff in commit messages, --help text and emails :>
^ permalink raw reply [relevance 90%]
* Re: [PATCH 1/3] lei p2q: patch-to-query generator for "lei q --stdin"
@ 2021-02-28 21:40 90% ` Kyle Meyer
0 siblings, 0 replies; 200+ results
From: Kyle Meyer @ 2021-02-28 21:40 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
Eric Wong writes:
> Instead of teaching the to-be-implemented "lei show" to search
> threads/messages based commits, this orthogonal sub-command is
> designed to generate queries for use with "lei q --stdin".
>
> URI-escaped query parameters may be generated with --uri for
> HTTP(S) public-inbox instances, but otherwise the output is
> designed for "lei q --stdin".
>
> To find threads for a given git commit from a git worktree:
>
> lei p2q $COMMIT_OID | lei q --stdin -t ...
>
> It can also read via --stdin|-
>
> curl $INBOX_URL/$MSGID/raw | lei p2q - | lei q --stdin -t
>
> Or from the filesystem:
>
> lei p2q $(git format-patch -1) | lei q --stdin -t
Very nice :)
> diff --git a/lib/PublicInbox/LeiP2q.pm b/lib/PublicInbox/LeiP2q.pm
[...]
> + } elsif (m!^(?:---|\+{3}) ("?[^/]+/.+)!) {
> + my $fn = (split(m!/!, git_unquote($1.''), 2))[1];
> + push @{$lei->{qterms}->{dfn}}, xphrase($fn);
> + } elsif ($in_diff && s/^\+//) { # diff added
> + push @{$lei->{qterms}->{dfb}}, xphrase($_);
> + } elsif ($in_diff && s/^-//) { # diff removed
> + push @{$lei->{qterms}->{dfa}}, xphrase($_);
I noticed an unexpected term when trying dfa:
$ curl -fSs \
https://public-inbox.org/meta/20210228122528.18552-2-e@80x24.org/raw >msg
$ lei p2q --want=dfa msg
dfa:my @WQ_KEYS = qw dfa:"lxs l2m imp mrr cnv" dfa:"internal workers" dfa:dev/null
So I think the upstream "--- " filename regexp needs to be adjusted to
account for "/dev/null".
^ permalink raw reply [relevance 90%]
* [PATCH 2/4] lei help: move "lei help" into LeiHelp.pm
2021-03-27 11:45 90% [PATCH 0/4] lei blob (formerly known as "lei show") Eric Wong
@ 2021-03-27 11:45 90% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-27 11:45 UTC (permalink / raw)
To: meta
We need to load LeiHelp.pm anyways if somebody calls "lei help",
so save a few kB RAM for users who don't need help.
---
lib/PublicInbox/LEI.pm | 2 --
lib/PublicInbox/LeiHelp.pm | 3 +++
2 files changed, 3 insertions(+), 2 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index eb3ad9e2..e680f5f0 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -787,8 +787,6 @@ sub lei_daemon_kill {
kill($sig, $$) or fail($self, "kill($sig, $$): $!");
}
-sub lei_help { _help($_[0]) }
-
# Shell completion helper. Used by lei-completion.bash and hopefully
# other shells. Try to do as much here as possible to avoid redundancy
# and improve maintainability.
diff --git a/lib/PublicInbox/LeiHelp.pm b/lib/PublicInbox/LeiHelp.pm
index be31c2a8..9c1b30a1 100644
--- a/lib/PublicInbox/LeiHelp.pm
+++ b/lib/PublicInbox/LeiHelp.pm
@@ -97,4 +97,7 @@ EOF
undef;
}
+# the "lei help" command
+sub lei_help { $_[0]->_help }
+
1;
^ permalink raw reply related [relevance 90%]
* [PATCH 0/4] lei blob (formerly known as "lei show")
@ 2021-03-27 11:45 90% Eric Wong
2021-03-27 11:45 90% ` [PATCH 2/4] lei help: move "lei help" into LeiHelp.pm Eric Wong
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2021-03-27 11:45 UTC (permalink / raw)
To: meta
It's still a "git show" with blob reconstruction (aka "solver")
I'm not sure if lei <add|ls|forget>-coderepo is necessary,
since this supports multiple --git-dir args (and uses
the current working directory).
Maybe submodule directories could be scanned...
Eric Wong (4):
lei_ale: do not create store unnecessarily
lei help: move "lei help" into LeiHelp.pm
lei_query: hoist out lxs_prepare
lei blob: aka "git-show-harder" for blobs
MANIFEST | 1 +
lib/PublicInbox/LEI.pm | 26 ++++----
lib/PublicInbox/LeiALE.pm | 3 +-
lib/PublicInbox/LeiBlob.pm | 119 ++++++++++++++++++++++++++++++++++++
lib/PublicInbox/LeiHelp.pm | 3 +
lib/PublicInbox/LeiQuery.pm | 38 +++++++-----
t/lei-import.t | 2 +
t/solver_git.t | 14 ++++-
8 files changed, 173 insertions(+), 33 deletions(-)
create mode 100644 lib/PublicInbox/LeiBlob.pm
^ permalink raw reply [relevance 90%]
* [PATCH 7/8] doc lei overview: note that lei-init is usually unnecessary
@ 2021-03-29 3:11 90% ` Kyle Meyer
0 siblings, 0 replies; 200+ results
From: Kyle Meyer @ 2021-03-29 3:11 UTC (permalink / raw)
To: meta
cf. https://public-inbox.org/meta/20210325083207.GA30551@dcvr
---
Documentation/lei-overview.pod | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/Documentation/lei-overview.pod b/Documentation/lei-overview.pod
index 7c7337ab..55e84254 100644
--- a/Documentation/lei-overview.pod
+++ b/Documentation/lei-overview.pod
@@ -9,8 +9,10 @@ provides some basic examples.
=head1 LEI STORE
-L<lei-init(1)> initializes writable local storage based on
-L<public-inbox-v2-format(5)>.
+lei has writable local storage based on L<public-inbox-v2-format(5)>.
+Commands will automatically initialize the store behind the scenes if
+needed, but you can call L<lei-init(1)> directly if you want to use a
+store location other than the default C<$XDG_DATA_HOME/lei/store>.
=head2 EXAMPLES
--
2.31.0
^ permalink raw reply related [relevance 90%]
* [PATCH 0/3] doc: lei updates around lei-q
@ 2021-04-28 4:51 90% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-28 4:51 UTC (permalink / raw)
To: meta
1/3 is probably a game changer and will help anybody supporting
this project sleep easier at night :)
Eric Wong (3):
doc: lei: use /tmp for search results pathnames
doc: lei q: split =item aliases onto separate lines
doc: lei q: split --output and --format, note "text"
Documentation/lei-overview.pod | 4 +-
Documentation/lei-q.pod | 88 +++++++++++++++++++++++-----------
2 files changed, 63 insertions(+), 29 deletions(-)
^ permalink raw reply [relevance 90%]
* [PATCH 1/3] lei export-kw: use lei->abs_path instead of rel2abs
@ 2021-05-25 11:01 90% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-05-25 11:01 UTC (permalink / raw)
To: meta
This is to match LeiInput behavior when mail-sync is
enabled, since rel2abs won't resolve symlinks.
---
lib/PublicInbox/LeiExportKw.pm | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/lib/PublicInbox/LeiExportKw.pm b/lib/PublicInbox/LeiExportKw.pm
index d8ba8fd5..fabc01f8 100644
--- a/lib/PublicInbox/LeiExportKw.pm
+++ b/lib/PublicInbox/LeiExportKw.pm
@@ -130,7 +130,7 @@ EOM
for (@folders) {
next if $all{$_}; # ok
if (-d "$_/new" && -d "$_/cur") {
- my $d = 'maildir:'.$lei->rel2abs($_);
+ my $d = 'maildir:'.$lei->abs_path($_);
push(@no, $_) unless $all{$d};
$_ = $d;
} elsif (m!\Aimaps?://!i) {
^ permalink raw reply related [relevance 90%]
* [PATCH] lei up: only show finmsg in top-level lei-daemon
@ 2021-08-31 19:38 90% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-08-31 19:38 UTC (permalink / raw)
To: meta
->DESTROY can get triggered in child processes, which
unnecessarily duplicates messages queued up for display
when lei spawns extra workers.
---
lib/PublicInbox/LeiFinmsg.pm | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/lib/PublicInbox/LeiFinmsg.pm b/lib/PublicInbox/LeiFinmsg.pm
index 0ef5f070..395e7d3c 100644
--- a/lib/PublicInbox/LeiFinmsg.pm
+++ b/lib/PublicInbox/LeiFinmsg.pm
@@ -9,13 +9,13 @@ use v5.10.1;
sub new {
my ($cls, $io) = @_;
- bless [ $io ], $cls;
+ bless [ $io, $$ ], $cls;
}
sub DESTROY {
my ($self) = @_;
my $io = shift @$self;
- print $io @$self;
+ shift(@$self) == $$ and print $io @$self;
}
1;
^ permalink raw reply related [relevance 90%]
* lei import on epochs [was: make menuconfig interface for lei / grok-pull]
@ 2021-09-16 21:09 90% ` Eric Wong
2021-09-16 23:53 90% ` Luis Chamberlain
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2021-09-16 21:09 UTC (permalink / raw)
To: Luis Chamberlain; +Cc: meta
Luis Chamberlain <mcgrof@kernel.org> wrote:
> # The next two don't work
> lei import import https://lore.kernel.org/linux-fsdevel/git/0.git
Btw, extra "import" aside; I never intended to support the above
case. However, your post got me thinking we could and probably
should... :>
(epochs are git storage with a loose limit of ~1G packed data,
"slices" are public-inbox-imapd folders capped at 50K to
accomodate limitations of existing clients/filesystems)
^ permalink raw reply [relevance 90%]
* Re: lei import on epochs [was: make menuconfig interface for lei / grok-pull]
2021-09-16 21:09 90% ` lei import on epochs [was: make menuconfig interface for lei / grok-pull] Eric Wong
@ 2021-09-16 23:53 90% ` Luis Chamberlain
2021-09-17 0:22 89% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Luis Chamberlain @ 2021-09-16 23:53 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
On Thu, Sep 16, 2021 at 09:09:02PM +0000, Eric Wong wrote:
> Luis Chamberlain <mcgrof@kernel.org> wrote:
> > # The next two don't work
> > lei import import https://lore.kernel.org/linux-fsdevel/git/0.git
>
> Btw, extra "import" aside; I never intended to support the above
> case. However, your post got me thinking we could and probably
> should... :>
>
> (epochs are git storage with a loose limit of ~1G packed data,
> "slices" are public-inbox-imapd folders capped at 50K to
> accomodate limitations of existing clients/filesystems)
So... by default all epochs are pulled? If so, yeah geesh.
I don't want to pull all linux-kernel epochs. Only if doing
R&D on that list would I need it.
Maybe a --query-epochs and then a --use-epoch 0 or whatever
as well?
Luis
^ permalink raw reply [relevance 90%]
* Re: lei import on epochs [was: make menuconfig interface for lei / grok-pull]
2021-09-17 0:22 89% ` Eric Wong
@ 2021-09-17 0:48 90% ` Luis Chamberlain
0 siblings, 0 replies; 200+ results
From: Luis Chamberlain @ 2021-09-17 0:48 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
On Thu, Sep 16, 2021 at 5:22 PM Eric Wong <e@80x24.org> wrote:
> Since epochs expose the gittyness of lei, perhaps a limited form
> of commitish-like ranges used by git would work:
>
> lei add-external --mirror $URL --epoch=0..9
> lei add-external --mirror $URL --epoch=5..
> lei add-external --mirror $URL --epoch=~2.. # like HEAD~2..
That looks sexy.
I mentioned the query option so that I could end up displaying on
'make menuconfig' the epochs dynamically, it would be a script that
wants the list of epochs, and I'd use this to dynamically populate
kconfig. Dynamic population of kconfig is not a yet well understood
thing, but I've figured out a way to do it :)
> That only works for contiguous ranges, though; but I'm also
> unsure if discontiguous ranges make sense to anyone.
>
> There's other syntaxes (e.g. (sed|Perl)-like), of course,
> but I think git-like here would be most natural.
>
> Thoughts?
I think the above would suffice for sure!
Luis
^ permalink raw reply [relevance 90%]
* Re: lei: incorrect quoting on saved searches (was Re: lore+lei: getting started)
2021-11-08 20:22 86% ` lei: incorrect quoting on saved searches (was Re: lore+lei: getting started) Konstantin Ryabitsev
@ 2021-11-08 20:53 90% ` Rob Herring
2021-11-08 20:49 90% ` Eric Wong
1 sibling, 0 replies; 200+ results
From: Rob Herring @ 2021-11-08 20:53 UTC (permalink / raw)
To: Konstantin Ryabitsev; +Cc: meta
On Mon, Nov 8, 2021 at 2:22 PM Konstantin Ryabitsev
<konstantin@linuxfoundation.org> wrote:
>
> On Mon, Nov 08, 2021 at 01:49:07PM -0600, Rob Herring wrote:
>
> Moving this to meta.
>
> > > lei q -I https://lore.kernel.org/all/ -o ~/Mail/floppy \
> > > --threads --dedupe=mid \
> > > '(dfn:drivers/block/floppy.c OR dfhh:floppy_* OR s:floppy \
> > > OR ((nq:bug OR nq:regression) AND nq:floppy)) \
> > > AND rt:1.month.ago..'
> >
> > I tried a similar one which I had working as a bookmark:
> >
> > $ lei q -I https://lore.kernel.org/all/ -o ~/Mail/my-patches
> > --threads --dedupe=mid '(dfn:drivers OR dfn:arch OR dfn:Documentation
> > OR dfn:include OR dfn:scripts) AND f:robh@kernel.org'
> > # /home/rob/.local/share/lei/store 0/0
> > # /usr/bin/curl -Sf -s -d ''
> > https://lore.kernel.org/all/?x=m&t=1&q=(dfn%3A%22drivers+OR+dfn%3Aarch+OR+dfn%3ADocumentation+OR+dfn%3Ainclude+OR+dfn%3Ascripts)+AND+f%3Arobh%40kernel.org%22
> > # 0 written to /home/rob/Mail/my-patches/ (0 matches)
>
> It's true, I get the same thing if I omit "AND rt:" at the end.
>
> $ lei q -I https://lore.kernel.org/all/ -o ~/work/temp/lei/robh-patches --threads --dedupe=mid '(dfn:drivers OR dfn:arch OR dfn:Documentation OR dfn:include OR dfn:scripts) AND f:robh@kernel.org'
> # /home/user/.local/share/lei/store 0/0
> # /usr/bin/curl -Sf -s -d '' https://lore.kernel.org/all/?x=m&t=1&q=(dfn%3A%22drivers+OR+dfn%3Aarch+OR+dfn%3ADocumentation+OR+dfn%3Ainclude+OR+dfn%3Ascripts)+AND+f%3Arobh%40kernel.org%22
> # 0 written to /home/user/work/temp/lei/robh-patches/ (0 matches)
> $ lei forget-search ~/work/temp/lei/robh-patches
> $ lei q -I https://lore.kernel.org/all/ -o ~/work/temp/lei/robh-patches --threads --dedupe=mid '(dfn:drivers OR dfn:arch OR dfn:Documentation OR dfn:include OR dfn:scripts) AND f:robh@kernel.org AND rt:1.month.ago..'
> # /usr/bin/curl -Sf -s -d '' https://lore.kernel.org/all/?x=m&t=1&q=(dfn%3Adrivers+OR+dfn%3Aarch+OR+dfn%3ADocumentation+OR+dfn%3Ainclude+OR+dfn%3Ascripts)+AND+f%3Arobh%40kernel.org+AND+rt%3A1633724105..
> # /home/user/.local/share/lei/store 13/13
> # https://lore.kernel.org/all/ 65/?
> # https://lore.kernel.org/all/ 75/75
> # 45 written to /home/user/work/temp/lei/robh-patches/ (88 matches)
>
> > It seems there is some problem in quoting. Notice the '%22' that's
> > inserted in the url.
>
> Deferring to Eric here.
>
> > Also, the above query is a bit of a work-around as what I really want
> > is just all patches from me. I haven't been able to get something to
> > work. I've tried things like 'dfn:*' or 'dfn:/' or 'dfn:b/'.
>
> I think 's:patch AND nq:diff' is a good option here.
Not even close really. That mainly finds my replies with 'diff' in
them. I'm not sure why, but it misses most actual patches:
https://lore.kernel.org/all/?q=s%3Apatch+nq%3Adiff+f%3Arobh%40kernel.org
Rob
^ permalink raw reply [relevance 90%]
* Re: lei: incorrect quoting on saved searches (was Re: lore+lei: getting started)
2021-11-08 20:49 90% ` Eric Wong
@ 2021-11-08 21:36 90% ` Konstantin Ryabitsev
2021-11-08 21:48 90% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Konstantin Ryabitsev @ 2021-11-08 21:36 UTC (permalink / raw)
To: Eric Wong; +Cc: Rob Herring, meta, workflows
On Mon, Nov 08, 2021 at 08:49:23PM +0000, Eric Wong wrote:
> Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> > On Mon, Nov 08, 2021 at 01:49:07PM -0600, Rob Herring wrote:
> >
> > Moving this to meta.
>
> I don't think workflows should've been dropped, though.
>
> > > > lei q -I https://lore.kernel.org/all/ -o ~/Mail/floppy \
> > > > --threads --dedupe=mid \
> > > > '(dfn:drivers/block/floppy.c OR dfhh:floppy_* OR s:floppy \
> > > > OR ((nq:bug OR nq:regression) AND nq:floppy)) \
> > > > AND rt:1.month.ago..'
> > >
> > > I tried a similar one which I had working as a bookmark:
>
> That's actually treating the entire single-quoted section as
> a phrase search for Xapian.
Hmm... I noticed that when I `lei edit-search` the initial query that was
causing quoting issues, I get the following:
[lei]
q = (dfn:drivers OR dfn:arch OR dfn:Documentation OR dfn:include OR dfn:scripts) AND f:robh@kernel.org
So, the extra quotes didn't get added to the config file. Running `lei up` on
that saved search seems to do the right thing, so the erroneous quotes are
only added during the initial `lei q` call.
> The correct way to use '(', ')', and '*' on the command-line for
> Xapian is to shell escape them:
But putting them into single quotes should accomplish the same result, no? At
least, that's how I've always understood shell escaping.
-K
^ permalink raw reply [relevance 90%]
* Re: lei: incorrect quoting on saved searches (was Re: lore+lei: getting started)
2021-11-08 21:36 90% ` Konstantin Ryabitsev
@ 2021-11-08 21:48 90% ` Eric Wong
2021-11-08 22:36 88% ` Konstantin Ryabitsev
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2021-11-08 21:48 UTC (permalink / raw)
To: Konstantin Ryabitsev; +Cc: Rob Herring, meta, workflows
Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> On Mon, Nov 08, 2021 at 08:49:23PM +0000, Eric Wong wrote:
> > Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> > > On Mon, Nov 08, 2021 at 01:49:07PM -0600, Rob Herring wrote:
> >
> > > > > lei q -I https://lore.kernel.org/all/ -o ~/Mail/floppy \
> > > > > --threads --dedupe=mid \
> > > > > '(dfn:drivers/block/floppy.c OR dfhh:floppy_* OR s:floppy \
> > > > > OR ((nq:bug OR nq:regression) AND nq:floppy)) \
> > > > > AND rt:1.month.ago..'
> > > >
> > > > I tried a similar one which I had working as a bookmark:
> >
> > That's actually treating the entire single-quoted section as
> > a phrase search for Xapian.
>
> Hmm... I noticed that when I `lei edit-search` the initial query that was
> causing quoting issues, I get the following:
>
> [lei]
> q = (dfn:drivers OR dfn:arch OR dfn:Documentation OR dfn:include OR dfn:scripts) AND f:robh@kernel.org
>
> So, the extra quotes didn't get added to the config file. Running `lei up` on
> that saved search seems to do the right thing, so the erroneous quotes are
> only added during the initial `lei q` call.
Right, each entry in lei.q is actually an entry in argv[].
So the correct query should look something like:
[lei]
q = (
q = dfn:drivers/block/floppy.c
q = OR
...
> > The correct way to use '(', ')', and '*' on the command-line for
> > Xapian is to shell escape them:
>
> But putting them into single quotes should accomplish the same result, no? At
> least, that's how I've always understood shell escaping.
Yeah, that works, too. As long as spaces/tabs don't show up
within each argv[] element, it won't be interpreted as a phrase.
I really wanted: lei q s:"a quick brown fox"
to work from a shell like it would in the WWW UI;
and thus deprioritized '(' and ')' working properly :x
^ permalink raw reply [relevance 90%]
* Re: lei: incorrect quoting on saved searches (was Re: lore+lei: getting started)
2021-11-08 22:36 88% ` Konstantin Ryabitsev
@ 2021-11-08 22:57 90% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-11-08 22:57 UTC (permalink / raw)
To: Konstantin Ryabitsev; +Cc: Rob Herring, meta, workflows
Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> On Mon, Nov 08, 2021 at 09:48:36PM +0000, Eric Wong wrote:
> > > Hmm... I noticed that when I `lei edit-search` the initial query that was
> > > causing quoting issues, I get the following:
> > >
> > > [lei]
> > > q = (dfn:drivers OR dfn:arch OR dfn:Documentation OR dfn:include OR dfn:scripts) AND f:robh@kernel.org
> > >
> > > So, the extra quotes didn't get added to the config file. Running `lei up` on
> > > that saved search seems to do the right thing, so the erroneous quotes are
> > > only added during the initial `lei q` call.
> >
> > Right, each entry in lei.q is actually an entry in argv[].
> > So the correct query should look something like:
>
> So, to be clear here... the following doesn't work because instead of multiple
> query parameters to 'lei q' the single-quoted string becomes a single
> parameter?
>
> lei q -I https://lore.kernel.org/all/ -o ~/work/temp/lei/robh-patches \
> --threads --dedupe=mid \
> '(dfn:drivers OR dfn:arch OR dfn:Documentation OR dfn:include OR dfn:scripts) AND f:robh@kernel.org'
>
> Any way to make this work? I find that it's more easily readable than the
> "echo | lei q" version.
I can't think of a way to make it work w/o breaking phrase searches
(or asking users to use both single and double-quotes):
lei q 's:"a quick brown fox"' # yuck
> For bash users, the following should work as well:
>
> lei q -I https://lore.kernel.org/all/ -o ~/work/temp/lei/robh-patches \
> --threads --dedupe=mid <<< \
> '(dfn:drivers OR dfn:arch OR dfn:Documentation OR dfn:include OR dfn:scripts) AND f:robh@kernel.org'
Oh, not sure about bash and <<<; but this heredoc should work with any
POSIX sh:
lei q -I https://lore.kernel.org/all/ \
-o ~/work/temp/lei/robh-patches \
--threads --dedupe=mid <<'EOM'
(dfn:drivers OR dfn:arch OR dfn:Documentation OR dfn:include OR dfn:scripts)
AND f:robh@kernel.org
EOM
> Suggestion, can -I accept the URL containing the query, so that the command
> becomes:
>
> lei q -o ~/mail/foo --threads --dedupe=mid -I \
> https://lore.kernel.org/all/?q=f%3Atorvalds+AND+nq%3Agarbage
>
> This way we pass both the location of the extindex to query AND the parameters
> we should use, avoiding shell quoting problems?
Maybe, but URI escaping bothers the heck out of me, too.
I guess the heredoc example is actually good...
^ permalink raw reply [relevance 90%]
* Re: Failed 'lei q' blocks 'lei init' from working
2022-03-12 21:04 87% Failed 'lei q' blocks 'lei init' from working Nícolas F. R. A. Prado
@ 2022-03-13 0:06 90% ` Kyle Meyer
0 siblings, 0 replies; 200+ results
From: Kyle Meyer @ 2022-03-13 0:06 UTC (permalink / raw)
To: Nícolas F. R. A. Prado; +Cc: meta
Nícolas F. R. A. Prado writes:
> Hi,
>
> Right after building lei from source, I get an error running the 'q' command:
>
> [nfraprado@notapiano public-inbox]$ ./lei.sh q -I https://lore.kernel.org/all/ -o ~/.mail/lei/test --threads --dedupe=mid 'dfn:mt8192.dtsi rt:1.month.ago..'
> open(/home/nfraprado/.local/share/lei/store/mail_sync.sqlite3): No such file or directory at /home/nfraprado/ext/git/public-inbox/lib/PublicInbox/LeiQuery.pm line 37.
>
> That's somewhat reasonable since I haven't issued an init first. But if I then
> try to init [...]:
Hmm, an explicit lei-init call shouldn't be needed (i.e. lei-q should
already behave as you propose at the end of your message, initializing
the store/config if necessary).
Eric has even considered dropping lei-init entirely:
https://public-inbox.org/meta/20210325083207.GA30551@dcvr/
> Not quite sure if it's a Perl thing or a bug in public-inbox, but since it's
> unusual behavior (and took me some time to find out) I figured I'd report it.
This reminded me a bit of
https://public-inbox.org/meta/20211120153054.xq6jfad2m2buoic6@gmail.com/
Perhaps a lei-daemon process was running from some previous
experimentation?
^ permalink raw reply [relevance 90%]
* Re: [PATCH 4/4] lei reindex: new command to reindex lei/store
@ 2022-08-18 7:22 90% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2022-08-18 7:22 UTC (permalink / raw)
To: meta
Eric Wong <e@80x24.org> wrote:
> index d49746cb..277ed6bd 100644
> --- a/lib/PublicInbox/LeiStore.pm
> +++ b/lib/PublicInbox/LeiStore.pm
> @@ -335,6 +335,36 @@ sub _docids_and_maybe_kw ($$) {
> ($docids, [ sort keys %$kw ]);
> }
>
> +sub _reindex_1 { # git->cat_async callback
> + my ($bref, $hex, $type, $size, $smsg) = @_;
> + my ($self, $eidx, $tl) = delete @$smsg{qw(-self -eidx -tl)};
> + $bref //= _lms_rw($self)->local_blob($hex, 1);
> + if ($bref) {
> + my $eml = PublicInbox::Eml->new($bref);
> + $smsg->{-merge_vmd} = 1; # preserve existing keywords
> + $eidx->idx_shard($smsg->{num})->index_eml($eml, $smsg);
> + } else {
> + warn("E: $type $hex\n");
This path has been worrying me a bit, I hit it quite a bit on
one of my systems since there was a time when external-only
messages were fully-indexed inside lei/store. Nowadays,
duplicate indexing is avoided...
^ permalink raw reply [relevance 90%]
* [PATCH 1/4] lei reindex: account for parallel lei/store users
2022-08-19 9:07 71% [PATCH 0/4] lei reindex-related stuff Eric Wong
@ 2022-08-19 9:07 90% ` Eric Wong
2022-08-19 9:07 71% ` [PATCH 4/4] lei/store: reindex culls over-indexed messages Eric Wong
1 sibling, 0 replies; 200+ results
From: Eric Wong @ 2022-08-19 9:07 UTC (permalink / raw)
To: meta
We need to call eidx_init in each git->cat_async callback
since another requestor may've stopped the shard processes.
---
lib/PublicInbox/LeiStore.pm | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/lib/PublicInbox/LeiStore.pm b/lib/PublicInbox/LeiStore.pm
index 277ed6bd..8e710540 100644
--- a/lib/PublicInbox/LeiStore.pm
+++ b/lib/PublicInbox/LeiStore.pm
@@ -337,7 +337,8 @@ sub _docids_and_maybe_kw ($$) {
sub _reindex_1 { # git->cat_async callback
my ($bref, $hex, $type, $size, $smsg) = @_;
- my ($self, $eidx, $tl) = delete @$smsg{qw(-self -eidx -tl)};
+ my $self = delete $smsg->{-sto};
+ my ($eidx, $tl) = eidx_init($self);
$bref //= _lms_rw($self)->local_blob($hex, 1);
if ($bref) {
my $eml = PublicInbox::Eml->new($bref);
@@ -353,7 +354,7 @@ sub reindex_art {
my ($eidx, $tl) = eidx_init($self);
my $smsg = $eidx->{oidx}->get_art($art) // return;
return if $smsg->{bytes} == 0; # external-only message
- @$smsg{qw(-self -eidx -tl)} = ($self, $eidx, $tl);
+ $smsg->{-sto} = $self;
$eidx->git->cat_async($smsg->{blob} // die("no blob (#$art)"),
\&_reindex_1, $smsg);
}
^ permalink raw reply related [relevance 90%]
* [PATCH] lei up: improve error for multiple lei.q values
@ 2022-10-31 21:52 90% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2022-10-31 21:52 UTC (permalink / raw)
To: meta
Point users towards the lei.internal.rawstr variable which
may be tripping up handling of lei.q after `lei edit-search'.
---
lib/PublicInbox/LeiUp.pm | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/lib/PublicInbox/LeiUp.pm b/lib/PublicInbox/LeiUp.pm
index 5ad21451..49917339 100644
--- a/lib/PublicInbox/LeiUp.pm
+++ b/lib/PublicInbox/LeiUp.pm
@@ -32,8 +32,10 @@ sub up1 ($$) {
my $rawstr = $lss->{-cfg}->{'lei.internal.rawstr'} //
(scalar(@$q) == 1 && substr($q->[0], -1) eq "\n");
if ($rawstr) {
- scalar(@$q) > 1 and
- die "$f: lei.q has multiple values (@$q) (out=$out)\n";
+ die <<EOM if scalar(@$q) > 1;
+$f: lei.q has multiple values (@$q) (out=$out)
+$f: while lei.internal.rawstr is set
+EOM
$lse->query_approxidate($lse->git, $mset_opt->{qstr} = $q->[0]);
} else {
$mset_opt->{qstr} = $lse->query_argv_to_string($lse->git, $q);
^ permalink raw reply related [relevance 90%]
* [PATCH 6/6] txt2pre: linkify lei/store => lei-store-format.html
@ 2022-11-03 0:48 90% ` Eric Wong
2022-11-03 0:48 90% ` [PATCH 5/6] doc: lei-import: link to lei-store-format(5) Eric Wong
2022-11-03 0:48 71% ` [PATCH 3/6] doc: txt2pre: linkify "lei COMMAND" form Eric Wong
2 siblings, 0 replies; 200+ results
From: Eric Wong @ 2022-11-03 0:48 UTC (permalink / raw)
To: meta
Linking to the manpage probably helps clarify what `lei/store'
refers to without too much clutter in the raw POD source.
---
Documentation/txt2pre | 2 ++
1 file changed, 2 insertions(+)
diff --git a/Documentation/txt2pre b/Documentation/txt2pre
index b9d74fb7..62175f34 100755
--- a/Documentation/txt2pre
+++ b/Documentation/txt2pre
@@ -80,6 +80,8 @@ for (qw[lei(1)
/\Alei-(.+?)\(1\)\z/ and $xurls{"lei $1"} = "$n.html";
}
+$xurls{'lei/store'} = 'lei-store-format.html';
+
for (qw[make(1) flock(2) setrlimit(2) vfork(2) tmpfs(5) inotify(7) unix(7)
syslog(3)]) {
my ($n, $s) = (/([\w\-]+)\((\d)\)/);
^ permalink raw reply related [relevance 90%]
* [PATCH 5/6] doc: lei-import: link to lei-store-format(5)
2022-11-03 0:48 90% ` [PATCH 6/6] txt2pre: linkify lei/store => lei-store-format.html Eric Wong
@ 2022-11-03 0:48 90% ` Eric Wong
2022-11-03 2:03 90% ` Eric Wong
2022-11-03 0:48 71% ` [PATCH 3/6] doc: txt2pre: linkify "lei COMMAND" form Eric Wong
2 siblings, 1 reply; 200+ results
From: Eric Wong @ 2022-11-03 0:48 UTC (permalink / raw)
To: meta
Users should know where `lei import' writes to.
---
Documentation/lei-import.pod | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/Documentation/lei-import.pod b/Documentation/lei-import.pod
index 4ac7dccd..25ef75c3 100644
--- a/Documentation/lei-import.pod
+++ b/Documentation/lei-import.pod
@@ -10,7 +10,8 @@ lei import [OPTIONS] (--stdin|-)
=head1 DESCRIPTION
-Import messages into the local storage of L<lei(1)>. C<LOCATION> is a
+Import messages into the local storage of L<lei(1)>
+(aka L<lei/store|lei-store-format(5)>). C<LOCATION> is a
source of messages: a directory (Maildir), a file, or a URL
(C<imap://>, C<imaps://>, C<nntp://>, or C<nntps://>). URLs requiring
authentication use L<git-credential(1)> to
@@ -102,4 +103,4 @@ License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>
=head1 SEE ALSO
-L<lei-index(1)>
+L<lei-index(1)>, L<lei-store-format(5)>
^ permalink raw reply related [relevance 90%]
* Re: [PATCH 5/6] doc: lei-import: link to lei-store-format(5)
2022-11-03 0:48 90% ` [PATCH 5/6] doc: lei-import: link to lei-store-format(5) Eric Wong
@ 2022-11-03 2:03 90% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2022-11-03 2:03 UTC (permalink / raw)
To: meta
Eric Wong <e@80x24.org> wrote:
> +(aka L<lei/store|lei-store-format(5)>). C<LOCATION> is a
Can't have unescaped `/' like that, will squash this in:
diff --git a/Documentation/lei-import.pod b/Documentation/lei-import.pod
index 25ef75c3..69ec6497 100644
--- a/Documentation/lei-import.pod
+++ b/Documentation/lei-import.pod
@@ -11,7 +11,7 @@ lei import [OPTIONS] (--stdin|-)
=head1 DESCRIPTION
Import messages into the local storage of L<lei(1)>
-(aka L<lei/store|lei-store-format(5)>). C<LOCATION> is a
+(aka L<leiE<sol>store|lei-store-format(5)>). C<LOCATION> is a
source of messages: a directory (Maildir), a file, or a URL
(C<imap://>, C<imaps://>, C<nntp://>, or C<nntps://>). URLs requiring
authentication use L<git-credential(1)> to
^ permalink raw reply related [relevance 90%]
* [PATCH 0/2] lei - expanding relative paths for `lei up'
@ 2022-12-01 11:21 90% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2022-12-01 11:21 UTC (permalink / raw)
To: meta
I ran `lei q --only ./ -o $MAILDIR $QUERY' at some point from
inside an inboxdir.
Then I got confused when `lei up --all' running from $HOME
started complaining about $HOME/objects being missing from ALE.
It turned out $HOME/public-inbox (one of my worktrees) was
causing $HOME to false-positive as a v1 public-inbox for lei :x.
So this is a two-pronged fix to prevent some weird stuff from
happening.
Eric Wong (2):
lei: stricter external checks for valid $GIT_DIR/objects
lei_saved_search: expand only/include/exclude to absolute paths
lib/PublicInbox/LeiQuery.pm | 23 ++++++++++++++++++++---
lib/PublicInbox/LeiXSearch.pm | 14 ++++++++++----
t/lei-q-save.t | 13 +++++++++----
t/lei.t | 3 ++-
4 files changed, 41 insertions(+), 12 deletions(-)
^ permalink raw reply [relevance 90%]
* [PATCH 1/4] lei blob|rediff: fix usage of lei->fail
2023-09-22 21:13 71% [PATCH 0/4] small lei fixes Eric Wong
@ 2023-09-22 21:13 90% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2023-09-22 21:13 UTC (permalink / raw)
To: meta
lei->fail only takes one message argument, presently;
but it's probably a good idea to change the API...
---
lib/PublicInbox/LeiBlob.pm | 2 +-
lib/PublicInbox/LeiRediff.pm | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/lib/PublicInbox/LeiBlob.pm b/lib/PublicInbox/LeiBlob.pm
index 1692289c..5fc6d902 100644
--- a/lib/PublicInbox/LeiBlob.pm
+++ b/lib/PublicInbox/LeiBlob.pm
@@ -158,7 +158,7 @@ sub lei_blob {
if ($lxs->remotes) {
require PublicInbox::LeiRemote;
$lei->{curl} //= which('curl') or return
- $lei->fail('curl needed for', $lxs->remotes);
+ $lei->fail('curl needed for '.join(', ',$lxs->remotes));
$lei->_lei_store(1)->write_prepare($lei);
}
require PublicInbox::SolverGit;
diff --git a/lib/PublicInbox/LeiRediff.pm b/lib/PublicInbox/LeiRediff.pm
index c312d90f..9cf95c08 100644
--- a/lib/PublicInbox/LeiRediff.pm
+++ b/lib/PublicInbox/LeiRediff.pm
@@ -268,7 +268,7 @@ sub lei_rediff {
if ($lxs->remotes) {
require PublicInbox::LeiRemote;
$lei->{curl} //= which('curl') or return
- $lei->fail('curl needed for', $lxs->remotes);
+ $lei->fail('curl needed for '.join(', ',$lxs->remotes));
}
$lei->ale->refresh_externals($lxs, $lei);
my $self = bless {
^ permalink raw reply related [relevance 90%]
* [PATCH 5/8] lei: workers exit after they tell lei-daemon
@ 2023-10-03 6:43 90% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2023-10-03 6:43 UTC (permalink / raw)
To: meta
We don't want workers continuing after their stdout has triggered
EPIPE or some other write error.
This fixes xt/lei-onion-convert.t to ensure the quit_waiter_pipe
is fully-closed at daemon teardown during tests. Using the
`exit' perlop still ensures OnDestroy callbacks will fire.
---
lib/PublicInbox/LEI.pm | 1 +
1 file changed, 1 insertion(+)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 817772f7..10c08b90 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -485,6 +485,7 @@ sub x_it ($$) {
stop_pager($self);
if ($self->{pkt_op_p}) { # worker => lei-daemon
$self->{pkt_op_p}->pkt_do('x_it', $code);
+ exit($code >> 8);
} elsif ($self->{sock}) { # lei->daemon => lei(1) client
send($self->{sock}, "x_it $code", 0);
} elsif ($quit == \&CORE::exit) { # an admin (one-shot) command
^ permalink raw reply related [relevance 90%]
* [PATCH 09/10] lei: fix error reporting from lei/store -> lei-daemon
@ 2021-08-31 11:21 94% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-08-31 11:21 UTC (permalink / raw)
To: meta
We must set autoflush to ensure timely notification of clients;
and lei-daemon must not block when waiting on reads in case of
spurious wakeups.
Finally, if no clients are connected to lei-daemon, write to
syslog to ensure the error is visible.
---
lib/PublicInbox/LeiStore.pm | 2 ++
lib/PublicInbox/LeiStoreErr.pm | 14 ++++++++++++--
2 files changed, 14 insertions(+), 2 deletions(-)
diff --git a/lib/PublicInbox/LeiStore.pm b/lib/PublicInbox/LeiStore.pm
index 5a48c064..0652137e 100644
--- a/lib/PublicInbox/LeiStore.pm
+++ b/lib/PublicInbox/LeiStore.pm
@@ -28,6 +28,7 @@ use PublicInbox::Spawn qw(spawn);
use List::Util qw(max);
use File::Temp ();
use POSIX ();
+use IO::Handle (); # ->autoflush
sub new {
my (undef, $dir, $opt) = @_;
@@ -514,6 +515,7 @@ sub write_prepare {
$self->ipc_lock_init("$dir/ipc.lock");
substr($dir, -length('/lei/store'), 10, '');
pipe(my ($r, $w)) or die "pipe: $!";
+ $w->autoflush(1);
# Mail we import into lei are private, so headers filtered out
# by -mda for public mail are not appropriate
local @PublicInbox::MDA::BAD_HEADERS = ();
diff --git a/lib/PublicInbox/LeiStoreErr.pm b/lib/PublicInbox/LeiStoreErr.pm
index 68ce96d6..5f9ba24d 100644
--- a/lib/PublicInbox/LeiStoreErr.pm
+++ b/lib/PublicInbox/LeiStoreErr.pm
@@ -8,22 +8,32 @@ use strict;
use v5.10.1;
use parent qw(PublicInbox::DS);
use PublicInbox::Syscall qw(EPOLLIN EPOLLONESHOT);
+use Sys::Syslog qw(openlog syslog closelog);
+use IO::Handle (); # ->blocking
sub new {
my ($cls, $rd, $lei) = @_;
my $self = bless { sock => $rd, store_path => $lei->store_path }, $cls;
+ $rd->blocking(0);
$self->SUPER::new($rd, EPOLLIN | EPOLLONESHOT);
}
sub event_step {
my ($self) = @_;
- $self->do_read(\(my $rbuf), 4096) or return;
+ my $rbuf = $self->{rbuf} // \(my $x = '');
+ $self->do_read($rbuf, 8192, length($$rbuf)) or return;
my $cb;
+ my $printed;
for my $lei (values %PublicInbox::DS::DescriptorMap) {
$cb = $lei->can('store_path') // next;
next if $cb->($lei) ne $self->{store_path};
my $err = $lei->{2} // next;
- print $err $rbuf;
+ print $err $$rbuf and $printed = 1;
+ }
+ if (!$printed) {
+ openlog('lei-store', 'pid,nowait,nofatal,ndelay', 'user');
+ for my $l (split(/\n/, $$rbuf)) { syslog('warning', '%s', $l) }
+ closelog(); # don't share across fork
}
}
^ permalink raw reply related [relevance 94%]
* [PATCH] t/lei-mirror: skip lei comparisons if lei missing
@ 2021-11-22 7:42 96% ` Eric Wong
2021-11-22 13:47 99% ` Jörg Rödel
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2021-11-22 7:42 UTC (permalink / raw)
To: Jörg Rödel; +Cc: meta
Eric Wong <e@80x24.org> wrote:
> Will work on a patch to ensure lei ran successfully.
---------8<--------
Subject: [PATCH] t/lei-mirror: skip lei comparisons if lei missing
We can't compare created_at times with lei if lei tests are
skipped due to Inline::C or Socket::MsgHdr unavailability.
Reported-by: Jörg Rödel <joro@8bytes.org>
Link: https://public-inbox.org/meta/YZebmAxlFJy4lqAw@8bytes.org/
---
t/lei-mirror.t | 17 +++++++++++------
1 file changed, 11 insertions(+), 6 deletions(-)
diff --git a/t/lei-mirror.t b/t/lei-mirror.t
index 646ff2b19f98..d6fa6db5e3cb 100644
--- a/t/lei-mirror.t
+++ b/t/lei-mirror.t
@@ -170,12 +170,17 @@ SKIP: {
local $ENV{HOME} = $tmpdir;
ok(run_script([qw(-index -Lbasic), "$d/t1"]), 'index v1');
ok(run_script([qw(-index -Lbasic), "$d/t2"]), 'index v2');
- my $f = "$d/t1/public-inbox/msgmap.sqlite3";
- my $ca = PublicInbox::Msgmap->new_file($f)->created_at;
- is($ca, $created{v1}, 'clone + index v1 synced ->created_at');
- $f = "$d/t2/msgmap.sqlite3";
- $ca = PublicInbox::Msgmap->new_file($f)->created_at;
- is($ca, $created{v2}, 'clone + index v1 synced ->created_at');
+
+ SKIP: {
+ skip "lei didn't run", 2 if join(sort keys %created) ne 'v1v2';
+ my $f = "$d/t1/public-inbox/msgmap.sqlite3";
+ my $ca = PublicInbox::Msgmap->new_file($f)->created_at;
+ is($ca, $created{v1}, 'clone + index v1 synced ->created_at');
+
+ $f = "$d/t2/msgmap.sqlite3";
+ $ca = PublicInbox::Msgmap->new_file($f)->created_at;
+ is($ca, $created{v2}, 'clone + index v2 synced ->created_at');
+ }
test_lei(sub {
lei_ok qw(inspect num:1 --dir), "$d/t1";
ok(ref(json_utf8->decode($lei_out)), 'inspect num: on v1');
^ permalink raw reply related [relevance 96%]
* lei stuff that should be in a lei(1) or lei-overview(7)
@ 2021-02-18 20:28 99% Eric Wong
2021-02-22 3:42 99% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2021-02-18 20:28 UTC (permalink / raw)
To: meta
More random scattered thoughts
* Note bash completions in contrib/, encourage contributions for
other shells
* Note UI/UX is a deeply personal choice, lei can't satisfy everyone
* Note Inline::C + "mkdir -p ~/.cache/public-inbox/inline-c"
significantly improves performance. Socket::MsgHdr
(libsocket-msghdr-perl in Debian) further improves startup
performance (most noticeable for bash completion)
* Don't impose or aggressively promote lei in your projects or
communities. That defeats the point of open standards. Until
AGPL acceptance grows, it's likely many users can't use lei.
* Primary author is scatter-brained, and getting worse :<
More to come...
^ permalink raw reply [relevance 99%]
* Re: lei stuff that should be in a lei(1) or lei-overview(7)
2021-02-18 20:28 99% lei stuff that should be in a lei(1) or lei-overview(7) Eric Wong
@ 2021-02-22 3:42 99% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-02-22 3:42 UTC (permalink / raw)
To: meta
Eric Wong <e@80x24.org> wrote:
> More random scattered thoughts
If "lei" conflicts with an existing script or alias on your
system, consider naming it "lorelei"(*). Of course, the
shorter name is preferred to save keystrokes.
(*) partly named after a well-known instance of public-inbox
^ permalink raw reply [relevance 99%]
* Re: [PATCH] t/lei-mirror: skip lei comparisons if lei missing
2021-11-22 7:42 96% ` [PATCH] t/lei-mirror: skip lei comparisons if lei missing Eric Wong
@ 2021-11-22 13:47 99% ` Jörg Rödel
2021-11-22 17:24 99% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Jörg Rödel @ 2021-11-22 13:47 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
On Mon, Nov 22, 2021 at 07:42:41AM +0000, Eric Wong wrote:
> t/lei-mirror.t | 17 +++++++++++------
> 1 file changed, 11 insertions(+), 6 deletions(-)
This seems to cause warnings:
[ 56s] t/lei-lcat.t ................. ok
[ 57s] Useless use of sort in scalar context at t/lei-mirror.t line 175.
[ 59s] Use of uninitialized value in join or string at t/lei-mirror.t line 175.
[ 59s] t/lei-mirror.t ............... ok
Regards,
Joerg
^ permalink raw reply [relevance 99%]
* Re: [PATCH] t/lei-mirror: skip lei comparisons if lei missing
2021-11-22 13:47 99% ` Jörg Rödel
@ 2021-11-22 17:24 99% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-11-22 17:24 UTC (permalink / raw)
To: Jörg Rödel; +Cc: meta
Jörg Rödel <joro@8bytes.org> wrote:
> This seems to cause warnings:
>
> [ 56s] t/lei-lcat.t ................. ok
> [ 57s] Useless use of sort in scalar context at t/lei-mirror.t line 175.
> [ 59s] Use of uninitialized value in join or string at t/lei-mirror.t line 175.
> [ 59s] t/lei-mirror.t ............... ok
Oops :x I'll squash this in before pushing:
diff --git a/t/lei-mirror.t b/t/lei-mirror.t
index d6fa6db5e3cb..32a5b0390714 100644
--- a/t/lei-mirror.t
+++ b/t/lei-mirror.t
@@ -172,7 +172,8 @@ SKIP: {
ok(run_script([qw(-index -Lbasic), "$d/t2"]), 'index v2');
SKIP: {
- skip "lei didn't run", 2 if join(sort keys %created) ne 'v1v2';
+ join('', sort(keys %created)) eq 'v1v2' or
+ skip "lei didn't run", 2;
my $f = "$d/t1/public-inbox/msgmap.sqlite3";
my $ca = PublicInbox::Msgmap->new_file($f)->created_at;
is($ca, $created{v1}, 'clone + index v1 synced ->created_at');
^ permalink raw reply related [relevance 99%]
Results 1-200 of ~1311 next [<= 71%] | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2021-02-18 20:28 99% lei stuff that should be in a lei(1) or lei-overview(7) Eric Wong
2021-02-22 3:42 99% ` Eric Wong
2021-03-27 11:45 90% [PATCH 0/4] lei blob (formerly known as "lei show") Eric Wong
2021-03-27 11:45 90% ` [PATCH 2/4] lei help: move "lei help" into LeiHelp.pm Eric Wong
2022-12-01 11:21 90% [PATCH 0/2] lei - expanding relative paths for `lei up' Eric Wong
2022-10-31 21:52 90% [PATCH] lei up: improve error for multiple lei.q values Eric Wong
2021-04-28 4:51 90% [PATCH 0/3] doc: lei updates around lei-q Eric Wong
2021-04-13 10:54 90% [PATCH 0/5] "lei q --save" + "lei up" Eric Wong
2021-04-13 10:54 75% ` [PATCH 5/5] lei: add "lei up" to complement "lei q --save" Eric Wong
2021-08-31 19:38 90% [PATCH] lei up: only show finmsg in top-level lei-daemon Eric Wong
2021-04-24 9:28 89% [PATCH 0/7] lei sync preparations, "lei inspect" Eric Wong
2023-10-27 1:14 88% [PATCH] lei: don't exit lei-daemon on ovv_begin failure Eric Wong
2022-03-12 21:04 87% Failed 'lei q' blocks 'lei init' from working Nícolas F. R. A. Prado
2022-03-13 0:06 90% ` Kyle Meyer
2021-05-31 10:20 86% [PATCH] lei import: reduce writes to lei/store on IMAP sync Eric Wong
[not found] <lorelei.part1.202111051304.mdtebsxahljcrxak@meerkat.local>
[not found] ` <CAL_JsqJBh1O3H2-P07AHzVq0x89BoP_N6P=rT5up6=3QyF_B0Q@mail.gmail.com>
2021-11-08 20:22 86% ` lei: incorrect quoting on saved searches (was Re: lore+lei: getting started) Konstantin Ryabitsev
2021-11-08 20:53 90% ` Rob Herring
2021-11-08 20:49 90% ` Eric Wong
2021-11-08 21:36 90% ` Konstantin Ryabitsev
2021-11-08 21:48 90% ` Eric Wong
2021-11-08 22:36 88% ` Konstantin Ryabitsev
2021-11-08 22:57 90% ` Eric Wong
2021-02-04 2:10 83% [PATCH] t/lei: skip "lei q" tests on missing dependencies Eric Wong
2021-11-14 20:41 71% lei spawns mua before results are written Leah Neukirchen
2021-11-15 0:31 71% ` Eric Wong
2022-04-18 9:50 71% [PATCH 0/4] lei: finish wiring up pure-Perl stuff for Linux Eric Wong
2022-04-18 9:50 71% ` [PATCH 1/4] lei: clobber recvmsg buffer on errors Eric Wong
2021-11-19 20:49 71% RFC: should lei inject its own "Received:" header? Konstantin Ryabitsev
2021-11-21 10:13 71% ` Eric Wong
2023-09-30 16:17 71% [PATCH] t/lei-convert: fix uninitialized variable w/o pigz Eric Wong
2023-11-15 9:21 71% [PATCH 0/4] lei convert: support idempotent v2 outputs Eric Wong
2023-11-15 9:21 71% ` [PATCH 1/4] lei: fix idempotent STDERR redirect in workers Eric Wong
2023-03-23 22:05 71% repeat `lei import' users? Eric Wong
2021-11-10 10:33 71% [PATCH] t/lei-watch: test with with higher sleep Eric Wong
2021-11-03 1:28 71% Initial Fedora packaging for lei Konstantin Ryabitsev
2021-11-10 10:25 71% [PATCH 0/4] "lei q" rawstr fallout + fixes Eric Wong
2021-11-10 10:28 71% ` [PATCH 4/4] lei q: make HTTP(S) query strings even less ugly Eric Wong
2021-11-01 19:00 71% [PATCH] doc: lei-config: fix missing =back Eric Wong
2023-03-28 10:53 71% [PATCH] t/lei-refresh-mail-sync: improve test reliability Eric Wong
2023-10-17 10:11 71% [PATCH 0/3] lei: stdin handling improvements Eric Wong
2021-10-30 8:11 71% [PATCH 0/5] lei: fix various SIGPIPE problems Eric Wong
2021-10-30 8:11 71% ` [PATCH 5/5] doc: lei-security: add a note about core dumps Eric Wong
2021-10-30 8:11 71% ` [PATCH 1/5] lei: do not access {sock} after SIGPIPE Eric Wong
2023-12-16 11:13 71% [PATCH 0/2] lei bugfixes Eric Wong
2024-04-12 18:04 71% [PATCH 0/3] some lei fixes Eric Wong
2024-04-12 18:04 71% ` [PATCH 3/3] lei: remove leftover debugging message Eric Wong
2021-11-02 18:14 71% [PATCH 0/3] lei: more coherent input handling Eric Wong
2021-11-02 18:14 71% ` [PATCH 1/3] lei mail-diff: do not default to 'eml' Eric Wong
2021-11-02 18:14 ` [PATCH 2/3] lei: simplify common LeiInput users with ->wq1_start Eric Wong
2021-11-02 18:47 71% ` Eric Wong
2023-06-15 0:08 71% [PATCH] doc: lei q: document v2:$INBOX_DIR output format Eric Wong
2023-03-28 1:00 71% Issues with `lei` as non-root Louis DeLosSantos
2023-03-28 1:32 71% ` Eric Wong
2023-03-28 2:30 ` Louis DeLosSantos
2023-03-28 2:52 71% ` Eric Wong
2023-03-28 3:05 ` Louis DeLosSantos
2023-03-28 3:38 71% ` Eric Wong
2023-03-28 4:08 71% ` Louis DeLosSantos
2022-05-02 18:10 71% [PATCH] lei import: add label completions (+L:$LABEL) Eric Wong
2021-10-31 9:10 71% [PATCH 0/2] lei: mail-sync docs + compat fix Eric Wong
2021-10-31 9:10 ` [PATCH 1/2] doc: add lei-mail-sync-overview manpage Eric Wong
2021-10-31 16:32 71% ` Kyle Meyer
2023-04-29 7:18 71% [PATCH] t/lei-import-nntp: dump $lei_err on failure Eric Wong
2022-08-17 9:33 71% [PATCH 0/4] lei reindex, minor tweaks Eric Wong
2022-08-17 9:33 71% ` [PATCH 3/4] lei/store: reduce work when accessing mail_sync.sqlite3 Eric Wong
2022-08-17 9:33 71% ` [PATCH 2/4] lei inspect: less scary exception for invalid "docid:" inspect Eric Wong
2022-08-17 9:33 ` [PATCH 4/4] lei reindex: new command to reindex lei/store Eric Wong
2022-08-18 7:22 90% ` Eric Wong
2024-03-13 15:35 71% Lei exception Gonsolo
2024-03-13 19:20 71% ` Eric Wong
[not found] ` <CANL0fFSMQ1YL1a8PEpU39pYQ7d6vmmndughvJVue=SWNYNdqGQ@mail.gmail.com>
2024-03-14 14:46 71% ` Eric Wong
2024-03-15 17:36 71% ` Gonsolo
2022-07-07 9:40 71% [PATCH 0/2] lei: minor diagnostic improvement Eric Wong
2022-07-07 9:40 71% ` [PATCH 1/2] lei_xsearch: simplify lei/store import check Eric Wong
2023-09-30 0:36 71% [PATCH 0/2] lei: support reading inboxes & extindex w/o search Eric Wong
2022-04-30 21:04 71% [PATCH] lei: improve diagnosis of errors from children Eric Wong
2022-08-19 9:07 71% [PATCH 0/4] lei reindex-related stuff Eric Wong
2022-08-19 9:07 90% ` [PATCH 1/4] lei reindex: account for parallel lei/store users Eric Wong
2022-08-19 9:07 71% ` [PATCH 4/4] lei/store: reindex culls over-indexed messages Eric Wong
2022-09-02 18:26 71% [PATCH 0/2] lei-related import tweaks Eric Wong
2024-04-16 20:56 71% [PATCH 0/4] lei parallelism fixes Eric Wong
2023-09-14 12:12 71% [PATCH] t/lei-mirror: do not bail out on `make help' failure Eric Wong
2022-09-10 1:35 71% [PATCH] lei: fix --help for --jobs with `up' and `q' Eric Wong
2022-09-30 9:21 71% [PATCH 0/4] fixes noticed while diagnosing t/lei-up.t Eric Wong
2024-04-04 19:04 71% v1.9.0 : `ls-search' is not an lei command Josh Steadmon
2024-04-04 19:55 71% ` Eric Wong
2024-04-09 19:59 71% ` Josh Steadmon
2022-04-30 21:29 71% [PATCH] lei refresh-mail-sync: filter NNTP(S) from --all Eric Wong
2022-06-29 16:15 71% lei missing mails Rob Herring
2022-06-29 16:30 71% ` Eric Wong
2022-06-29 16:53 ` Rob Herring
2022-06-29 17:27 71% ` Eric Wong
2022-06-29 22:01 ` Rob Herring
2022-06-30 8:55 71% ` Eric Wong
2022-07-07 9:48 71% ` Eric Wong
2022-07-11 21:17 71% ` Rob Herring
2022-07-11 21:59 ` Rob Herring
2022-07-18 23:41 71% ` Eric Wong
2023-12-12 11:17 71% lei up without creating xapian database Aneesh Kumar K.V (IBM)
2023-12-12 11:41 71% ` Eric Wong
2021-11-12 11:08 71% [PATCH] lei forget-search: add help for --prune Eric Wong
2024-01-30 6:31 71% [PATCH 0/2] watch: add MH support + lei doc Eric Wong
2024-04-11 18:58 71% [PATCH] lei blob: fix attachment extraction for unimported||inflight Eric Wong
2023-09-22 21:13 71% [PATCH 0/4] small lei fixes Eric Wong
2023-09-22 21:13 90% ` [PATCH 1/4] lei blob|rediff: fix usage of lei->fail Eric Wong
2024-04-18 6:24 71% Sharing lei searches Gonsolo
2024-04-18 10:26 71% ` Eric Wong
2024-04-18 15:17 71% ` Gonsolo
2022-07-19 22:42 71% [PATCH 0/2] lei inotify/EVFILT_VNODE deadlock fix Eric Wong
2022-07-19 22:42 71% ` [PATCH 2/2] lei note-event: inline note_event_arm_done Eric Wong
2023-01-29 22:58 71% [PATCH 0/2] fix xt/lei-auth-fail.t Eric Wong
2021-11-22 18:38 71% [PATCH] lei: always use 3-arg open perlop Eric Wong
2024-01-10 11:18 71% [PATCH 0/3] lei NNTP + error handling fixes Eric Wong
2023-11-02 21:16 71% lei - dfn filters for net/* catching drivers/net/* David Wei
2023-11-02 21:27 71% ` Eric Wong
2023-11-03 18:29 71% ` David Wei
2023-06-08 18:26 71% [PATCH] t/lei.t: quiet newline warning on older Perls Eric Wong
2023-10-01 9:54 [PATCH 00/13] various warning/diagnostic fixes Eric Wong
2023-10-01 9:54 71% ` [PATCH 07/13] lei: correct exit signal Eric Wong
2023-10-01 9:54 71% ` [PATCH 12/13] lei: ->fail only allows integer exit codes Eric Wong
2023-10-01 9:54 71% ` [PATCH 08/13] lei mail-diff: don't remove temporary subdirectory Eric Wong
2023-10-03 6:43 [PATCH 0/8] IMAP/NNTP client improvements Eric Wong
2023-10-03 6:43 90% ` [PATCH 5/8] lei: workers exit after they tell lei-daemon Eric Wong
2021-05-17 3:35 [PATCH 0/9] doc: lei manpages, round 5 Kyle Meyer
2021-05-17 3:35 90% ` [PATCH 4/9] doc lei: resort lei-tag entries Kyle Meyer
2021-05-17 3:35 82% ` [PATCH 3/9] doc lei blob: point to lei-q for shared options Kyle Meyer
2021-10-25 22:24 [PATCH] t/index-git-times: support non-master default branch Thomas Weißschuh
2021-10-25 22:58 ` Eric Wong
2021-10-26 5:05 ` Thomas Weißschuh
2021-10-26 5:28 ` Eric Wong
2021-10-26 18:43 ` Thomas Weißschuh
2021-10-27 21:15 ` lei-q-remote-import failures [was: [PATCH] t/index-git-times: support non-master default branch] Eric Wong
2021-10-27 21:24 ` Thomas Weißschuh
2021-10-27 23:48 ` Eric Wong
2021-10-28 14:03 71% ` Thomas Weißschuh
2021-10-28 19:16 ` [PATCH] test_common: clear XDG_CACHE_HOME before lei tests Eric Wong
2021-10-28 19:22 71% ` Thomas Weißschuh
2023-10-04 3:49 [PATCH 00/21] lei + IPC related stuff Eric Wong
2023-10-04 3:49 71% ` [PATCH 10/21] lei: reuse PublicInbox::Config::noop Eric Wong
2023-11-11 22:44 [Bug] lei: extra quotes inserted into query with AND/OR Henrik Grimler
2023-11-12 0:10 71% ` Eric Wong
2023-11-12 8:23 71% ` Henrik Grimler
2023-11-12 9:02 ` Eric Wong
2023-11-12 11:59 71% ` Henrik Grimler
2023-11-12 13:24 71% ` Eric Wong
2021-02-09 8:09 [PATCH 00/11] Maildir code consolidation, test updates Eric Wong
2021-02-09 8:09 81% ` [PATCH 11/11] tests|lei: fixes for TEST_RUN_MODE=0 and lei oneshot Eric Wong
2021-10-19 9:33 [PATCH 00/11] refining lei up+inspect Eric Wong
2021-10-19 9:33 80% ` [PATCH 08/11] doc: lei: describe lei-daemon-kill and upgrades Eric Wong
2023-09-24 21:08 [PATCH 0/4] various CentOS 7.x related fixes Eric Wong
2023-09-24 21:08 71% ` [PATCH 3/4] lei: use scalar %SIG assignment Eric Wong
2021-12-08 1:07 Test failures with 1.7.0 Julien Moutinho
2021-12-08 4:08 ` Eric Wong
2021-12-08 10:56 ` Dominique Martinet
2021-12-09 1:37 ` Julien Moutinho
2022-02-17 21:02 ` [PATCH] t/lei-sigpipe: attempt to improve diagnostics for stuck test Eric Wong
2022-02-20 1:38 ` Julien Moutinho
2022-02-22 6:44 71% ` Eric Wong
2022-02-27 4:15 71% ` Julien Moutinho
2022-02-27 6:41 71% ` Julien Moutinho
2022-02-27 7:23 71% ` Dominique Martinet
2022-02-27 8:04 71% ` Julien Moutinho
2023-03-09 19:28 [PATCH 0/6] various doc updates Eric Wong
2023-03-09 19:28 71% ` [PATCH 4/6] doc: lei import: add hints about nntp.* and imap.* config options Eric Wong
2021-11-19 12:42 'make test' failures while packaging for openSUSE Jörg Rödel
2021-11-19 18:54 ` Eric Wong
2021-11-19 21:46 ` Jörg Rödel
2021-11-22 6:55 ` [PATCH] searchidx: add some extra diagnostics for odd message Eric Wong
2021-11-22 7:42 96% ` [PATCH] t/lei-mirror: skip lei comparisons if lei missing Eric Wong
2021-11-22 13:47 99% ` Jörg Rödel
2021-11-22 17:24 99% ` Eric Wong
2021-08-17 8:52 [PATCH 0/3] lei: some mail sync stuff Eric Wong
2021-08-17 8:52 88% ` [PATCH 3/3] lei forget-mail-sync: rely on lei/store process Eric Wong
2022-05-03 11:15 Trouble running lei Filipe Manana
2022-05-03 11:37 71% ` Eric Wong
2022-05-03 12:50 71% ` Filipe Manana
2022-05-03 15:24 71% ` Konstantin Ryabitsev
2022-05-03 20:32 71% ` Eric Wong
2021-03-29 3:11 [PATCH 0/8] doc: lei manpages, round 4 Kyle Meyer
2021-03-29 3:11 90% ` [PATCH 7/8] doc lei overview: note that lei-init is usually unnecessary Kyle Meyer
2021-02-01 5:57 [PATCH 0/2] doc: initial lei manpages Kyle Meyer
2021-02-01 5:57 ` [PATCH 1/2] doc: start manpages for lei commands Kyle Meyer
2021-02-06 9:01 90% ` lei-q doc thoughts... [was: doc: start manpages for lei commands] Eric Wong
2021-02-06 19:57 90% ` Kyle Meyer
2021-02-07 3:33 90% ` Eric Wong
2021-02-07 19:58 90% ` lei q --output vs --mfolder [was: [PATCH 1/2] " Eric Wong
2021-02-07 20:33 90% ` Kyle Meyer
2021-02-07 20:59 90% ` Eric Wong
2021-02-07 21:47 90% ` Kyle Meyer
2021-02-07 21:55 90% ` Eric Wong
2024-04-12 2:01 [PATCH] lei q: support --thread-id=$MSGID || -T $MSGID Eric Wong
2024-04-12 8:03 71% ` Štěpán Němec
2024-04-12 9:43 71% ` Eric Wong
2023-09-22 20:33 lei interactive TUIs (ncurses/vim/emacs) Eric Wong
2023-11-09 4:14 71% ` Kyle Meyer
2022-08-30 9:10 [PATCH 0/3] misc doc updates I missed for 1.9 :x Eric Wong
2022-08-30 9:10 71% ` [PATCH 1/3] Makefile.PL: add lei-reindex manpage Eric Wong
2021-04-19 8:52 [PATCH 0/6] lei saved search improvements Eric Wong
2021-04-19 8:52 73% ` [PATCH 5/6] lei_saved_search: split "lei q --save" and "lei up" init paths Eric Wong
2023-10-17 23:37 [PATCH 00/30] autodie-ification and code simplifications Eric Wong
2023-10-17 23:38 71% ` [PATCH 18/30] t/lei-up: additional diagnostics for match failures Eric Wong
2022-09-09 10:09 imap: "Can't use an undefined value as a subroutine reference" Ricardo Ribalda
2022-09-09 17:44 ` [PATCH] lei: add diagnostics for IMAP writer failures Eric Wong
2022-09-09 18:00 86% ` [PATCH] doc: document --jobs for `lei q' and `lei up' Eric Wong
2022-09-09 20:35 71% ` [PATCH] lei: add diagnostics for IMAP writer failures Ricardo Ribalda
2022-09-10 1:18 ` [PATCH v2] lei: bail out earlier on " Eric Wong
2022-09-10 19:34 ` Ricardo Ribalda
2022-09-10 19:50 71% ` Eric Wong
2022-09-10 19:53 ` Ricardo Ribalda
2022-09-10 20:19 71% ` Eric Wong
2021-04-16 23:10 [PATCH 0/9] lei saved search usability improvements Eric Wong
2021-04-16 23:10 90% ` [PATCH 8/9] lei q --save: avoid lei.q.format Eric Wong
2021-09-03 8:54 [PATCH 0/8] lei: fix IMAP R/W; L/kw false positives Eric Wong
2021-09-03 8:54 83% ` [PATCH 4/8] lei: use lei->lms in place of lse->lms in a few places Eric Wong
2023-10-11 7:20 [PATCH 0/9] lei + import-related updates Eric Wong
2023-10-11 7:20 86% ` [PATCH 8/9] lei blob: run cat_blob on lei/store for pending blobs Eric Wong
2023-02-13 16:06 lei q -tt doesn't work properly? Maxim Mikityanskiy
2023-02-14 2:42 ` [PATCH] lei q: do not collapse threads with `-tt' Eric Wong
2023-02-26 12:17 71% ` Maxim Mikityanskiy
2023-02-26 17:09 ` Eric Wong
2023-02-26 17:15 71% ` [PATCH] doc: note "lei q -tt" is broken with HTTP(S) remotes Eric Wong
2022-08-04 7:23 [PATCH] TODO: remove done items, adjust/add/abandon some Eric Wong
2022-12-09 1:41 ` FUSE3 vs read-write IMAP for lei Eric Wong
2023-02-20 19:27 71% ` Eric Wong
2022-10-30 4:03 [Need Help] lei add quotes at the search Hangbin Liu
2022-10-30 5:13 71% ` Eric Wong
2022-10-30 7:08 ` Hangbin Liu
2022-10-30 23:06 ` Eric Wong
2022-10-31 7:47 71% ` Hangbin Liu
2022-10-31 7:36 71% ` Hangbin Liu
2021-10-24 0:20 [PATCH 0/7] misc tweaks and fixes Eric Wong
2021-10-24 0:20 75% ` [PATCH 1/7] lei: always pass $lei to LeiAuth->op_merge Eric Wong
2023-10-16 11:33 [PATCH 1/3] doc: fix some typos and grammar Štěpán Němec
2023-10-16 11:33 71% ` [PATCH 2/3] doc: lei-q: drop stale TODO comment (fixed in 1f1b1f0e22f7) Štěpán Němec
2023-10-16 21:17 71% ` Eric Wong
2021-02-17 10:06 [PATCH 00/11] lei IMAP read support Eric Wong
2021-02-17 10:07 85% ` [PATCH 05/11] lei import: move check_input_format to lei Eric Wong
2021-11-02 23:55 [PATCH] doc: lei-q: document SEARCH TERMS prefixes Eric Wong
2021-11-03 0:28 71% ` Kyle Meyer
2021-11-20 15:30 How to delete caches Johannes Altmanninger
2021-11-20 16:36 71% ` lei q error after deleting cache (can't find mail_sync.sqlite3) Johannes Altmanninger
2021-05-25 11:01 [PATCH 0/3] lei forget-mail-sync: drop sync information Eric Wong
2021-05-25 11:01 90% ` [PATCH 1/3] lei export-kw: use lei->abs_path instead of rel2abs Eric Wong
2021-04-01 12:10 [PATCH 0/5] quieter and less noisy Eric Wong
2021-04-01 12:10 84% ` [PATCH 2/5] lei q: reduce lei/store work for kw changes to stored mail Eric Wong
2023-12-13 0:50 [PATCH 00/14] Alpine Linux support Eric Wong
2023-12-13 0:50 71% ` [PATCH 05/14] lei inspect: drop unneeded strftime import Eric Wong
2022-02-27 11:17 [PATCH] t/lei-sigpipe: ensure SIGPIPE is unblocked for this test Eric Wong
2022-03-11 10:42 ` [PATCH] t/lei-sigpipe.t: ensure SIGPIPE is not ignored instead of not blocked Julien Moutinho
2022-03-14 22:14 71% ` Eric Wong
2022-03-15 2:56 71% ` Julien Moutinho
2022-11-03 0:48 [PATCH 0/6] doc: linkify HTML harder Eric Wong
2022-11-03 0:48 90% ` [PATCH 6/6] txt2pre: linkify lei/store => lei-store-format.html Eric Wong
2022-11-03 0:48 90% ` [PATCH 5/6] doc: lei-import: link to lei-store-format(5) Eric Wong
2022-11-03 2:03 90% ` Eric Wong
2022-11-03 0:48 71% ` [PATCH 3/6] doc: txt2pre: linkify "lei COMMAND" form Eric Wong
2024-04-11 22:46 lei-up doesn't output replies to matching thread Josh Steadmon
2024-04-12 2:07 71% ` Eric Wong
2021-02-28 12:25 [PATCH 0/3] lei p2q (patch-to-query) Eric Wong
2021-02-28 12:25 ` [PATCH 1/3] lei p2q: patch-to-query generator for "lei q --stdin" Eric Wong
2021-02-28 21:40 90% ` Kyle Meyer
2021-05-05 10:46 [PATCH 0/2] lei rediff + solver-related fix Eric Wong
2021-05-05 10:46 87% ` [PATCH 2/2] lei blob: support "lei index"-ed mail Eric Wong
2024-02-09 16:06 lei up can't fetch new thread messages when searching by mid Pratyush Yadav
2024-02-09 17:35 71% ` Eric Wong
2024-02-12 11:11 71% ` Pratyush Yadav
2021-02-01 8:28 [PATCH 00/21] lei2mail worker segfault finally fixed Eric Wong
2021-02-01 8:28 82% ` [PATCH 08/21] lei: keep $lei around until workers are reaped Eric Wong
2021-08-31 11:21 [PATCH 00/10] lei: several bug fixes and refinements Eric Wong
2021-08-31 11:21 94% ` [PATCH 09/10] lei: fix error reporting from lei/store -> lei-daemon Eric Wong
2023-12-16 13:09 [PATCH] lei: support reading MH for convert+import+index Eric Wong
2023-12-16 16:15 71% ` Konstantin Ryabitsev
2023-12-16 18:17 71% ` Eric Wong
2021-10-21 21:10 [PATCH 00/15] use RENAME_NOREPLACE on Linux 3.15+ Eric Wong
2021-10-21 21:10 ` [PATCH 03/15] t/lei-p2q: extra diagnostics Eric Wong
2023-09-21 10:23 71% ` Eric Wong
2021-05-01 6:21 [PATCH 0/5] lei: more UI/UX tweaks Eric Wong
2021-05-01 6:21 75% ` [PATCH 5/5] lei edit-search: support relocating lei.q.output Eric Wong
2021-03-26 9:51 [PATCH 0/4] lei minor things Eric Wong
2021-03-26 9:51 90% ` [PATCH 2/4] lei: do not blindly commit to lei/store on close Eric Wong
2021-09-15 21:34 make menuconfig interface for lei / grok-pull Luis Chamberlain
2021-09-16 21:09 90% ` lei import on epochs [was: make menuconfig interface for lei / grok-pull] Eric Wong
2021-09-16 23:53 90% ` Luis Chamberlain
2021-09-17 0:22 89% ` Eric Wong
2021-09-17 0:48 90% ` Luis Chamberlain
2023-10-07 21:24 [PATCH 0/9] more process-related cleanups Eric Wong
2023-10-07 21:24 71% ` [PATCH 2/9] lei: do not issue sto->done if socket is inactive Eric Wong
2023-10-07 21:24 ` [PATCH 3/9] lei: always use async `done' requests to store Eric Wong
2023-10-08 1:58 71% ` Eric Wong
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).