* [PATCH 0/5] lei: TEST_LEI_DAEMON_PERSIST bugfixes @ 2021-09-14 2:39 Eric Wong 2021-09-14 2:39 ` [PATCH 1/5] lei: warn on event loop errors Eric Wong ` (4 more replies) 0 siblings, 5 replies; 6+ messages in thread From: Eric Wong @ 2021-09-14 2:39 UTC (permalink / raw) To: meta The 5/5 "lei up" fix has real-world implications if you're using it in parallel. Otherwise, it's test-only stuff, and `make check-run TEST_LEI_DAEMON_PERSIST=1 N=$(nproc)` seems to fail or stall less often than before (but it still can...). (persisting the lei-daemon is around 5% faster with check-run) Eric Wong (5): lei: warn on event loop errors lei: sto_done_request: add eval guard t/run: TEST_LEI_DAEMON_PERSIST: die if pid changes test_common: remove non-hidden files, first lei up: fix env/cwd mismatches with multiple folders lib/PublicInbox/LEI.pm | 33 ++++++++++++++++----------------- lib/PublicInbox/LeiUp.pm | 15 +++++++-------- lib/PublicInbox/TestCommon.pm | 5 ++++- t/run.perl | 3 +++ 4 files changed, 30 insertions(+), 26 deletions(-) ^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH 1/5] lei: warn on event loop errors 2021-09-14 2:39 [PATCH 0/5] lei: TEST_LEI_DAEMON_PERSIST bugfixes Eric Wong @ 2021-09-14 2:39 ` Eric Wong 2021-09-14 2:39 ` [PATCH 2/5] lei: sto_done_request: add eval guard Eric Wong ` (3 subsequent siblings) 4 siblings, 0 replies; 6+ messages in thread From: Eric Wong @ 2021-09-14 2:39 UTC (permalink / raw) To: meta This should help us notice (and fix) bugs more easily. --- lib/PublicInbox/LEI.pm | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm index 784e679d..f0caac03 100644 --- a/lib/PublicInbox/LEI.pm +++ b/lib/PublicInbox/LEI.pm @@ -1347,7 +1347,8 @@ sub lazy_start { open STDERR, '>&STDIN' or die "redirect stderr failed: $!"; open STDOUT, '>&STDIN' or die "redirect stdout failed: $!"; # $daemon pipe to `lei' closed, main loop begins: - PublicInbox::DS->EventLoop; + eval { PublicInbox::DS->EventLoop }; + warn "event loop error: $@\n" if $@; dump_and_clear_log(); exit($exit_code // 0); } ^ permalink raw reply related [flat|nested] 6+ messages in thread
* [PATCH 2/5] lei: sto_done_request: add eval guard 2021-09-14 2:39 [PATCH 0/5] lei: TEST_LEI_DAEMON_PERSIST bugfixes Eric Wong 2021-09-14 2:39 ` [PATCH 1/5] lei: warn on event loop errors Eric Wong @ 2021-09-14 2:39 ` Eric Wong 2021-09-14 2:39 ` [PATCH 3/5] t/run: TEST_LEI_DAEMON_PERSIST: die if pid changes Eric Wong ` (2 subsequent siblings) 4 siblings, 0 replies; 6+ messages in thread From: Eric Wong @ 2021-09-14 2:39 UTC (permalink / raw) To: meta Failures here can cause the lei-daemon event loop to break since PktOp doesn't guard dispatch. Add a guard here (and not deeper in the stack) so we can use the $lei object to report errors. --- lib/PublicInbox/LEI.pm | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm index f0caac03..e529c86a 100644 --- a/lib/PublicInbox/LEI.pm +++ b/lib/PublicInbox/LEI.pm @@ -1500,12 +1500,15 @@ sub lms { # read-only LeiMailSync sub sto_done_request { # only call this from lei-daemon process (not workers) my ($lei, $sock) = @_; - if ($sock //= $lei->{sock}) { - $LIVE_SOCK{"$sock"} = $sock; - $lei->{sto}->ipc_do('done', "$sock"); # issue, async wait - } else { # forcibly wait - my $wait = $lei->{sto}->ipc_do('done'); - } + eval { + if ($sock //= $lei->{sock}) { # issue, async wait + $LIVE_SOCK{"$sock"} = $sock; + $lei->{sto}->ipc_do('done', "$sock"); + } else { # forcibly wait + my $wait = $lei->{sto}->ipc_do('done'); + } + }; + $lei->err($@) if $@; } sub sto_done_complete { # called in lei-daemon when LeiStore->done is complete ^ permalink raw reply related [flat|nested] 6+ messages in thread
* [PATCH 3/5] t/run: TEST_LEI_DAEMON_PERSIST: die if pid changes 2021-09-14 2:39 [PATCH 0/5] lei: TEST_LEI_DAEMON_PERSIST bugfixes Eric Wong 2021-09-14 2:39 ` [PATCH 1/5] lei: warn on event loop errors Eric Wong 2021-09-14 2:39 ` [PATCH 2/5] lei: sto_done_request: add eval guard Eric Wong @ 2021-09-14 2:39 ` Eric Wong 2021-09-14 2:39 ` [PATCH 4/5] test_common: remove non-hidden files, first Eric Wong 2021-09-14 2:39 ` [PATCH 5/5] lei up: fix env/cwd mismatches with multiple folders Eric Wong 4 siblings, 0 replies; 6+ messages in thread From: Eric Wong @ 2021-09-14 2:39 UTC (permalink / raw) To: meta While persisting lei-daemon across different test cases isn't the default anymore, we can notice problems more quickly if the daemon PID changes since the daemon gets auto-restarted after failures. --- t/run.perl | 3 +++ 1 file changed, 3 insertions(+) diff --git a/t/run.perl b/t/run.perl index b0ee611a..e5ee0ade 100755 --- a/t/run.perl +++ b/t/run.perl @@ -254,5 +254,8 @@ for (my $i = $repeat; $i != 0; $i--) { print $OLDOUT "1..".($repeat * scalar(@tests))."\n" if $repeat >= 0; if ($lei_env && $$ == $owner_pid) { my $opt = { 1 => $OLDOUT, 2 => $OLDERR }; + my $cur_daemon_pid; + run_script([qw(lei daemon-pid)], $lei_env, { 1 => \$cur_daemon_pid }); run_script([qw(lei daemon-kill)], $lei_env, $opt); + DIE "lei daemon restarted\n" if $cur_daemon_pid != $lei_daemon_pid; } ^ permalink raw reply related [flat|nested] 6+ messages in thread
* [PATCH 4/5] test_common: remove non-hidden files, first 2021-09-14 2:39 [PATCH 0/5] lei: TEST_LEI_DAEMON_PERSIST bugfixes Eric Wong ` (2 preceding siblings ...) 2021-09-14 2:39 ` [PATCH 3/5] t/run: TEST_LEI_DAEMON_PERSIST: die if pid changes Eric Wong @ 2021-09-14 2:39 ` Eric Wong 2021-09-14 2:39 ` [PATCH 5/5] lei up: fix env/cwd mismatches with multiple folders Eric Wong 4 siblings, 0 replies; 6+ messages in thread From: Eric Wong @ 2021-09-14 2:39 UTC (permalink / raw) To: meta We want to remove any inotify-watched files before removing ~/.local/lei/store/ipc.lock, since sto_done_request was failing on attempts to lock a non-existent lei/store/ipc.lock file. --- lib/PublicInbox/TestCommon.pm | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/lib/PublicInbox/TestCommon.pm b/lib/PublicInbox/TestCommon.pm index d8346673..0ee4b228 100644 --- a/lib/PublicInbox/TestCommon.pm +++ b/lib/PublicInbox/TestCommon.pm @@ -567,7 +567,10 @@ SKIP: { } local $ENV{XDG_RUNTIME_DIR} = $daemon_xrd; $cb->(); - unless ($persist) { + if ($persist) { # remove before ~/.local gets removed + File::Path::rmtree([glob("$home/*")]); + File::Path::rmtree("$home/.config"); + } else { lei_ok(qw(daemon-pid), \"daemon-pid after $t"); chomp($daemon_pid = $lei_out); if (!$daemon_pid) { ^ permalink raw reply related [flat|nested] 6+ messages in thread
* [PATCH 5/5] lei up: fix env/cwd mismatches with multiple folders 2021-09-14 2:39 [PATCH 0/5] lei: TEST_LEI_DAEMON_PERSIST bugfixes Eric Wong ` (3 preceding siblings ...) 2021-09-14 2:39 ` [PATCH 4/5] test_common: remove non-hidden files, first Eric Wong @ 2021-09-14 2:39 ` Eric Wong 4 siblings, 0 replies; 6+ messages in thread From: Eric Wong @ 2021-09-14 2:39 UTC (permalink / raw) To: meta By moving %ENV localization and fchdir into ->dispatch, we can maintain a consistent environment across multiple dispatches while having different clients. --- lib/PublicInbox/LEI.pm | 15 +++++---------- lib/PublicInbox/LeiUp.pm | 15 +++++++-------- 2 files changed, 12 insertions(+), 18 deletions(-) diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm index e529c86a..0a30bc36 100644 --- a/lib/PublicInbox/LEI.pm +++ b/lib/PublicInbox/LEI.pm @@ -769,6 +769,8 @@ sub lazy_cb ($$$) { sub dispatch { my ($self, $cmd, @argv) = @_; + fchdir($self) or return; + local %ENV = %{$self->{env}}; local $current_lei = $self; # for __WARN__ $self->{2}->autoflush(1); # keep stdout buffered until x_it|DESTROY return _help($self, 'no command given') unless defined($cmd); @@ -1104,14 +1106,9 @@ sub accept_dispatch { # Listener {post_accept} callback my ($argc, @argv) = split(/\0/, $buf, -1); undef $buf; my %env = map { split(/=/, $_, 2) } splice(@argv, $argc); - if (chdir($self->{3})) { - local %ENV = %env; - $self->{env} = \%env; - eval { dispatch($self, @argv) }; - send($sock, $@, MSG_EOR) if $@; - } else { - send($sock, "fchdir: $!", MSG_EOR); # implicit close - } + $self->{env} = \%env; + eval { dispatch($self, @argv) }; + send($sock, $@, MSG_EOR) if $@; } sub dclose { @@ -1181,7 +1178,6 @@ sub cfg2lei ($) { open($lei->{1}, '>>&', \*STDOUT) or die "dup 1: $!"; open($lei->{2}, '>>&', \*STDERR) or die "dup 2: $!"; open($lei->{3}, '/') or die "open /: $!"; - chdir($lei->{3}) or die "chdir /': $!"; my ($x, $y); socketpair($x, $y, AF_UNIX, SOCK_SEQPACKET, 0) or die "socketpair: $!"; $lei->{sock} = $x; @@ -1199,7 +1195,6 @@ sub dir_idle_handler ($) { # PublicInbox::DirIdle callback for my $f (keys %{$MDIR2CFGPATH->{$mdir} // {}}) { my $cfg = $PATH2CFG{$f} // next; eval { - local %ENV = %{$cfg->{-env}}; my $lei = cfg2lei($cfg); $lei->dispatch('note-event', "maildir:$mdir", $nc, $bn, $fn); diff --git a/lib/PublicInbox/LeiUp.pm b/lib/PublicInbox/LeiUp.pm index 53f06dbc..4637cb46 100644 --- a/lib/PublicInbox/LeiUp.pm +++ b/lib/PublicInbox/LeiUp.pm @@ -11,7 +11,6 @@ use PublicInbox::LeiSavedSearch; use PublicInbox::DS; use PublicInbox::PktOp; use PublicInbox::LeiFinmsg; -use PublicInbox::LEI; my $REMOTE_RE = qr!\A(?:imap|http)s?://!i; # http(s) will be for JMAP sub up1 ($$) { @@ -60,19 +59,16 @@ sub up1_redispatch { } else { # multiple outputs $l = bless { %$lei }, ref($lei); $l->{opt} = { %{$l->{opt}} }; # deep copy + delete $l->{opt}->{all}; delete $l->{sock}; # do not close # make close($l->{1}) happy in lei->dclose open my $fh, '>&', $l->{1} or return $l->child_error(0, "dup: $!"); $l->{1} = $fh; + $l->qerr("# updating $out"); } - local $PublicInbox::LEI::current_lei = $l; - local %ENV = %{$l->{env}}; $l->{''} = $op_p; # daemon only ($l => $lei => script/lei) - eval { - $l->qerr("# updating $out"); - up1($l, $out); - }; + eval { $l->dispatch('up', $out) }; $lei->child_error(0, $@) if $@ || $l->{failed}; # lei->fail() } @@ -94,7 +90,6 @@ sub lei_up { my ($lei, @outs) = @_; my $opt = $lei->{opt}; my $self = bless { -mail_sync => 1 }, __PACKAGE__; - $lei->{lse} = $lei->_lei_store(1)->write_prepare($lei)->search; if (defined(my $all = $opt->{all})) { return $lei->fail("--all and @outs incompatible") if @outs; defined($opt->{mua}) and return @@ -110,10 +105,14 @@ sub lei_up { } else { $lei->fail("only --all=$all not understood"); } + } elsif ($lei->{lse}) { + scalar(@outs) == 1 or die "BUG: lse set w/ >1 out[@outs]"; + return up1($lei, $outs[0]); } else { $self->{remote} = [ grep(/$REMOTE_RE/, @outs) ]; $self->{local} = [ grep(!/$REMOTE_RE/, @outs) ]; } + $lei->{lse} = $lei->_lei_store(1)->write_prepare($lei)->search; ((@{$self->{local} // []} + @{$self->{remote} // []}) > 1 && defined($opt->{mua})) and return $lei->fail(<<EOM); multiple outputs and --mua= are incompatible ^ permalink raw reply related [flat|nested] 6+ messages in thread
end of thread, other threads:[~2021-09-14 2:39 UTC | newest] Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2021-09-14 2:39 [PATCH 0/5] lei: TEST_LEI_DAEMON_PERSIST bugfixes Eric Wong 2021-09-14 2:39 ` [PATCH 1/5] lei: warn on event loop errors Eric Wong 2021-09-14 2:39 ` [PATCH 2/5] lei: sto_done_request: add eval guard Eric Wong 2021-09-14 2:39 ` [PATCH 3/5] t/run: TEST_LEI_DAEMON_PERSIST: die if pid changes Eric Wong 2021-09-14 2:39 ` [PATCH 4/5] test_common: remove non-hidden files, first Eric Wong 2021-09-14 2:39 ` [PATCH 5/5] lei up: fix env/cwd mismatches with multiple folders Eric Wong
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).