unofficial mirror of meta@public-inbox.org
 help / color / mirror / Atom feed
* [PATCH 0/5] lei: TEST_LEI_DAEMON_PERSIST bugfixes
@ 2021-09-14  2:39 Eric Wong
  2021-09-14  2:39 ` [PATCH 1/5] lei: warn on event loop errors Eric Wong
                   ` (4 more replies)
  0 siblings, 5 replies; 6+ messages in thread
From: Eric Wong @ 2021-09-14  2:39 UTC (permalink / raw)
  To: meta

The 5/5 "lei up" fix has real-world implications if you're
using it in parallel.  Otherwise, it's test-only stuff, and
`make check-run TEST_LEI_DAEMON_PERSIST=1 N=$(nproc)`
seems to fail or stall less often than before (but it
still can...).

(persisting the lei-daemon is around 5% faster with check-run)

Eric Wong (5):
  lei: warn on event loop errors
  lei: sto_done_request: add eval guard
  t/run: TEST_LEI_DAEMON_PERSIST: die if pid changes
  test_common: remove non-hidden files, first
  lei up: fix env/cwd mismatches with multiple folders

 lib/PublicInbox/LEI.pm        | 33 ++++++++++++++++-----------------
 lib/PublicInbox/LeiUp.pm      | 15 +++++++--------
 lib/PublicInbox/TestCommon.pm |  5 ++++-
 t/run.perl                    |  3 +++
 4 files changed, 30 insertions(+), 26 deletions(-)

^ permalink raw reply	[flat|nested] 6+ messages in thread

* [PATCH 1/5] lei: warn on event loop errors
  2021-09-14  2:39 [PATCH 0/5] lei: TEST_LEI_DAEMON_PERSIST bugfixes Eric Wong
@ 2021-09-14  2:39 ` Eric Wong
  2021-09-14  2:39 ` [PATCH 2/5] lei: sto_done_request: add eval guard Eric Wong
                   ` (3 subsequent siblings)
  4 siblings, 0 replies; 6+ messages in thread
From: Eric Wong @ 2021-09-14  2:39 UTC (permalink / raw)
  To: meta

This should help us notice (and fix) bugs more easily.
---
 lib/PublicInbox/LEI.pm | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 784e679d..f0caac03 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -1347,7 +1347,8 @@ sub lazy_start {
 	open STDERR, '>&STDIN' or die "redirect stderr failed: $!";
 	open STDOUT, '>&STDIN' or die "redirect stdout failed: $!";
 	# $daemon pipe to `lei' closed, main loop begins:
-	PublicInbox::DS->EventLoop;
+	eval { PublicInbox::DS->EventLoop };
+	warn "event loop error: $@\n" if $@;
 	dump_and_clear_log();
 	exit($exit_code // 0);
 }

^ permalink raw reply	[flat|nested] 6+ messages in thread

* [PATCH 2/5] lei: sto_done_request: add eval guard
  2021-09-14  2:39 [PATCH 0/5] lei: TEST_LEI_DAEMON_PERSIST bugfixes Eric Wong
  2021-09-14  2:39 ` [PATCH 1/5] lei: warn on event loop errors Eric Wong
@ 2021-09-14  2:39 ` Eric Wong
  2021-09-14  2:39 ` [PATCH 3/5] t/run: TEST_LEI_DAEMON_PERSIST: die if pid changes Eric Wong
                   ` (2 subsequent siblings)
  4 siblings, 0 replies; 6+ messages in thread
From: Eric Wong @ 2021-09-14  2:39 UTC (permalink / raw)
  To: meta

Failures here can cause the lei-daemon event loop to break
since PktOp doesn't guard dispatch.  Add a guard here (and
not deeper in the stack) so we can use the $lei object to
report errors.
---
 lib/PublicInbox/LEI.pm | 15 +++++++++------
 1 file changed, 9 insertions(+), 6 deletions(-)

diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index f0caac03..e529c86a 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -1500,12 +1500,15 @@ sub lms { # read-only LeiMailSync
 
 sub sto_done_request { # only call this from lei-daemon process (not workers)
 	my ($lei, $sock) = @_;
-	if ($sock //= $lei->{sock}) {
-		$LIVE_SOCK{"$sock"} = $sock;
-		$lei->{sto}->ipc_do('done', "$sock"); # issue, async wait
-	} else { # forcibly wait
-		my $wait = $lei->{sto}->ipc_do('done');
-	}
+	eval {
+		if ($sock //= $lei->{sock}) { # issue, async wait
+			$LIVE_SOCK{"$sock"} = $sock;
+			$lei->{sto}->ipc_do('done', "$sock");
+		} else { # forcibly wait
+			my $wait = $lei->{sto}->ipc_do('done');
+		}
+	};
+	$lei->err($@) if $@;
 }
 
 sub sto_done_complete { # called in lei-daemon when LeiStore->done is complete

^ permalink raw reply	[flat|nested] 6+ messages in thread

* [PATCH 3/5] t/run: TEST_LEI_DAEMON_PERSIST: die if pid changes
  2021-09-14  2:39 [PATCH 0/5] lei: TEST_LEI_DAEMON_PERSIST bugfixes Eric Wong
  2021-09-14  2:39 ` [PATCH 1/5] lei: warn on event loop errors Eric Wong
  2021-09-14  2:39 ` [PATCH 2/5] lei: sto_done_request: add eval guard Eric Wong
@ 2021-09-14  2:39 ` Eric Wong
  2021-09-14  2:39 ` [PATCH 4/5] test_common: remove non-hidden files, first Eric Wong
  2021-09-14  2:39 ` [PATCH 5/5] lei up: fix env/cwd mismatches with multiple folders Eric Wong
  4 siblings, 0 replies; 6+ messages in thread
From: Eric Wong @ 2021-09-14  2:39 UTC (permalink / raw)
  To: meta

While persisting lei-daemon across different test cases isn't
the default anymore, we can notice problems more quickly if
the daemon PID changes since the daemon gets auto-restarted
after failures.
---
 t/run.perl | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/t/run.perl b/t/run.perl
index b0ee611a..e5ee0ade 100755
--- a/t/run.perl
+++ b/t/run.perl
@@ -254,5 +254,8 @@ for (my $i = $repeat; $i != 0; $i--) {
 print $OLDOUT "1..".($repeat * scalar(@tests))."\n" if $repeat >= 0;
 if ($lei_env && $$ == $owner_pid) {
 	my $opt = { 1 => $OLDOUT, 2 => $OLDERR };
+	my $cur_daemon_pid;
+	run_script([qw(lei daemon-pid)], $lei_env, { 1 => \$cur_daemon_pid });
 	run_script([qw(lei daemon-kill)], $lei_env, $opt);
+	DIE "lei daemon restarted\n" if $cur_daemon_pid != $lei_daemon_pid;
 }

^ permalink raw reply	[flat|nested] 6+ messages in thread

* [PATCH 4/5] test_common: remove non-hidden files, first
  2021-09-14  2:39 [PATCH 0/5] lei: TEST_LEI_DAEMON_PERSIST bugfixes Eric Wong
                   ` (2 preceding siblings ...)
  2021-09-14  2:39 ` [PATCH 3/5] t/run: TEST_LEI_DAEMON_PERSIST: die if pid changes Eric Wong
@ 2021-09-14  2:39 ` Eric Wong
  2021-09-14  2:39 ` [PATCH 5/5] lei up: fix env/cwd mismatches with multiple folders Eric Wong
  4 siblings, 0 replies; 6+ messages in thread
From: Eric Wong @ 2021-09-14  2:39 UTC (permalink / raw)
  To: meta

We want to remove any inotify-watched files before removing
~/.local/lei/store/ipc.lock, since sto_done_request was failing
on attempts to lock a non-existent lei/store/ipc.lock file.
---
 lib/PublicInbox/TestCommon.pm | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/lib/PublicInbox/TestCommon.pm b/lib/PublicInbox/TestCommon.pm
index d8346673..0ee4b228 100644
--- a/lib/PublicInbox/TestCommon.pm
+++ b/lib/PublicInbox/TestCommon.pm
@@ -567,7 +567,10 @@ SKIP: {
 		}
 		local $ENV{XDG_RUNTIME_DIR} = $daemon_xrd;
 		$cb->();
-		unless ($persist) {
+		if ($persist) { # remove before ~/.local gets removed
+			File::Path::rmtree([glob("$home/*")]);
+			File::Path::rmtree("$home/.config");
+		} else {
 			lei_ok(qw(daemon-pid), \"daemon-pid after $t");
 			chomp($daemon_pid = $lei_out);
 			if (!$daemon_pid) {

^ permalink raw reply	[flat|nested] 6+ messages in thread

* [PATCH 5/5] lei up: fix env/cwd mismatches with multiple folders
  2021-09-14  2:39 [PATCH 0/5] lei: TEST_LEI_DAEMON_PERSIST bugfixes Eric Wong
                   ` (3 preceding siblings ...)
  2021-09-14  2:39 ` [PATCH 4/5] test_common: remove non-hidden files, first Eric Wong
@ 2021-09-14  2:39 ` Eric Wong
  4 siblings, 0 replies; 6+ messages in thread
From: Eric Wong @ 2021-09-14  2:39 UTC (permalink / raw)
  To: meta

By moving %ENV localization and fchdir into ->dispatch,
we can maintain a consistent environment across multiple
dispatches while having different clients.
---
 lib/PublicInbox/LEI.pm   | 15 +++++----------
 lib/PublicInbox/LeiUp.pm | 15 +++++++--------
 2 files changed, 12 insertions(+), 18 deletions(-)

diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index e529c86a..0a30bc36 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -769,6 +769,8 @@ sub lazy_cb ($$$) {
 
 sub dispatch {
 	my ($self, $cmd, @argv) = @_;
+	fchdir($self) or return;
+	local %ENV = %{$self->{env}};
 	local $current_lei = $self; # for __WARN__
 	$self->{2}->autoflush(1); # keep stdout buffered until x_it|DESTROY
 	return _help($self, 'no command given') unless defined($cmd);
@@ -1104,14 +1106,9 @@ sub accept_dispatch { # Listener {post_accept} callback
 	my ($argc, @argv) = split(/\0/, $buf, -1);
 	undef $buf;
 	my %env = map { split(/=/, $_, 2) } splice(@argv, $argc);
-	if (chdir($self->{3})) {
-		local %ENV = %env;
-		$self->{env} = \%env;
-		eval { dispatch($self, @argv) };
-		send($sock, $@, MSG_EOR) if $@;
-	} else {
-		send($sock, "fchdir: $!", MSG_EOR); # implicit close
-	}
+	$self->{env} = \%env;
+	eval { dispatch($self, @argv) };
+	send($sock, $@, MSG_EOR) if $@;
 }
 
 sub dclose {
@@ -1181,7 +1178,6 @@ sub cfg2lei ($) {
 	open($lei->{1}, '>>&', \*STDOUT) or die "dup 1: $!";
 	open($lei->{2}, '>>&', \*STDERR) or die "dup 2: $!";
 	open($lei->{3}, '/') or die "open /: $!";
-	chdir($lei->{3}) or die "chdir /': $!";
 	my ($x, $y);
 	socketpair($x, $y, AF_UNIX, SOCK_SEQPACKET, 0) or die "socketpair: $!";
 	$lei->{sock} = $x;
@@ -1199,7 +1195,6 @@ sub dir_idle_handler ($) { # PublicInbox::DirIdle callback
 		for my $f (keys %{$MDIR2CFGPATH->{$mdir} // {}}) {
 			my $cfg = $PATH2CFG{$f} // next;
 			eval {
-				local %ENV = %{$cfg->{-env}};
 				my $lei = cfg2lei($cfg);
 				$lei->dispatch('note-event',
 						"maildir:$mdir", $nc, $bn, $fn);
diff --git a/lib/PublicInbox/LeiUp.pm b/lib/PublicInbox/LeiUp.pm
index 53f06dbc..4637cb46 100644
--- a/lib/PublicInbox/LeiUp.pm
+++ b/lib/PublicInbox/LeiUp.pm
@@ -11,7 +11,6 @@ use PublicInbox::LeiSavedSearch;
 use PublicInbox::DS;
 use PublicInbox::PktOp;
 use PublicInbox::LeiFinmsg;
-use PublicInbox::LEI;
 my $REMOTE_RE = qr!\A(?:imap|http)s?://!i; # http(s) will be for JMAP
 
 sub up1 ($$) {
@@ -60,19 +59,16 @@ sub up1_redispatch {
 	} else { # multiple outputs
 		$l = bless { %$lei }, ref($lei);
 		$l->{opt} = { %{$l->{opt}} }; # deep copy
+		delete $l->{opt}->{all};
 		delete $l->{sock}; # do not close
 		# make close($l->{1}) happy in lei->dclose
 		open my $fh, '>&', $l->{1} or
 			return $l->child_error(0, "dup: $!");
 		$l->{1} = $fh;
+		$l->qerr("# updating $out");
 	}
-	local $PublicInbox::LEI::current_lei = $l;
-	local %ENV = %{$l->{env}};
 	$l->{''} = $op_p; # daemon only ($l => $lei => script/lei)
-	eval {
-		$l->qerr("# updating $out");
-		up1($l, $out);
-	};
+	eval { $l->dispatch('up', $out) };
 	$lei->child_error(0, $@) if $@ || $l->{failed}; # lei->fail()
 }
 
@@ -94,7 +90,6 @@ sub lei_up {
 	my ($lei, @outs) = @_;
 	my $opt = $lei->{opt};
 	my $self = bless { -mail_sync => 1 }, __PACKAGE__;
-	$lei->{lse} = $lei->_lei_store(1)->write_prepare($lei)->search;
 	if (defined(my $all = $opt->{all})) {
 		return $lei->fail("--all and @outs incompatible") if @outs;
 		defined($opt->{mua}) and return
@@ -110,10 +105,14 @@ sub lei_up {
 		} else {
 			$lei->fail("only --all=$all not understood");
 		}
+	} elsif ($lei->{lse}) {
+		scalar(@outs) == 1 or die "BUG: lse set w/ >1 out[@outs]";
+		return up1($lei, $outs[0]);
 	} else {
 		$self->{remote} = [ grep(/$REMOTE_RE/, @outs) ];
 		$self->{local} = [ grep(!/$REMOTE_RE/, @outs) ];
 	}
+	$lei->{lse} = $lei->_lei_store(1)->write_prepare($lei)->search;
 	((@{$self->{local} // []} + @{$self->{remote} // []}) > 1 &&
 		defined($opt->{mua})) and return $lei->fail(<<EOM);
 multiple outputs and --mua= are incompatible

^ permalink raw reply	[flat|nested] 6+ messages in thread

end of thread, other threads:[~2021-09-14  2:39 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-09-14  2:39 [PATCH 0/5] lei: TEST_LEI_DAEMON_PERSIST bugfixes Eric Wong
2021-09-14  2:39 ` [PATCH 1/5] lei: warn on event loop errors Eric Wong
2021-09-14  2:39 ` [PATCH 2/5] lei: sto_done_request: add eval guard Eric Wong
2021-09-14  2:39 ` [PATCH 3/5] t/run: TEST_LEI_DAEMON_PERSIST: die if pid changes Eric Wong
2021-09-14  2:39 ` [PATCH 4/5] test_common: remove non-hidden files, first Eric Wong
2021-09-14  2:39 ` [PATCH 5/5] lei up: fix env/cwd mismatches with multiple folders Eric Wong

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).