unofficial mirror of meta@public-inbox.org
 help / color / mirror / Atom feed
From: Eric Wong <e@80x24.org>
To: meta@public-inbox.org
Subject: [PATCH 33/36] lei: avoid Spawn package when starting daemon
Date: Thu, 31 Dec 2020 13:51:51 +0000	[thread overview]
Message-ID: <20201231135154.6070-34-e@80x24.org> (raw)
In-Reply-To: <20201231135154.6070-1-e@80x24.org>

Spawn was designed to speed up process spawning inside
long-lived daemons with largish memory usage.  It does not help
for short-lived scripts which only exist to start and connect to
a daemon.

This change actually speeds up initial lei startup from
~190ms to ~140ms(!).  Normal usage once the daemon is running
is unaffected, at <20ms for help text.

While we're in the area, simplify Cwd error message generation,
too.
---
 lib/PublicInbox/LEI.pm | 10 +++++-----
 script/lei             | 17 ++++++-----------
 2 files changed, 11 insertions(+), 16 deletions(-)

diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 03302f8a..b84e24ef 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -675,7 +675,7 @@ sub lazy_start {
 	require IO::FDPass;
 	require PublicInbox::Listener;
 	require PublicInbox::EOFpipe;
-	(-p STDOUT && -p STDERR) or die "E: stdout+stderr must be pipes\n";
+	(-p STDOUT) or die "E: stdout must be a pipe\n";
 	open(STDIN, '+<', '/dev/null') or die "redirect stdin failed: $!";
 	POSIX::setsid() > 0 or die "setsid: $!";
 	my $pid = fork // die "fork: $!";
@@ -740,10 +740,9 @@ sub lazy_start {
 		$n; # true: continue, false: stop
 	});
 
-	# STDIN was redirected to /dev/null above, closing STDOUT and
-	# STDERR will cause the calling `lei' client process to finish
-	# reading <$daemon> pipe.
-	open STDOUT, '>&STDIN' or die "redirect stdout failed: $!";
+	# STDIN was redirected to /dev/null above, closing STDERR and
+	# STDOUT will cause the calling `lei' client process to finish
+	# reading the <$daemon> pipe.
 	openlog($path, 'pid', 'user');
 	local $SIG{__WARN__} = sub { syslog('warning', "@_") };
 	my $owner_pid = $$;
@@ -751,6 +750,7 @@ sub lazy_start {
 		syslog('crit', "$@") if $@ && $$ == $owner_pid;
 	});
 	open STDERR, '>&STDIN' or die "redirect stderr failed: $!";
+	open STDOUT, '>&STDIN' or die "redirect stdout failed: $!";
 	# $daemon pipe to `lei' closed, main loop begins:
 	PublicInbox::DS->EventLoop;
 	@$on_destroy = (); # cancel on_destroy if we get here
diff --git a/script/lei b/script/lei
index ceaf1e00..0457adfd 100755
--- a/script/lei
+++ b/script/lei
@@ -21,18 +21,13 @@ if (my ($sock, $pwd) = eval {
 	my $addr = pack_sockaddr_un($path);
 	socket(my $sock, AF_UNIX, SOCK_STREAM, 0) or die "socket: $!";
 	unless (connect($sock, $addr)) { # start the daemon if not started
-		my $cmd = [ $^X, qw[-MPublicInbox::LEI
+		local $ENV{PERL5LIB} = join(':', @INC);
+		open(my $daemon, '-|', $^X, qw[-MPublicInbox::LEI
 			-E PublicInbox::LEI::lazy_start(@ARGV)],
-			$path, $! + 0 ];
-		my $env = { PERL5LIB => join(':', @INC) };
-		pipe(my ($daemon, $w)) or die "pipe: $!";
-		my $opt = { 1 => $w, 2 => $w };
-		require PublicInbox::Spawn;
-		my $pid = PublicInbox::Spawn::spawn($cmd, $env, $opt);
-		$opt = $w = undef;
+			$path, $! + 0) or die "popen: $!";
 		while (<$daemon>) { warn $_ } # EOF when STDERR is redirected
-		waitpid($pid, 0) or warn <<"";
-lei-daemon could not start, PID:$pid exited with \$?=$?
+		close($daemon) or warn <<"";
+lei-daemon could not start, exited with \$?=$?
 
 		# try connecting again anyways, unlink+bind may be racy
 		unless (connect($sock, $addr)) {
@@ -43,8 +38,8 @@ Falling back to (slow) one-shot mode
 		}
 	}
 	require Cwd;
-	my $cwd = Cwd::fastcwd() // die "fastcwd(PWD=".($ENV{PWD}//'').": $!";
 	my $pwd = $ENV{PWD} // '';
+	my $cwd = Cwd::fastcwd() // die "fastcwd(PWD=$pwd): $!";
 	if ($pwd ne $cwd) { # prefer ENV{PWD} if it's a symlink to real cwd
 		my @st_cwd = stat($cwd) or die "stat(cwd=$cwd): $!";
 		my @st_pwd = stat($pwd); # PWD invalid, use cwd

  parent reply	other threads:[~2020-12-31 13:51 UTC|newest]

Thread overview: 37+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-12-31 13:51 [PATCH 00/36] another round of lei stuff Eric Wong
2020-12-31 13:51 ` [PATCH 01/36] import: respect init.defaultBranch Eric Wong
2020-12-31 13:51 ` [PATCH 02/36] lei_store: use per-machine refname as git HEAD Eric Wong
2020-12-31 13:51 ` [PATCH 03/36] revert "lei_store: use per-machine refname as git HEAD" Eric Wong
2020-12-31 13:51 ` [PATCH 04/36] lei_to_mail: initial implementation for writing mbox formats Eric Wong
2020-12-31 13:51 ` [PATCH 05/36] sharedkv: fork()-friendly key-value store Eric Wong
2020-12-31 13:51 ` [PATCH 06/36] sharedkv: split out index_values Eric Wong
2020-12-31 13:51 ` [PATCH 07/36] lei_to_mail: start atomic and compressed mbox writing Eric Wong
2020-12-31 13:51 ` [PATCH 08/36] mboxreader: new class for reading various mbox formats Eric Wong
2020-12-31 13:51 ` [PATCH 09/36] lei_to_mail: start --augment, dedupe, bz2 and xz Eric Wong
2020-12-31 13:51 ` [PATCH 10/36] lei: implement various deduplication strategies Eric Wong
2020-12-31 13:51 ` [PATCH 11/36] lei_to_mail: lazy-require LeiDedupe Eric Wong
2020-12-31 13:51 ` [PATCH 12/36] lei_to_mail: support for non-seekable outputs Eric Wong
2020-12-31 13:51 ` [PATCH 13/36] lei_to_mail: support Maildir, fix+test --augment Eric Wong
2020-12-31 13:51 ` [PATCH 14/36] ipc: generic IPC dispatch based on Storable Eric Wong
2020-12-31 13:51 ` [PATCH 15/36] ipc: support Sereal Eric Wong
2020-12-31 13:51 ` [PATCH 16/36] lei_store: add ->set_eml, ->add_eml can return smsg Eric Wong
2020-12-31 13:51 ` [PATCH 17/36] lei: rename "extinbox" => "external" Eric Wong
2020-12-31 13:51 ` [PATCH 18/36] mid: use defined-or with `push' for uniqueness check Eric Wong
2020-12-31 13:51 ` [PATCH 19/36] mid: hoist out mids_in sub Eric Wong
2020-12-31 13:51 ` [PATCH 20/36] lei_store: handle messages without Message-ID at all Eric Wong
2020-12-31 13:51 ` [PATCH 21/36] ipc: use shutdown(2), base atfork* callback Eric Wong
2020-12-31 13:51 ` [PATCH 22/36] lei_to_mail: unlink mboxes if not augmenting Eric Wong
2020-12-31 13:51 ` [PATCH 23/36] lei: add --mfolder as an --output alias Eric Wong
2020-12-31 13:51 ` [PATCH 24/36] spawn: move run_die here from PublicInbox::Import Eric Wong
2020-12-31 13:51 ` [PATCH 25/36] init: remove embedded UnlinkMe package Eric Wong
2020-12-31 13:51 ` [PATCH 26/36] t/run: avoid uninitialized var on incomplete test Eric Wong
2020-12-31 13:51 ` [PATCH 27/36] gcf2client: reap process on DESTROY Eric Wong
2020-12-31 13:51 ` [PATCH 28/36] lei_to_mail: open FIFOs O_WRONLY so we block Eric Wong
2020-12-31 13:51 ` [PATCH 29/36] searchidxshard: call DS->Reset at worker start Eric Wong
2020-12-31 13:51 ` [PATCH 30/36] t/ipc.t: test for references via `die' Eric Wong
2020-12-31 13:51 ` [PATCH 31/36] use PublicInbox::DS for dwaitpid Eric Wong
2020-12-31 13:51 ` [PATCH 32/36] syscall: SFD_NONBLOCK can be a constant, again Eric Wong
2020-12-31 13:51 ` Eric Wong [this message]
2020-12-31 13:51 ` [PATCH 34/36] avoid calling waitpid from children in DESTROY Eric Wong
2020-12-31 13:51 ` [PATCH 35/36] ds: clobber $in_loop first at reset Eric Wong
2020-12-31 13:51 ` [PATCH 36/36] on_destroy: support PID owner guard Eric Wong

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: https://public-inbox.org/README

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20201231135154.6070-34-e@80x24.org \
    --to=e@80x24.org \
    --cc=meta@public-inbox.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).