[PATCH 0/5] more deps and updates from portability

[PATCH 0/5] more deps and updates from portability

[Eric Wong]

install/deps.perl could become a useful tool for new users
since basically every tool is optional.

CentOS partitions the Perl standard library into having
circular dependencies which make no sense to me the more
I look at it.

5/5 was found to only be a problem my NetBSD VM.

Eric Wong (5):
  INSTALL: update for 2023, NetBSD and OpenBSD
  move deps.perl into new install/ directory
  updates around RPM packages on CentOS 7.x
  install/deps: safer defaults for regular users
  t/xap_helper: improve reliability of TTIN/TTOU tests

 INSTALL                   |  27 +++---
 MANIFEST                  |   4 +-
 Makefile.PL               |  25 +++---
 ci/README                 |  15 ++--
 ci/profiles.perl          |  66 +-------------
 ci/run.sh                 |   3 +-
 install/README            |  12 +++
 {ci => install}/deps.perl | 182 ++++++++++++++++++++++++--------------
 install/os.perl           |  78 ++++++++++++++++
 t/xap_helper.t            |  14 +--
 10 files changed, 252 insertions(+), 174 deletions(-)
 create mode 100644 install/README
 rename {ci => install}/deps.perl (53%)
 create mode 100644 install/os.perl

[PATCH 1/5] INSTALL: update for 2023, NetBSD and OpenBSD

[Eric Wong]

We'll also note that curl is used by lei and -clone.
---
 INSTALL | 24 +++++++++++++-----------
 1 file changed, 13 insertions(+), 11 deletions(-)

diff --git a/INSTALL b/INSTALL
index 13a1bdc6..4f473b78 100644
--- a/INSTALL
+++ b/INSTALL
@@ -5,17 +5,17 @@ This is for folks who want to set up their own public-inbox instance.
 Clients should use normal git-clone/git-fetch, IMAP or NNTP clients
 if they want to import mail into their personal inboxes.
 
-As of 2022, public-inbox is packaged by several OS distributions,
+As of 2023, public-inbox is packaged by several OS distributions,
 listed in alphabetical order: Debian, GNU Guix, NixOS, and Void Linux.
 
 public-inbox is developed on Debian GNU/Linux systems and will
 never depend on packages outside of the "main" component of
-the "stable" distribution, currently Debian 10.x ("buster"),
-but older versions of Debian remain supported.
+the "oldstable" distribution, currently Debian 11.x ("bullseye"),
+but older versions of Debian remain supported (as are newer ones).
 
-Most packages are available in other GNU/Linux distributions
-and FreeBSD.  CentOS 7.x users will likely want newer git and
-Xapian packages for better performance and v2 inbox support:
+Most packages are available in other GNU/Linux distributions,
+FreeBSD, NetBSD, and OpenBSD.  CentOS 7.x users will likely want
+newer git and Xapian for better performance and v2 inbox support:
 https://public-inbox.org/meta/20210421151308.yz5hzkgm75klunpe@nitro.local/
 
 TODO: this still needs to be documented better,
@@ -57,7 +57,7 @@ but required for older releases:
                                    (for broken, mostly historical emails)
 
 Where "deb" indicates package names for Debian-derived distributions,
-"pkg" is for the FreeBSD package (maybe other common BSDs, too), and
+"pkg" is for the FreeBSD package (and some other common BSDs, too), and
 "rpm" is for RPM-based distributions (only known to work on Fedora).
 
 Numerous optional modules are likely to be useful as well:
@@ -71,7 +71,7 @@ Numerous optional modules are likely to be useful as well:
                                    pkg: p5-Xapian (FreeBSD, NetBSD)
                                         xapian-bindings-perl (OpenBSD)
                                    rpm: perl-Search-Xapian
-                                   (HTTP and IMAP search)
+                                   (lei; HTTP and IMAP search)
 
 - Inline::C                        deb: libinline-c-perl
                                    pkg: p5-Inline-C
@@ -94,7 +94,8 @@ Numerous optional modules are likely to be useful as well:
 - Mail::IMAPClient                 deb: libmail-imapclient-perl
                                    pkg: p5-Mail-IMAPClient
                                    rpm: perl-Mail-IMAPClient
-                                   (only for lei and public-inbox-watch)
+                                   (only for lei and public-inbox-watch
+                                    when reading from IMAP)
 
 - BSD::Resource                    deb: libbsd-resource-perl
                                    pkg: p5-BSD-Resource
@@ -115,10 +116,11 @@ Numerous optional modules are likely to be useful as well:
                                    pkg: xapian-core
                                    rpm: xapian-core
                                    (for public-inbox-compact(1) and
-				    public-inbox-cindex(1))
+                                    public-inbox-cindex(1))
 
 * curl (tool)                      deb, pkg, rpm: curl
-                                   (for HTTP(S) externals with curl)
+                                   (for lei HTTP(S) externals with curl and
+                                    public-inbox-clone(1))
 
 - Linux::Inotify2                  deb: liblinux-inotify2-perl
                                    rpm: perl-Linux-Inotify2

[PATCH 2/5] move deps.perl into new install/ directory

[Eric Wong]

deps.perl can be useful for non-CI purposes as long as it's not
blindly removing packages.  Thus, a --allow-remove flag now
exists for CI use and removals are disabled by default.

deps.perl also gets easier-to-use in that now install/os.perl
is split off from from ci/profiles.perl so OS-supplied packaged
manager.
---
 MANIFEST                  |  4 +-
 ci/README                 | 15 +++-----
 ci/profiles.perl          | 66 +--------------------------------
 ci/run.sh                 |  2 +-
 install/README            | 12 ++++++
 {ci => install}/deps.perl | 52 ++++++++++++++++----------
 install/os.perl           | 78 +++++++++++++++++++++++++++++++++++++++
 7 files changed, 134 insertions(+), 95 deletions(-)
 create mode 100644 install/README
 rename {ci => install}/deps.perl (84%)
 create mode 100644 install/os.perl

diff --git a/MANIFEST b/MANIFEST
index 63287bad..21f718ec 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -112,7 +112,6 @@ TODO
 certs/.gitignore
 certs/create-certs.perl
 ci/README
-ci/deps.perl
 ci/profiles.perl
 ci/run.sh
 contrib/completion/lei-completion.bash
@@ -153,6 +152,9 @@ examples/unsubscribe-psgi@.service
 examples/unsubscribe.milter
 examples/unsubscribe.psgi
 examples/varnish-4.vcl
+install/README
+install/deps.perl
+install/os.perl
 lei.sh
 lib/PublicInbox/Address.pm
 lib/PublicInbox/AddressPP.pm
diff --git a/ci/README b/ci/README
index 728d82a0..c57c510c 100644
--- a/ci/README
+++ b/ci/README
@@ -2,9 +2,10 @@ various scripts for automated testing in chroots/VMs/jails
 
 TL;DR: ./ci/run.sh
 
-By default, `sudo' is used to install/uninstall packages.  It may be
-overridden with the `SUDO' environment variable.  These scripts should
-run in the top-level source tree, that is, as `./ci/run.sh'.
+By default, `sudo' is used to run install/deps.perl to install/uninstall
+packages.  It may be overridden with the `SUDO' environment variable.
+These scripts should run in the top-level source tree, that is, as
+`./ci/run.sh'.
 
 * ci/run.sh - runs tests against all profiles for the current OS
 
@@ -19,15 +20,11 @@ run in the top-level source tree, that is, as `./ci/run.sh'.
 	* PERL - default: "perl"
 	* SUDO - default: "sudo"
 
-* ci/deps.perl - script to mass-install/remove packages (requires root/sudo)
+* install/deps.perl - see install/README
 
 	Called automatically by ci/run.sh
 
-	There is no need to run this manually unless you are debugging
-	or doing development.  However, it can be convenient to for
-	users to mass-install several packages.
-
-* ci/profiles.sh - prints to-be-tested package profile for the current OS
+* ci/profiles.perl - prints to-be-tested package profile for the current OS
 
 	Called automatically by ci/run.sh
 	The output is read by ci/run.sh
diff --git a/ci/profiles.perl b/ci/profiles.perl
index e18f01fa..6f90a0e4 100755
--- a/ci/profiles.perl
+++ b/ci/profiles.perl
@@ -5,69 +5,7 @@
 eval 'exec perl -wS $0 ${1+"$@"}' # no shebang
 if 0; # running under some shell
 use v5.12;
-our ($ID, $PRETTY_NAME, $VERSION_ID); # same vars as os-release(5)
-my ($release, $version); # from uname
-if ($^O eq 'linux') { # try using os-release(5)
-	for my $f (qw(/etc/os-release /usr/lib/os-release)) {
-		next unless -f $f;
-		my @echo = map {
-			qq{echo "\$"$_" = qq[\$$_];"; }
-		} qw(ID PRETTY_NAME VERSION_ID);
-		# rely on sh(1) to handle interpolation and such:
-		my $vars = `sh -c '. $f; @echo'`;
-		die "sh \$?=$?" if $?;
-		eval $vars;
-		die $@ if $@;
-		$VERSION_ID //= '';
-		$ID //= '';
-		if ($ID eq 'debian') {
-			if ($PRETTY_NAME =~ m!/sid\z!) {
-				$VERSION_ID = 'sid';
-			} else {
-				open my $fh, '<', $f or die "open($f): $!";
-				my $msg = do { local $/; <$fh> };
-				die <<EOM;
-ID=$ID, but no VERSION_ID
-==> $f <==
-$msg
-EOM
-			}
-		}
-		last if $ID ne '' && $VERSION_ID ne '';
-	}
-	$ID = 'linux' if $ID eq ''; # cf. os-release(5)
-} elsif ($^O =~ m!\A(?:free|net|open)bsd\z!) { # TODO: net? dragonfly?
-	$ID = $^O;
-	require POSIX;
-	(undef, undef, $release, $version) = POSIX::uname();
-	$VERSION_ID = lc $release;
-	$VERSION_ID =~ s/[^0-9a-z\.\_\-]//sg; # cf. os-release(5)
-} else { # only support POSIX-like and Free systems:
-	die "$^O unsupported";
-}
-$VERSION_ID //= 0; # numeric? could be 'sid', actually...
-my %MIN_VER = (freebsd => v11, openbsd => v7.3, netbsd => v9.3);
-
-if (defined(my $min_ver = $MIN_VER{$^O})) {
-	my $vid = $VERSION_ID;
-	$vid =~ s/-.*\z//s; # no dashes in v-strings
-	my $vstr = eval "v$vid";
-	die "can't convert VERSION_ID=$VERSION_ID to v-string" if $@;
-	die <<EOM if $vstr lt $min_ver;
-ID=$ID VERSION_ID=$VERSION_ID release=$release ($version) too old to support
-EOM
-}
-my $PKG_FMT = do {
-	if ($ID eq 'freebsd') { 'pkg' }
-	# *shrug*, as long as the (Net|Open)BSD names don't conflict w/ FreeBSD
-	elsif ($ID eq 'netbsd') { 'pkgin' }
-	elsif ($ID eq 'openbsd') { 'pkg_add' }
-	elsif ($ID =~ m!\A(?:debian|ubuntu)\z!) { 'deb' }
-	elsif ($ID =~ m!\A(?:centos|redhat|fedora)\z!) { 'rpm' }
-	else { die "PKG_FMT undefined for ID=$ID" }
-};
-
-# these package group names and '-' syntax are passed to ci/deps.perl
+BEGIN { require './install/os.perl' }
 my $TASKS = do {
 	if ($ID =~ /\A(?:free|net|open)bsd\z/) { <<EOM
 all devtest Xapian-
@@ -92,5 +30,5 @@ EOM
 	} else { die "TODO: support ID=$ID VERSION_ID=$VERSION_ID" }
 };
 
-$TASKS =~ s/^/$PKG_FMT /gms;
+# this output is read by ci/run.sh and fed to install/deps.perl:
 print $TASKS;
diff --git a/ci/run.sh b/ci/run.sh
index 8f717508..5aa22491 100755
--- a/ci/run.sh
+++ b/ci/run.sh
@@ -15,7 +15,7 @@ NPROC=${NPROC-$({ getconf _NPROCESSORS_ONLN || getconf NPROCESSORS_ONLN ||
 
 $PERL -w ci/profiles.perl | while read args
 do
-	$DO $SUDO $PERL -w ci/deps.perl $args
+	$DO $SUDO $PERL -w install/deps.perl --allow-remove $args
 	$DO $PERL Makefile.PL
 	$DO $MAKE -j${BUILD_JOBS-$NPROC}
 	$DO $MAKE -j${TEST_JOBS-1} ${TEST_TARGET-test}
diff --git a/install/README b/install/README
new file mode 100644
index 00000000..981e014b
--- /dev/null
+++ b/install/README
@@ -0,0 +1,12 @@
+tooling for mass package installation
+-------------------------------------
+
+TL;DR:	sudo ./install/deps.perl all
+
+Files in this directory are designed for:
+
+* users using from git or tarballs (and not the OS package manager)
+
+* lazy users who can't be bothered to read all of INSTALL
+
+* automated testing scripts (see ci/README)
diff --git a/ci/deps.perl b/install/deps.perl
similarity index 84%
rename from ci/deps.perl
rename to install/deps.perl
index 3fa8f642..e888cf30 100755
--- a/ci/deps.perl
+++ b/install/deps.perl
@@ -1,14 +1,32 @@
-#!/usr/bin/perl -w
 # Copyright (C) all contributors <meta@public-inbox.org>
 # License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
-# Helper script for installing/uninstalling packages for CI use
-# Intended for use on non-production chroots or VMs since it
-# changes installed packages
+# Helper script for mass installing/uninstalling with the OS package manager
+eval 'exec perl -S $0 ${1+"$@"}' # no shebang
+if 0; # running under some shell
 use v5.12;
-my $usage = "$0 PKG_FMT PROFILE [PROFILE_MOD]";
-my $pkg_fmt = shift;
-@ARGV or die $usage, "\n";
+my $help = <<EOM;
+usage: $^X $0 [-f PKG_FMT] [--allow-remove] PROFILE [PROFILE_MOD]
 
+  -f PKG_FMT      package format (`deb', `pkg', `pkg_add', `pkgin' or `rpm')
+  --allow-remove  allow removing packages (for development use only)
+  --dry-run | -n  show commands that would be run
+
+PROFILE is typically `all'.  Other profiles are subject to change.
+PROFILE_MOD is only for developers checking dependencies
+
+OS package installation typically requires administrative privileges
+EOM
+use Getopt::Long qw(:config gnu_getopt no_ignore_case auto_abbrev);
+BEGIN { require './install/os.perl' };
+my $opt = {};
+GetOptions($opt, qw(pkg-fmt|f=s allow-remove dry-run|n help|h)) or die $help;
+if ($opt->{help}) { print $help; exit }
+my $pkg_fmt = $opt->{'pkg-fmt'} // do {
+	my $fmt = pkg_fmt;
+	warn "# using detected --pkg-fmt=$fmt on $ID/$VERSION_ID\n";
+	$fmt;
+};
+@ARGV or die $help;
 my @test_essential = qw(Test::Simple); # we actually use Test::More
 
 # package profiles.  Note we specify packages at maximum granularity,
@@ -125,19 +143,14 @@ for (qw(Digest::SHA Encode ExtUtils::MakeMaker IO::Compress Test::Simple)) {
 }
 
 # NetBSD and OpenBSD package names are similar to FreeBSD in most cases
-if ($pkg_fmt eq 'pkg_add') {
+if ($pkg_fmt =~ /\A(?:pkg_add|pkgin)\z/) {
 	for my $name (keys %$non_auto) {
 		my $fbsd_pkg = $non_auto->{$name}->{pkg};
-		$non_auto->{$name}->{pkg_add} //= $fbsd_pkg if $fbsd_pkg;
-	}
-} elsif ($pkg_fmt eq 'pkgin') {
-	for my $name (keys %$non_auto) {
-		my $fbsd_pkg = $non_auto->{$name}->{pkg};
-		$non_auto->{$name}->{pkgin} //= $fbsd_pkg if $fbsd_pkg;
+		$non_auto->{$name}->{$pkg_fmt} //= $fbsd_pkg if $fbsd_pkg;
 	}
 }
 
-my %inst_check = (
+my %inst_check = ( # subs which return true if a package is intalled
 	pkg => sub { system(qw(pkg info -q), $_[0]) == 0 },
 	deb => sub { system("dpkg -s $_[0] >/dev/null 2>&1") == 0 },
 	pkg_add => sub { system(qw(pkg_info -q -e), "$_[0]->=0") == 0 },
@@ -173,14 +186,13 @@ while (my ($pkg, $dst_pkg_list) = each %all) {
 }
 
 my %inst = map { $_ => 1 } @pkg_install;
-@pkg_remove = grep { !$inst{$_} } @pkg_remove;
+@pkg_remove = $opt->{'allow-remove'} ? grep { !$inst{$_} } @pkg_remove : ();
 @pkg_install = grep { !$INST_CHECK->($_) } @pkg_install;
 
 my @apt_opts =
 	qw(-o APT::Install-Recommends=false -o APT::Install-Suggests=false);
 
 # OS-specific cleanups appreciated
-
 if ($pkg_fmt eq 'deb') {
 	my @quiet = $ENV{V} ? () : ('-q');
 	root('apt-get', @apt_opts, qw(install --purge -y), @quiet,
@@ -282,10 +294,10 @@ sub exclude_uninstalled {
 }
 
 sub root {
-	print join(' ', @_), "\n";
-	return if $ENV{DRY_RUN};
+	warn "# @_\n";
+	return if $opt->{'dry-run'};
 	return if system(@_) == 0;
-	warn 'command failed: ', join(' ', @_), "\n";
+	warn "E: command failed: @_\n";
 	exit($? >> 8);
 }
 
diff --git a/install/os.perl b/install/os.perl
new file mode 100644
index 00000000..4fcbcbe4
--- /dev/null
+++ b/install/os.perl
@@ -0,0 +1,78 @@
+# Copyright (C) all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+
+# Helper library for detecting distro info and mapping to package manager.
+# This should NOT be installed via `make install'.
+# This is used by install/deps.perl and ci/profiles.perl
+package PublicInbox::InstallOS;
+use v5.12;
+use parent qw(Exporter);
+our ($ID, $PRETTY_NAME, $VERSION_ID); # same vars as os-release(5)
+our @EXPORT = qw($ID $VERSION_ID pkg_fmt);
+
+my ($release, $version); # from uname
+if ($^O eq 'linux') { # try using os-release(5)
+	for my $f (qw(/etc/os-release /usr/lib/os-release)) {
+		next unless -f $f;
+		my @echo = map {
+			qq{echo "\$"$_" = qq[\$$_];"; }
+		} qw(ID PRETTY_NAME VERSION_ID);
+		# rely on sh(1) to handle interpolation and such:
+		my $vars = `sh -c '. $f; @echo'`;
+		die "sh \$?=$?" if $?;
+		eval $vars;
+		die $@ if $@;
+		$VERSION_ID //= '';
+		$ID //= '';
+		if ($ID eq 'debian' && $VERSION_ID eq '') {
+			if ($PRETTY_NAME =~ m!/sid\z!) {
+				$VERSION_ID = 'sid';
+			} else {
+				open my $fh, '<', $f or die "open($f): $!";
+				my $msg = do { local $/; <$fh> };
+				die <<EOM;
+ID=$ID, but no VERSION_ID
+==> $f <==
+$msg
+EOM
+			}
+		}
+		last if $ID ne '' && $VERSION_ID ne '';
+	}
+	$ID = 'linux' if $ID eq ''; # cf. os-release(5)
+} elsif ($^O =~ m!\A(?:free|net|open)bsd\z!) { # TODO: net? dragonfly?
+	$ID = $^O;
+	require POSIX;
+	(undef, undef, $release, $version) = POSIX::uname();
+	$VERSION_ID = lc $release;
+	$VERSION_ID =~ s/[^0-9a-z\.\_\-]//sg; # cf. os-release(5)
+} else { # only support POSIX-like and Free systems:
+	die "$^O unsupported";
+}
+$VERSION_ID //= 0; # numeric? could be 'sid', actually...
+my %MIN_VER = (freebsd => v11, openbsd => v7.3, netbsd => v9.3);
+
+if (defined(my $min_ver = $MIN_VER{$^O})) {
+	my $vid = $VERSION_ID;
+	$vid =~ s/-.*\z//s; # no dashes in v-strings
+	my $vstr = eval "v$vid";
+	die "can't convert VERSION_ID=$VERSION_ID to v-string" if $@;
+	die <<EOM if $vstr lt $min_ver;
+ID=$ID VERSION_ID=$VERSION_ID release=$release ($version) too old to support
+EOM
+}
+
+sub pkg_fmt () {
+	if ($ID eq 'freebsd') { 'pkg' }
+	# *shrug*, as long as the (Net|Open)BSD names don't conflict w/ FreeBSD
+	elsif ($ID eq 'netbsd') { 'pkgin' }
+	elsif ($ID eq 'openbsd') { 'pkg_add' }
+	elsif ($ID =~ m!\A(?:debian|ubuntu)\z!) { 'deb' }
+	elsif ($ID =~ m!\A(?:centos|redhat|fedora)\z!) { 'rpm' }
+	else { warn "PKG_FMT undefined for ID=$ID"; undef }
+}
+
+package main;
+PublicInbox::InstallOS->import;
+
+1;

[PATCH 3/5] updates around RPM packages on CentOS 7.x

[Eric Wong]

Add explicit dependencies on packages that are possible to
remove without removing the `perl' RPM from CentOS 7.x.
However, simplify deps.perl to avoid naming essential Perl
modules (e.g. `perl-Exporter') that cannot be removed
without removing the `perl' RPM entirely.

We can also drop `Socket6' from deps.perl since it's pulled
in as-needed by other packages and we try to avoid it in favor
of the (now-standard) IO::Socket::IP.
---
 INSTALL           |  3 +--
 Makefile.PL       | 25 ++++++++++++++-----------
 install/deps.perl | 18 ++++++++++++------
 3 files changed, 27 insertions(+), 19 deletions(-)

diff --git a/INSTALL b/INSTALL
index 4f473b78..52bc9447 100644
--- a/INSTALL
+++ b/INSTALL
@@ -208,9 +208,8 @@ RPM-based distros split them out into separate packages:
 * autodie                          rpm: perl-autodie
 * Digest::SHA                      rpm: perl-Digest-SHA
 * Data::Dumper                     rpm: perl-Data-Dumper
-* Encode                           rpm: perl-Encode
 * IO::Compress                     rpm: perl-IO-Compress
-* Storable                         rpm: perl-Storable
+* Sys::Syslog                      rpm: perl-Sys-Syslog
 * Text::ParseWords                 rpm: perl-Text-Parsewords
 
 Copyright
diff --git a/Makefile.PL b/Makefile.PL
index d0652410..a04a3b75 100644
--- a/Makefile.PL
+++ b/Makefile.PL
@@ -150,15 +150,14 @@ WriteMakefile(
 
 		# perl-modules-5.xx or libperl5.xx in Debian-based
 		# part of "perl5" on FreeBSD
-		'autodie' => 0,
-		'Compress::Raw::Zlib' => 0,
-		'Compress::Zlib' => 0,
-		'Data::Dumper' => 0,
+		'autodie' => 0, # rpm: perl-autodie
+		'Compress::Raw::Zlib' => 0, # rpm: perl-Compress-Raw-Zlib
+		'Compress::Zlib' => 0, # rpm: perl-IO-Compress
+		'Data::Dumper' => 0, # rpm: perl-Data-Dumper
 		'Digest::SHA' => 0, # rpm: perl-Digest-SHA
-		'Encode' => 2.35, # 2.35 shipped with 5.10.1
-		'IO::Compress::Gzip' => 0,
-		'IO::Uncompress::Gunzip' => 0,
-		'Storable' => 0, # rpm: perl-Storable
+		'IO::Compress::Gzip' => 0, # rpm: perl-IO-Compress
+		'IO::Uncompress::Gunzip' => 0, # rpm: perl-IO-Compress
+		'Sys::Syslog' => 0, # rpm: perl-Sys-Syslog
 		'Text::ParseWords' => 0, # rpm: perl-Text-ParseWords
 
 		# Plack is needed for public-inbox-httpd and PublicInbox::WWW
@@ -170,12 +169,16 @@ WriteMakefile(
 		# users to install them.  See INSTALL
 
 		# All Perl installs I know about have these, but RH-based
-		# distros make them separate even though 'perl' pulls them in
+		# distros can separate these even if `perl' depends on them:
+		'constant' => 0, # rpm: perl-constant
+		'Encode' => 2.35, # rpm: perl-Encode # 2.35 shipped with 5.10.1
 		'File::Path' => 0,
 		'File::Temp' => '0.19', # for ->tmpdir support
-		'Getopt::Long' => 0,
-		'Exporter' => 0,
+		'Getopt::Long' => 0, # rpm: perl-Getopt-Long
+		'Exporter' => 0, # rpm: perl-Exporter
 		'IO::Poll' => 0,
+		'Storable' => 0, # rpm: perl-Storable
+		'Time::HiRes' => 0, # rpm: perl-Time-HiRes
 		# ExtUtils::MakeMaker # this file won't run w/o it...
 	},
 	MAN3PODS => \%man3,
diff --git a/install/deps.perl b/install/deps.perl
index e888cf30..a7b42607 100755
--- a/install/deps.perl
+++ b/install/deps.perl
@@ -35,12 +35,13 @@ my @test_essential = qw(Test::Simple); # we actually use Test::More
 my $profiles = {
 	# the smallest possible profile for testing
 	essential => [ qw(
+		autodie
 		git
 		perl
 		Digest::SHA
-		Encode
 		ExtUtils::MakeMaker
 		IO::Compress
+		Text::ParseWords
 		URI
 		), @test_essential ],
 
@@ -58,7 +59,6 @@ my $profiles = {
 		Plack::Test
 		Plack::Middleware::ReverseProxy
 		Xapian
-		Socket6
 		highlight.pm
 		xapian-tools
 		) ],
@@ -79,9 +79,9 @@ $profiles->{v2essential} = [ @{$profiles->{essential}}, qw(DBD::SQLite DBI) ];
 # dependencies to prevent essential package removal:
 my $non_auto = {
 	git => {
-		pkg => [ qw(curl p5-Socket6 p5-TimeDate git) ],
+		pkg => [ qw(curl p5-TimeDate git) ],
 		rpm => [ qw(curl git) ],
-		pkg_add => [ qw(curl p5-Socket6 p5-Time-TimeDate git) ],
+		pkg_add => [ qw(curl p5-Time-TimeDate git) ],
 	},
 	perl => {
 		pkg => 'perl5',
@@ -132,8 +132,14 @@ my $non_auto = {
 	},
 };
 
-# standard library stuff that CentOS 7.x (and presumably other RPM) split out:
-for (qw(Digest::SHA Encode ExtUtils::MakeMaker IO::Compress Test::Simple)) {
+# standard library stuff that CentOS 7.x (and presumably other RPM)
+# split out and can be removed without removing the `perl' RPM:
+for (qw(autodie Digest::SHA ExtUtils::MakeMaker IO::Compress Sys::Syslog
+		Test::Simple Text::ParseWords)) {
+	# n.b.: Compress::Raw::Zlib is pulled in by IO::Compress
+	# qw(constant Encode Getopt::Long Exporter Storable Time::HiRes)
+	# don't need to be here since it's impossible to have `perl'
+	# on CentOS 7.x without them.
 	$non_auto->{$_} = {
 		deb => 'perl', # libperl5.XX, but the XX varies
 		pkg => 'perl5',

[PATCH 4/5] install/deps: safer defaults for regular users

[Eric Wong]

We'll leave forcing `--yes' to ci/run.sh and remove --purge
usage with apt-get(1) entirely.  Also start defining some
more profiles aimed at users who want a minimal install for
the subset of public-inbox they wish to use.

There'll be some more built-in dependency handling to work
across different distros, but the $always_deps thing is a
start.
---
 ci/run.sh         |   3 +-
 install/deps.perl | 122 ++++++++++++++++++++++++++++------------------
 2 files changed, 77 insertions(+), 48 deletions(-)

diff --git a/ci/run.sh b/ci/run.sh
index 5aa22491..54563f1c 100755
--- a/ci/run.sh
+++ b/ci/run.sh
@@ -1,6 +1,7 @@
 #!/bin/sh
 # Copyright (C) all contributors <meta@public-inbox.org>
 # License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+# Beware, this alters system-wide package installation.
 set -e
 SUDO=${SUDO-'sudo'} PERL=${PERL-'perl'} MAKE=${MAKE-'make'}
 DO=${DO-''}
@@ -15,7 +16,7 @@ NPROC=${NPROC-$({ getconf _NPROCESSORS_ONLN || getconf NPROCESSORS_ONLN ||
 
 $PERL -w ci/profiles.perl | while read args
 do
-	$DO $SUDO $PERL -w install/deps.perl --allow-remove $args
+	$DO $SUDO $PERL -w install/deps.perl -y --allow-remove $args
 	$DO $PERL Makefile.PL
 	$DO $MAKE -j${BUILD_JOBS-$NPROC}
 	$DO $MAKE -j${TEST_JOBS-1} ${TEST_TARGET-test}
diff --git a/install/deps.perl b/install/deps.perl
index a7b42607..6a60c592 100755
--- a/install/deps.perl
+++ b/install/deps.perl
@@ -4,14 +4,16 @@
 eval 'exec perl -S $0 ${1+"$@"}' # no shebang
 if 0; # running under some shell
 use v5.12;
-my $help = <<EOM;
+my $help = <<EOM; # make sure this fits in 80x24 terminals
 usage: $^X $0 [-f PKG_FMT] [--allow-remove] PROFILE [PROFILE_MOD]
 
   -f PKG_FMT      package format (`deb', `pkg', `pkg_add', `pkgin' or `rpm')
-  --allow-remove  allow removing packages (for development use only)
+  --allow-remove  allow removing packages (DANGEROUS, non-production use only)
   --dry-run | -n  show commands that would be run
+  --yes | -y      non-interactive mode / assume yes to package manager
 
-PROFILE is typically `all'.  Other profiles are subject to change.
+PROFILE is typically `www-search', `lei', or `nntpd'
+Some profile names are intended for developer use only and subject to change.
 PROFILE_MOD is only for developers checking dependencies
 
 OS package installation typically requires administrative privileges
@@ -19,7 +21,8 @@ EOM
 use Getopt::Long qw(:config gnu_getopt no_ignore_case auto_abbrev);
 BEGIN { require './install/os.perl' };
 my $opt = {};
-GetOptions($opt, qw(pkg-fmt|f=s allow-remove dry-run|n help|h)) or die $help;
+GetOptions($opt, qw(pkg-fmt|f=s allow-remove dry-run|n yes|y help|h))
+	or die $help;
 if ($opt->{help}) { print $help; exit }
 my $pkg_fmt = $opt->{'pkg-fmt'} // do {
 	my $fmt = pkg_fmt;
@@ -31,7 +34,7 @@ my @test_essential = qw(Test::Simple); # we actually use Test::More
 
 # package profiles.  Note we specify packages at maximum granularity,
 # which is typically deb for most things, but rpm seems to have the
-# highest granularity for things in the Prl standard library.
+# highest granularity for things in the Perl standard library.
 my $profiles = {
 	# the smallest possible profile for testing
 	essential => [ qw(
@@ -47,10 +50,10 @@ my $profiles = {
 
 	# everything optional for normal use
 	optional => [ qw(
+		curl
 		Date::Parse
 		BSD::Resource
 		DBD::SQLite
-		DBI
 		Inline::C
 		Mail::IMAPClient
 		Net::Server
@@ -66,14 +69,44 @@ my $profiles = {
 	# optional developer stuff
 	devtest => [ qw(
 		XML::TreePP
-		curl
 		w3m
 		Plack::Test::ExternalServer
 		) ],
 };
 
+# only for distro-agnostic dependencies which are always true:
+my $always_deps = {
+	'DBD::SQLite' => [ qw(DBI) ],
+	'Mail::IMAPClient' => [ qw(Parse::RecDescent) ],
+	'Plack::Middleware::ReverseProxy' => [ qw(Plack) ],
+};
+
 # bare minimum for v2
-$profiles->{v2essential} = [ @{$profiles->{essential}}, qw(DBD::SQLite DBI) ];
+$profiles->{v2essential} = [ @{$profiles->{essential}}, qw(DBD::SQLite) ];
+
+# for old v1 installs
+$profiles->{'www-v1'} = [ @{$profiles->{essential}}, qw(Plack) ];
+$profiles->{'www-thread'} = [ @{$profiles->{v2essential}}, qw(Plack) ];
+
+# common profile for PublicInbox::WWW
+$profiles->{'www-search'} = [ @{$profiles->{'www-thread'}}, qw(Xapian) ];
+
+# bare mininum for lei
+$profiles->{'lei-core'} = [ @{$profiles->{v2essential}}, qw(Xapian) ];
+push @{$profiles->{'lei-core'}}, 'Inline::C' if $^O ne 'linux';
+
+# common profile for lei:
+$profiles->{lei} = [ @{$profiles->{'lei-core'}}, qw(Mail::IMAPClient curl) ];
+
+$profiles->{nntpd} = [ @{$profiles->{v2essential}} ];
+$profiles->{pop3d} = [ @{$profiles->{v2essential}} ];
+$profiles->{'imapd-bare'} = [ @{$profiles->{v2essential}},
+				qw(Parse::RecDescent) ];
+$profiles->{imapd} = [ @{$profiles->{'imapd-bare'}}, qw(Xapian) ];
+$profiles->{pop3d} = [ @{$profiles->{v2essential}} ];
+$profiles->{watch} = [ @{$profiles->{v2essential}}, qw(Mail::IMAPClient) ];
+$profiles->{'watch-v1'} = [ @{$profiles->{essential}} ];
+$profiles->{'watch-maildir'} = [ @{$profiles->{v2essential}} ];
 
 # package names which can't be mapped automatically and explicit
 # dependencies to prevent essential package removal:
@@ -169,6 +202,12 @@ don't know how to check install status for $pkg_fmt
 
 my (@pkg_install, @pkg_remove, %all);
 for my $ary (values %$profiles) {
+	my @extra;
+	for my $pkg (@$ary) {
+		my $deps = $always_deps->{$pkg} // next;
+		push @extra, @$deps;
+	}
+	push @$ary, @extra;
 	$all{$_} = \@pkg_remove for @$ary;
 }
 if ($^O =~ /\A(?:free|net|open)bsd\z/) {
@@ -177,7 +216,8 @@ if ($^O =~ /\A(?:free|net|open)bsd\z/) {
 $profiles->{all} = [ keys %all ]; # pseudo-profile for all packages
 
 # parse the profile list from the command-line
-for my $profile (@ARGV) {
+my @profiles = @ARGV;
+while (defined(my $profile = shift @profiles)) {
 	if ($profile =~ s/-\z//) {
 		# like apt-get, trailing "-" means remove
 		profile2dst($profile, \@pkg_remove);
@@ -191,57 +231,54 @@ while (my ($pkg, $dst_pkg_list) = each %all) {
 	push @$dst_pkg_list, list(pkg2ospkg($pkg, $pkg_fmt));
 }
 
-my %inst = map { $_ => 1 } @pkg_install;
-@pkg_remove = $opt->{'allow-remove'} ? grep { !$inst{$_} } @pkg_remove : ();
-@pkg_install = grep { !$INST_CHECK->($_) } @pkg_install;
-
-my @apt_opts =
-	qw(-o APT::Install-Recommends=false -o APT::Install-Suggests=false);
+my (%add, %rm); # uniquify lists
+@pkg_install = grep { !$add{$_}++ && !$INST_CHECK->($_) } @pkg_install;
+@pkg_remove = $opt->{'allow-remove'} ? grep {
+		!$add{$_} && !$rm{$_}++ && $INST_CHECK->($_)
+	} @pkg_remove : ();
 
 # OS-specific cleanups appreciated
 if ($pkg_fmt eq 'deb') {
-	my @quiet = $ENV{V} ? () : ('-q');
-	root('apt-get', @apt_opts, qw(install --purge -y), @quiet,
+	my @apt_opt = qw(-o APT::Install-Recommends=false
+			-o APT::Install-Suggests=false);
+	push @apt_opt, '-y' if $opt->{yes};
+	root('apt-get', @apt_opt, qw(install),
 		@pkg_install,
 		# apt-get lets you suffix a package with "-" to
 		# remove it in an "install" sub-command:
 		map { "$_-" } @pkg_remove);
-	root('apt-get', @apt_opts, qw(autoremove --purge -y), @quiet);
+	root('apt-get', @apt_opt, qw(autoremove)) if $opt->{'allow-remove'};
 } elsif ($pkg_fmt eq 'pkg') { # FreeBSD
-	my @quiet = $ENV{V} ? () : ('-q');
+	my @pkg_opt = $opt->{yes} ? qw(-y) : ();
 
 	# don't remove stuff that isn't installed:
-	exclude_uninstalled(\@pkg_remove);
-	root(qw(pkg remove -y), @quiet, @pkg_remove) if @pkg_remove;
-	root(qw(pkg install -y), @quiet, @pkg_install) if @pkg_install;
-	root(qw(pkg autoremove -y), @quiet);
+	root(qw(pkg remove), @pkg_opt, @pkg_remove) if @pkg_remove;
+	root(qw(pkg install), @pkg_opt, @pkg_install) if @pkg_install;
+	root(qw(pkg autoremove), @pkg_opt) if $opt->{'allow-remove'};
 } elsif ($pkg_fmt eq 'pkgin') { # NetBSD
-	my @quiet = $ENV{V} ? ('-'.('V'x$ENV{V})) : ();
-	exclude_uninstalled(\@pkg_remove);
-	root(qw(pkgin -y), @quiet, 'remove', @pkg_remove) if @pkg_remove;
-	root(qw(pkgin -y), @quiet, 'install', @pkg_install) if @pkg_install;
-	root(qw(pkgin -y), @quiet, 'autoremove');
+	my @pkg_opt = $opt->{yes} ? qw(-y) : ();
+	root(qw(pkgin), @pkg_opt, 'remove', @pkg_remove) if @pkg_remove;
+	root(qw(pkgin), @pkg_opt, 'install', @pkg_install) if @pkg_install;
+	root(qw(pkgin), @pkg_opt, 'autoremove') if $opt->{'allow-remove'};
 # TODO: yum / rpm support
 } elsif ($pkg_fmt eq 'rpm') {
-	my @quiet = $ENV{V} ? () : ('-q');
-	exclude_uninstalled(\@pkg_remove);
-	root(qw(yum remove -y), @quiet, @pkg_remove) if @pkg_remove;
-	root(qw(yum install -y), @quiet, @pkg_install) if @pkg_install;
+	my @pkg_opt = $opt->{yes} ? qw(-y) : ();
+	root(qw(yum remove), @pkg_opt, @pkg_remove) if @pkg_remove;
+	root(qw(yum install), @pkg_opt, @pkg_install) if @pkg_install;
 } elsif ($pkg_fmt eq 'pkg_add') { # OpenBSD
-	exclude_uninstalled(\@pkg_remove);
-	my @quiet = $ENV{V} ? ('-'.('v'x$ENV{V})) : qw(-x); # -x : no progress
+	my @pkg_opt = $opt->{yes} ? qw(-I) : (); # -I means non-interactive
 	if (@pkg_remove) {
 		my @lifo = qw(xapian-bindings-perl);
 		for my $dep (@lifo) {
 			grep(/\A\Q$dep\E\z/, @pkg_remove) or next;
-			root(qw(pkg_delete -I), @quiet, $dep);
+			root(qw(pkg_delete), @pkg_opt, $dep);
 			@pkg_remove = grep(!/\A\Q$dep\E\z/, @pkg_remove);
 		}
-		root(qw(pkg_delete -I), @quiet, @pkg_remove);
+		root(qw(pkg_delete), @pkg_opt, @pkg_remove);
 	}
-	root(qw(pkg_delete -a), @quiet);
+	root(qw(pkg_delete -a), @pkg_opt); # autoremove unspecified
 	@pkg_install = map { "$_--" } @pkg_install; # disambiguate w3m
-	root(qw(pkg_add), @quiet, @pkg_install) if @pkg_install;
+	root(qw(pkg_add), @pkg_opt, @pkg_install) if @pkg_install;
 } else {
 	die "unsupported package format: $pkg_fmt\n";
 }
@@ -290,15 +327,6 @@ sub profile2dst {
 	}
 }
 
-sub exclude_uninstalled {
-	my ($list) = @_;
-	my (@tmp, %seen);
-	for my $pkg (@$list) {
-		push @tmp, $pkg if !$seen{$pkg}++ && $INST_CHECK->($pkg);
-	}
-	@$list = @tmp;
-}
-
 sub root {
 	warn "# @_\n";
 	return if $opt->{'dry-run'};

[PATCH 5/5] t/xap_helper: improve reliability of TTIN/TTOU tests

[Eric Wong]

We need to send more parallel requests and possibly retry more
to ensure SIGTTIN/TTOU are handled due to OS scheduling delays.

$doreq was also getting too noisy with extra is() calls, so just
xbail on mismatches if somehow we get a truncated write.
---
 t/xap_helper.t | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)

diff --git a/t/xap_helper.t b/t/xap_helper.t
index 54bef191..2303301d 100644
--- a/t/xap_helper.t
+++ b/t/xap_helper.t
@@ -54,9 +54,8 @@ my $doreq = sub {
 	push @fds, fileno($err) if $err;
 	my $n = PublicInbox::IPC::send_cmd($s, \@fds, $buf, 0);
 	$n // xbail "send: $!";
-	my $arg = "@arg";
-	$arg =~ s/\Q$tmp\E/\$TMP/gs;
-	is(length($buf), $n, "req $arg sent");
+	my $exp = length($buf);
+	$exp == $n or xbail "req @arg sent short ($n != $exp)";
 	$x;
 };
 
@@ -109,15 +108,16 @@ my $test = sub {
 	$tries = 0;
 	my @ins = ($s, qw(test_inspect -d), $ibx_idx[0]);
 	kill('TTIN', $pid);
-	until (scalar(keys %pids) >= 2 || ++$tries > 10) {
+	until (scalar(keys %pids) >= 2 || ++$tries > 100) {
 		tick;
-		my @r = map { $doreq->(@ins) } (0..5);
+		my @r = map { $doreq->(@ins) } (0..100);
 		for my $fh (@r) {
 			my $buf = do { local $/; <$fh> } // die "read: $!";
 			$buf =~ /\bpid=(\d+)/ and $pids{$1} = undef;
 		}
 	}
-	is(scalar keys %pids, 2, 'have two pids');
+	is(scalar keys %pids, 2, 'have two pids') or
+		diag 'pids='.explain(\%pids);
 
 	kill('TTOU', $pid);
 	%pids = ();
@@ -127,7 +127,7 @@ my $test = sub {
 	tick($delay);
 	until (scalar(keys %pids) == 1 || ++$tries > 100) {
 		%pids = ();
-		my @r = map { $doreq->(@ins) } (0..5);
+		my @r = map { $doreq->(@ins) } (0..100);
 		for my $fh (@r) {
 			my $buf = do { local $/; <$fh> } // die "read: $!";
 			$buf =~ /\bpid=(\d+)/ and $pids{$1} = undef;