"make test" for 1.9.0 on centos-7

"make test" for 1.9.0 on centos-7

[Konstantin Ryabitsev]

Hello:

I'm starting to work on upgrading lore.kernel.org to 1.9.0. For a number of
yak-shavey reasons we are still on centos-7, though hopefully we'll be able to
move on to something newer soon. Right now, I'm having difficulty running
"make test":

	... [skipping OK tests] ...
	t/cmd_ipc.t .................. 1/? sleeping on sendmsg: Too many references: cannot splice (#1)
	sleeping on sendmsg: Too many references: cannot splice (#2)
	... (repeats 49 times) ...

	t/cmd_ipc.t .................. 24/? # sent 2097152, retrying with more
	sleeping on sendmsg: No buffer space available (#1)
	... (repeats 49 times, but much slower) ...

	t/cmd_ipc.t .................. 29/? Use of uninitialized value $_[1] in vec at /home/mricon/public-inbox/blib/lib/PublicInbox/Syscall.pm line 457.
	Use of uninitialized value $_[1] in scalar assignment at /home/mricon/public-inbox/blib/lib/PublicInbox/Syscall.pm line 457.
	Use of uninitialized value $_[1] in vec at /home/mricon/public-inbox/blib/lib/PublicInbox/Syscall.pm line 457.
	Use of uninitialized value $_[1] in scalar assignment at /home/mricon/public-inbox/blib/lib/PublicInbox/Syscall.pm line 457.
	1 at /home/mricon/public-inbox/blib/lib/PublicInbox/Syscall.pm line 445.
	... (repeats a lot of times) ...

	t/cmd_ipc.t .................. 66/? # sent 2097152, retrying with more
	1 at /home/mricon/public-inbox/blib/lib/PublicInbox/Syscall.pm line 445.
	... (repeats a lot of times) ...

	... [skipped some OK tests] ...

	t/extsearch.t ................ 5/? # inherited [::1]:45080 fd=3
	Can't call method "xpath" on an undefined value at t/extsearch.t line 132.
    # Tests were run but no plan was declared and done_testing() was not seen.
	t/extsearch.t ................ Dubious, test returned 254 (wstat 65024, 0xfe00)
	... [skipped some OK tests] ...

	t/httpd-corner.t ............. 1/? # inherited [::1]:46301 fd=3
    # inherited /tmp/pi-httpd-corner-12549-8Xh4/s fd=4
    # inherited [::1]:45630 fd=5
    # http://[::1]:45630 psgi=t/alt.psgi
    # http://[::1]:45630 err=/tmp/pi-httpd-corner-12549-8Xh4/alt.err
    # inherited [::1]:46301 fd=3
    # inherited /tmp/pi-httpd-corner-12549-8Xh4/s fd=4
    # inherited [::1]:45630 fd=5
    # http://[::1]:45630 psgi=t/alt.psgi
    # http://[::1]:45630 err=/tmp/pi-httpd-corner-12549-8Xh4/alt.err
    # inherited [::1]:46301 fd=3
    # inherited /tmp/pi-httpd-corner-12549-8Xh4/s fd=4
    # inherited [::1]:45630 fd=5
    # http://[::1]:45630 psgi=t/alt.psgi
    # http://[::1]:45630 err=/tmp/pi-httpd-corner-12549-8Xh4/alt.err
	... [seems to hang here forever] ...

Any pointers where I should start looking?

Best wishes,
-Konstantin

Re: "make test" for 1.9.0 on centos-7

[Eric Wong]

Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> Hello:
> 
> I'm starting to work on upgrading lore.kernel.org to 1.9.0. For a number of
> yak-shavey reasons we are still on centos-7, though hopefully we'll be able to
> move on to something newer soon. Right now, I'm having difficulty running
> "make test":

Eep, ok there's a lot of breakages on CentOS 7, likely going
back to our 1.7/1.8 releases and my C7 VM was broken/forgotten :x
Working on fixes now...

Re: "make test" for 1.9.0 on centos-7

[Konstantin Ryabitsev]

On Wed, Sep 28, 2022 at 07:59:39PM +0000, Eric Wong wrote:
> > I'm starting to work on upgrading lore.kernel.org to 1.9.0. For a number of
> > yak-shavey reasons we are still on centos-7, though hopefully we'll be able to
> > move on to something newer soon. Right now, I'm having difficulty running
> > "make test":
> 
> Eep, ok there's a lot of breakages on CentOS 7, likely going
> back to our 1.7/1.8 releases and my C7 VM was broken/forgotten :x
> Working on fixes now...

Ok, no worries -- I know that Centos-7 is getting pretty old. :) Note, that in
our particular case most perl dependencies are installed with cpanm, not with
RPMs, and we're using newer git and xapian-1.4.

Best regards,
Konstantin

[PATCH 0/4] CentOS 7 fixes + fix Gcf2 everywhere

[Eric Wong]

A bunch of CentOS 7.x fixes noted by Konstantin
I also just noticed I broke Gcf2 everywhere :x

I'd much rather support old enterprise distros directly than
direct people towards bloated Docker containers and what-not.

Eric Wong (4):
  syscall: initialize buffer for vec()
  treewide: use --globoff with curl(1)
  gcf2: fix syntax error and require PublicInbox::Git
  tests: no IPv6 on old Net::NNTP, Mail::IMAPClient, HTTP::Tiny

 lib/PublicInbox/Gcf2.pm       | 6 ++++--
 lib/PublicInbox/LeiCurl.pm    | 4 ++--
 lib/PublicInbox/Syscall.pm    | 2 +-
 lib/PublicInbox/TestCommon.pm | 9 +++++++++
 t/extsearch.t                 | 4 ++--
 t/httpd-corner.t              | 8 ++++----
 t/nntpd.t                     | 3 +--
 t/v2writable.t                | 2 +-
 8 files changed, 24 insertions(+), 14 deletions(-)

[PATCH 1/4] syscall: initialize buffer for vec()

[Eric Wong]

This is needed for older Perls (tested perl 5.16.3 on CentOS 7).
---
 lib/PublicInbox/Syscall.pm | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/PublicInbox/Syscall.pm b/lib/PublicInbox/Syscall.pm
index 46496bca..412ca64f 100644
--- a/lib/PublicInbox/Syscall.pm
+++ b/lib/PublicInbox/Syscall.pm
@@ -454,7 +454,7 @@ no warnings 'once';
 
 *recv_cmd4 = sub ($$$) {
 	my ($sock, undef, $len) = @_;
-	vec($_[1], ($len + 1) * 8, 1) = 0;
+	vec($_[1] //= '', ($len + 1) * 8, 1) = 0;
 	my $cmsghdr = "\0" x msg_controllen; # 10 * sizeof(int)
 	my $iov = pack('P'.TMPL_size_t, $_[1], $len);
 	my $mh = pack('PL' . # msg_name, msg_namelen (socklen_t (U32))

[PATCH 2/4] treewide: use --globoff with curl(1)

[Eric Wong]

curl 7.29.0 (on CentOS 7.x) seems to mishandle square-bracketed
IPv6 addresses, at least.  Furthermore, we don't actually need
nor use the globbing in curl for lei when forwarding requests
from the lei command-line.  lei has its own globbing and
`--globoff' behavior for externals and none of it is intended
for curl.
---
 lib/PublicInbox/LeiCurl.pm | 4 ++--
 t/httpd-corner.t           | 8 ++++----
 2 files changed, 6 insertions(+), 6 deletions(-)

diff --git a/lib/PublicInbox/LeiCurl.pm b/lib/PublicInbox/LeiCurl.pm
index 5ffade99..4e602a48 100644
--- a/lib/PublicInbox/LeiCurl.pm
+++ b/lib/PublicInbox/LeiCurl.pm
@@ -1,4 +1,4 @@
-# Copyright (C) 2021 all contributors <meta@public-inbox.org>
+# Copyright (C) all contributors <meta@public-inbox.org>
 # License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
 
 # common option and torsocks(1) wrapping for curl(1)
@@ -27,7 +27,7 @@ sub new {
 	my ($cls, $lei, $curl) = @_;
 	$curl //= which('curl') // return $lei->fail('curl not found');
 	my $opt = $lei->{opt};
-	my @cmd = ($curl, qw(-Sf));
+	my @cmd = ($curl, qw(-gSf));
 	$cmd[-1] .= 's' if $opt->{quiet}; # already the default for "lei q"
 	$cmd[-1] .= 'v' if $opt->{verbose}; # we use ourselves, too
 	for my $o ($lei->curl_opt) {
diff --git a/t/httpd-corner.t b/t/httpd-corner.t
index 973cc55d..e16e8dc8 100644
--- a/t/httpd-corner.t
+++ b/t/httpd-corner.t
@@ -341,7 +341,7 @@ SKIP: {
 	my $url = "$base/sha1";
 	my ($r, $w);
 	pipe($r, $w) or die "pipe: $!";
-	my $cmd = [$curl, qw(--tcp-nodelay -T- -HExpect: -sSN), $url];
+	my $cmd = [$curl, qw(--tcp-nodelay -T- -HExpect: -gsSN), $url];
 	open my $cout, '+>', undef or die;
 	open my $cerr, '>', undef or die;
 	my $rdr = { 0 => $r, 1 => $cout, 2 => $cerr };
@@ -358,7 +358,7 @@ SKIP: {
 	seek($cout, 0, SEEK_SET);
 	is(<$cout>, sha1_hex($str), 'read expected body');
 
-	my $fh = popen_rd([$curl, '-sS', "$base/async-big"]);
+	my $fh = popen_rd([$curl, '-gsS', "$base/async-big"]);
 	my $n = 0;
 	my $non_zero = 0;
 	while (1) {
@@ -372,12 +372,12 @@ SKIP: {
 	is($non_zero, 0, 'read all zeros');
 
 	require_mods(@zmods, 4);
-	my $buf = xqx([$curl, '-sS', "$base/psgi-return-gzip"]);
+	my $buf = xqx([$curl, '-gsS', "$base/psgi-return-gzip"]);
 	is($?, 0, 'curl succesful');
 	IO::Uncompress::Gunzip::gunzip(\$buf => \(my $out));
 	is($out, "hello world\n");
 	my $curl_rdr = { 2 => \(my $curl_err = '') };
-	$buf = xqx([$curl, qw(-sSv --compressed),
+	$buf = xqx([$curl, qw(-gsSv --compressed),
 			"$base/psgi-return-compressible"], undef, $curl_rdr);
 	is($?, 0, 'curl --compressed successful');
 	is($buf, "goodbye world\n", 'gzipped response as expected');

[PATCH 3/4] gcf2: fix syntax error and require PublicInbox::Git

[Eric Wong]

I failed to notice these since I uninstalled libgit2 for
benchmarking and kept it uninstalled since my git(1) install
is faster.

Fixes: 1c0ec857d041 "gcf2: support worktree $GIT_DIR"
---
 lib/PublicInbox/Gcf2.pm | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/lib/PublicInbox/Gcf2.pm b/lib/PublicInbox/Gcf2.pm
index 2ba2efff..5c353095 100644
--- a/lib/PublicInbox/Gcf2.pm
+++ b/lib/PublicInbox/Gcf2.pm
@@ -1,4 +1,4 @@
-# Copyright (C) 2020-2021 all contributors <meta@public-inbox.org>
+# Copyright (C) all contributors <meta@public-inbox.org>
 # License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
 
 # backend for a git-cat-file-workalike based on libgit2,
@@ -11,6 +11,8 @@ use Fcntl qw(LOCK_EX SEEK_SET);
 use Time::HiRes qw(clock_gettime CLOCK_MONOTONIC);
 use IO::Handle; # autoflush
 use File::Path qw(make_path);
+use PublicInbox::Git;
+
 BEGIN {
 	my (%CFG, $c_src);
 	# PublicInbox::Spawn will set PERL_INLINE_DIRECTORY
@@ -129,7 +131,7 @@ sub loop (;$) {
 			warn "I: $$ $oid missing, retrying in $git_dir\n";
 
 			$gcf2 = new();
-			%seen = ($git_dir => add_alt($gcf2, $git_dir);
+			%seen = ($git_dir => add_alt($gcf2, $git_dir));
 			$check_at = clock_gettime(CLOCK_MONOTONIC) + $exp;
 
 			if ($gcf2->cat_oid(1, $oid)) {

[PATCH 4/4] tests: no IPv6 on old Net::NNTP, Mail::IMAPClient, HTTP::Tiny

[Eric Wong]

The versions of these modules which ship with CentOS 7.x did not
support IPv6 properly.
---
 lib/PublicInbox/TestCommon.pm | 9 +++++++++
 t/extsearch.t                 | 4 ++--
 t/nntpd.t                     | 3 +--
 t/v2writable.t                | 2 +-
 4 files changed, 13 insertions(+), 5 deletions(-)

diff --git a/lib/PublicInbox/TestCommon.pm b/lib/PublicInbox/TestCommon.pm
index 55d82fc0..333791b4 100644
--- a/lib/PublicInbox/TestCommon.pm
+++ b/lib/PublicInbox/TestCommon.pm
@@ -117,6 +117,12 @@ sub require_git ($;$) {
 	1;
 }
 
+my %IPv6_VERSION = (
+	'Net::NNTP' => 3.00,
+	'Mail::IMAPClient' => 3.40,
+	'HTTP::Tiny' => 0.042,
+);
+
 sub require_mods {
 	my @mods = @_;
 	my $maybe = pop @mods if $mods[-1] =~ /\A[0-9]+\z/;
@@ -167,6 +173,9 @@ sub require_mods {
 				!eval{ IO::Socket::SSL->VERSION(2.007); 1 }) {
 			push @need, $@;
 		}
+		if (defined(my $v = $IPv6_VERSION{$mod})) {
+			$ENV{TEST_IPV4_ONLY} = 1 if !eval { $mod->VERSION($v) };
+		}
 	}
 	return unless @need;
 	my $m = join(', ', @need)." missing for $0";
diff --git a/t/extsearch.t b/t/extsearch.t
index 2d7375d6..6cbe5262 100644
--- a/t/extsearch.t
+++ b/t/extsearch.t
@@ -13,8 +13,6 @@ require PublicInbox::Search;
 use_ok 'PublicInbox::ExtSearch';
 use_ok 'PublicInbox::ExtSearchIdx';
 use_ok 'PublicInbox::OverIdx';
-my $sock = tcp_server();
-my $host_port = tcp_host_port($sock);
 my ($home, $for_destroy) = tmpdir();
 local $ENV{HOME} = $home;
 mkdir "$home/.public-inbox" or BAIL_OUT $!;
@@ -125,6 +123,8 @@ EOF
 
 SKIP: {
 	require_mods(qw(Net::NNTP), 1);
+	my $sock = tcp_server();
+	my $host_port = tcp_host_port($sock);
 	my ($out, $err) = ("$home/nntpd.out.log", "$home/nntpd.err.log");
 	my $cmd = [ '-nntpd', '-W0', "--stdout=$out", "--stderr=$err" ];
 	my $td = start_script($cmd, undef, { 3 => $sock });
diff --git a/t/nntpd.t b/t/nntpd.t
index 34e9e1b4..d352c3c1 100644
--- a/t/nntpd.t
+++ b/t/nntpd.t
@@ -2,10 +2,9 @@
 # Copyright (C) all contributors <meta@public-inbox.org>
 # License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
 use strict; use v5.10.1; use PublicInbox::TestCommon;
-require_mods(qw(DBD::SQLite));
+require_mods(qw(DBD::SQLite Net::NNTP));
 use PublicInbox::Eml;
 use Socket qw(IPPROTO_TCP TCP_NODELAY);
-use Net::NNTP;
 use Sys::Hostname;
 use POSIX qw(_exit);
 use Digest::SHA;
diff --git a/t/v2writable.t b/t/v2writable.t
index 477621e2..ad946338 100644
--- a/t/v2writable.t
+++ b/t/v2writable.t
@@ -149,7 +149,7 @@ SELECT COUNT(*) FROM over WHERE num > 0
 }
 
 {
-	use Net::NNTP;
+	require_mods('Net::NNTP', 1);
 	my $err = "$inboxdir/stderr.log";
 	my $out = "$inboxdir/stdout.log";
 	my $group = 'inbox.comp.test.v2writable';

Re: "make test" for 1.9.0 on centos-7

[Eric Wong]

Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> 	t/cmd_ipc.t .................. 1/? sleeping on sendmsg: Too many references: cannot splice (#1)
> 	sleeping on sendmsg: Too many references: cannot splice (#2)
> 	... (repeats 49 times) ...
> 
> 	t/cmd_ipc.t .................. 24/? # sent 2097152, retrying with more
> 	sleeping on sendmsg: No buffer space available (#1)
> 	... (repeats 49 times, but much slower) ...

I wasn't able to reproduce that, but it's likely RLIMIT_NOFILE
being too low (ulimit -n).

Re: [PATCH 0/4] CentOS 7 fixes + fix Gcf2 everywhere

[Konstantin Ryabitsev]

On Thu, Sep 29, 2022 at 05:48:27PM +0000, Eric Wong wrote:
> A bunch of CentOS 7.x fixes noted by Konstantin
> I also just noticed I broke Gcf2 everywhere :x

The tests pass now, thanks!

I still noticed this error (master + this series):

    1 at /home/mricon/public-inbox/blib/lib/PublicInbox/Syscall.pm line 445.
    (repeated a bunch of times)

Cheers,
-K

Re: [PATCH 0/4] CentOS 7 fixes + fix Gcf2 everywhere

[Eric Wong]

Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> On Thu, Sep 29, 2022 at 05:48:27PM +0000, Eric Wong wrote:
> > A bunch of CentOS 7.x fixes noted by Konstantin
> > I also just noticed I broke Gcf2 everywhere :x
> 
> The tests pass now, thanks!
> 
> I still noticed this error (master + this series):
> 
>     1 at /home/mricon/public-inbox/blib/lib/PublicInbox/Syscall.pm line 445.
>     (repeated a bunch of times)

OK, so that's the "do {" line in a do/while loop for send_cmd4, right?

What's before and after?  Which *.t is that coming from?

This is single-process "make test"?

I'm not seeing where that warning is coming from at all...
Removing Inline::C and ~/.cache/public-inbox/inline-c doesn't
make the warning appear, either.

Of course I don't have Socket::MsgHdr installed on CentOS 7,
either, since it's not available via yum.