make test in a debian-11 container

make test in a debian-11 container

[Konstantin Ryabitsev]

Hello:

I can't seem to get a clean "make test" in the Debian container. It's possible
that I'm missing some of the packages, as the official Debian container image
is very minimal.

The dockerfile is here: https://gist.github.com/mricon/046ba7c8b03bd92176dbe83e04f2466c

The pertinent section is:
    RUN apt-get -y install git liburi-perl libemail-mime-perl libplack-perl libtimedate-perl \
                           libdbd-sqlite3-perl libsearch-xapian-perl libnet-server-perl \
                           libinline-c-perl libemail-address-xs-perl libparse-recdescent-perl \
                           xapian-tools libencode-perl libdbi-perl liblinux-inotify2-perl \
                           libio-compress-perl curl libmail-imapclient-perl sqlite3 \
                           libgit2-dev make eatmydata

When I run "make test", I get the following failures:

    ...
    t/gcf2.t ..................... Use of uninitialized value $file in index at /home/user/work/temp/pi/blib/lib/PublicInbox/Spawn.pm line 348.
    Use of uninitialized value $file in concatenation (.) or string at /home/user/work/temp/pi/blib/lib/PublicInbox/Spawn.pm line 350.
    Use of uninitialized value in subroutine entry at /home/user/work/temp/pi/blib/lib/PublicInbox/Spawn.pm line 388.
    Use of uninitialized value in join or string at /home/user/work/temp/pi/blib/lib/PublicInbox/Spawn.pm line 389.
    t/gcf2.t ..................... skipped: PublicInbox::Gcf2 missing for t/gcf2.t
    t/gcf2_client.t .............. Use of uninitialized value $file in index at /home/user/work/temp/pi/blib/lib/PublicInbox/Spawn.pm line 348.
    Use of uninitialized value $file in concatenation (.) or string at /home/user/work/temp/pi/blib/lib/PublicInbox/Spawn.pm line 350.
    Use of uninitialized value in subroutine entry at /home/user/work/temp/pi/blib/lib/PublicInbox/Spawn.pm line 388.
    Use of uninitialized value in join or string at /home/user/work/temp/pi/blib/lib/PublicInbox/Spawn.pm line 389.
    t/gcf2_client.t .............. skipped: PublicInbox::Gcf2 missing for t/gcf2_client.t
    t/git.t ...................... 1/? fatal: Needed a single revision
    t/git.t ...................... ok
    t/gzip_filter.t .............. ok
    t/hl_mod.t ................... skipped: failed to load highlight.pm for t/hl_mod.t
    t/httpd-corner.t ............. ok
    t/httpd-https.t .............. skipped: certs/ missing for t/httpd-https.t, run /usr/bin/perl ./create-certs.perl in certs/
    t/httpd-unix.t ............... ok
    t/httpd.t .................... ok
    t/hval.t ..................... ok
    t/idx_stack.t ................ ok
    t/imap.t ..................... ok
    t/imap_searchqp.t ............ ok
    t/imap_tracker.t ............. ok
    t/imapd-tls.t ................ skipped: certs/ missing for t/imapd-tls.t, run /usr/bin/perl ./create-certs.perl in certs/
    t/imapd.t .................... 1/? # waiting for initial fetch...
    # inbox unlocked on initial fetch, waiting for IDLE
    t/imapd.t .................... 180/? # waiting for IMAP IDLE wakeup
    # inbox unlocked on IDLE wakeup
    # waiting for -watch reload + initial fetch
    # waiting for PollInterval wakeup
    # inbox unlocked (poll)

    #   Failed test 'no uninitialized warnings'
    #   at t/imapd.t line 543.
    #                   'Use of uninitialized value $file in index at /home/user/work/temp/pi/blib/lib/PublicInbox/Spawn.pm line 348.
    # Use of uninitialized value $file in concatenation (.) or string at /home/user/work/temp/pi/blib/lib/PublicInbox/Spawn.pm line 350.
    # Use of uninitialized value in subroutine entry at /home/user/work/temp/pi/blib/lib/PublicInbox/Spawn.pm line 388.
    # Use of uninitialized value in join or string at /home/user/work/temp/pi/blib/lib/PublicInbox/Spawn.pm line 389.
    # '
    #           matches '(?^i:uninitialized)'
    # Looks like you failed 1 test of 186.
    t/imapd.t .................... Dubious, test returned 1 (wstat 256, 0x100)
    Failed 1/186 subtests
    ...

What am I missing?

-K

[PATCH] gcf2: die if pkg-config is missing

[Eric Wong]

Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> Hello:
> 
> I can't seem to get a clean "make test" in the Debian container. It's possible
> that I'm missing some of the packages, as the official Debian container image
> is very minimal.

Not sure, it's taking forever to update one of my VMs...

> The dockerfile is here: https://gist.github.com/mricon/046ba7c8b03bd92176dbe83e04f2466c
> 
> The pertinent section is:
>     RUN apt-get -y install git liburi-perl libemail-mime-perl libplack-perl libtimedate-perl \
>                            libdbd-sqlite3-perl libsearch-xapian-perl libnet-server-perl \
>                            libinline-c-perl libemail-address-xs-perl libparse-recdescent-perl \
>                            xapian-tools libencode-perl libdbi-perl liblinux-inotify2-perl \
>                            libio-compress-perl curl libmail-imapclient-perl sqlite3 \
>                            libgit2-dev make eatmydata

Oops, pkg-config is needed for libgit2-dev.  I thought most
*-dev packages would pull it in...

------------8<------------
Subject: [PATCH] gcf2: die if pkg-config is missing

We can't link properly to libgit2 without pkg-config telling
us which libraries and headers to use.
---
 lib/PublicInbox/Gcf2.pm | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/lib/PublicInbox/Gcf2.pm b/lib/PublicInbox/Gcf2.pm
index 99f4ae04..0f967579 100644
--- a/lib/PublicInbox/Gcf2.pm
+++ b/lib/PublicInbox/Gcf2.pm
@@ -17,7 +17,8 @@ BEGIN {
 		die 'PERL_INLINE_DIRECTORY not defined';
 	my $f = "$inline_dir/.public-inbox.lock";
 	open $lockfh, '>', $f or die "failed to open $f: $!\n";
-	my $pc = which($ENV{PKG_CONFIG} // 'pkg-config');
+	my $pc = which($ENV{PKG_CONFIG} // 'pkg-config') //
+		die "pkg-config missing for libgit2";
 	my ($dir) = (__FILE__ =~ m!\A(.+?)/[^/]+\z!);
 	my $rdr = {};
 	open $rdr->{2}, '>', '/dev/null' or die "open /dev/null: $!";


> When I run "make test", I get the following failures:
> 
>     ...
>     t/gcf2.t ..................... Use of uninitialized value $file in index at /home/user/work/temp/pi/blib/lib/PublicInbox/Spawn.pm line 348.
>     Use of uninitialized value $file in concatenation (.) or string at /home/user/work/temp/pi/blib/lib/PublicInbox/Spawn.pm line 350.
>     Use of uninitialized value in subroutine entry at /home/user/work/temp/pi/blib/lib/PublicInbox/Spawn.pm line 388.
>     Use of uninitialized value in join or string at /home/user/work/temp/pi/blib/lib/PublicInbox/Spawn.pm line 389.

Otherwise, maybe printing a stacktrace can shine a light on the problem:

diff --git a/lib/PublicInbox/Spawn.pm b/lib/PublicInbox/Spawn.pm
index fe7aa0a8..473d0716 100644
--- a/lib/PublicInbox/Spawn.pm
+++ b/lib/PublicInbox/Spawn.pm
@@ -343,8 +343,10 @@ unless ($set_nodatacow) {
 undef $set_nodatacow;
 undef $all_libc;
 
+use Carp;
 sub which ($) {
 	my ($file) = @_;
+	warn "file undefined ",Carp::longmess() unless defined($file);
 	return $file if index($file, '/') >= 0;
 	for my $p (split(/:/, $ENV{PATH})) {
 		$p .= "/$file";

Re: [PATCH] gcf2: die if pkg-config is missing

[Konstantin Ryabitsev]

On Thu, Sep 09, 2021 at 08:45:41PM +0000, Eric Wong wrote:
> Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> > Hello:
> > 
> > I can't seem to get a clean "make test" in the Debian container. It's possible
> > that I'm missing some of the packages, as the official Debian container image
> > is very minimal.
> 
> Not sure, it's taking forever to update one of my VMs...
> 
> > The dockerfile is here: https://gist.github.com/mricon/046ba7c8b03bd92176dbe83e04f2466c
> > 
> > The pertinent section is:
> >     RUN apt-get -y install git liburi-perl libemail-mime-perl libplack-perl libtimedate-perl \
> >                            libdbd-sqlite3-perl libsearch-xapian-perl libnet-server-perl \
> >                            libinline-c-perl libemail-address-xs-perl libparse-recdescent-perl \
> >                            xapian-tools libencode-perl libdbi-perl liblinux-inotify2-perl \
> >                            libio-compress-perl curl libmail-imapclient-perl sqlite3 \
> >                            libgit2-dev make eatmydata
> 
> Oops, pkg-config is needed for libgit2-dev.  I thought most
> *-dev packages would pull it in...

Yep, that was the culprit. After installing pkg-config, gcf2 builds and tests
pass. I still have this while running "make test":

t/git.t ...................... 1/? fatal: Needed a single revision

But I do get "All tests successful" in the end, so I assume that's the
expected condition.

Thanks,
-K

Re: [PATCH] gcf2: die if pkg-config is missing

[Eric Wong]

Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> Yep, that was the culprit. After installing pkg-config, gcf2 builds and tests
> pass. I still have this while running "make test":

Good to know.  We'll have to document that when adding libgit2
stuff to the already-huge INSTALL file.

> t/git.t ...................... 1/? fatal: Needed a single revision
> 
> But I do get "All tests successful" in the end, so I assume that's the
> expected condition.

Yes, it was making sure errors got propagated properly.
It always got lost in the "make -j4 check" parallel output for me :x

--------8<---------
Subject: [PATCH] t/git.t: quiet intentional git-rev-parse failure

It can get confusing, especially when running non-parallel "make test"

Link: https://public-inbox.org/meta/20210909210138.ssiv5tri65mf4l4o@meerkat.local/
---
 t/git.t | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/t/git.t b/t/git.t
index bc1dea50..844e0111 100644
--- a/t/git.t
+++ b/t/git.t
@@ -98,8 +98,10 @@ if (1) {
 
 	$gcf->qx(qw(repack -adq));
 	ok($gcf->packed_bytes > 0, 'packed size is positive');
-	$gcf->qx(qw(rev-parse --verify bogus));
-	isnt($?, 0, '$? set on failure'.$?);
+	my $rdr;
+	open $rdr->{2}, '+>', '/dev/null' or xbail "open $!";
+	$gcf->qx([qw(rev-parse --verify bogus)], undef, $rdr);
+	isnt($?, 0, '$? set on failure: '.$?);
 }
 
 SKIP: {

Re: [PATCH] gcf2: die if pkg-config is missing

[Konstantin Ryabitsev]

On Thu, Sep 09, 2021 at 09:14:36PM +0000, Eric Wong wrote:
> Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> > Yep, that was the culprit. After installing pkg-config, gcf2 builds and tests
> > pass. I still have this while running "make test":
> 
> Good to know.  We'll have to document that when adding libgit2
> stuff to the already-huge INSTALL file.

Annoyingly, after I blow away the container to start fresh, I get:

    t/gcf2.t ..................... skipped: PublicInbox::Gcf2 missing for t/gcf2.t

That means gcf2 isn't getting built, right?

-K

Re: [PATCH] gcf2: die if pkg-config is missing

[Eric Wong]

Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> On Thu, Sep 09, 2021 at 09:14:36PM +0000, Eric Wong wrote:
> > Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> > > Yep, that was the culprit. After installing pkg-config, gcf2 builds and tests
> > > pass. I still have this while running "make test":
> > 
> > Good to know.  We'll have to document that when adding libgit2
> > stuff to the already-huge INSTALL file.
> 
> Annoyingly, after I blow away the container to start fresh, I get:
> 
>     t/gcf2.t ..................... skipped: PublicInbox::Gcf2 missing for t/gcf2.t
> 
> That means gcf2 isn't getting built, right?

Yeah, assuming all dependencies are there;
is PERL_INLINE_DIRECTORY set (or ~/.cache/public-inbox/inline-c exists?)

Re: [PATCH] gcf2: die if pkg-config is missing

[Konstantin Ryabitsev]

On Thu, Sep 09, 2021 at 09:23:48PM +0000, Eric Wong wrote:
> > Annoyingly, after I blow away the container to start fresh, I get:
> > 
> >     t/gcf2.t ..................... skipped: PublicInbox::Gcf2 missing for t/gcf2.t
> > 
> > That means gcf2 isn't getting built, right?
> 
> Yeah, assuming all dependencies are there;
> is PERL_INLINE_DIRECTORY set (or ~/.cache/public-inbox/inline-c exists?)

Ah, yes, when I build as part of the dockerfile image, that dir is absent.
It shouldn't matter for actual usage, though -- so we can ignore this.

Sorry for a lot of noise about it, just want to make sure I got everything
working. :)

I'll follow up with my draft instructions on how to get things running with
toolbox.

Best,
-K

Re: [PATCH] gcf2: die if pkg-config is missing

[Eric Wong]

Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> On Thu, Sep 09, 2021 at 09:23:48PM +0000, Eric Wong wrote:
> > > Annoyingly, after I blow away the container to start fresh, I get:
> > > 
> > >     t/gcf2.t ..................... skipped: PublicInbox::Gcf2 missing for t/gcf2.t
> > > 
> > > That means gcf2 isn't getting built, right?
> > 
> > Yeah, assuming all dependencies are there;
> > is PERL_INLINE_DIRECTORY set (or ~/.cache/public-inbox/inline-c exists?)
> 
> Ah, yes, when I build as part of the dockerfile image, that dir is absent.
> It shouldn't matter for actual usage, though -- so we can ignore this.
> 
> Sorry for a lot of noise about it, just want to make sure I got everything
> working. :)
> 
> I'll follow up with my draft instructions on how to get things running with
> toolbox.

No worries, I think the test can show diagnostic info without
being too alarming for optional dependencies:

--------8<--------
Subject: [PATCH] test_common: print diagnostic for missing dependencies

The "#" prefix should prevent it from being too alarming if
a dependency is expected to be missing, but still useful if a
dependency is misconfigured.
---
 lib/PublicInbox/TestCommon.pm | 1 +
 1 file changed, 1 insertion(+)

diff --git a/lib/PublicInbox/TestCommon.pm b/lib/PublicInbox/TestCommon.pm
index 83dcf650..2e3e9ecc 100644
--- a/lib/PublicInbox/TestCommon.pm
+++ b/lib/PublicInbox/TestCommon.pm
@@ -144,6 +144,7 @@ sub require_mods {
 			eval "require $mod";
 		}
 		if ($@) {
+			diag "require $mod: $@";
 			push @need, $mod;
 		} elsif ($mod eq 'IO::Socket::SSL' &&
 			# old versions of IO::Socket::SSL aren't supported