[PATCH] test suite: don't consider skipped individual tests as failing

[PATCH] test suite: don't consider skipped individual tests as failing

[David Bremner]

It isn't completely clear what we want to do here, but

1) We currently don't fail if we skip a whole test file (mainly because
we neglect to count those skipped tests properly). This change at least
makes the two kinds of skipping consistent.

2) Automated build environments may have good reasons for building with
a minimal set of prereqs, and we don't want to discourage running our
test suite by breaking builds.
---
 test/aggregate-results.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/test/aggregate-results.sh b/test/aggregate-results.sh
index b016edb..b30da44 100755
--- a/test/aggregate-results.sh
+++ b/test/aggregate-results.sh
@@ -82,7 +82,7 @@ if [ "$skipped" != "0" ]; then
     echo "$skipped $tests skipped."
 fi
 
-if [ $success -gt 0 -a $fixed -eq 0 -a $failed -eq 0 -a $skipped -eq 0 ]
+if [ $success -gt 0 -a $fixed -eq 0 -a $failed -eq 0 ]
 then
     exit 0
 else
-- 
2.5.1

Re: [PATCH] test suite: don't consider skipped individual tests as failing

[Tomi Ollila]

On Thu, Sep 24 2015, David Bremner <david@tethera.net> wrote:

> It isn't completely clear what we want to do here, but
>
> 1) We currently don't fail if we skip a whole test file (mainly because
> we neglect to count those skipped tests properly). This change at least
> makes the two kinds of skipping consistent.
>
> 2) Automated build environments may have good reasons for building with
> a minimal set of prereqs, and we don't want to discourage running our
> test suite by breaking builds.
> ---
>  test/aggregate-results.sh | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/test/aggregate-results.sh b/test/aggregate-results.sh
> index b016edb..b30da44 100755
> --- a/test/aggregate-results.sh
> +++ b/test/aggregate-results.sh
> @@ -82,7 +82,7 @@ if [ "$skipped" != "0" ]; then
>      echo "$skipped $tests skipped."
>  fi
>  

Looks OK, but I think there should be (some good) comment here informing
any potential viewer that we're not checking $skipped...

> -if [ $success -gt 0 -a $fixed -eq 0 -a $failed -eq 0 -a $skipped -eq 0 ]
> +if [ $success -gt 0 -a $fixed -eq 0 -a $failed -eq 0 ]
>  then
>      exit 0
>  else
> -- 
> 2.5.1

Tomi

Re: [PATCH] test suite: don't consider skipped individual tests as failing

[David Bremner]

Tomi Ollila <tomi.ollila@iki.fi> writes:

>
> Looks OK, but I think there should be (some good) comment here informing
> any potential viewer that we're not checking $skipped...
>

I put some vaguely threatening comment in test/README, and a brief
comment in the file. I've merged the commit, but feel free to send
suggestions / patches for further doc changes. I more or less made up a
hypothetical future policy on the spot...

d

Re: [PATCH] test suite: don't consider skipped individual tests as failing

[Tomi Ollila]

On Fri, Sep 25 2015, David Bremner <david@tethera.net> wrote:

> Tomi Ollila <tomi.ollila@iki.fi> writes:
>
>>
>> Looks OK, but I think there should be (some good) comment here informing
>> any potential viewer that we're not checking $skipped...
>>
>
> I put some vaguely threatening comment in test/README, and a brief
> comment in the file. I've merged the commit, but feel free to send
> suggestions / patches for further doc changes. I more or less made up a
> hypothetical future policy on the spot...

ack. earlier today I though of make target test-strict -- let's see
whethere I come up something with that...

>
> d