From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <tomi.ollila@iki.fi>
Received: from localhost (localhost [127.0.0.1])
 by arlo.cworth.org (Postfix) with ESMTP id 86BBF6DE0B7C
 for <notmuch@notmuchmail.org>; Thu, 24 Sep 2015 12:54:08 -0700 (PDT)
X-Virus-Scanned: Debian amavisd-new at cworth.org
X-Spam-Flag: NO
X-Spam-Score: 1.158
X-Spam-Level: *
X-Spam-Status: No, score=1.158 tagged_above=-999 required=5 tests=[AWL=-0.238, 
 SPF_NEUTRAL=0.652, URIBL_SBL=0.644, URIBL_SBL_A=0.1]
 autolearn=disabled
Received: from arlo.cworth.org ([127.0.0.1])
 by localhost (arlo.cworth.org [127.0.0.1]) (amavisd-new, port 10024)
 with ESMTP id dj4nBKrkqUws for <notmuch@notmuchmail.org>;
 Thu, 24 Sep 2015 12:54:06 -0700 (PDT)
Received: from guru.guru-group.fi (guru.guru-group.fi [46.183.73.34])
 by arlo.cworth.org (Postfix) with ESMTP id EFEC26DE0B64
 for <notmuch@notmuchmail.org>; Thu, 24 Sep 2015 12:54:05 -0700 (PDT)
Received: from guru.guru-group.fi (localhost [IPv6:::1])
 by guru.guru-group.fi (Postfix) with ESMTP id E056910008D;
 Thu, 24 Sep 2015 22:54:08 +0300 (EEST)
From: Tomi Ollila <tomi.ollila@iki.fi>
To: David Bremner <david@tethera.net>, notmuch@notmuchmail.org
Subject: Re: [PATCH] test suite: don't consider skipped individual tests as
 failing
In-Reply-To: <1443093210-6511-1-git-send-email-david@tethera.net>
References: <1443093210-6511-1-git-send-email-david@tethera.net>
User-Agent: Notmuch/0.20.2+68~g0c35549 (http://notmuchmail.org) Emacs/24.3.1
 (x86_64-unknown-linux-gnu)
X-Face: HhBM'cA~<r"^Xv\KRN0P{vn'Y"Kd;zg_y3S[4)KSN~s?O\"QPoL
 $[Xv_BD:i/F$WiEWax}R(MPS`^UaptOGD`*/=@\1lKoVa9tnrg0TW?"r7aRtgk[F
 !)g;OY^,BjTbr)Np:%c_o'jj,Z
Date: Thu, 24 Sep 2015 22:54:08 +0300
Message-ID: <m27fnf4mqn.fsf@guru.guru-group.fi>
MIME-Version: 1.0
Content-Type: text/plain
X-BeenThere: notmuch@notmuchmail.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: "Use and development of the notmuch mail system."
 <notmuch.notmuchmail.org>
List-Unsubscribe: <http://notmuchmail.org/mailman/options/notmuch>,
 <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>
List-Archive: <http://notmuchmail.org/pipermail/notmuch/>
List-Post: <mailto:notmuch@notmuchmail.org>
List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>
List-Subscribe: <http://notmuchmail.org/mailman/listinfo/notmuch>,
 <mailto:notmuch-request@notmuchmail.org?subject=subscribe>
X-List-Received-Date: Thu, 24 Sep 2015 19:54:08 -0000

On Thu, Sep 24 2015, David Bremner <david@tethera.net> wrote:

> It isn't completely clear what we want to do here, but
>
> 1) We currently don't fail if we skip a whole test file (mainly because
> we neglect to count those skipped tests properly). This change at least
> makes the two kinds of skipping consistent.
>
> 2) Automated build environments may have good reasons for building with
> a minimal set of prereqs, and we don't want to discourage running our
> test suite by breaking builds.
> ---
>  test/aggregate-results.sh | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/test/aggregate-results.sh b/test/aggregate-results.sh
> index b016edb..b30da44 100755
> --- a/test/aggregate-results.sh
> +++ b/test/aggregate-results.sh
> @@ -82,7 +82,7 @@ if [ "$skipped" != "0" ]; then
>      echo "$skipped $tests skipped."
>  fi
>  

Looks OK, but I think there should be (some good) comment here informing
any potential viewer that we're not checking $skipped...

> -if [ $success -gt 0 -a $fixed -eq 0 -a $failed -eq 0 -a $skipped -eq 0 ]
> +if [ $success -gt 0 -a $fixed -eq 0 -a $failed -eq 0 ]
>  then
>      exit 0
>  else
> -- 
> 2.5.1

Tomi