From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0 ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id 8JhzJjQlomDjZAEAgWs5BA (envelope-from ) for ; Mon, 17 May 2021 10:11:32 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0 with LMTPS id sGYmIjQlomCBQAAA1q6Kng (envelope-from ) for ; Mon, 17 May 2021 08:11:32 +0000 Received: from mail.notmuchmail.org (nmbug.tethera.net [144.217.243.247]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id B3F511AA04 for ; Mon, 17 May 2021 10:11:31 +0200 (CEST) Received: from nmbug.tethera.net (localhost [127.0.0.1]) by mail.notmuchmail.org (Postfix) with ESMTP id 432482A9B8; Mon, 17 May 2021 04:11:27 -0400 (EDT) Received: from lahtoruutu.iki.fi (unknown [IPv6:2a0b:5c81:1c1::37]) by mail.notmuchmail.org (Postfix) with ESMTPS id 4ACC92A9A9 for ; Mon, 17 May 2021 04:11:24 -0400 (EDT) Received: from guru.guru-group.fi (guru.guru-group.fi [46.183.73.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) (Authenticated sender: too) by lahtoruutu.iki.fi (Postfix) with ESMTPSA id D9A801B001B4; Mon, 17 May 2021 11:11:12 +0300 (EEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=iki.fi; s=lahtoruutu; t=1621239072; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding; bh=7KdlYNCqsPfAluJ8cF4GEDun7oiD//kbH4r8KEpHFTc=; b=FfTvS8DF6oG/hfknlLIsA3zMhkIpJfwEeBbxQYTvZOYTCzA+Jokmy6QBnnV0Xkt8QN+1Bl LK3kLpOVfMTmtXVrw6cezQ72Nj3LN5XDZHw55bs+J3HaetYNQ5/cEt7HETCjFeqKNMU5he 38ZeCVh2yzzaBPcD5n1HXeoy134AHPAE3TTFMG2foR3lQQG1ER4KLZSKqwWpq4VsbVFect xXLv9HzFznrN4d7zpmop1AcLa3u6tZ1SzsVEV2I8trNSvoF/ZdBXvIhuRrnenhhQ5NebKJ Y6hXvJLQ1Sr9m60wkwYUVg5QnWn71FRCo+uZ0ltx+/tSxZp9jlALBI/hbx1TdA== From: Tomi Ollila To: notmuch@notmuchmail.org Subject: [PATCH] test: aggregate-results updates Date: Mon, 17 May 2021 11:11:09 +0300 Message-Id: <20210517081109.31897-1-tomi.ollila@iki.fi> X-Mailer: git-send-email 2.30.0 MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=iki.fi; s=lahtoruutu; t=1621239072; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding; bh=7KdlYNCqsPfAluJ8cF4GEDun7oiD//kbH4r8KEpHFTc=; b=dbA9NWeOA5McKfNdm61gVq1ro2H27lkm5vVl3KpkTI+Y/UQVAtjM4uDlOIze7mcvb6CRer ZSyTlPi6AYxV1NUoGdWjYqQ/HkHZ3zy7etV3WfMFWuFKkfzHYCI0mm2GVNto3tb5tI9exD orj2hWU5PfxRaYcMjgJaqK6Iv9sairM1QZdDFdObugGudrkTVqY6iVVrf/IZELZRrw1Qyb +e+E2Gzs6GW2eTTD/XSyFkaDOg2PNrmsVNKibvnTShaDpktaRTNjd2BQxWkoIUX5YvHdLL Ds1BQ0Fc4FiM+XstwhSeZPM2soKwcewjwufF8nR1hAr1povyXXobac/muuDx2g== ARC-Authentication-Results: i=1; ORIGINATING; auth=pass smtp.auth=too smtp.mailfrom=tomi.ollila@iki.fi ARC-Seal: i=1; s=lahtoruutu; d=iki.fi; t=1621239072; a=rsa-sha256; cv=none; b=McSedPykMIo5ITrpqTHp8CSvAam5AWnMT2gE0RLa9ewK5t9mdcDfqyWwrU0gerZ8TzlCUA FT3t9q84yyZFqT3XU+rp1L+ydog//u6SzV1r2BB81AlxOKfFoJUs8P22+7tO36MAFsTpW+ QFrnfj5YLGEW/1E1J1M0Mcu4hhmHa10rG0Ol3smRv9w1CH36grNpIgz1CA5NVXyT2tr8f7 5l8uDqP5LbxR5J3UmSnhrHZM1qKKqFhzt2Q3RfphpmXVVEnR3L3JUhoFFlHf2isfNNred1 e3h3IoWFR28ygjhLQUQOYVGLzX9QQUDttY+JZVqu21sg8/V3nC1znH7jY179AQ== Message-ID-Hash: YFPZHPKGZOPHFN3I4AW4ATCFJISBZTIQ X-Message-ID-Hash: YFPZHPKGZOPHFN3I4AW4ATCFJISBZTIQ X-MailFrom: tomi.ollila@iki.fi X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; header-match-notmuch.notmuchmail.org-0; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; suspicious-header CC: tomi.ollila@iki.fi X-Mailman-Version: 3.2.1 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Help: List-Post: List-Subscribe: List-Unsubscribe: Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1621239091; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding:list-id:list-help: list-unsubscribe:list-subscribe:list-post:dkim-signature; bh=2Ek2oVGia0OAK0lWtPiMR7LMt2a1aMjY+q+Gc7hUbVg=; b=c9p/OwOizFz7TCaf7EqQu1tO9ZZTJjmhVq1ZM47UgOXMlkXGX9WpCGSe5B1eBbNrkobu4p iBP7gMU/DZGLYIpGf5zf8nJZPYhnbzPOE+hp99llbV0PKQX4MWgfBGsakSwnAIAg+A/v/n PVPY1u7tQ6WwW5GOHqa9qyeCrTtwnj8TD66EPUF3yCIwYtaey/xt8tjVRooWR2w40yiY6S Yb3CdDTodbcJWj48PTVJR9pNRSG3nbjIKstU5r09BStVT0EVlTql3iao2VKkihcwXVGMlw 6nBdlsA6GB15tOydAQ4KOkSSBivUFzgBcqs6h5x9DxzLTaW7ONHtPkiUkgXL2w== ARC-Seal: i=2; s=key1; d=yhetil.org; t=1621239091; a=rsa-sha256; cv=fail; b=rSWpyx/iSzzg7asLCd3U9uj617onnBy28WQ3rXYL74eDf7QPyay5w7QYHGm6ILRuBI4CWb OkRw4GpqWp5mTbXb8/56Ooq5slX2yCoW4teErQDx4n5IJm3zF3Ee+IKlcKMi+PuCLZvBoP Aal13G5k6v6zDx54QoMhoHBONHxZb+xn3ZXCN7BcoqRyEoMVuvPedIV8uMkUhvAnPXSI9W fs0bCt0+MwIytCkF90RmTiEW2WSyUWvDqsQJttEYqJQhmt2ltUt8PclDUlPqJD9njr/EwV 735NJT02/0xjUPRUYuYK4JzfZdXv95xRAsdkJEnULPUMrXsckjOTOyBcPMz8oA== ARC-Authentication-Results: i=2; aspmx1.migadu.com; dkim=fail ("body hash did not verify") header.d=iki.fi header.s=lahtoruutu header.b=FfTvS8DF; arc=reject ("signature check failed: fail, {[1] = sig:iki.fi:reject}"); dmarc=none; spf=pass (aspmx1.migadu.com: domain of notmuch-bounces@notmuchmail.org designates 144.217.243.247 as permitted sender) smtp.mailfrom=notmuch-bounces@notmuchmail.org X-Migadu-Spam-Score: 1.93 Authentication-Results: aspmx1.migadu.com; dkim=fail ("body hash did not verify") header.d=iki.fi header.s=lahtoruutu header.b=FfTvS8DF; arc=reject ("signature check failed: fail, {[1] = sig:iki.fi:reject}"); dmarc=none; spf=pass (aspmx1.migadu.com: domain of notmuch-bounces@notmuchmail.org designates 144.217.243.247 as permitted sender) smtp.mailfrom=notmuch-bounces@notmuchmail.org X-Migadu-Queue-Id: B3F511AA04 X-Spam-Score: 1.93 X-Migadu-Scanner: scn0.migadu.com X-TUID: y8WITHKVey87 notmuch-test will now call aggregate-results.sh with file list that it compiles based on the test ran, and aggregate-results will report failure is any of the test files are missing. With this notmuch-test no longer has to exit in non-parallel run if some test fail to write its report file -- so it works as parallel tests in this sense. Changed test_done() in test-lib.sh write report file in one write(2), so there is (even) less chance it being partially written. Also, now it writes 'total' last and aggregate-results.sh expects this line to exist in all report files for reporting to be successful. Added 'set -eu' to notmuch-test and modified code to work with these settings. That makes it harder to get mistakes slipped into committed code. --- testing all changed branches is somewhat hard; I did by adding sleep 20 and file what effective removed created results files; another change was to change 'total' to 'toxtal' in test_done, then I run (at least) the following command lines: make test NOTMUCH_TESTS='T140-excludes.sh T740-body.sh' make test NOTMUCH_TESTS='T140-excludes.sh T740-body.sh' NOTMUCH_TEST_SERIALIZE=t make test NOTMUCH_TESTS=T160-json.sh NOTMUCH_TEST_SERIALIZE=t test/notmuch-test test/aggregate-results.sh | 22 ++++++++++++++++-- test/notmuch-test | 47 ++++++++++++++++++++++----------------- test/test-lib.sh | 15 +++++++------ 3 files changed, 54 insertions(+), 30 deletions(-) diff --git a/test/aggregate-results.sh b/test/aggregate-results.sh index 75400e6e..6845fcf0 100755 --- a/test/aggregate-results.sh +++ b/test/aggregate-results.sh @@ -8,9 +8,16 @@ failed=0 broken=0 total=0 all_skipped=0 +rep_failed=0 for file do + if [ ! -f "$file" ]; then + echo "'$file' does not exist!" + rep_failed=$((rep_failed + 1)) + continue + fi + has_total=0 while read type value do case $type in @@ -24,18 +31,23 @@ do broken=$((broken + value)) ;; total) total=$((total + value)) + has_total=1 if [ "$value" -eq 0 ]; then all_skipped=$((all_skipped + 1)) fi esac done <"$file" + if [ "$has_total" -eq 0 ]; then + echo "'$file' lacks 'total ...'; results may be inconsistent." + failed=$((failed + 1)) + fi done pluralize_s () { [ "$1" -eq 1 ] && s='' || s='s'; } echo "Notmuch test suite complete." -if [ "$fixed" -eq 0 ] && [ "$failed" -eq 0 ]; then +if [ "$fixed" -eq 0 ] && [ "$failed" -eq 0 ] && [ "$rep_failed" -eq 0 ]; then pluralize_s "$total" printf "All $total test$s " if [ "$broken" -eq 0 ]; then @@ -70,10 +82,16 @@ if [ "$all_skipped" -ne 0 ]; then echo "All tests in $all_skipped file$s skipped." fi +if [ "$rep_failed" -ne 0 ]; then + pluralize_s "$rep_failed" + echo "$rep_failed test$s failed to report results." +fi + # Note that we currently do not consider skipped tests as failing the # build. -if [ "$success" -gt 0 ] && [ "$fixed" -eq 0 ] && [ "$failed" -eq 0 ] +if [ "$success" -gt 0 ] && [ "$fixed" -eq 0 ] && + [ "$failed" -eq 0 ] && [ "$rep_failed" -eq 0 ] then exit 0 else diff --git a/test/notmuch-test b/test/notmuch-test index cbd33f93..ce142f7c 100755 --- a/test/notmuch-test +++ b/test/notmuch-test @@ -18,12 +18,14 @@ fi # Ensure NOTMUCH_SRCDIR and NOTMUCH_BUILDDIR are set. . $(dirname "$0")/export-dirs.sh || exit 1 +set -eu + TESTS= -for test in $NOTMUCH_TESTS; do +for test in ${NOTMUCH_TESTS-}; do TESTS="$TESTS $NOTMUCH_SRCDIR/test/$test" done -if [[ -z "$TESTS" ]]; then +if [ -z "$TESTS" ]; then TESTS="$NOTMUCH_SRCDIR/test/T[0-9][0-9][0-9]-*.sh" fi @@ -44,43 +46,46 @@ else TEST_TIMEOUT_CMD="" fi -trap 'e=$?; kill $!; exit $e' HUP INT TERM - META_FAILURE= +RES=0 # Run the tests -if test -z "$NOTMUCH_TEST_SERIALIZE" && command -v parallel >/dev/null ; then +if test -z "${NOTMUCH_TEST_SERIALIZE-}" && command -v parallel >/dev/null ; then test -t 1 && export COLORS_WITHOUT_TTY=t || : if parallel --version 2>&1 | grep -q GNU ; then echo "INFO: running tests with GNU parallel" - printf '%s\n' $TESTS | $TEST_TIMEOUT_CMD parallel + printf '%s\n' $TESTS | $TEST_TIMEOUT_CMD parallel || RES=$? else echo "INFO: running tests with moreutils parallel" - $TEST_TIMEOUT_CMD parallel -- $TESTS + $TEST_TIMEOUT_CMD parallel -- $TESTS || RES=$? fi - RES=$? - if [[ $RES != 0 ]]; then + if [ $RES != 0 ]; then META_FAILURE="parallel test suite returned error code $RES" fi else + trap 'e=$?; trap - 0; kill ${!-}; exit $e' 0 HUP INT TERM for test in $TESTS; do $TEST_TIMEOUT_CMD $test "$@" & - wait $! - # If the test failed without producing results, then it aborted, - # so we should abort, too. - RES=$? - testname=$(basename $test .sh) - if [[ $RES != 0 && ! -e "$NOTMUCH_BUILDDIR/test/test-results/$testname" ]]; then - META_FAILURE="Aborting on $testname (returned $RES)" - break - fi + wait $! && ev=0 || ev=$? + test $ev = 0 || RES=$ev done + trap - 0 HUP INT TERM + if [ $RES != 0 ]; then + META_FAILURE="some tests failed; first failed returned error code $RES" + fi fi -trap - HUP INT TERM # Report results +RESULT_FILES= +for file in $TESTS +do + file=${file##*/} # drop leading path components + file=${file%.sh} # drop trailing '.sh' + RESULT_FILES="$RESULT_FILES $NOTMUCH_BUILDDIR/test/test-results/$file" +done + echo -$NOTMUCH_SRCDIR/test/aggregate-results.sh $NOTMUCH_BUILDDIR/test/test-results/* -ev=$? +$NOTMUCH_SRCDIR/test/aggregate-results.sh $RESULT_FILES && ev=0 || ev=$? + if [ -n "$META_FAILURE" ]; then printf 'ERROR: %s\n' "$META_FAILURE" if [ $ev = 0 ]; then diff --git a/test/test-lib.sh b/test/test-lib.sh index d7c96a93..2dceb497 100644 --- a/test/test-lib.sh +++ b/test/test-lib.sh @@ -1021,15 +1021,16 @@ test_when_finished () { test_done () { GIT_EXIT_OK=t test_results_dir="$TEST_DIRECTORY/test-results" - mkdir -p "$test_results_dir" + test -d "$test_results_dir" || mkdir "$test_results_dir" test_results_path="$test_results_dir/$this_test" - echo "total $test_count" >> $test_results_path - echo "success $test_success" >> $test_results_path - echo "fixed $test_fixed" >> $test_results_path - echo "broken $test_broken" >> $test_results_path - echo "failed $test_failure" >> $test_results_path - echo "" >> $test_results_path + printf %s\\n \ + "success $test_success" \ + "fixed $test_fixed" \ + "broken $test_broken" \ + "failed $test_failure" \ + "total $test_count" \ + > $test_results_path [ -n "$EMACS_SERVER" ] && test_emacs '(kill-emacs)' -- 2.25.1