From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from localhost (localhost [127.0.0.1]) by arlo.cworth.org (Postfix) with ESMTP id 8F7826DE0EEA for ; Sun, 26 May 2019 08:06:24 -0700 (PDT) X-Virus-Scanned: Debian amavisd-new at cworth.org X-Spam-Flag: NO X-Spam-Score: -0.16 X-Spam-Level: X-Spam-Status: No, score=-0.16 tagged_above=-999 required=5 tests=[AWL=0.041, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001] autolearn=disabled Received: from arlo.cworth.org ([127.0.0.1]) by localhost (arlo.cworth.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ExE17novYpfU for ; Sun, 26 May 2019 08:06:23 -0700 (PDT) Received: from che.mayfirst.org (che.mayfirst.org [162.247.75.118]) by arlo.cworth.org (Postfix) with ESMTPS id 27A2E6DE0F9E for ; Sun, 26 May 2019 08:06:23 -0700 (PDT) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/simple; d=fifthhorseman.net; i=@fifthhorseman.net; q=dns/txt; s=2019; t=1558883181; h=from : to : subject : in-reply-to : references : date : message-id : mime-version : content-type : from; bh=4fsaZu0Tfe1pNks5IAy08ngQypQ5DXWsNEyHgnMVF50=; b=YeCB/rhLvqO1fkQWeX3736JmZrVVVd43LAEfeqZGPwfhlwCLOHeJx3DX sp3yQqqKvUgjn6gpIJg8MZOQXFAbAg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=fifthhorseman.net; i=@fifthhorseman.net; q=dns/txt; s=2019rsa; t=1558883180; h=from : to : subject : in-reply-to : references : date : message-id : mime-version : content-type : from; bh=4fsaZu0Tfe1pNks5IAy08ngQypQ5DXWsNEyHgnMVF50=; b=wTPDzNjmlOCCMrbmHlnwSvAdxE26QNQTY9G6C5+rHGedan14vQQQOJCO dCZw41ITueT5noK3TxkMozWIVDDPCjYlDKkVi1BoQSLqWj2R7ZyaHLMT83 gHNYNWy4SasE/5LvF0kwAd38wnI3VnmehR9EbV222bJRW63oKYPTsW6D94 Fv+IlZCiDTjkKav6xcuvSEWHRxIyYCKlmhg+4z1Q1pcPMGDd8Nzr3Qfu6l O/mgsVOYjcL9V7zDtlKXzMeT/GrfYX8AKCMkL+lg0aXvQUd4+uQWiK5BBE emUvnzYd+ZzMy8JdcbScESCoPiZ/+y24ZZoQKOZVl63bQDavZJASYA== Received: from fifthhorseman.net (cpe-74-71-53-242.nyc.res.rr.com [74.71.53.242]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by che.mayfirst.org (Postfix) with ESMTPSA id DAEE0F99E; Sun, 26 May 2019 11:06:19 -0400 (EDT) Received: by fifthhorseman.net (Postfix, from userid 1000) id BC4C2202A0; Sun, 26 May 2019 11:06:16 -0400 (EDT) From: Daniel Kahn Gillmor To: Tomi Ollila , Notmuch Mail Subject: Re: [PATCH] test: report summary even when aborting In-Reply-To: References: <20190525173050.8707-1-dkg@fifthhorseman.net> Autocrypt: addr=dkg@fifthhorseman.net; prefer-encrypt=mutual; keydata= mDMEXEK/AhYJKwYBBAHaRw8BAQdAr/gSROcn+6m8ijTN0DV9AahoHGafy52RRkhCZVwxhEe0K0Rh bmllbCBLYWhuIEdpbGxtb3IgPGRrZ0BmaWZ0aGhvcnNlbWFuLm5ldD6ImQQTFggAQQIbAQUJA8Jn AAULCQgHAgYVCgkICwIEFgIDAQIeAQIXgBYhBMS8Lds4zOlkhevpwvIGkReQOOXGBQJcQsbzAhkB AAoJEPIGkReQOOXG4fkBAO1joRxqAZY57PjdzGieXLpluk9RkWa3ufkt3YUVEpH/AP9c+pgIxtyW +FwMQRjlqljuj8amdN4zuEqaCy4hhz/1DbgzBFxCv4sWCSsGAQQB2kcPAQEHQERSZxSPmgtdw6nN u7uxY7bzb9TnPrGAOp9kClBLRwGfiPUEGBYIACYWIQTEvC3bOMzpZIXr6cLyBpEXkDjlxgUCXEK/ iwIbAgUJAeEzgACBCRDyBpEXkDjlxnYgBBkWCAAdFiEEyQ5tNiAKG5IqFQnndhgZZSmuX/gFAlxC v4sACgkQdhgZZSmuX/iVWgD/fCU4ONzgy8w8UCHGmrmIZfDvdhg512NIBfx+Mz9ls5kA/Rq97vz4 z48MFuBdCuu0W/fVqVjnY7LN5n+CQJwGC0MIA7QA/RyY7Sz2gFIOcrns0RpoHr+3WI+won3xCD8+ sVXSHZvCAP98HCjDnw/b0lGuCR7coTXKLIM44/LFWgXAdZjm1wjODbg4BFxCv50SCisGAQQBl1UB BQEBB0BG4iXnHX/fs35NWKMWQTQoRI7oiAUt0wJHFFJbomxXbAMBCAeIfgQYFggAJhYhBMS8Lds4 zOlkhevpwvIGkReQOOXGBQJcQr+dAhsMBQkB4TOAAAoJEPIGkReQOOXGe/cBAPlek5d9xzcXUn/D kY6jKmxe26CTws3ZkbK6Aa5Ey/qKAP0VuPQSCRxA7RKfcB/XrEphfUFkraL06Xn/xGwJ+D0hCw== Date: Sun, 26 May 2019 11:06:16 -0400 Message-ID: <87woid45nb.fsf@fifthhorseman.net> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" X-BeenThere: notmuch@notmuchmail.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 26 May 2019 15:06:24 -0000 --=-=-= Content-Type: text/plain Content-Transfer-Encoding: quoted-printable Thanks for the feedback, Tomi! On Sat 2019-05-25 22:41:58 +0300, Tomi Ollila wrote: > On Sat, May 25 2019, Daniel Kahn Gillmor wrote: > >> In certain cases of test suite failure, the summary report was not >> being printed. In particular, any failure on the parallel test suite, >> and any aborted test in the serialized test suite would end up hiding >> the summary. >> >> It's better to always show the summary where we can (while preserving >> the return code). > > This is an improvement, but it looks like it is possible > aggregate-results.sh prints everything good (like all 3 tests passed) > since it may be that there are only good logs in test-results/. > > That might, in some cases, be confusing (exit value of notmuch-test > is not always visible).=20 Yes, i share your concern about confusion, and i tried to stave off that confusion by making sure that there was something printed in each of the fallthrough cases. But you're right that i didn't do that well enough. v2 of my patch also moves the printed warning to after aggregate-results.sh output, and prefixes it with "ERROR:", so that it's more obvious that this is why we are terminating non-zero. > One option could be giving $RES to aggregate-results.sh and if it is > not zero then that could print different text -- just that it requires > a bit more coding... I'm not convinced that's a worthwhile tradeoff. i don't want to add an option parser to aggregate-results.sh, and it uses all of "$@" already. So in the simplest implementation we'd be passing some sort of environment variable to it -- but that variable wouldn't do anything other than print what notmuch-test is already printing. certainly if we just passed $RES we'd have even less information about the context of where/how the meta-failure happened. My v2 patch does a better job of this, hopefully. So if someone has strong feelings about this and wants to update aggregate-results.sh instead, feel free to do so; i wouldn't block it, but hopefully the current proposal is acceptable. thanks for the review, --dkg --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iHUEARYKAB0WIQTJDm02IAobkioVCed2GBllKa5f+AUCXOqraAAKCRB2GBllKa5f +HFLAQDVOl2oUk4mPn3MgmdLtVHymkwp6zIDPOWUtRuqfzAumgEA2J7Erd25ymOo zjrpN586y39F7g3nYXF1JVtZdf7Q5QM= =DwLZ -----END PGP SIGNATURE----- --=-=-=--