From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp10.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id WDEZF4DvBGL8RAEAgWs5BA (envelope-from ) for ; Thu, 10 Feb 2022 11:57:04 +0100 Received: from aspmx1.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp10.migadu.com with LMTPS id OCu6D4DvBGJlIgEAG6o9tA (envelope-from ) for ; Thu, 10 Feb 2022 11:57:04 +0100 Received: from mail.notmuchmail.org (yantan.tethera.net [IPv6:2a01:4f9:c011:7a79::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id D0FA81B0B4 for ; Thu, 10 Feb 2022 11:57:03 +0100 (CET) Received: from yantan.tethera.net (localhost [127.0.0.1]) by mail.notmuchmail.org (Postfix) with ESMTP id A31155F718; Thu, 10 Feb 2022 10:57:01 +0000 (UTC) X-Greylist: delayed 3073 seconds by postgrey-1.36 at yantan; Thu, 10 Feb 2022 10:56:58 UTC Received: from mailproxy05.manitu.net (mailproxy05.manitu.net [IPv6:2a00:1828:1000:1110::2]) by mail.notmuchmail.org (Postfix) with ESMTPS id 391395F714 for ; Thu, 10 Feb 2022 10:56:58 +0000 (UTC) Received: from localhost (200116b8605b9100a390552ff161dcf3.dip.versatel-1u1.de [IPv6:2001:16b8:605b:9100:a390:552f:f161:dcf3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: michael@grubix.eu) by mailproxy05.manitu.net (Postfix) with ESMTPSA id 13D691B60650; Thu, 10 Feb 2022 11:56:57 +0100 (CET) MIME-Version: 1.0 In-Reply-To: <87pmnvzo3b.fsf@tethera.net> References: <8d483901f3cf9bee86d7359ff1d665943adcd31b.1644406756.git.git@grubix.eu> <87pmnvzo3b.fsf@tethera.net> Subject: Re: [PATCH 2/4] test: due not pass T380.1 for the wrong reasons From: Michael J Gruber To: David Bremner , notmuch@notmuchmail.org Message-ID: <164449061694.6325.1611477846768874524.git@grubix.eu> Date: Thu, 10 Feb 2022 11:56:56 +0100 User-Agent: alot/0.10 Message-ID-Hash: KTSECS4VBTKBVGHUK2OLHLPCDSBZCYFD X-Message-ID-Hash: KTSECS4VBTKBVGHUK2OLHLPCDSBZCYFD X-MailFrom: michael@grubix.eu X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; header-match-notmuch.notmuchmail.org-0; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.3 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Migadu-Flow: FLOW_IN X-Migadu-Country: DE ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1644490624; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-owner:list-unsubscribe:list-subscribe:list-post; bh=aJx8Yjnb1SdTdNzpyqQdh92rpFL2nfRNyNbqlyCi6pI=; b=mpWcspSOWjKR2rFcPEx67cIv4WjP6UIKSrRRW9DQy4JEvarnk79lcRiYyVnjY20xw3uO9A lYWYC4VvAt4aHR77J7luhCEIfRsxXpd7+ByT24JxZMrsVeUpuddEjlssew1yp8K6oFgDOa Ww0OIjZVINB9sZmvZ19pQ5h721UskofvpdPYTd/nD/Gn98nqZjq+VWLDWhvv+f+tsGEg4n Aq8r1Nvp7/rniA+8eub7yAKz6P+VlHh5acs9tSiX3tg+OjdTCV/U/Vm63u2FyQAbLe3mPo V69xlE0GcN+C+ykwwtLAb+NQ1pZaE4l+HXoQhA6SS52cWU5UDaUQKqUVrGvzJQ== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1644490624; a=rsa-sha256; cv=none; b=ALt3G4lVAsqYdSytTDTw9799Hwj/XfVwcwGDNbOhbBso76x3h6o+vJOmgtCGYXYmpx0rh2 cswvfPw0xq4s0j7fbjVfxweFNTjG6+1rzlUqXd2vgUnFCBzcL3MfgsHGRADykVwcYEyk9t 4wUJlAEEfbNhq8iS5z0FVtKKTPVnZ3f1R9sgKS4sRvjSvaCrs/h8RxGMu4ZZfVWRhh1xIF sD7kGYD/hn9sU12w4oLMeWxySiZtal7LXWUOlGKz8p07ZA/EX3TK6onYcdQKJ2ywDIzgb+ sc+PqfhgLhNciCZECybmwHn+aLcxwTpgBz7oJ/OQ9MByhudyMe5y7Qb8VgOgiw== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of notmuch-bounces@notmuchmail.org designates 2a01:4f9:c011:7a79::1 as permitted sender) smtp.mailfrom=notmuch-bounces@notmuchmail.org X-Migadu-Spam-Score: -1.33 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of notmuch-bounces@notmuchmail.org designates 2a01:4f9:c011:7a79::1 as permitted sender) smtp.mailfrom=notmuch-bounces@notmuchmail.org X-Migadu-Queue-Id: D0FA81B0B4 X-Spam-Score: -1.33 X-Migadu-Scanner: scn1.migadu.com X-TUID: EriO0HvCatNf David Bremner venit, vidit, dixit 2022-02-09 23:12:24: > Michael J Gruber writes: > > > If gdb is missing then some files are never written to so that the > > comparisons of non-existing files succeeds for the wrong reason, > > claiming that `notmch new` is idempotent when it was in fact never run. > > > > Catch this and (for lack of a better spot) set up the files with a > > reason for the FAIL. > > > > I'm a bit confused by this. For me if gdb is missing I get > > missing prerequisites: gdb(1) > SKIP all tests in T380-atomicity > > ... > > All 1842 tests behaved as expected (8 expected failures). > All tests in 1 file skipped. > > Do I misunderstand something, or is prerequisite detection not working > for you? It does now, my understanding has been mislead by a few things. The issues were: On one of Fedora's arches which is emulated in the COPR test environment, gdb was present (just like in all of them) but apparantly failed to set any break points. T380 spew out something like cat: outcount: No such file or directory ./T380-atomicity.sh: line 79: ((: i < : syntax error: operand expected (error token is "< ") followed by wrong PASS/FAIL etc. When analysing this, I was confused by the way test_require_external_prereq works and the "if" in T380 (as opposed to how test_require_external_prereq is used in other tests). Over at git.git, we have test setup code in functions which don't get executed if prerequisites fail. I guess the "if" emulates that, but then the actual tests in T380 are outside the if block and use files and variables which are created in the if block. So, this is something to fix anyways. In order to try things out locally, I changed if test_require_external_prereq gdb; then into if test_require_external_prereq gggdb; then I know now that `test_require_external_prereq binary` does not test for the presence of binary, nor does it abort the test script. It merely checks among the set of "predefined external prerequisites", and this would be much clearer if "binary" wasn't an argument to that function but a suffix: `test_require_external_prereq_gdb` immediately says "I'm something predefined". Until now, I wondered how `test_require_external_prereq` could skip the PASS/FAIL outout if it could not skip the previous part of the test script. This is not clear from its implementation. I know now that indeed text_expect-success etc. call test_check_missing_external_prereqs_ for the current subtest and skip the output. Add to this the fact that the tests needing sfsexp or asan (and probably others) do things yet differently and call "test_done" immediately, so that no SKIP appears. And those were the only ones skipped at all here ... In effect, I knoew SKIP only from manually skipping tests (NOTMUCH_SKIP_TESTS). I'm not complaining, on the other hand I don't fail too bad about myself for being confused by this ;) So, in the long run I think test_setup() is worth thinking about. In the short run, initialising variables and files which are used is still a good thing, but I would have to rewrite some commit messages. I'll wait until it's clear how to handle style, though: switch to printf from echo whenever I touch those lines (leading to mixed use) or keeing style and leaving the style change for another series. Cheers, Michael