From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id CF0wEF6bxV5qWwAA0tVLHw (envelope-from ) for ; Wed, 20 May 2020 21:04:30 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2 with LMTPS id OJMUDF6bxV4KWwAAB5/wlQ (envelope-from ) for ; Wed, 20 May 2020 21:04:30 +0000 Received: from arlo.cworth.org (arlo.cworth.org [50.126.95.6]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 8AF94940363 for ; Wed, 20 May 2020 21:04:28 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by arlo.cworth.org (Postfix) with ESMTP id 3FB536DE1395; Wed, 20 May 2020 14:04:25 -0700 (PDT) X-Virus-Scanned: Debian amavisd-new at cworth.org Received: from arlo.cworth.org ([127.0.0.1]) by localhost (arlo.cworth.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Kd-GJVM4A2Vr; Wed, 20 May 2020 14:04:24 -0700 (PDT) Received: from arlo.cworth.org (localhost [IPv6:::1]) by arlo.cworth.org (Postfix) with ESMTP id EE27E6DE13BE; Wed, 20 May 2020 14:04:23 -0700 (PDT) Received: from localhost (localhost [127.0.0.1]) by arlo.cworth.org (Postfix) with ESMTP id 6E51B6DE13BE for ; Wed, 20 May 2020 14:04:22 -0700 (PDT) X-Virus-Scanned: Debian amavisd-new at cworth.org Received: from arlo.cworth.org ([127.0.0.1]) by localhost (arlo.cworth.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id HaiOkJ63-Zhw for ; Wed, 20 May 2020 14:04:21 -0700 (PDT) Received: from lahtoruutu.iki.fi (lahtoruutu.iki.fi [212.16.98.55]) by arlo.cworth.org (Postfix) with ESMTPS id DBB596DE1395 for ; Wed, 20 May 2020 14:04:18 -0700 (PDT) Received: from guru.guru-group.fi (unknown [IPv6:2a02:2380:1:9:5054:ff:feb7:a4bc]) (using TLSv1.2 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: too) by lahtoruutu.iki.fi (Postfix) with ESMTPSA id 102831B00120; Thu, 21 May 2020 00:04:13 +0300 (EEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=iki.fi; s=lahtoruutu; t=1590008653; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=Xe81Z8bjEMnBwoO34kQRZ7sa7KvdGMziKGN4rOz9CjU=; b=necl9D36X2SujiqmTAItUzynZJ1RRHutLPuIv/S9en2PUQuc7XszFNCCROHJvQpEexu9Ox kre3sUW+lY36lIGV5wHtKUupedl3WfDmO8O2EZ65gJ6W3vdJ/LXmMapGwlusXWUkdbEAiP tVDlwj7pPa0RY6w5sQPUyqyNYIe7oSNo84AtIv23tSj1cnk8DcONSSY7WRUMw9GOV16lF2 i/CfJEL/WtalfqQDpxTCR1jc31pTAQy8KegyOa+m7T8bAeU2IPrxAJVlWjo8mglKVSEyXf sf4TBO2P7iSQfPqrc/ZNQb6rGQu+KnnBQT/mBJyb5PlZG+Tkwnd3FtNgXg18kg== From: Tomi Ollila To: Daniel Kahn Gillmor , notmuch@notmuchmail.org Subject: Re: test suite: FIXED messages are misordered with tests In-Reply-To: <87eeron74q.fsf@fifthhorseman.net> References: <87eeron74q.fsf@fifthhorseman.net> User-Agent: Notmuch/0.28.3+84~g41389bb (https://notmuchmail.org) Emacs/26.3 (x86_64-pc-linux-gnu) X-Face: HhBM'cA~ MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=iki.fi; s=lahtoruutu; t=1590008653; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=Xe81Z8bjEMnBwoO34kQRZ7sa7KvdGMziKGN4rOz9CjU=; b=dMk2QeB0tt57+OQC/ieEtpitpNt5axfjiVUsSMYReM4scCrIBS41OWmq8cDNhBn66IUzwV 2+AkefXxP3O+U7Ci/+Jrk+Yi7bOdhHfNUe90lCvxiDXLAgYn6KiaOxrd8vYYNzOXkdnSY8 5uvkj0OJ1A+meKRvSu+jo8x5Q47QJH1aYARCWEnVrehHQoI6oZdYNiTJGfYH4QGB0kq6kd 5LY/ZPD/t+jRjUfhff3Mz/04Wkbf0wf0pQ44ghaOPv+Luk6DfQeBFtmnuJUMpVkRrdaYrh n8b+IbiZrwecxySPL8Km7kYmMmBvFbrOSR0YqzbZe7BZ7Tj7OQ+k/g523HP2rA== ARC-Seal: i=1; s=lahtoruutu; d=iki.fi; t=1590008653; a=rsa-sha256; cv=none; b=U0hDSRUmmERB5avMa32aNAUmcj/JnfXHkc7ujK3z+G6GZygH4UdlfgaLeikizxhfMZ3EGP a4AIIUitKNCG3yqD5s+et2ofbOk8h0ZrUCvpkdASKcvCcmFI+qmQW3EAvZsnh0hrab66M/ ZgzCjIQIfw/baXWdUq+OqRVsSetZ1oujjWHGC/0XZXpnOfhn1Ff0Jcb2UHL3ql5quPjS3+ hJbmRJMTekHElncOs/2ATykMHwoeAHCWLH/BtsHAbrI5UV9y9tukrzxTSuakDNFTcREvGD V9R2XbqVbExs6ocZe3zMl9PkvFUxm9uNLylK8IAR5qAn3EL3xPgmMTcIqcZKTg== ARC-Authentication-Results: i=1; ORIGINATING; auth=pass smtp.auth=too smtp.mailfrom=tomi.ollila@iki.fi X-BeenThere: notmuch@notmuchmail.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: notmuch-bounces@notmuchmail.org Sender: "notmuch" X-Scanner: scn0 Authentication-Results: aspmx1.migadu.com; dkim=fail (body hash did not verify) header.d=iki.fi header.s=lahtoruutu header.b=necl9D36; dmarc=none; spf=pass (aspmx1.migadu.com: domain of notmuch-bounces@notmuchmail.org designates 50.126.95.6 as permitted sender) smtp.mailfrom=notmuch-bounces@notmuchmail.org X-Spam-Score: 1.99 X-TUID: FwGcX2+fImoq On Tue, May 12 2020, Daniel Kahn Gillmor wrote: > I'm debugging/diagnosing/trying to clean up some "FIXED" known-broken > tests right now. > > Sometimes, depending on circumstances i can't predict (race > conditions?), I see funny output like: > > ~~~ > Use "make V=1" to see the details for passing and known broken tests. > INFO: using 2m timeout for tests > INFO: running tests with moreutils parallel > FIXED verify signed PKCS#7 subject (onepart-signed) signer User ID > > T356-protected-headers: Testing Message decryption with protected headers > BROKEN confirm signed and encrypted PKCS#7 subject (sign+enc) > BROKEN confirm signed and encrypted PKCS#7 subject (sign+enc) signer User ID > BROKEN confirm signed and encrypted PKCS#7 subject (sign+enc+legacy-disp) > BROKEN confirm signed and encrypted PKCS#7 subject (sign+enc+legacy-disp) signer User ID > BROKEN confirm encryption-protected PKCS#7 subject (enc+legacy-disp) > ~~~ > > Clearly, that FIXED should come *after* the "T356-protected-headers:" > separator. After your second mail I tried to reproduce by taking a test file, marking one known_broken, even it is not to get FIXED, and then broke next one. And I got it reproduced. the message that prints FIXED, does not execute print_test_description but the ones that do BROKEN or FAIL does (did not fully check but yes). And: print_test_description () { test -z "$test_description_printed" || return 0 echo echo $this_test: "Testing ${test_description}" test_description_printed=1 } if [ -z "$NOTMUCH_TEST_QUIET" ] then print_test_description fi Tomi > > This is a minor bug, i suppose, but i confess i don't understand the > maze of shell functions in test-lib.sh well enough to see why this is > happening, let alone to fix it. > > Anyone interested in fixing it should be able to do so by marking a good > test "known broken" and then re-running the test suite. The above > output is taken from: > > make -j4 check NOTMUCH_TESTS=T356-protected-headers.sh > > Sorry to send a bug report with no fixes! > > --dkg > _______________________________________________ > notmuch mailing list > notmuch@notmuchmail.org > https://notmuchmail.org/mailman/listinfo/notmuch