From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from localhost (localhost [127.0.0.1]) by arlo.cworth.org (Postfix) with ESMTP id 2F5FB6DE01E6 for ; Sun, 27 Jan 2019 14:04:39 -0800 (PST) X-Virus-Scanned: Debian amavisd-new at cworth.org X-Spam-Flag: NO X-Spam-Score: 0.04 X-Spam-Level: X-Spam-Status: No, score=0.04 tagged_above=-999 required=5 tests=[AWL=0.040, DKIMWL_WL_MED=0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001] autolearn=disabled Received: from arlo.cworth.org ([127.0.0.1]) by localhost (arlo.cworth.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id oPvBaX2f_f4h for ; Sun, 27 Jan 2019 14:04:38 -0800 (PST) Received: from mail-lj1-f193.google.com (mail-lj1-f193.google.com [209.85.208.193]) by arlo.cworth.org (Postfix) with ESMTPS id 318B46DE0134 for ; Sun, 27 Jan 2019 14:04:38 -0800 (PST) Received: by mail-lj1-f193.google.com with SMTP id n18-v6so12556778lji.7 for ; Sun, 27 Jan 2019 14:04:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nikula-org.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=5bywg7/INUT9V7Yrt0JUcfY1c4lOWDaKYeFZ+Gt+zss=; b=LXh85Epg58aRAiQUupoSeYeaPXjmhRQZOIEHtlAOjf4v0/nP5u5+9F6VTihezb42u0 PymW+MMxcgShIzsMx/HUyiLU219vzofgE7VHq3mG/yxgpQZ5RxSv+0N77QTrIzCQmaQo gYxjUhCSzYQrbuESTZFL4hkwFlSaJwlSqctQpNbSC0P1zvCwc/nRRPcoU1tTbuOPCn3D Gs3z7g+aBu2HPkA8UURlg8RpEnqgylDBnG07H8vIX3lLgkiAL0BRGsVqD2duVy7VUClQ gI2VbYPoi8vQ2p64KRGJwNmx902V2X567N6saGMNbqihhkGCK0ch/cXfx0W5lxrkbCM9 Cnkg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=5bywg7/INUT9V7Yrt0JUcfY1c4lOWDaKYeFZ+Gt+zss=; b=oFcFNxXqi4iLH2AvvGsjYJ1wXGhUyQFuR5k8Wpvpc9WXP7Uk0mfcsiFiGBiugssLEH f7p7QMU/n35uPxdgoRhPPMigz/0JBU80UbLmhWhbmC8FMA2wsef+JCbuj1YaYn6ZuFKe Kq/sx4PmRq8v5daXnBMMcoo06Jk1fZiPEAK0DIVxr7UJpDI2k12u0vmzwH/QzQ5dz4EH JDMf3K31Ivc0TVi5/2KQLfefj4yXP2b+pdqoeTuksgqUwOe5TV0lb7f/Mj90M4UGliZD 1sjim8TmSVeIhtKNxrFvyKr4K67DtSNijLg/8/RgNW3ip8cN7V/6HdYokaO+hH8okl5V 8mZw== X-Gm-Message-State: AJcUukfxg9wDaifv/ix5Lzq7DEV+gNOD2XcfXlxkZ1DOjPHzJmSv8sOI fYgVPNxQbAHspbVR9SlKmwM0gE6xnycw+Q== X-Google-Smtp-Source: ALg8bN7f5pu2NWfATxD4RyvMroVl6B8Shqy+d4I8y0PBT13kJWUhvQBahc8FKMoqZ5K7ayV7RbbF8w== X-Received: by 2002:a2e:89d7:: with SMTP id c23-v6mr15607231ljk.0.1548626676256; Sun, 27 Jan 2019 14:04:36 -0800 (PST) Received: from localhost (82-203-142-74.bb.dnainternet.fi. [82.203.142.74]) by smtp.gmail.com with ESMTPSA id o14-v6sm2705640lji.70.2019.01.27.14.04.35 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Sun, 27 Jan 2019 14:04:35 -0800 (PST) From: Jani Nikula To: notmuch@notmuchmail.org Subject: [PATCH v2 3/4] test: add support for flagging expected failures in testcase options Date: Mon, 28 Jan 2019 00:04:28 +0200 Message-Id: <20190127220429.9206-3-jani@nikula.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190127220429.9206-1-jani@nikula.org> References: <20190127220429.9206-1-jani@nikula.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-BeenThere: notmuch@notmuchmail.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 27 Jan 2019 22:04:39 -0000 Since our tests are dynamically created, we also need to decorate expected failures dynamically. Use the testcase options file as the source. Only pass known directive options to the directive in the cautodoc test. Add a meta test to verify this works, with result "OK (expected failures=2)". --- test/meta-expected-failure.c | 3 +++ test/meta-expected-failure.options | 1 + test/meta-expected-failure.stdout | 3 +++ test/test_cautodoc.py | 2 +- test/testenv.py | 8 ++++++++ 5 files changed, 16 insertions(+), 1 deletion(-) create mode 100644 test/meta-expected-failure.c create mode 100644 test/meta-expected-failure.options create mode 100644 test/meta-expected-failure.stdout diff --git a/test/meta-expected-failure.c b/test/meta-expected-failure.c new file mode 100644 index 000000000000..18065136572f --- /dev/null +++ b/test/meta-expected-failure.c @@ -0,0 +1,3 @@ +/** + * Meta test: This fails. Always. + */ diff --git a/test/meta-expected-failure.options b/test/meta-expected-failure.options new file mode 100644 index 000000000000..68c905946fce --- /dev/null +++ b/test/meta-expected-failure.options @@ -0,0 +1 @@ +test-expected-failure diff --git a/test/meta-expected-failure.stdout b/test/meta-expected-failure.stdout new file mode 100644 index 000000000000..dbd71dcb308f --- /dev/null +++ b/test/meta-expected-failure.stdout @@ -0,0 +1,3 @@ + +Meta test: This fails. + diff --git a/test/test_cautodoc.py b/test/test_cautodoc.py index 848f2105a1da..0a8e5bb2815f 100755 --- a/test/test_cautodoc.py +++ b/test/test_cautodoc.py @@ -17,7 +17,7 @@ def _get_output(input_filename, app, status, warning, **options): with open(os.path.join(app.srcdir, 'index.rst'), 'w') as file: fmt = '.. c:autodoc:: {source}\n' file.write(fmt.format(source=os.path.basename(input_filename))) - for key in options.keys(): + for key in [k for k in options.keys() if k in testenv.directive_options]: fmt = ' :{key}: {value}\n' file.write(fmt.format(key=key, value=options[key])) diff --git a/test/testenv.py b/test/testenv.py index cc80ef2218ed..b6842a81b375 100644 --- a/test/testenv.py +++ b/test/testenv.py @@ -26,6 +26,11 @@ def get_testcases(path): if f.endswith(testext): yield os.path.join(path, f) +directive_options = [ + 'compat', + 'clang', +] + def get_testcase_options(testcase): options_filename = modify_filename(testcase, ext='options') @@ -80,4 +85,7 @@ def assign_test_methods(cls, get_output, get_expected): options = get_testcase_options(f) method = _test_generator(get_output, get_expected, f, **options) + if options.get('test-expected-failure') is not None: + method = unittest.expectedFailure(method) + setattr(cls, _testcase_name(f), method) -- 2.20.1