From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from localhost (localhost [127.0.0.1]) by arlo.cworth.org (Postfix) with ESMTP id 1C25C6DE01E6 for ; Sun, 27 Jan 2019 14:04:38 -0800 (PST) X-Virus-Scanned: Debian amavisd-new at cworth.org X-Spam-Flag: NO X-Spam-Score: 0.042 X-Spam-Level: X-Spam-Status: No, score=0.042 tagged_above=-999 required=5 tests=[AWL=0.041, DKIMWL_WL_MED=0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, RCVD_IN_DNSWL_NONE=-0.0001] autolearn=disabled Received: from arlo.cworth.org ([127.0.0.1]) by localhost (arlo.cworth.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 74lHZ_izN5CR for ; Sun, 27 Jan 2019 14:04:36 -0800 (PST) Received: from mail-lf1-f47.google.com (mail-lf1-f47.google.com [209.85.167.47]) by arlo.cworth.org (Postfix) with ESMTPS id DDECC6DE0134 for ; Sun, 27 Jan 2019 14:04:35 -0800 (PST) Received: by mail-lf1-f47.google.com with SMTP id f5so10421626lfc.13 for ; Sun, 27 Jan 2019 14:04:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nikula-org.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=1g5FUhdVZE83pFAVqQvIJmM8VVyv/8s6DwSvWc9q7XU=; b=ww4i3PN/oblmPoDLvCUcz3RYQMV0jSG6eWeHh27jG4JZIkrACmWZwrUgUdFMzG+Aei fwpxBajGl6V9ySP8nUE1a0t9nS8mNjj0MONyxjoh0u6WsRHbsuOw0SpXjhUC8GhareCv XkPsd9qjEpF6El39EIHQl8SrV+zyaLb+mjDImLKgZgLuqx3MvgHj88c1omNmDjnHuzqb Rm3Pd5n55/xFLkClo3QbSLCRMUrlfe5grcmbkXr00QxWdP2mHXQo93DUBkC4Bu6HcH0u W6lSxV4fBf6jMF6Qblsmw8LtebLz7y06i6ffMMcQtkSh5ZAgis4pobYVb601jxJKdU1s Qc+Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=1g5FUhdVZE83pFAVqQvIJmM8VVyv/8s6DwSvWc9q7XU=; b=U/ugi24bSvTAcYndQtGeNHtwyJEl0yFTQyoKja6vkTEIA63GHH+K5xLojBOJziIBHd J6CD8YoFH9eY7dEh7NaOUsoZZyL/pygnCGVXT70y45sWVATvaX+Vm9d9hCez1R+HblwC 0Xw0lJA3FSXoEiEx6SwOmfAg2JNeZfAawHQA2/sv6tv8tAFWNW/KOBZUEjPZzyw9ZTGX KyIhDC3/ARAv7EmwfdYjWnP/1ZJUxOxCeTtVn3X079pwjeHZt5z89W83c4FW5ktvIi+l 8cy0zo4I4oaj1q9K7qUcZRMNQUG/wL3iZhhOiXeloIv4XAOV32TyHwdxzQ9W+ubf6Nmy g4uA== X-Gm-Message-State: AJcUukfFVfyV8Sz+SNazLnm9ik2xqrTliV9tlAlMRMIFirpZW7NQe369 poxBxd5PRpTFMyiheYgB8Tb/RAN4kgYv0w== X-Google-Smtp-Source: ALg8bN5UyHIR5I8oEnSDdJuy2ef/n2GGYBVEA2xaJ25ZAHFHfhxtspkLpDGM9soTj+BtxfHPEWQY0g== X-Received: by 2002:a19:ec16:: with SMTP id b22mr15203262lfa.65.1548626673359; Sun, 27 Jan 2019 14:04:33 -0800 (PST) Received: from localhost (82-203-142-74.bb.dnainternet.fi. [82.203.142.74]) by smtp.gmail.com with ESMTPSA id c22sm2635997lfd.88.2019.01.27.14.04.32 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Sun, 27 Jan 2019 14:04:32 -0800 (PST) From: Jani Nikula To: notmuch@notmuchmail.org Subject: [PATCH v2 1/4] test: dynamically generate parser tests Date: Mon, 28 Jan 2019 00:04:26 +0200 Message-Id: <20190127220429.9206-1-jani@nikula.org> X-Mailer: git-send-email 2.20.1 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-BeenThere: notmuch@notmuchmail.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 27 Jan 2019 22:04:38 -0000 It's impossible to have expected failures or other unittest decorators at subtest granularity. They only work at the test method level. On the other hand we don't want to be manually adding test methods when all of the tests are defined in terms of input files and expected results. Generate test methods dynamically from the input files, and assign to the test class. Running code at import time to do this is less than stellar, but it needs to be done early to have unittest test discovery find the methods. The alternative would be to add a load_tests protocol function [1], but that seems like more boilerplate. Can be added later as needed. Finally, one massive upside to this is the ability to run individual named testcases. For example, to test enum.c and typedef-enum.c, use: $ test/test_hawkmoth.py ParserTest.test_enum ParserTest.test_typedef_enum [1] https://docs.python.org/3/library/unittest.html#load-tests-protocol --- test/test_hawkmoth.py | 26 +++++++------------------- test/testenv.py | 29 +++++++++++++++++++++++++++++ 2 files changed, 36 insertions(+), 19 deletions(-) diff --git a/test/test_hawkmoth.py b/test/test_hawkmoth.py index 1fe02efc004d..75eebbe35eef 100755 --- a/test/test_hawkmoth.py +++ b/test/test_hawkmoth.py @@ -8,28 +8,16 @@ import unittest import testenv from hawkmoth import hawkmoth -class ParserTest(unittest.TestCase): - def _run_test(self, input_filename): - # sanity check - self.assertTrue(os.path.isfile(input_filename)) - - options = testenv.get_testcase_options(input_filename) - output = hawkmoth.parse_to_string(input_filename, False, **options) - expected = testenv.read_file(input_filename, ext='stdout') +def _get_output(input_filename, **options): + return hawkmoth.parse_to_string(input_filename, False, **options) - self.assertEqual(expected, output) +def _get_expected(input_filename, **options): + return testenv.read_file(input_filename, ext='stdout') - def _run_dir(self, path): - # sanity check - self.assertTrue(os.path.isdir(path)) - - with self.subTest(path=path): - for f in testenv.get_testcases(path): - with self.subTest(source=os.path.basename(f)): - self._run_test(f) +class ParserTest(unittest.TestCase): + pass - def test_parser(self): - self._run_dir(testenv.testdir) +testenv.assign_test_methods(ParserTest, _get_output, _get_expected) if __name__ == '__main__': unittest.main() diff --git a/test/testenv.py b/test/testenv.py index f026aead8c07..cc80ef2218ed 100644 --- a/test/testenv.py +++ b/test/testenv.py @@ -3,6 +3,7 @@ import sys import os +import unittest testext = '.c' testdir = os.path.dirname(os.path.abspath(__file__)) @@ -10,6 +11,16 @@ rootdir = os.path.dirname(testdir) sys.path.insert(0, rootdir) +def _testcase_name(testcase): + """Convert a testcase filename into a test case identifier.""" + name = os.path.splitext(os.path.basename(testcase))[0] + name = name.replace('-', '_') + name = 'test_{name}'.format(name=name) + + assert name.isidentifier() + + return name + def get_testcases(path): for f in sorted(os.listdir(path)): if f.endswith(testext): @@ -52,3 +63,21 @@ def read_file(filename, **kwargs): expected = file.read() return expected + +def _test_generator(get_output, get_expected, input_filename, **options): + """Return a function that compares output/expected results on input_filename.""" + def test(self): + output = get_output(input_filename, **options) + expected = get_expected(input_filename, **options) + + self.assertEqual(expected, output) + + return test + +def assign_test_methods(cls, get_output, get_expected): + """Assign test case functions to the given class.""" + for f in get_testcases(testdir): + options = get_testcase_options(f) + method = _test_generator(get_output, get_expected, f, **options) + + setattr(cls, _testcase_name(f), method) -- 2.20.1