From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from localhost (localhost [127.0.0.1]) by arlo.cworth.org (Postfix) with ESMTP id 946176DE16F3 for ; Sun, 25 Mar 2018 12:14:18 -0700 (PDT) X-Virus-Scanned: Debian amavisd-new at cworth.org X-Spam-Flag: NO X-Spam-Score: 0.478 X-Spam-Level: X-Spam-Status: No, score=0.478 tagged_above=-999 required=5 tests=[AWL=-0.174, SPF_NEUTRAL=0.652] autolearn=disabled Received: from arlo.cworth.org ([127.0.0.1]) by localhost (arlo.cworth.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id BOGliGqogCPS for ; Sun, 25 Mar 2018 12:14:17 -0700 (PDT) Received: from guru.guru-group.fi (guru.guru-group.fi [46.183.73.34]) by arlo.cworth.org (Postfix) with ESMTP id 2A86E6DE16CE for ; Sun, 25 Mar 2018 12:14:16 -0700 (PDT) Received: from guru.guru-group.fi (localhost [IPv6:::1]) by guru.guru-group.fi (Postfix) with ESMTP id 358AE100033; Sun, 25 Mar 2018 22:14:05 +0300 (EEST) From: Tomi Ollila To: David Bremner , Justus Winter , Floris Bruynooghe , David Bremner , "W. Trevor King" Cc: notmuch@notmuchmail.org Subject: Re: pytest integration for the notmuch test suite In-Reply-To: <20180325174034.17912-1-david@tethera.net> References: <87po3x4tuc.fsf@thinkbox.jade-hamburg.de> <20180325174034.17912-1-david@tethera.net> User-Agent: Notmuch/0.26~rc0+5~gaec2eb0 (https://notmuchmail.org) Emacs/25.2.1 (x86_64-unknown-linux-gnu) X-Face: HhBM'cA~ MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: notmuch@notmuchmail.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 25 Mar 2018 19:14:18 -0000 On Sun, Mar 25 2018, David Bremner wrote: > Here's one approach. A given pytest "file" can be embedded in a normal > (for us) test script. As I write this, it occurs to me you might be > thinking of embedding unit tests in the bindings source files; that > would be easy to add, something along the lines of > > test_begin_subtest "python bindings embedded unit tests" > test_expect_success "${NOTMUCH_PYTEST} ${NOTMUCH_SRCDIR}/bindings/python/= notmuch" Hmm. Looks a bit strange to embed the pytest snippets into shell script and then execute each of these individiually. The only thing py.test seems to do her= e is "visualizing" assert output. We could just use normal python otherwise, and just not (necessarily) drop things into functions. If we had pytest, I'd suggest the tests were written and executed separately (from one test script) and then results collected somehow to the final aggregator. (Recently I've been working with py.test-3 (and am not too happy with it, perhaps it is better when testing python code, we use it for testing c code with help of ctypes...) so I have some knowledge of how it works... BTW: in case of pytest. be careful there is no `conftest.py` in any of the directories starting / !!!, to mess up with your tests >;/=20 > You could also run one source file of tests with > > test_begin_subtest "python bindings foo tests" > test_expect_success "${NOTMUCH_PYTEST} ${NOTMUCH_SRCDIR}/bindings/python/= notmuch/test_foo.py" notmuch b=C3=A4tter... ;/ > > that would give a less granular result, at the cost of more boilerplate Tomi