From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from localhost (localhost [127.0.0.1]) by arlo.cworth.org (Postfix) with ESMTP id C66816DE2A89 for ; Mon, 26 Mar 2018 14:26:19 -0700 (PDT) X-Virus-Scanned: Debian amavisd-new at cworth.org X-Spam-Flag: NO X-Spam-Score: 0 X-Spam-Level: X-Spam-Status: No, score=0 tagged_above=-999 required=5 tests=[AWL=0.011, SPF_PASS=-0.001, T_RP_MATCHES_RCVD=-0.01] autolearn=disabled Received: from arlo.cworth.org ([127.0.0.1]) by localhost (arlo.cworth.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id T7UPZLNH3OyB for ; Mon, 26 Mar 2018 14:26:15 -0700 (PDT) Received: from fethera.tethera.net (fethera.tethera.net [198.245.60.197]) by arlo.cworth.org (Postfix) with ESMTPS id 6F1196DE2935 for ; Mon, 26 Mar 2018 14:26:12 -0700 (PDT) Received: from remotemail by fethera.tethera.net with local (Exim 4.89) (envelope-from ) id 1f0Zct-0000Na-6N; Mon, 26 Mar 2018 17:25:59 -0400 Received: (nullmailer pid 11181 invoked by uid 1000); Mon, 26 Mar 2018 21:25:58 -0000 From: David Bremner To: Floris Bruynooghe , Justus Winter , "W. Trevor King" Cc: notmuch@notmuchmail.org Subject: Re: pytest integration for the notmuch test suite In-Reply-To: References: <87po3x4tuc.fsf@thinkbox.jade-hamburg.de> <20180325174034.17912-1-david@tethera.net> Date: Mon, 26 Mar 2018 18:25:58 -0300 Message-ID: <874ll2a5qx.fsf@tethera.net> MIME-Version: 1.0 Content-Type: text/plain X-BeenThere: notmuch@notmuchmail.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Mar 2018 21:26:19 -0000 Floris Bruynooghe writes: > On Sun, Mar 25 2018, David Bremner wrote: > >> Here's one approach. A given pytest "file" can be embedded in a normal >> (for us) test script. As I write this, it occurs to me you might be >> thinking of embedding unit tests in the bindings source files; that >> would be easy to add, something along the lines of >> >> test_begin_subtest "python bindings embedded unit tests" >> test_expect_success "${NOTMUCH_PYTEST} ${NOTMUCH_SRCDIR}/bindings/python/notmuch" > > I was trying to construct something where a full pytest run on one > python version was one subtest. For granularity I think treating an > entire pytest run as a subtest with just checking the return code should > be sufficient, > e.g. `python2.7 -m pytest ${NOTMUCH_SRCDIR}/bindings/python/notmuch`. > > But the whole test in this case would be this same subtest but once with > python2.7, python3.5, python3.6 etc. > > What do you think of this? The notmuch build system currently looks for a single version of python. I'm pretty sure we don't want to require multiple python versions to build notmuch, although I guess it wouldn't be that hard to detect whatever versions exist in the environment and run pytest with each of them. I just need a list of names to look for (or a better way to find all the pythons installed). Debian has the handy comand "py3versions --installed", but I suppose that's completely unportable. Then each one can be checked to see if it finds the pytest module. It sounds doable to me; it needs a bit more work in configure, but I can drop the stuff in test-lib.sh d