From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from localhost (localhost [127.0.0.1]) by arlo.cworth.org (Postfix) with ESMTP id 603536DE31DA for ; Mon, 26 Mar 2018 04:31:40 -0700 (PDT) X-Virus-Scanned: Debian amavisd-new at cworth.org X-Spam-Flag: NO X-Spam-Score: 0 X-Spam-Level: X-Spam-Status: No, score=0 tagged_above=-999 required=5 tests=[AWL=0.011, SPF_PASS=-0.001, T_RP_MATCHES_RCVD=-0.01] autolearn=disabled Received: from arlo.cworth.org ([127.0.0.1]) by localhost (arlo.cworth.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 2L9NLvKA56u3 for ; Mon, 26 Mar 2018 04:31:38 -0700 (PDT) Received: from fethera.tethera.net (fethera.tethera.net [198.245.60.197]) by arlo.cworth.org (Postfix) with ESMTPS id CA94A6DE31D9 for ; Mon, 26 Mar 2018 04:31:37 -0700 (PDT) Received: from remotemail by fethera.tethera.net with local (Exim 4.89) (envelope-from ) id 1f0QLS-0006Wt-VK; Mon, 26 Mar 2018 07:31:22 -0400 Received: (nullmailer pid 956 invoked by uid 1000); Mon, 26 Mar 2018 11:31:21 -0000 From: David Bremner To: Tomi Ollila , Justus Winter , Floris Bruynooghe , "W. Trevor King" Cc: notmuch@notmuchmail.org Subject: Re: pytest integration for the notmuch test suite In-Reply-To: References: <87po3x4tuc.fsf@thinkbox.jade-hamburg.de> <20180325174034.17912-1-david@tethera.net> Date: Mon, 26 Mar 2018 08:31:21 -0300 Message-ID: <87sh8n9ipi.fsf@tethera.net> MIME-Version: 1.0 Content-Type: text/plain X-BeenThere: notmuch@notmuchmail.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Mar 2018 11:31:40 -0000 Tomi Ollila writes: > On Sun, Mar 25 2018, David Bremner wrote: > >> Here's one approach. A given pytest "file" can be embedded in a normal >> (for us) test script. As I write this, it occurs to me you might be >> thinking of embedding unit tests in the bindings source files; that >> would be easy to add, something along the lines of >> >> test_begin_subtest "python bindings embedded unit tests" >> test_expect_success "${NOTMUCH_PYTEST} ${NOTMUCH_SRCDIR}/bindings/python/notmuch" > > Hmm. > > Looks a bit strange to embed the pytest snippets into shell script and then > execute each of these individiually. The only thing py.test seems to do here is > "visualizing" assert output. We could just use normal python otherwise, and > just not (necessarily) drop things into functions. Yes, I had pretty much the same thought. It's also true that pytest allows mixing tests with regular definition, if desired (i.e. it can search for all the functions starting test_ or some other marker). It's also true this doesn't really cost more at run time than the current test_python; they both exec the python interpreter. > If we had pytest, I'd suggest the tests were written and executed > separately (from one test script) and then results collected somehow > to the final aggregator. Sure, that sounds nicer. I just don't want to invest a lot in writing glue before it's clear what is needed. If start by running the pytest files the crude way, we can always improve the glue later. That does argue against doing many embedded snippets, I agree. d