From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from localhost (localhost [127.0.0.1]) by arlo.cworth.org (Postfix) with ESMTP id 10B8E6DE32CD for ; Mon, 26 Mar 2018 14:01:50 -0700 (PDT) X-Virus-Scanned: Debian amavisd-new at cworth.org X-Spam-Flag: NO X-Spam-Score: 0.168 X-Spam-Level: X-Spam-Status: No, score=0.168 tagged_above=-999 required=5 tests=[AWL=-0.062, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, FREEMAIL_FORGED_FROMDOMAIN=0.249, FREEMAIL_FROM=0.001, HEADER_FROM_DIFFERENT_DOMAINS=0.001, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Received: from arlo.cworth.org ([127.0.0.1]) by localhost (arlo.cworth.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id HjI__i1MQnwh for ; Mon, 26 Mar 2018 14:01:48 -0700 (PDT) Received: from mail-wm0-f45.google.com (mail-wm0-f45.google.com [74.125.82.45]) by arlo.cworth.org (Postfix) with ESMTPS id C31B16DE329C for ; Mon, 26 Mar 2018 14:01:47 -0700 (PDT) Received: by mail-wm0-f45.google.com with SMTP id x82so17891637wmg.1 for ; Mon, 26 Mar 2018 14:01:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:cc:subject:in-reply-to:references:date:message-id :mime-version; bh=ttK4w74ohSRbVa53eV44argJbDwPYIXqYxoOfqIRWsI=; b=I/vSPMV1F8iIOrPNk3O01pDP1b30qwJLzlGaP4P5siwTJ2/NfyOboWWQSeo54FYSt9 7w0iFrFLdqUlibLLUI3dbXZRDHB/zA9TfUkt7Ns9+8pW2lBuOzVvpzoLIbUpSd1Ddn6o eYoke5rmpAuuSzg0x1wh673B1Vgn7qbSUS3pEwTmZJULLV2vma2dA2zpJnBYHopubwin G+WTSGILezpr31x3xNQCXn+WsT3zbSO+kPz3eUA/J27A6wE91zORA6j9gHLGMcZn34I9 5bG2qPeOpVz8RNJ2J00PIWWt/sgcbcxhxIoOhi/D0JHe+hKgnSrYtb1VWbeBW0yOvo2x 0X/w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:cc:subject:in-reply-to:references :date:message-id:mime-version; bh=ttK4w74ohSRbVa53eV44argJbDwPYIXqYxoOfqIRWsI=; b=nzcenXIfOcrGDMToN87hZJtJHvxcdr/2WoNmU3v1a+J5VtmQfy0BqOfQoRoV7CMN+h 7c9heIjypkYtN8cegIiFp450If+GnRLrH/Xs9LriKLbJIMDnhUHkR7qGMs9k8jrc8tnY SPrtWpw0uDohA66VKybODLutgAA7sFW3bTCvfiGoVccNTRhRTAiVY3CTzrAfIoODStLL N4V8OtMVOX+Ru6RD/KCf1hxsilGVZnY0Az0wXsh6q1JNl9F/HigLVl62uxhmSkrZbR5W HrqjstQfXlRdPfb4fyO4Li5Mb9dFBY7uh3mrL7RnjzSvkbZDJLJFhGOc6LqsEaCX5XWv o0Lg== X-Gm-Message-State: AElRT7GLmrGnKi+s2+c1CxemKn0qnoe5zPlTZLcYv2aDZ9DLQuf47366 YI8toga50Ait8SHCVF6fiJMndIEb X-Google-Smtp-Source: AG47ELthz89QS+JZElfe9So7ZcLLL+gYTmYL8EuhS8YD5swLL+Kz5HImHcSz0JxSz709qm7yyirJmQ== X-Received: by 10.80.190.133 with SMTP id b5mr40869961edk.246.1522098106461; Mon, 26 Mar 2018 14:01:46 -0700 (PDT) Received: from devork.be ([2a02:168:581b::b89]) by smtp.gmail.com with ESMTPSA id e12sm9740493edi.29.2018.03.26.14.01.45 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 26 Mar 2018 14:01:45 -0700 (PDT) Sender: Floris Bruynooghe Received: (nullmailer pid 199000 invoked by uid 469025); Mon, 26 Mar 2018 21:01:45 -0000 From: Floris Bruynooghe To: David Bremner , Justus Winter , David Bremner , "W. Trevor King" Cc: notmuch@notmuchmail.org Subject: Re: pytest integration for the notmuch test suite In-Reply-To: <20180325174034.17912-1-david@tethera.net> References: <87po3x4tuc.fsf@thinkbox.jade-hamburg.de> <20180325174034.17912-1-david@tethera.net> Date: Mon, 26 Mar 2018 23:01:45 +0200 Message-ID: MIME-Version: 1.0 Content-Type: text/plain X-BeenThere: notmuch@notmuchmail.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Mar 2018 21:01:50 -0000 On Sun, Mar 25 2018, David Bremner wrote: > Here's one approach. A given pytest "file" can be embedded in a normal > (for us) test script. As I write this, it occurs to me you might be > thinking of embedding unit tests in the bindings source files; that > would be easy to add, something along the lines of > > test_begin_subtest "python bindings embedded unit tests" > test_expect_success "${NOTMUCH_PYTEST} ${NOTMUCH_SRCDIR}/bindings/python/notmuch" I was trying to construct something where a full pytest run on one python version was one subtest. For granularity I think treating an entire pytest run as a subtest with just checking the return code should be sufficient, e.g. `python2.7 -m pytest ${NOTMUCH_SRCDIR}/bindings/python/notmuch`. But the whole test in this case would be this same subtest but once with python2.7, python3.5, python3.6 etc. What do you think of this? Cheers, Floris