From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from localhost (localhost [127.0.0.1]) by arlo.cworth.org (Postfix) with ESMTP id 6AE696DE1039 for ; Sun, 5 May 2019 09:44:09 -0700 (PDT) X-Virus-Scanned: Debian amavisd-new at cworth.org X-Spam-Flag: NO X-Spam-Score: 0.44 X-Spam-Level: X-Spam-Status: No, score=0.44 tagged_above=-999 required=5 tests=[AWL=-0.212, SPF_NEUTRAL=0.652] autolearn=disabled Received: from arlo.cworth.org ([127.0.0.1]) by localhost (arlo.cworth.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id jVazAKQqSniY for ; Sun, 5 May 2019 09:44:07 -0700 (PDT) Received: from guru.guru-group.fi (guru.guru-group.fi [46.183.73.34]) by arlo.cworth.org (Postfix) with ESMTP id 4685B6DE0F4F for ; Sun, 5 May 2019 09:44:06 -0700 (PDT) Received: from guru.guru-group.fi (localhost [IPv6:::1]) by guru.guru-group.fi (Postfix) with ESMTP id D11B41000E3; Sun, 5 May 2019 19:44:01 +0300 (EEST) From: Tomi Ollila To: Daniel Kahn Gillmor , David Bremner , "Rollins\, Jameson" , Notmuch Mail Subject: Re: parallelize test suite In-Reply-To: <87lfzllyd1.fsf@fifthhorseman.net> References: <20190504205738.21504-1-jrollins@caltech.edu> <87r29dnbl0.fsf@fifthhorseman.net> <878svlhlwc.fsf@tethera.net> <87lfzllyd1.fsf@fifthhorseman.net> User-Agent: Notmuch/0.28.3+84~g41389bb (https://notmuchmail.org) Emacs/25.2.1 (x86_64-unknown-linux-gnu) X-Face: HhBM'cA~ MIME-Version: 1.0 Content-Type: text/plain X-BeenThere: notmuch@notmuchmail.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 05 May 2019 16:44:09 -0000 On Sun, May 05 2019, Daniel Kahn Gillmor wrote: > On Sat 2019-05-04 19:53:23 -0300, David Bremner wrote: >> Last time we discussed parallel test running, there we concerns about >> multiple versions of certain servers colliding with each other. This >> still seems to be at least a theoretical issue with smtp-dummy, although >> a glance suggests that it might only currently be used in T310-emacs.sh. we,d need to change 25025 with something else -- configured in the test script ? -- same could be used as part of emacs server socket ? >> I'm not sure what a robust solution is here. >> >> - gpg-agent >> - emacs >> - dtach >> >> Did I miss any other background processes run by the test suite? >> >> I can imagine gpg-agent is managed OK these days since it's started >> automagically by gpg. > > gpg-agent is fine, because it's isolated by $GNUPGHOME, and each test > uses a distinct $GNUPGHOME (see GNUPGHOME="${TEST_TMPDIR}/gnupg in > test-lib.sh) > >> emacs seems to use the current process id in the socket name, so that >> also should be OK, although it should maybe be replaced with something >> more robust to avoid problems with pid rollover. I _think_ including the >> test name in the emacs server would do the trick > > I would have no objection to this improvement in isolation of the emacs > server processes, but i think the pid rollover race condition is so > minor that i don't think it sholud be a blocker for the adoption of this > series. when test code done correctly (test_done ends it), the shell launching emacs holds the pid $$ until emacs exits, so the pid is not reused in name for any emacs process (but if bash died leaving emacs running then we'd have chance for collisions...) > >> The dtach socket is in the tmp.T* directory, so that should be OK. >> >> I wonder if a good solution would be to make running the test suite in >> parallel be opt-in (e.g. by configuration option). Or at least have a >> way to disable it for situations like CI and autobuilders. > > I agree that making it possible to force serialized tests would be good. > > I'd prefer that running tests in parallel be the default, though i > wouldn't object to a ./configure --serialize-tests option if someone > wants to implement it. no need for configure option, one can just remove parallel(1) from system ;) ... well, we could use ${PARALLEL:=parallel} in script for users to shadow it out in some other way... > I actually think that CI and autobuilders *should* exercise the parallel > tests, as annoying as that might be initially, because it seems likely > to catch any other potential entanglements. ... after it has been proven a bit more to work... One thing more, the "perverse" pre-caching done in add_email_corpus can be retained -- just do that step *before* going to parallelism... > > thanks for the review! > > --dkg Tomi