From: Tomi Ollila <tomi.ollila@iki.fi>
To: Daniel Kahn Gillmor <dkg@fifthhorseman.net>,
David Bremner <david@tethera.net>, "Rollins\,
Jameson" <jrollins@caltech.edu>,
Notmuch Mail <notmuch@notmuchmail.org>
Subject: Re: parallelize test suite
Date: Sun, 05 May 2019 19:44:01 +0300 [thread overview]
Message-ID: <m2v9yo4zse.fsf@guru.guru-group.fi> (raw)
In-Reply-To: <87lfzllyd1.fsf@fifthhorseman.net>
On Sun, May 05 2019, Daniel Kahn Gillmor wrote:
> On Sat 2019-05-04 19:53:23 -0300, David Bremner wrote:
>> Last time we discussed parallel test running, there we concerns about
>> multiple versions of certain servers colliding with each other. This
>> still seems to be at least a theoretical issue with smtp-dummy, although
>> a glance suggests that it might only currently be used in T310-emacs.sh.
we,d need to change 25025 with something else -- configured in the test
script ? -- same could be used as part of emacs server socket ?
>> I'm not sure what a robust solution is here.
>>
>> - gpg-agent
>> - emacs
>> - dtach
>>
>> Did I miss any other background processes run by the test suite?
>>
>> I can imagine gpg-agent is managed OK these days since it's started
>> automagically by gpg.
>
> gpg-agent is fine, because it's isolated by $GNUPGHOME, and each test
> uses a distinct $GNUPGHOME (see GNUPGHOME="${TEST_TMPDIR}/gnupg in
> test-lib.sh)
>
>> emacs seems to use the current process id in the socket name, so that
>> also should be OK, although it should maybe be replaced with something
>> more robust to avoid problems with pid rollover. I _think_ including the
>> test name in the emacs server would do the trick
>
> I would have no objection to this improvement in isolation of the emacs
> server processes, but i think the pid rollover race condition is so
> minor that i don't think it sholud be a blocker for the adoption of this
> series.
when test code done correctly (test_done ends it), the shell launching
emacs holds the pid $$ until emacs exits, so the pid is not reused
in name for any emacs process (but if bash died leaving emacs running
then we'd have chance for collisions...)
>
>> The dtach socket is in the tmp.T* directory, so that should be OK.
>>
>> I wonder if a good solution would be to make running the test suite in
>> parallel be opt-in (e.g. by configuration option). Or at least have a
>> way to disable it for situations like CI and autobuilders.
>
> I agree that making it possible to force serialized tests would be good.
>
> I'd prefer that running tests in parallel be the default, though i
> wouldn't object to a ./configure --serialize-tests option if someone
> wants to implement it.
no need for configure option, one can just remove parallel(1) from system ;)
... well, we could use ${PARALLEL:=parallel} in script for users to shadow
it out in some other way...
> I actually think that CI and autobuilders *should* exercise the parallel
> tests, as annoying as that might be initially, because it seems likely
> to catch any other potential entanglements.
... after it has been proven a bit more to work...
One thing more, the "perverse" pre-caching done in add_email_corpus
can be retained -- just do that step *before* going to parallelism...
>
> thanks for the review!
>
> --dkg
Tomi
next prev parent reply other threads:[~2019-05-05 16:44 UTC|newest]
Thread overview: 22+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-05-04 20:57 parallelize test suite Rollins, Jameson
2019-05-04 20:57 ` [PATCH 1/3] tests: remove some redundant pre-cleanup of the corpus MAIL_DIR Rollins, Jameson
2019-05-04 20:57 ` [PATCH 2/3] tests: remove entangling corpus index optimization Rollins, Jameson
2019-05-04 20:57 ` [PATCH 3/3] tests: run all tests in parallel, if available Rollins, Jameson
2019-05-06 4:44 ` [PATCH] tests: environment variable to specify that tests should be serialized Rollins, Jameson
2019-05-06 10:32 ` [PATCH] test: add configurable port to smtp-dummy David Bremner
2019-05-06 19:39 ` Tomi Ollila
2019-05-06 20:55 ` Daniel Kahn Gillmor
2019-05-07 10:20 ` [PATCH] test: let the OS choose a port for smtp-dummy David Bremner
2019-05-07 12:38 ` Daniel Kahn Gillmor
2019-05-08 15:59 ` Tomi Ollila
2019-05-10 10:16 ` David Bremner
2019-05-06 19:15 ` [PATCH] tests: environment variable to specify that tests should be serialized Tomi Ollila
2019-05-06 20:58 ` Daniel Kahn Gillmor
2019-05-05 10:22 ` [PATCH 1/3] tests: remove some redundant pre-cleanup of the corpus MAIL_DIR Tomi Ollila
2019-05-04 21:33 ` parallelize test suite Rollins, Jameson
2019-05-04 21:39 ` Daniel Kahn Gillmor
2019-05-04 22:53 ` David Bremner
2019-05-05 15:22 ` Daniel Kahn Gillmor
2019-05-05 16:44 ` Tomi Ollila [this message]
2019-05-06 21:39 ` Daniel Kahn Gillmor
2019-05-07 10:26 ` David Bremner
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://notmuchmail.org/
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=m2v9yo4zse.fsf@guru.guru-group.fi \
--to=tomi.ollila@iki.fi \
--cc=david@tethera.net \
--cc=dkg@fifthhorseman.net \
--cc=jrollins@caltech.edu \
--cc=notmuch@notmuchmail.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://yhetil.org/notmuch.git/
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).