From: Lars Ingebrigtsen <larsi@gnus.org>
To: Eli Zaretskii <eliz@gnu.org>
Cc: stefan@marxist.se, emacs-devel@gnu.org, michael.albinus@gmx.de,
mardani29@yahoo.es
Subject: Re: master 262d0c6: Mark some tests as expensive
Date: Sun, 13 Sep 2020 14:30:07 +0200 [thread overview]
Message-ID: <87zh5tzwog.fsf@gnus.org> (raw)
In-Reply-To: <83363ndvql.fsf@gnu.org> (Eli Zaretskii's message of "Sat, 12 Sep 2020 15:29:06 +0300")
Eli Zaretskii <eliz@gnu.org> writes:
>> Yes, if they take more than a second to run.
>
> So how do we make sure some change didn't introduce a bug in those
> features?
We either say "make check-expensive", or we live on in uncertainty, as
with all those other functions we've marked as expensive.
>> We've already made this decision with all the tests previously marked as
>> expensive, so there's nothing new here.
>
> There's a large gap between what is currently marked as "expensive"
> tests and having entire packages not tested at all. The latter sounds
> too radical to me. E.g., auto-revert is an important feature, used by
> many people. Not having it in regression testing sounds like a step
> backward to me.
It's still being tested -- just not as often.
> I'm talking about a balance here. Losing the tests of complete
> features just because we want tests to finish quickly sounds
> sub-optimal to me. Can we make a smarter balance?
Sure, there's a balance, and I think the current balance is a bit too
skewed towards having too many slow tests in the "make check". I think
that, ideally, "make check" should be so fast that people run it as
their standard workflow before pushing a change, and we're not quite
there.
Having people do "make check" as a matter of routine, and running 97% of
the tests is, in my opinion, better than people doing a "make check"
seldom, but running 98% of the tests.
Where the cutoff is a matter of balance, yes, but I think a test that
takes a second is way too slow to be run in the routine case.
--
(domestic pets only, the antidote for overdose, milk.)
bloggy blog: http://lars.ingebrigtsen.no
next prev parent reply other threads:[~2020-09-13 12:30 UTC|newest]
Thread overview: 36+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <20200910182904.20559.25935@vcs0.savannah.gnu.org>
[not found] ` <20200910182905.F0E4520A2E@vcs0.savannah.gnu.org>
2020-09-11 9:25 ` master 262d0c6: Mark some tests as expensive Michael Albinus
2020-09-11 18:06 ` Stefan Kangas
2020-09-12 10:25 ` Daniel Martín
2020-09-12 10:38 ` Eli Zaretskii
2020-09-12 11:15 ` Lars Ingebrigtsen
2020-09-12 11:24 ` Eli Zaretskii
2020-09-12 12:11 ` Lars Ingebrigtsen
2020-09-12 12:29 ` Eli Zaretskii
2020-09-13 12:30 ` Lars Ingebrigtsen [this message]
2020-09-13 15:21 ` Stefan Monnier
2020-09-13 15:30 ` Lars Ingebrigtsen
2020-09-13 17:22 ` Michael Albinus
2020-09-12 16:47 ` Michael Albinus
2020-09-13 12:33 ` Lars Ingebrigtsen
2020-09-13 14:37 ` Eli Zaretskii
2020-09-12 11:27 ` Michael Albinus
2020-09-12 12:15 ` Lars Ingebrigtsen
2020-09-12 12:30 ` Michael Albinus
2020-09-12 12:36 ` Lars Ingebrigtsen
2020-09-12 13:04 ` Dmitry Gutov
2020-09-12 14:23 ` Daniel Martín
2020-09-12 14:49 ` Michael Albinus
2020-09-12 16:47 ` Dmitry Gutov
2020-09-12 14:53 ` Lars Ingebrigtsen
2020-09-12 14:00 ` Daniel Martín
2020-09-12 14:14 ` Eli Zaretskii
2020-09-12 15:16 ` Daniel Martín
2020-09-12 14:43 ` Michael Albinus
2020-09-12 15:02 ` Daniel Martín
2020-09-12 10:52 ` Michael Albinus
2020-09-18 10:22 ` Stefan Kangas
2020-09-18 10:31 ` Michael Albinus
2020-10-18 18:15 ` Stefan Kangas
2020-10-19 12:40 ` Michael Albinus
2020-10-19 15:34 ` Stefan Kangas
2020-10-19 16:42 ` Michael Albinus
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=87zh5tzwog.fsf@gnus.org \
--to=larsi@gnus.org \
--cc=eliz@gnu.org \
--cc=emacs-devel@gnu.org \
--cc=mardani29@yahoo.es \
--cc=michael.albinus@gmx.de \
--cc=stefan@marxist.se \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this external index
https://git.savannah.gnu.org/cgit/emacs.git
https://git.savannah.gnu.org/cgit/emacs/org-mode.git
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.