From: Eli Zaretskii <eliz@gnu.org>
To: Lars Ingebrigtsen <larsi@gnus.org>
Cc: stefan@marxist.se, emacs-devel@gnu.org, michael.albinus@gmx.de,
mardani29@yahoo.es
Subject: Re: master 262d0c6: Mark some tests as expensive
Date: Sat, 12 Sep 2020 15:29:06 +0300 [thread overview]
Message-ID: <83363ndvql.fsf@gnu.org> (raw)
In-Reply-To: <877dsz2o03.fsf@gnus.org> (message from Lars Ingebrigtsen on Sat, 12 Sep 2020 14:11:40 +0200)
> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: mardani29@yahoo.es, michael.albinus@gmx.de, stefan@marxist.se,
> emacs-devel@gnu.org
> Date: Sat, 12 Sep 2020 14:11:40 +0200
>
> Eli Zaretskii <eliz@gnu.org> writes:
>
> > How will that work if the change was in Tramp, or was related to
> > Tramp? In my case, for example, it would most probably mean none of
> > the Tramp tests will ever run at all, because they all take more than
> > 1 sec down here.
> >
> > Also, there are some tests with inherent delays (auto-revert tests are
> > one example, but there are others) -- would that mean we don't run
> > these, either?
>
> Yes, if they take more than a second to run.
So how do we make sure some change didn't introduce a bug in those
features?
> We've already made this decision with all the tests previously marked as
> expensive, so there's nothing new here.
There's a large gap between what is currently marked as "expensive"
tests and having entire packages not tested at all. The latter sounds
too radical to me. E.g., auto-revert is an important feature, used by
many people. Not having it in regression testing sounds like a step
backward to me.
> > What I do for a "fast test" use case is to run only the tests directly
> > related to the change, usually the FOO-tests.el tests that correspond
> > to the file FOO I've changed. Sometimes, there are related tests in
> > other files as well. A simple "make FOO-tests" command is all that's
> > needed.
>
> I do that, too, when working on something specific. But being able to
> say "make check" gives me greater confidence that I've not messed up
> something in a related area (and let's face it, the entirety of Emacs is
> "a related area" :-)).
I'm talking about a balance here. Losing the tests of complete
features just because we want tests to finish quickly sounds
sub-optimal to me. Can we make a smarter balance? After all, what
matters is not how long a single test runs, but how long the entire
suite runs. Having a few tests that take more than a second doesn't
necessarily enlarge the total time significantly, especially since
quite a few of the tests are much shorter. So maybe a better
criterion would be the time it takes to run the entire suite?
next prev parent reply other threads:[~2020-09-12 12:29 UTC|newest]
Thread overview: 36+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <20200910182904.20559.25935@vcs0.savannah.gnu.org>
[not found] ` <20200910182905.F0E4520A2E@vcs0.savannah.gnu.org>
2020-09-11 9:25 ` master 262d0c6: Mark some tests as expensive Michael Albinus
2020-09-11 18:06 ` Stefan Kangas
2020-09-12 10:25 ` Daniel Martín
2020-09-12 10:38 ` Eli Zaretskii
2020-09-12 11:15 ` Lars Ingebrigtsen
2020-09-12 11:24 ` Eli Zaretskii
2020-09-12 12:11 ` Lars Ingebrigtsen
2020-09-12 12:29 ` Eli Zaretskii [this message]
2020-09-13 12:30 ` Lars Ingebrigtsen
2020-09-13 15:21 ` Stefan Monnier
2020-09-13 15:30 ` Lars Ingebrigtsen
2020-09-13 17:22 ` Michael Albinus
2020-09-12 16:47 ` Michael Albinus
2020-09-13 12:33 ` Lars Ingebrigtsen
2020-09-13 14:37 ` Eli Zaretskii
2020-09-12 11:27 ` Michael Albinus
2020-09-12 12:15 ` Lars Ingebrigtsen
2020-09-12 12:30 ` Michael Albinus
2020-09-12 12:36 ` Lars Ingebrigtsen
2020-09-12 13:04 ` Dmitry Gutov
2020-09-12 14:23 ` Daniel Martín
2020-09-12 14:49 ` Michael Albinus
2020-09-12 16:47 ` Dmitry Gutov
2020-09-12 14:53 ` Lars Ingebrigtsen
2020-09-12 14:00 ` Daniel Martín
2020-09-12 14:14 ` Eli Zaretskii
2020-09-12 15:16 ` Daniel Martín
2020-09-12 14:43 ` Michael Albinus
2020-09-12 15:02 ` Daniel Martín
2020-09-12 10:52 ` Michael Albinus
2020-09-18 10:22 ` Stefan Kangas
2020-09-18 10:31 ` Michael Albinus
2020-10-18 18:15 ` Stefan Kangas
2020-10-19 12:40 ` Michael Albinus
2020-10-19 15:34 ` Stefan Kangas
2020-10-19 16:42 ` Michael Albinus
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=83363ndvql.fsf@gnu.org \
--to=eliz@gnu.org \
--cc=emacs-devel@gnu.org \
--cc=larsi@gnus.org \
--cc=mardani29@yahoo.es \
--cc=michael.albinus@gmx.de \
--cc=stefan@marxist.se \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this external index
https://git.savannah.gnu.org/cgit/emacs.git
https://git.savannah.gnu.org/cgit/emacs/org-mode.git
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.