bug#25618: 25.1.91; More precise limit for tagging expensive tests

bug#25618: 25.1.91; More precise limit for tagging expensive tests

[Tino Calancha]

Prefer set a numeric limit of seconds than using the word 'few'.

For instance, in my box the test `cl-seq-test-bug24264' in
test/lisp/emacs-lisp/cl-seq-tests.el
takes 7 s.
With an explicit numeric threshold is easier to decide whether this test
must be tagged as expensive or not.

In following patch i define a few as 2-3 seconds.  Other people might
prefer 4-5 or even higher.
Please, suggest a proper value to consider a test as expensive.

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
From 5671609ff5120ac7572b8a103a9aeb899542d47f Mon Sep 17 00:00:00 2001
From: Tino Calancha <tino.calancha@gmail.com>
Date: Sat, 4 Feb 2017 22:29:24 +0900
Subject: [PATCH] More precise limit for tagging expensive tests

* CONTRIBUTE (Testing your changes): Tag a test as expensive
when lasts longer than 2-3 seconds.
---
 CONTRIBUTE | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/CONTRIBUTE b/CONTRIBUTE
index e9a0341980..155b7ef762 100644
--- a/CONTRIBUTE
+++ b/CONTRIBUTE
@@ -273,7 +273,7 @@ http://www.gnu.org/software/emacs/manual/html_node/ert/
 or run 'info "(ert)"' for for more information on writing and running
 tests.
 
-If your test lasts longer than some few seconds, mark it in its
+If your test lasts longer than 2-3 seconds, mark it in its
 'ert-deftest' definition with ":tags '(:expensive-test)".
 
 To run tests on the entire Emacs tree, run "make check" from the
-- 
2.11.0

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
In GNU Emacs 25.1.91.1 (x86_64-pc-linux-gnu, GTK+ Version 3.22.7)
 of 2017-02-04
Repository revision: 5e921112680f96d3ed94faf7dcf257e07f92d74b

bug#25618: 25.1.91; More precise limit for tagging expensive tests

[Noam Postavsky]

On Sat, Feb 4, 2017 at 8:42 AM, Tino Calancha <tino.calancha@gmail.com> wrote:
>
> Prefer set a numeric limit of seconds than using the word 'few'.
>
> For instance, in my box the test `cl-seq-test-bug24264' in
> test/lisp/emacs-lisp/cl-seq-tests.el
> takes 7 s.
> With an explicit numeric threshold is easier to decide whether this test
> must be tagged as expensive or not.
>
> In following patch i define a few as 2-3 seconds.  Other people might
> prefer 4-5 or even higher.
> Please, suggest a proper value to consider a test as expensive.

Is it worth specifying this so precisely? The time elapsed depends on
the speed of the computer you run it on (also, whether Emacs is
compiled with optimization or not). A test that takes 2 seconds on
your computer might easily take 5 seconds on mine.

bug#25618: 25.1.91; More precise limit for tagging expensive tests

[Tino Calancha]

On Sat, 4 Feb 2017, Noam Postavsky wrote:

> On Sat, Feb 4, 2017 at 8:42 AM, Tino Calancha <tino.calancha@gmail.com> wrote:
>>
>> Prefer set a numeric limit of seconds than using the word 'few'.
>>
>> For instance, in my box the test `cl-seq-test-bug24264' in
>> test/lisp/emacs-lisp/cl-seq-tests.el
>> takes 7 s.
>> With an explicit numeric threshold is easier to decide whether this test
>> must be tagged as expensive or not.
>>
>> In following patch i define a few as 2-3 seconds.  Other people might
>> prefer 4-5 or even higher.
>> Please, suggest a proper value to consider a test as expensive.
>
> Is it worth specifying this so precisely? The time elapsed depends on
> the speed of the computer you run it on (also, whether Emacs is
> compiled with optimization or not). A test that takes 2 seconds on
> your computer might easily take 5 seconds on mine.
When i run the test suite, the test mentioned before takes significantly
longer than the others.  I feel like it could be considered as 
slightly expensive.
After reading 'some few seconds' in CONTRIBUTE i am still in doubt on 
that.  It could be more clear.

bug#25618: 25.1.91; More precise limit for tagging expensive tests

[Eli Zaretskii]

> From: Tino Calancha <tino.calancha@gmail.com>
> Date: Sat, 04 Feb 2017 22:42:44 +0900
> 
> 
> Prefer set a numeric limit of seconds than using the word 'few'.
> 
> For instance, in my box the test `cl-seq-test-bug24264' in
> test/lisp/emacs-lisp/cl-seq-tests.el
> takes 7 s.
> With an explicit numeric threshold is easier to decide whether this test
> must be tagged as expensive or not.
> 
> In following patch i define a few as 2-3 seconds.  Other people might
> prefer 4-5 or even higher.
> Please, suggest a proper value to consider a test as expensive.

I think 2-3 sec is negligible, and 7 sec is not long enough to annoy.
10 sec or more is beginning to sound like it comes close to "too
long", and 20 sec is definitely too long.

bug#25618: 25.1.91; More precise limit for tagging expensive tests

[Tino Calancha]

On Sat, 4 Feb 2017, Eli Zaretskii wrote:

>> From: Tino Calancha <tino.calancha@gmail.com>
>> Date: Sat, 04 Feb 2017 22:42:44 +0900
>>
>>
>> Prefer set a numeric limit of seconds than using the word 'few'.
>>
>> For instance, in my box the test `cl-seq-test-bug24264' in
>> test/lisp/emacs-lisp/cl-seq-tests.el
>> takes 7 s.
>> With an explicit numeric threshold is easier to decide whether this test
>> must be tagged as expensive or not.
>>
>> In following patch i define a few as 2-3 seconds.  Other people might
>> prefer 4-5 or even higher.
>> Please, suggest a proper value to consider a test as expensive.
>
> I think 2-3 sec is negligible, and 7 sec is not long enough to annoy.
> 10 sec or more is beginning to sound like it comes close to "too
> long", and 20 sec is definitely too long.
Thank you.  Now is clear.
Are you OK with keep the sentence as it is now?
I mean, keep: 'some few seconds'.

bug#25618: 25.1.91; More precise limit for tagging expensive tests

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Tino Calancha <tino.calancha@gmail.com>
>> Date: Sat, 04 Feb 2017 22:42:44 +0900
>> 
>> 
>> Prefer set a numeric limit of seconds than using the word 'few'.
>> 
>> For instance, in my box the test `cl-seq-test-bug24264' in
>> test/lisp/emacs-lisp/cl-seq-tests.el
>> takes 7 s.
>> With an explicit numeric threshold is easier to decide whether this test
>> must be tagged as expensive or not.
>> 
>> In following patch i define a few as 2-3 seconds.  Other people might
>> prefer 4-5 or even higher.
>> Please, suggest a proper value to consider a test as expensive.
>
> I think 2-3 sec is negligible, and 7 sec is not long enough to annoy.
> 10 sec or more is beginning to sound like it comes close to "too
> long", and 20 sec is definitely too long.

Unfortunately, there is no mean in ert to see how long a single test
runs. I often run expensive tests (well, I'm testing Tramp, filenotify,
autorevert :-) and it would help me to know the time for a single test
case run.

Maybe somebody could add it? I haven't the time just now; but if it
annoys me too much, I'll do it myself, later on, maybe.

This ticket might be a good place to remember this. Pls don't close it
after changing CONTRIBUTE.

Best regards, Michael.

bug#25618: 25.1.91; More precise limit for tagging expensive tests

[Noam Postavsky]

On Sun, Feb 5, 2017 at 4:53 AM, Michael Albinus <michael.albinus@gmx.de> wrote:
> Maybe somebody could add it? I haven't the time just now; but if it
> annoys me too much, I'll do it myself, later on, maybe.
>
> This ticket might be a good place to remember this. Pls don't close it
> after changing CONTRIBUTE.

Could you instead open a separate bug? I've been through a lot of old
bugs threads that are difficult to understand because they branch into
multiple issues.

bug#25618: AW: Re: bug#25618: 25.1.91; More precise limit for tagging expensive tests

[michael.albinus]

[-- Attachment #1: Type: text/plain, Size: 979 bytes --]

Will do later this week. I'm almost offline next days, only email works for me.
Best regards, Michael. 


Von meinem Samsung Galaxy Smartphone gesendet.
-------- Ursprüngliche Nachricht --------Von: Noam Postavsky <npostavs@users.sourceforge.net> Datum: 05.02.17  14:53  (GMT+01:00) An: Michael Albinus <michael.albinus@gmx.de> Cc: Eli Zaretskii <eliz@gnu.org>, 25618@debbugs.gnu.org, Tino Calancha <tino.calancha@gmail.com> Betreff: Re: bug#25618: 25.1.91; More precise limit for tagging expensive tests 
On Sun, Feb 5, 2017 at 4:53 AM, Michael Albinus <michael.albinus@gmx.de> wrote:
> Maybe somebody could add it? I haven't the time just now; but if it
> annoys me too much, I'll do it myself, later on, maybe.
>
> This ticket might be a good place to remember this. Pls don't close it
> after changing CONTRIBUTE.

Could you instead open a separate bug? I've been through a lot of old
bugs threads that are difficult to understand because they branch into
multiple issues.

[-- Attachment #2: Type: text/html, Size: 1501 bytes --]

bug#25618: 25.1.91; More precise limit for tagging expensive tests

[Eli Zaretskii]

> From: Tino Calancha <tino.calancha@gmail.com>
> Date: Sun, 5 Feb 2017 02:54:03 +0900 (JST)
> cc: Tino Calancha <tino.calancha@gmail.com>, 25618@debbugs.gnu.org
> 
> On Sat, 4 Feb 2017, Eli Zaretskii wrote:
> 
> >> From: Tino Calancha <tino.calancha@gmail.com>
> >> Date: Sat, 04 Feb 2017 22:42:44 +0900
> >>
> >>
> >> Prefer set a numeric limit of seconds than using the word 'few'.
> >>
> >> For instance, in my box the test `cl-seq-test-bug24264' in
> >> test/lisp/emacs-lisp/cl-seq-tests.el
> >> takes 7 s.
> >> With an explicit numeric threshold is easier to decide whether this test
> >> must be tagged as expensive or not.
> >>
> >> In following patch i define a few as 2-3 seconds.  Other people might
> >> prefer 4-5 or even higher.
> >> Please, suggest a proper value to consider a test as expensive.
> >
> > I think 2-3 sec is negligible, and 7 sec is not long enough to annoy.
> > 10 sec or more is beginning to sound like it comes close to "too
> > long", and 20 sec is definitely too long.
> Thank you.  Now is clear.
> Are you OK with keep the sentence as it is now?
> I mean, keep: 'some few seconds'.

It's okay to leave it as is.  I also don't object to saying something
more specific, but 2-3 sec sound just too few to me.

bug#25618: 25.1.91; More precise limit for tagging expensive tests

[Tino Calancha]

On 02/10/2017 05:49 PM, Eli Zaretskii wrote:
>> From: Tino Calancha <tino.calancha@gmail.com>
>> Date: Sun, 5 Feb 2017 02:54:03 +0900 (JST)
>> cc: Tino Calancha <tino.calancha@gmail.com>, 25618@debbugs.gnu.org
>>
>> On Sat, 4 Feb 2017, Eli Zaretskii wrote:
>>
>>>> From: Tino Calancha <tino.calancha@gmail.com>
>>>> Date: Sat, 04 Feb 2017 22:42:44 +0900
>>>>
>>>>
>>>> Prefer set a numeric limit of seconds than using the word 'few'.
>>>>
>>>> For instance, in my box the test `cl-seq-test-bug24264' in
>>>> test/lisp/emacs-lisp/cl-seq-tests.el
>>>> takes 7 s.
>>>> With an explicit numeric threshold is easier to decide whether this test
>>>> must be tagged as expensive or not.
>>>>
>>>> In following patch i define a few as 2-3 seconds.  Other people might
>>>> prefer 4-5 or even higher.
>>>> Please, suggest a proper value to consider a test as expensive.
>>> I think 2-3 sec is negligible, and 7 sec is not long enough to annoy.
>>> 10 sec or more is beginning to sound like it comes close to "too
>>> long", and 20 sec is definitely too long.
>> Thank you.  Now is clear.
>> Are you OK with keep the sentence as it is now?
>> I mean, keep: 'some few seconds'.
> It's okay to leave it as is.  I also don't object to saying something
> more specific, but 2-3 sec sound just too few to me.
Thanks.  It's not a big thing.  I will leave it as it is.