[PATCH] test: optionally print subtest number

[PATCH] test: optionally print subtest number

[David Bremner]

From: David Bremner <bremner@debian.org>

The idea is that $test_count could be used in tests to label
intermediate files. The output enabled by this patch (and --debug)
helps figure out which OUTPUT.nn file belongs to which test in case
several subtests write to OUTPUT.$test_count
---

Is there something that depends on the test format? I find it pretty
handy to have the subtest numbers, but I don't want to break some
other tools. I followed the existing style of conditionally defining
functions, but maybe someone with more bash-fu can improve that.

 test/test-lib.sh |   12 ++++++++++++
 1 files changed, 12 insertions(+), 0 deletions(-)

diff --git a/test/test-lib.sh b/test/test-lib.sh
index 6be93fe..aaaaf5c 100644
--- a/test/test-lib.sh
+++ b/test/test-lib.sh
@@ -116,6 +116,16 @@ do
 	esac
 done
 
+if test -n "$debug"; then
+    print_subtest () {
+	printf "\t[%d]\t" $(($test_count - 1))
+    }
+else
+    print_subtest () {
+	true
+    }
+fi
+
 if test -n "$color"; then
 	say_color () {
 		(
@@ -132,6 +142,7 @@ if test -n "$color"; then
 		printf " "
                 printf "$@"
 		tput sgr0
+		print_subtest
 		)
 	}
 else
@@ -140,6 +151,7 @@ else
 		shift
 		printf " "
                 printf "$@"
+		print_subtest
 	}
 fi
 
-- 
1.7.5.4

Re: [PATCH] test: optionally print subtest number

[Dmitry Kurochkin]

On Tue, 13 Dec 2011 15:56:47 -0400, David Bremner <david@tethera.net> wrote:
> From: David Bremner <bremner@debian.org>
> 
> The idea is that $test_count could be used in tests to label
> intermediate files. The output enabled by this patch (and --debug)
> helps figure out which OUTPUT.nn file belongs to which test in case
> several subtests write to OUTPUT.$test_count
> ---
> 
> Is there something that depends on the test format? I find it pretty
> handy to have the subtest numbers, but I don't want to break some
> other tools. I followed the existing style of conditionally defining
> functions, but maybe someone with more bash-fu can improve that.
> 

Looks good to me.  Except for tabs taking too much space.  Perhaps the
following would be better?

	printf "%-4s" "[$(($test_count - 1))]"

Regards,
  Dmitry

>  test/test-lib.sh |   12 ++++++++++++
>  1 files changed, 12 insertions(+), 0 deletions(-)
> 
> diff --git a/test/test-lib.sh b/test/test-lib.sh
> index 6be93fe..aaaaf5c 100644
> --- a/test/test-lib.sh
> +++ b/test/test-lib.sh
> @@ -116,6 +116,16 @@ do
>  	esac
>  done
>  
> +if test -n "$debug"; then
> +    print_subtest () {
> +	printf "\t[%d]\t" $(($test_count - 1))
> +    }
> +else
> +    print_subtest () {
> +	true
> +    }
> +fi
> +
>  if test -n "$color"; then
>  	say_color () {
>  		(
> @@ -132,6 +142,7 @@ if test -n "$color"; then
>  		printf " "
>                  printf "$@"
>  		tput sgr0
> +		print_subtest
>  		)
>  	}
>  else
> @@ -140,6 +151,7 @@ else
>  		shift
>  		printf " "
>                  printf "$@"
> +		print_subtest
>  	}
>  fi
>  
> -- 
> 1.7.5.4
> 
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH] test: optionally print subtest number

[Tomi Ollila]

On Wed, 14 Dec 2011 00:15:43 +0400, Dmitry Kurochkin <dmitry.kurochkin@gmail.com> wrote:
> On Tue, 13 Dec 2011 15:56:47 -0400, David Bremner <david@tethera.net> wrote:
> > From: David Bremner <bremner@debian.org>
> > 
> > The idea is that $test_count could be used in tests to label
> > intermediate files. The output enabled by this patch (and --debug)
> > helps figure out which OUTPUT.nn file belongs to which test in case
> > several subtests write to OUTPUT.$test_count
> > ---
> > 
> > Is there something that depends on the test format? I find it pretty
> > handy to have the subtest numbers, but I don't want to break some
> > other tools. I followed the existing style of conditionally defining
> > functions, but maybe someone with more bash-fu can improve that.
> > 
> 
> Looks good to me.  Except for tabs taking too much space.  Perhaps the
> following would be better?
> 
> 	printf "%-4s" "[$(($test_count - 1))]"

I attempted the same size reduction. Therefore I know that
this should to be either

 	printf " %-4s" "[$(($test_count - 1))]"
or
 	printf "\t%-4s" "[$(($test_count - 1))]"

(or something similar) so that there is space betweem BROKEN and [num]

This takes 4 bytes out from width (and drops tab as field separator)
(and only few lines goes over 80 char width (some goes even with this
reduction). So ...


> Regards,
>   Dmitry

Tomi

Re: [PATCH] test: optionally print subtest number

[Dmitry Kurochkin]

On Tue, 13 Dec 2011 22:55:18 +0200, Tomi Ollila <tomi.ollila@iki.fi> wrote:
> On Wed, 14 Dec 2011 00:15:43 +0400, Dmitry Kurochkin <dmitry.kurochkin@gmail.com> wrote:
> > On Tue, 13 Dec 2011 15:56:47 -0400, David Bremner <david@tethera.net> wrote:
> > > From: David Bremner <bremner@debian.org>
> > > 
> > > The idea is that $test_count could be used in tests to label
> > > intermediate files. The output enabled by this patch (and --debug)
> > > helps figure out which OUTPUT.nn file belongs to which test in case
> > > several subtests write to OUTPUT.$test_count
> > > ---
> > > 
> > > Is there something that depends on the test format? I find it pretty
> > > handy to have the subtest numbers, but I don't want to break some
> > > other tools. I followed the existing style of conditionally defining
> > > functions, but maybe someone with more bash-fu can improve that.
> > > 
> > 
> > Looks good to me.  Except for tabs taking too much space.  Perhaps the
> > following would be better?
> > 
> > 	printf "%-4s" "[$(($test_count - 1))]"
> 
> I attempted the same size reduction. Therefore I know that
> this should to be either
> 
>  	printf " %-4s" "[$(($test_count - 1))]"
> or
>  	printf "\t%-4s" "[$(($test_count - 1))]"
> 
> (or something similar) so that there is space betweem BROKEN and [num]
> 

Oh, apparently, not enough testing on my side.

I vote for the first version with a space.

Regards,
  Dmitry

> This takes 4 bytes out from width (and drops tab as field separator)
> (and only few lines goes over 80 char width (some goes even with this
> reduction). So ...
> 
> 
> > Regards,
> >   Dmitry
> 
> Tomi
> 

Re: [PATCH] test: optionally print subtest number

[Tomi Ollila]

On Wed, 14 Dec 2011 00:58:35 +0400, Dmitry Kurochkin <dmitry.kurochkin@gmail.com> wrote:
> On Tue, 13 Dec 2011 22:55:18 +0200, Tomi Ollila <tomi.ollila@iki.fi> wrote:
> > On Wed, 14 Dec 2011 00:15:43 +0400, Dmitry Kurochkin <dmitry.kurochkin@gmail.com> wrote:
> > > 
> > > Looks good to me.  Except for tabs taking too much space.  Perhaps the
> > > following would be better?
> > > 
> > > 	printf "%-4s" "[$(($test_count - 1))]"
> > 
> > I attempted the same size reduction. Therefore I know that
> > this should to be either
> > 
> >  	printf " %-4s" "[$(($test_count - 1))]"
> > or
> >  	printf "\t%-4s" "[$(($test_count - 1))]"
> > 
> > (or something similar) so that there is space betweem BROKEN and [num]
> > 
> 
> Oh, apparently, not enough testing on my side.
> 
> I vote for the first version with a space.

Have to agree, Now that I checked tabs were not used in output lines
before this addition.

> 
> Regards,
>   Dmitry

Tomi


> 
> > This takes 4 bytes out from width (and drops tab as field separator)
> > (and only few lines goes over 80 char width (some goes even with this
> > reduction). So ...
> > 
> > 
> > > Regards,
> > >   Dmitry
> > 
> > Tomi

Re: [PATCH] test: optionally print subtest number

[Tomi Ollila]

On Tue, 13 Dec 2011 15:56:47 -0400, David Bremner <david@tethera.net> wrote:
> From: David Bremner <bremner@debian.org>
> 
> The idea is that $test_count could be used in tests to label
> intermediate files. The output enabled by this patch (and --debug)
> helps figure out which OUTPUT.nn file belongs to which test in case
> several subtests write to OUTPUT.$test_count
> ---
> 
> Is there something that depends on the test format? I find it pretty
> handy to have the subtest numbers, but I don't want to break some
> other tools. I followed the existing style of conditionally defining
> functions, but maybe someone with more bash-fu can improve that.

+1. I would also say '+1' even non-debug format (and update dependend code,
if there is any).

Tomi

> 
>  test/test-lib.sh |   12 ++++++++++++
>  1 files changed, 12 insertions(+), 0 deletions(-)
> 
> diff --git a/test/test-lib.sh b/test/test-lib.sh
> index 6be93fe..aaaaf5c 100644
> --- a/test/test-lib.sh
> +++ b/test/test-lib.sh
> @@ -116,6 +116,16 @@ do
>  	esac
>  done
>  
> +if test -n "$debug"; then
> +    print_subtest () {
> +	printf "\t[%d]\t" $(($test_count - 1))
> +    }
> +else
> +    print_subtest () {
> +	true
> +    }
> +fi
> +
>  if test -n "$color"; then
>  	say_color () {
>  		(
> @@ -132,6 +142,7 @@ if test -n "$color"; then
>  		printf " "
>                  printf "$@"
>  		tput sgr0
> +		print_subtest
>  		)
>  	}
>  else
> @@ -140,6 +151,7 @@ else
>  		shift
>  		printf " "
>                  printf "$@"
> +		print_subtest
>  	}
>  fi
>  
> -- 
> 1.7.5.4
> 
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch
> 

-- 
Tomi Ollila !! tomi.ollila@nixu.fi !! +358 400 888 366
Nixu Oy !! http://www.nixu.fi/ !! Keilaranta 15, FI-02150 Espoo, Finland

Re: [PATCH] test: optionally print subtest number

[Jameson Graef Rollins]

[-- Attachment #1: Type: text/plain, Size: 650 bytes --]

On Tue, 13 Dec 2011 15:56:47 -0400, David Bremner <david@tethera.net> wrote:
> The idea is that $test_count could be used in tests to label
> intermediate files. The output enabled by this patch (and --debug)
> helps figure out which OUTPUT.nn file belongs to which test in case
> several subtests write to OUTPUT.$test_count

I've only been vaguely following this "test count" stuff, but I'm not
sure I understand what's the point of giving tests a number that is
ultimately mutable.  Why not just label things by the test name, instead
of the count?  That wouldn't require keeping track of number/name
mapping, which will change over time.

jamie.

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: [PATCH] test: optionally print subtest number

[David Bremner]

On Tue, 13 Dec 2011 14:22:21 -0800, Jameson Graef Rollins <jrollins@finestructure.net> wrote:

> I've only been vaguely following this "test count" stuff, but I'm not
> sure I understand what's the point of giving tests a number that is
> ultimately mutable.  Why not just label things by the test name, instead
> of the count?  That wouldn't require keeping track of number/name
> mapping, which will change over time.

We don't actually have test names, at least not ones directly suitable
for file names.  I guess we could encode them or something, is that what
you mean?

d

Re: [PATCH] test: optionally print subtest number

[Dmitry Kurochkin]

On Tue, 13 Dec 2011 19:18:16 -0400, David Bremner <david@tethera.net> wrote:
> On Tue, 13 Dec 2011 14:22:21 -0800, Jameson Graef Rollins <jrollins@finestructure.net> wrote:
> 
> > I've only been vaguely following this "test count" stuff, but I'm not
> > sure I understand what's the point of giving tests a number that is
> > ultimately mutable.  Why not just label things by the test name, instead
> > of the count?  That wouldn't require keeping track of number/name
> > mapping, which will change over time.
> 
> We don't actually have test names, at least not ones directly suitable
> for file names.  I guess we could encode them or something, is that what
> you mean?
> 

BTW I have some plans to introduce optional explicit test ids that can
be used for inter-test dependencies.  E.g.:

  test_begin_subtest test-id-1 "A subtest"
  ....
  ;; in another test requre that test-id-1 passed
  test_require_subtest test-id-1

Regards,
  Dmitry

> d
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH] test: optionally print subtest number

[Jameson Graef Rollins]

[-- Attachment #1: Type: text/plain, Size: 1065 bytes --]

On Wed, 14 Dec 2011 03:24:23 +0400, Dmitry Kurochkin <dmitry.kurochkin@gmail.com> wrote:
> BTW I have some plans to introduce optional explicit test ids that can
> be used for inter-test dependencies.  E.g.:
> 
>   test_begin_subtest test-id-1 "A subtest"
>   ....
>   ;; in another test requre that test-id-1 passed
>   test_require_subtest test-id-1

Would the required test need to be listed twice, both on the
begin_subtest line *and* in the require_subtest line?

And again, why would the test id have to be any different that the
existing test names?  The tests already have names, so I don't
understand why we would want to introduce some other kind of
identification.  Seems like it's just going to add extra confusion.

And speaking of which, I sometimes worry that the test infrastructure
itself is getting too complicated.  Pretty soon we're going to need
tests for the tests.  I don't necessarily see the need to all of these
extra features in the test suite, so I worry that it's just making
everything harder to debug.

jamie.

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: [PATCH] test: optionally print subtest number

[Dmitry Kurochkin]

On Tue, 13 Dec 2011 15:35:53 -0800, Jameson Graef Rollins <jrollins@finestructure.net> wrote:
> On Wed, 14 Dec 2011 03:24:23 +0400, Dmitry Kurochkin <dmitry.kurochkin@gmail.com> wrote:
> > BTW I have some plans to introduce optional explicit test ids that can
> > be used for inter-test dependencies.  E.g.:
> > 
> >   test_begin_subtest test-id-1 "A subtest"
> >   ....
> >   ;; in another test requre that test-id-1 passed
> >   test_require_subtest test-id-1
> 
> Would the required test need to be listed twice, both on the
> begin_subtest line *and* in the require_subtest line?
> 
> And again, why would the test id have to be any different that the
> existing test names?  The tests already have names, so I don't
> understand why we would want to introduce some other kind of
> identification.  Seems like it's just going to add extra confusion.
> 

What you listed in the other email are test scripts, each with many
subtests.  I was talking about dependencies between subtests, not test
scripts.

> And speaking of which, I sometimes worry that the test infrastructure
> itself is getting too complicated.  Pretty soon we're going to need
> tests for the tests.

We already have them :)  Though, pretty limited.

>  I don't necessarily see the need to all of these
> extra features in the test suite, so I worry that it's just making
> everything harder to debug.
> 

I hope we can keep balance here.

Without inter-subtest dependencies, we have unhealthy situation where
some tests may be skipped because of missing prerequisites, but test
that depend on them are failing.  The only alternative I see is to
rewrite these tests to remove the dependencies.  But that would
complicate test cases itself, so I believe inter-subtest dependencies is
a better option.

Regards,
  Dmitry

> jamie.

Re: [PATCH] test: optionally print subtest number

[Jameson Graef Rollins]

[-- Attachment #1: Type: text/plain, Size: 981 bytes --]

On Tue, 13 Dec 2011 19:18:16 -0400, David Bremner <david@tethera.net> wrote:
> We don't actually have test names, at least not ones directly suitable
> for file names.  I guess we could encode them or something, is that what
> you mean?

Don't we?  There is at least a word/phrase that's consistent between the
actual test script names, what is listed in the TESTS variable (below),
and what is output when the tests are run (ie. "basic:" or
"search-folder-coherence:").  Are those not test names?

jamie.


TESTS="
  basic
  help-test
  new
  count
  search
  search-output
  search-by-folder
  search-position-overlap-bug
  search-insufficient-from-quoting
  search-limiting
  tagging
  json
  multipart
  thread-naming
  raw
  reply
  dump-restore
  uuencode
  thread-order
  author-order
  from-guessing
  long-id
  encoding
  emacs
  emacs-large-search-buffer
  maildir-sync
  crypto
  symbol-hiding
  search-folder-coherence
  atomicity
  python
  hooks
  argument-parsing
"

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: [PATCH] test: optionally print subtest number

[David Bremner]

On Tue, 13 Dec 2011 15:25:14 -0800, Jameson Graef Rollins <jrollins@finestructure.net> wrote:
> On Tue, 13 Dec 2011 19:18:16 -0400, David Bremner <david@tethera.net> wrote:
> > We don't actually have test names, at least not ones directly suitable
> > for file names.  I guess we could encode them or something, is that what
> > you mean?
> 
> Don't we?  There is at least a word/phrase that's consistent between the
> actual test script names, what is listed in the TESTS variable (below),
> and what is output when the tests are run (ie. "basic:" or
> "search-folder-coherence:").  Are those not test names?

OK, we can call those test names. This thread/patch is about "subtest
numbers", so finer granularity. There are an average of 10 subtests per
test, those are are the things that don't have names.

d

Re: [PATCH] test: optionally print subtest number

[Tomi Ollila]

On Tue, 13 Dec 2011 15:56:47 -0400, David Bremner <david@tethera.net> wrote:
> From: David Bremner <bremner@debian.org>
> 
> The idea is that $test_count could be used in tests to label
> intermediate files. The output enabled by this patch (and --debug)
> helps figure out which OUTPUT.nn file belongs to which test in case
> several subtests write to OUTPUT.$test_count
> ---
> 
> Is there something that depends on the test format? I find it pretty
> handy to have the subtest numbers, but I don't want to break some
> other tools. I followed the existing style of conditionally defining
> functions, but maybe someone with more bash-fu can improve that.
> 
>  test/test-lib.sh |   12 ++++++++++++
>  1 files changed, 12 insertions(+), 0 deletions(-)
> 
> diff --git a/test/test-lib.sh b/test/test-lib.sh
> index 6be93fe..aaaaf5c 100644
> --- a/test/test-lib.sh
> +++ b/test/test-lib.sh
> @@ -116,6 +116,16 @@ do
>  	esac
>  done
>  
> +if test -n "$debug"; then
> +    print_subtest () {
> +	printf "\t[%d]\t" $(($test_count - 1))

Like this, as iterated by Dmitry & me

	printf " %-4s" "[$(($test_count - 1))]"
(or
	printf " %-4s" "[$((test_count - 1))]"
)

I think this is very useful feature; When searching reasons why
test failed the output files can be easily found as the test
number is show on screen. Also, when debugging problems with 
users online in distant locations, passing around the test
number in any particular moment eases the interaction considerably.

Tomi