[PATCH] Document the dependency on dtach in the test suite

[PATCH] Document the dependency on dtach in the test suite

[Ethan Glasser-Camp]

From: Ethan Glasser-Camp <ethan@betacantrips.com>

dtach (and before that, screen) are used to run emacs with more
verisimilitude, but this is only mentioned in the NEWS. Mention this
in the test/README file, to make it easier to find out why a bunch of
tests are failing/skipped.

Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
---

Just cutting my teeth on the codebase, and it took me a little while
to figure this out, so I thought I'd document it.

 test/README |    5 +++++
 1 files changed, 5 insertions(+), 0 deletions(-)

diff --git a/test/README b/test/README
index bde6db0..29afb3b 100644
--- a/test/README
+++ b/test/README
@@ -6,6 +6,11 @@ When fixing bugs or enhancing notmuch, you are strongly encouraged to
 add tests in this directory to cover what you are trying to fix or
 enhance.
 
+Prerequisites
+-------------
+emacs is run within dtach, so you need to have that installed to run
+emacs tests (or crypto tests, which rely on emacs to run).
+
 Running Tests
 -------------
 The easiest way to run tests is to say "make test", (or simply run the
-- 
1.7.5.4

Re: [PATCH] Document the dependency on dtach in the test suite

[Dmitry Kurochkin]

On Mon, 16 Jan 2012 12:16:08 -0500, Ethan Glasser-Camp <glasse@cs.rpi.edu> wrote:
> From: Ethan Glasser-Camp <ethan@betacantrips.com>
> 
> dtach (and before that, screen) are used to run emacs with more
> verisimilitude, but this is only mentioned in the NEWS. Mention this
> in the test/README file, to make it easier to find out why a bunch of
> tests are failing/skipped.
> 

Can you please add other prerequisites as well?  There is gpg and
probably more.

Can we just make it an alphabetically-sorted list?  That would make it
easier to read and add more dependencies in the future.  I.e.:

  Some tests require the following external dependencies to run:

    * emacs(1)
    * emacsclient(1)
    ...

Regards,
  Dmitry

> Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
> ---
> 
> Just cutting my teeth on the codebase, and it took me a little while
> to figure this out, so I thought I'd document it.
> 
>  test/README |    5 +++++
>  1 files changed, 5 insertions(+), 0 deletions(-)
> 
> diff --git a/test/README b/test/README
> index bde6db0..29afb3b 100644
> --- a/test/README
> +++ b/test/README
> @@ -6,6 +6,11 @@ When fixing bugs or enhancing notmuch, you are strongly encouraged to
>  add tests in this directory to cover what you are trying to fix or
>  enhance.
>  
> +Prerequisites
> +-------------
> +emacs is run within dtach, so you need to have that installed to run
> +emacs tests (or crypto tests, which rely on emacs to run).
> +
>  Running Tests
>  -------------
>  The easiest way to run tests is to say "make test", (or simply run the
> -- 
> 1.7.5.4
> 
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch

[PATCH] test: always report missing prereqs, independent of `--verbose' option

[Pieter Praet]

When tests are skipped due to missing prereqs, those prereqs are only
displayed when running with the `--verbose' option.  This is essential
information when troubleshooting, so always send to stdout.

---

 test/test-lib.sh |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/test/test-lib.sh b/test/test-lib.sh
index d1fbc05..6560628 100644
--- a/test/test-lib.sh
+++ b/test/test-lib.sh
@@ -673,8 +673,8 @@ test_skip () {
 
 test_check_missing_external_prereqs_ () {
 	if test -n "$test_subtest_missing_external_prereqs_"; then
-		say_color skip >&3 "missing prerequisites:"
-		echo "$test_subtest_missing_external_prereqs_" >&3
+		say_color skip >&1 "missing prerequisites:"
+		echo "$test_subtest_missing_external_prereqs_" >&1
 		test_report_skip_ "$@"
 	else
 		false
-- 
1.7.8.1

Re: [PATCH] test: always report missing prereqs, independent of `--verbose' option

[Tomi Ollila]

On Wed, 18 Jan 2012 13:19:41 +0100, Pieter Praet <pieter@praet.org> wrote:
> When tests are skipped due to missing prereqs, those prereqs are only
> displayed when running with the `--verbose' option.  This is essential
> information when troubleshooting, so always send to stdout.
> 
> ---

Instead of this the test suite could be enchanged so that
full log of the execution is stored somewhere and path
to that file is displayed at the end of test. Generally,
all output could stay available after tests exit so that
those could be investigated furtner.

(I tried to grep 'missing prerequisites' after running test
but did not find the info. the file opened for >&3 has been
deleted already?)

Now I just see huge amounts of missing prerequisites: emacs(1)
in my output (I presume that doesn't happen due to my
hack to "hide" emacs).

But well, I'm not against applying this patch -- just that
the test system starts to get so rotten touching it gives
shivers...

Tomi

> 
>  test/test-lib.sh |    4 ++--
>  1 files changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/test/test-lib.sh b/test/test-lib.sh
> index d1fbc05..6560628 100644
> --- a/test/test-lib.sh
> +++ b/test/test-lib.sh
> @@ -673,8 +673,8 @@ test_skip () {
>  
>  test_check_missing_external_prereqs_ () {
>  	if test -n "$test_subtest_missing_external_prereqs_"; then
> -		say_color skip >&3 "missing prerequisites:"
> -		echo "$test_subtest_missing_external_prereqs_" >&3
> +		say_color skip >&1 "missing prerequisites:"
> +		echo "$test_subtest_missing_external_prereqs_" >&1
>  		test_report_skip_ "$@"
>  	else
>  		false
> -- 
> 1.7.8.1

Re: [PATCH] test: always report missing prereqs, independent of `--verbose' option

[Pieter Praet]

On Wed, 18 Jan 2012 15:53:59 +0200, Tomi Ollila <tomi.ollila@iki.fi> wrote:
> On Wed, 18 Jan 2012 13:19:41 +0100, Pieter Praet <pieter@praet.org> wrote:
> > When tests are skipped due to missing prereqs, those prereqs are only
> > displayed when running with the `--verbose' option.  This is essential
> > information when troubleshooting, so always send to stdout.
> > 
> > ---
> 
> Instead of this the test suite could be enchanged so that
> full log of the execution is stored somewhere and path
> to that file is displayed at the end of test. Generally,
> all output could stay available after tests exit so that
> those could be investigated furtner.
> 

Indeed.  AFAIK, this used to be possible using the `--tee' option,
but the files it produces appear to be deleted when the tests finish.

> (I tried to grep 'missing prerequisites' after running test
> but did not find the info. the file opened for >&3 has been
> deleted already?)
> 

Hmmm, odd.

Are you sure you were using my patch and/or the `--verbose' option?

Works here, using this highly sophisticated Emacs-hiding wizardry:

  #+begin_src sh
    sudo mv /usr/bin/emacs /usr/bin/emacs.bak
    make test | grep 'missing prerequisites'
    sudo mv /usr/bin/emacs.bak /usr/bin/emacs
  #+end_src

(patent pending... ;)

> Now I just see huge amounts of missing prerequisites: emacs(1)
> in my output (I presume that doesn't happen due to my
> hack to "hide" emacs).
> 

When using my patch, you mean?

Most likely because "$test_subtest_missing_external_prereqs_" is only
reset at the beginning of every test series, so if an Emacs-dependent
subtest is skipped, all subsequent skipped subtests will report that
dependency as well, even if they were skipped due to a missing vi :).

> But well, I'm not against applying this patch -- just that
> the test system starts to get so rotten touching it gives
> shivers...
> 

I wouldn't say it's getting rotten, but yeah, "tread lightly" should
probably be mandatory advice to those aspiring to explore its bowels...

> Tomi
> 
> > 
> >  test/test-lib.sh |    4 ++--
> >  1 files changed, 2 insertions(+), 2 deletions(-)
> > 
> > diff --git a/test/test-lib.sh b/test/test-lib.sh
> > index d1fbc05..6560628 100644
> > --- a/test/test-lib.sh
> > +++ b/test/test-lib.sh
> > @@ -673,8 +673,8 @@ test_skip () {
> >  
> >  test_check_missing_external_prereqs_ () {
> >  	if test -n "$test_subtest_missing_external_prereqs_"; then
> > -		say_color skip >&3 "missing prerequisites:"
> > -		echo "$test_subtest_missing_external_prereqs_" >&3
> > +		say_color skip >&1 "missing prerequisites:"
> > +		echo "$test_subtest_missing_external_prereqs_" >&1
> >  		test_report_skip_ "$@"
> >  	else
> >  		false
> > -- 
> > 1.7.8.1


Peace

-- 
Pieter

Re: [PATCH] test: always report missing prereqs, independent of `--verbose' option

[Pieter Praet]

On Fri, 20 Jan 2012 01:24:02 +0100, Pieter Praet <pieter@praet.org> wrote:
> [...]
> Most likely because "$test_subtest_missing_external_prereqs_" is only
> reset at the beginning of every test series, so if an Emacs-dependent
> subtest is skipped, all subsequent skipped subtests will report that
> dependency as well, even if they were skipped due to a missing vi :).
> [...]

That's not right at all.

"$test_subtest_missing_external_prereqs_" is reset at every invocation
of `test_begin_subtest', using `test_reset_state_'.

Apologies for the misinformation.


Peace

-- 
Pieter

Re: [PATCH] test: always report missing prereqs, independent of `--verbose' option

[David Bremner]

On Wed, 18 Jan 2012 13:19:41 +0100, Pieter Praet <pieter@praet.org> wrote:
> When tests are skipped due to missing prereqs, those prereqs are only
> displayed when running with the `--verbose' option.  This is essential
> information when troubleshooting, so always send to stdout.

pushed,

d