bug#15991: 24.3.50; Wishlist: Make test suite user-friendlier

bug#15991: 24.3.50; Wishlist: Make test suite user-friendlier

[Eli Zaretskii]

This is a wishlist feature request: make the test suite easier to use
when some tests fail, and there's a need to investigate why and debug
them.

When you see any failures, the first thing you want to do is run only
one test that failed at a time.  But there's no way to do that, so it
seems, except by copying a (quite complex) command line from the
Makefile, while manually expanding every variable that's involved in
that.

Also, suppose one needs to run Emacs under a debugger for running a
test -- there doesn't seem to be a convenient way of doing that,
either.  Once again, start Emacs manually in GDB, then copy/paste the
command line there, and be sure to set the environment variables, like
EMACS_TEST_DIRECTORY before starting GDB.  Wait, and what if the test
forks, and you need to attach GDB to the correct Emacs process?

Another possible improvement would be for each test to automatically
save the log of its run, including any error messages, to a file.  As
things are now, one needs to do that manually as well, using shell
magic and/or 'tee', as the full log is extremely voluminous.

I wish all this could be easier, and I wish there would be a README in
that directory with instructions of how to do all that.

A good example of both the facilities and their README documentation
can be found in the Cairo distribution (look in the test/
subdirectory).

Thanks in advance.


In GNU Emacs 24.3.50.81 (i686-pc-mingw32)
 of 2013-11-28 on HOME-C4E4A596F7
Bzr revision: 115269 eliz@gnu.org-20131128194015-2eu4ceuanrqt0qwr
Windowing system distributor `Microsoft Corp.', version 5.1.2600
Configured using:
 `configure --prefix=/d/usr --enable-checking=yes,glyphs 'CFLAGS=-O0
 -gdwarf-2 -g3''

Important settings:
  value of $LANG: ENU
  locale-coding-system: cp1255
  default enable-multibyte-characters: t

Major mode: Lisp Interaction

Minor modes in effect:
  tooltip-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent input:
M-x r e p o r t - e m <tab> <return>

Recent messages:
For information about GNU Emacs and the GNU system, type C-h C-a.

Load-path shadows:
None found.

Features:
(shadow sort gnus-util mail-extr emacsbug message format-spec rfc822 mml
easymenu mml-sec mm-decode mm-bodies mm-encode mail-parse rfc2231
mailabbrev gmm-utils mailheader sendmail rfc2047 rfc2045 ietf-drums
mm-util mail-prsvr mail-utils time-date tooltip ediff-hook vc-hooks
lisp-float-type mwheel dos-w32 ls-lisp w32-common-fns disp-table w32-win
w32-vars tool-bar dnd fontset image regexp-opt fringe tabulated-list
newcomment lisp-mode prog-mode register page menu-bar rfn-eshadow timer
select scroll-bar mouse jit-lock font-lock syntax facemenu font-core
frame cham georgian utf-8-lang misc-lang vietnamese tibetan thai
tai-viet lao korean japanese hebrew greek romanian slovak czech european
ethiopic indian cyrillic chinese case-table epa-hook jka-cmpr-hook help
simple abbrev minibuffer nadvice loaddefs button faces cus-face macroexp
files text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote make-network-process
w32notify w32 multi-tty emacs)

bug#15991: 24.3.50; Wishlist: Make test suite user-friendlier

[Glenn Morris]

There was a patch for some of this (eg running one test).
http://lists.gnu.org/archive/html/emacs-devel/2013-08/msg00339.html

(It feels to me like one of things that will never get installed...)

bug#15991: 24.3.50; Wishlist: Make test suite user-friendlier

[Stefan Monnier]

> There was a patch for some of this (eg running one test).
> http://lists.gnu.org/archive/html/emacs-devel/2013-08/msg00339.html

Don't know it hasn't been installed yet, indeed,


        Stefan

bug#15991: 24.3.50; Wishlist: Make test suite user-friendlier

[Glenn Morris]

Eli Zaretskii wrote:

> When you see any failures, the first thing you want to do is run only
> one test that failed at a time.
[...]
> Another possible improvement would be for each test to automatically
> save the log of its run, including any error messages, to a file.

Now you can run `make foo.log' to run the foo.el tests.

> Also, suppose one needs to run Emacs under a debugger for running a
> test -- there doesn't seem to be a convenient way of doing that,
> either.  Once again, start Emacs manually in GDB, then copy/paste the
> command line there, and be sure to set the environment variables, like
> EMACS_TEST_DIRECTORY before starting GDB.  Wait, and what if the test
> forks, and you need to attach GDB to the correct Emacs process?

Haven't done anything about this.

bug#15991: 24.3.50; Wishlist: Make test suite user-friendlier

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Cc: 15991@debbugs.gnu.org
> Date: Thu, 26 Jun 2014 02:05:53 -0400
> 
> Eli Zaretskii wrote:
> 
> > When you see any failures, the first thing you want to do is run only
> > one test that failed at a time.
> [...]
> > Another possible improvement would be for each test to automatically
> > save the log of its run, including any error messages, to a file.
> 
> Now you can run `make foo.log' to run the foo.el tests.

Thank you.  Perhaps this should be mentioned in some README.

bug#15991: 24.3.50; Wishlist: Make test suite user-friendlier

[Michael Albinus]

Glenn Morris <rgm@gnu.org> writes:

> Now you can run `make foo.log' to run the foo.el tests.

Yes, but you must also force the target, or remove the log file prior
the make call:

~/src/emacs> make -B -C test/automated tramp-tests.log

bug#15991: 24.3.50; Wishlist: Make test suite user-friendlier

[Glenn Morris]

Michael Albinus wrote:

> Yes, but you must also force the target, or remove the log file prior
> the make call:
>
> ~/src/emacs> make -B -C test/automated tramp-tests.log

Yes, the issue is that the log depends only on the test source file,
not on "all of Emacs".

bug#15991: 24.3.50; Wishlist: Make test suite user-friendlier

[Glenn Morris]

Glenn Morris wrote:

>> ~/src/emacs> make -B -C test/automated tramp-tests.log
>
> Yes, the issue is that the log depends only on the test source file,
> not on "all of Emacs".

I mean, we could force it so that the tests are simply run every time,
but I wasn't sure if that was preferable.

bug#15991: 24.3.50; Wishlist: Make test suite user-friendlier

[Glenn Morris]

> Also, suppose one needs to run Emacs under a debugger for running a
> test -- there doesn't seem to be a convenient way of doing that,
> either.  Once again, start Emacs manually in GDB, then copy/paste the
> command line there, and be sure to set the environment variables, like
> EMACS_TEST_DIRECTORY before starting GDB.  Wait, and what if the test
> forks, and you need to attach GDB to the correct Emacs process?

(But none of the Emacs Makefiles provide any such gdb facility?)

All I can suggest here is that the Makefile define a GDB variable,
normally empty, that you can set to "gdb" on the make command-line when
you want to.

bug#15991: 24.3.50; Wishlist: Make test suite user-friendlier

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Cc: 15991@debbugs.gnu.org
> Date: Thu, 26 Jun 2014 12:44:10 -0400
> 
> 
> > Also, suppose one needs to run Emacs under a debugger for running a
> > test -- there doesn't seem to be a convenient way of doing that,
> > either.  Once again, start Emacs manually in GDB, then copy/paste the
> > command line there, and be sure to set the environment variables, like
> > EMACS_TEST_DIRECTORY before starting GDB.  Wait, and what if the test
> > forks, and you need to attach GDB to the correct Emacs process?
> 
> (But none of the Emacs Makefiles provide any such gdb facility?)

Most Emacs programs are interactive, so it's relatively easy to run
them under a debugger.  The test suite runs Emacs with a non-trivial
and quite long command line, so it's harder.

> All I can suggest here is that the Makefile define a GDB variable,
> normally empty, that you can set to "gdb" on the make command-line when
> you want to.

I guess it would be OK, if $GDB could be set to "gdb --args".  Thanks.

bug#15991: 24.3.50; Wishlist: Make test suite user-friendlier

[Glenn Morris]

Eli Zaretskii wrote:

>> All I can suggest here is that the Makefile define a GDB variable,
>> normally empty, that you can set to "gdb" on the make command-line when
>> you want to.
>
> I guess it would be OK, if $GDB could be set to "gdb --args".  Thanks.

Could you do it? :)
I confess I don't know quite where you want it to go...

bug#15991: 24.3.50; Wishlist: Make test suite user-friendlier

[Andreas Röhler]

On 26.06.2014 19:00, Glenn Morris wrote:
> Eli Zaretskii wrote:
>
>>> All I can suggest here is that the Makefile define a GDB variable,
>>> normally empty, that you can set to "gdb" on the make command-line when
>>> you want to.
>>
>> I guess it would be OK, if $GDB could be set to "gdb --args".  Thanks.
>
> Could you do it? :)
> I confess I don't know quite where you want it to go...
>
>
>
>

Hi,

testing and debugging are quite different things.
IMO it's not a good idea mixing these.

What about making ert offer an appropriate listing of available tests?

Thanks,

Andreas

bug#15991: 24.3.50; Wishlist: Make test suite user-friendlier

[Eli Zaretskii]

> Date: Fri, 27 Jun 2014 07:45:57 +0200
> From: Andreas Röhler <andreas.roehler@easy-emacs.de>
> 
> >> I guess it would be OK, if $GDB could be set to "gdb --args".  Thanks.
> >
> > Could you do it? :)
> > I confess I don't know quite where you want it to go...
> 
> testing and debugging are quite different things.
> IMO it's not a good idea mixing these.

I'm not sure I understand what you are saying.  When a test fails,
don't you want to debug Emacs to understand why and find a solution?

So yes, testing and debugging are different things, but when a test
fails, one frequently needs to debug Emacs.

bug#15991: 24.3.50; Wishlist: Make test suite user-friendlier

[Michael Albinus]

Glenn Morris <rgm@gnu.org> writes:

> Glenn Morris wrote:
>
>>> ~/src/emacs> make -B -C test/automated tramp-tests.log
>>
>> Yes, the issue is that the log depends only on the test source file,
>> not on "all of Emacs".
>
> I mean, we could force it so that the tests are simply run every time,
> but I wasn't sure if that was preferable.

I believe it is desirable to run the tests unconditionally when invoked
via make. People call "make check" for a reason.

Maybe you could even offer the test package name as target. Then one
could call "make -C test/automated tramp-tests",

There is another issue. By redirecting the output to a file, the
password prompt for tramp-tests isn't visible anymore. This is a real
regression, because these tests are intended to run for different remote
file names. Could you pipe the test output through tee or something like
this? I don't know how portable this would be, 'tho.

Best regards, Michael.

bug#15991: 24.3.50; Wishlist: Make test suite user-friendlier

[Eli Zaretskii]

> From: Michael Albinus <michael.albinus@gmx.de>
> Date: Fri, 27 Jun 2014 09:23:10 +0200
> Cc: 15991@debbugs.gnu.org
> 
> Could you pipe the test output through tee or something like this? I
> don't know how portable this would be, 'tho.

It should be portable for every platform that supports running the
test suite, since that requires a shell that can handle this.

bug#15991: 24.3.50; Wishlist: Make test suite user-friendlier

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

>> Could you pipe the test output through tee or something like this? I
>> don't know how portable this would be, 'tho.
>
> It should be portable for every platform that supports running the
> test suite, since that requires a shell that can handle this.

Sure. But if there is a platform which does not support it, this
additional pipe could be suppressed on that platform. To be decided
during configure.

Best regards, Michael.

bug#15991: 24.3.50; Wishlist: Make test suite user-friendlier

[Andreas Röhler]

On 27.06.2014 08:00, Eli Zaretskii wrote:
>> Date: Fri, 27 Jun 2014 07:45:57 +0200
>> From: Andreas Röhler <andreas.roehler@easy-emacs.de>
>>
>>>> I guess it would be OK, if $GDB could be set to "gdb --args".  Thanks.
>>>
>>> Could you do it? :)
>>> I confess I don't know quite where you want it to go...
>>
>> testing and debugging are quite different things.
>> IMO it's not a good idea mixing these.
>
> I'm not sure I understand what you are saying.  When a test fails,
> don't you want to debug Emacs to understand why and find a solution?
>
> So yes, testing and debugging are different things, but when a test
> fails, one frequently needs to debug Emacs.
>
>

That depends. IIUC it's about making it more user friendly.
Perspective of users is different from core developers.

So if, say 10 from 100 tests fail, it doesn't mean to start a debug.
Maybe just taking notice and look, if it matters for current task.

We had that matter already WRT steep learning curve.
Keeping things apart if possible is important.

Best,

Andreas

bug#15991: 24.3.50; Wishlist: Make test suite user-friendlier

[Eli Zaretskii]

> Date: Fri, 27 Jun 2014 12:00:19 +0200
> From: Andreas Röhler <andreas.roehler@easy-emacs.de>
> CC: 15991@debbugs.gnu.org
> 
> > I'm not sure I understand what you are saying.  When a test fails,
> > don't you want to debug Emacs to understand why and find a solution?
> >
> > So yes, testing and debugging are different things, but when a test
> > fails, one frequently needs to debug Emacs.
> >
> >
> 
> That depends. IIUC it's about making it more user friendly.
> Perspective of users is different from core developers.
> 
> So if, say 10 from 100 tests fail, it doesn't mean to start a debug.
> Maybe just taking notice and look, if it matters for current task.

If someone just wants to take a look, they can do just that.  No one
is forcing them to say "make foo.log DEBUG=1" and run the foo test
under a debugger.  This is an optional feature, as should be clear
from reading the original bug report and this discussion.

> We had that matter already WRT steep learning curve.
> Keeping things apart if possible is important.

Not sure how is this related to the issue at hand.  Maybe you should
read the whole discussion.

bug#15991: 24.3.50; Wishlist: Make test suite user-friendlier

[Glenn Morris]

Michael Albinus wrote:

> I believe it is desirable to run the tests unconditionally when invoked
> via make. People call "make check" for a reason.

I suppose you are right, and this is consistent with how it used to behave.

> Maybe you could even offer the test package name as target. Then one
> could call "make -C test/automated tramp-tests",

You can already do

   make -C test/automated tramp-tests.log

I don't think removing the ".log" is worth it.

> There is another issue. By redirecting the output to a file, the
> password prompt for tramp-tests isn't visible anymore. This is a real
> regression, because these tests are intended to run for different remote
> file names. Could you pipe the test output through tee or something like
> this? I don't know how portable this would be, 'tho.

Not sure I understand. Do you want to interact, ie type a password?
I think tests that require interaction are a Bad Thing.
Are these optional tests that do not normally run?

bug#15991: 24.3.50; Wishlist: Make test suite user-friendlier

[Michael Albinus]

Glenn Morris <rgm@gnu.org> writes:

>> I believe it is desirable to run the tests unconditionally when invoked
>> via make. People call "make check" for a reason.
>
> I suppose you are right, and this is consistent with how it used to behave.

Nope. When you cal "make check" a second time, no test runs again,
'cause the log files exist already.

>> Maybe you could even offer the test package name as target. Then one
>> could call "make -C test/automated tramp-tests",
>
> You can already do
>
>    make -C test/automated tramp-tests.log

Again, it works only when there is no log file. The second time you
call it, nothing happens:

~/src/emacs> make -C test/automated tramp-tests.log
make: Entering directory `/usr/local/src/emacs/test/automated'
make: `tramp-tests.log' is up to date.
make: Leaving directory `/usr/local/src/emacs/test/automated'

> I don't think removing the ".log" is worth it.

If you want to rerun a test, you must remove the log file
manually. Inconvenient.

>> There is another issue. By redirecting the output to a file, the
>> password prompt for tramp-tests isn't visible anymore. This is a real
>> regression, because these tests are intended to run for different remote
>> file names. Could you pipe the test output through tee or something like
>> this? I don't know how portable this would be, 'tho.
>
> Not sure I understand. Do you want to interact, ie type a password?
> I think tests that require interaction are a Bad Thing.
> Are these optional tests that do not normally run?

Nope. See the commentary section of tramp-test.el. Try

~/src/emacs> env REMOTE_TEMPORARY_FILE_DIRECTORY=/sudo::/tmp make -C test/automated tramp-tests.log

It will ask you for the sudo password, but you don't see the prompt.

Best regards, Michael.

bug#15991: 24.3.50; Wishlist: Make test suite user-friendlier

[Michael Albinus]

Michael Albinus <michael.albinus@gmx.de> writes:

> Glenn Morris <rgm@gnu.org> writes:
>
>>> I believe it is desirable to run the tests unconditionally when invoked
>>> via make. People call "make check" for a reason.
>>
>> I suppose you are right, and this is consistent with how it used to behave.
>
> Nope. When you cal "make check" a second time, no test runs again,
> 'cause the log files exist already.

Forget this. I haven't seen your latest change. Thanks!

Best regards, Michael.

bug#15991: 24.3.50; Wishlist: Make test suite user-friendlier

[Glenn Morris]

Michael Albinus wrote:

>>    make -C test/automated tramp-tests.log
>
> Again, it works only when there is no log file.

OK, now you can do `make tramp-tests' to always run the tests.

> ~/src/emacs> env REMOTE_TEMPORARY_FILE_DIRECTORY=/sudo::/tmp make -C test/automated tramp-tests.log
>
> It will ask you for the sudo password, but you don't see the prompt.

So, you do want to run interactive tests.
Is it not more natural to run such tests from an interactive Emacs,
with M-x ert?
We could add the "tee" thing, I suppose.

bug#15991: 24.3.50; Wishlist: Make test suite user-friendlier

[Glenn Morris]

I changed it so that `make tramp-tests': i) always runs the tests, and
ii) does no logging at all (ie, writes to stdout).

Whereas `make tramp-tests.log' writes to the .log file, only if that is
out of date.

I think that makes sense?

Almost anything is possible of course, but I don't want to add too many
different options for running things, in case it all gets too confusing.

bug#15991: 24.3.50; Wishlist: Make test suite user-friendlier

[Glenn Morris]

Eli Zaretskii wrote:

>> All I can suggest here is that the Makefile define a GDB variable,
>> normally empty, that you can set to "gdb" on the make command-line when
>> you want to.
>
> I guess it would be OK, if $GDB could be set to "gdb --args".  Thanks.

I added $GDB at what seems like the right place.

I think this issue is done now.

bug#15991: 24.3.50; Wishlist: Make test suite user-friendlier

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Cc: 15991@debbugs.gnu.org
> Date: Sat, 28 Jun 2014 13:19:33 -0400
> 
> Eli Zaretskii wrote:
> 
> >> All I can suggest here is that the Makefile define a GDB variable,
> >> normally empty, that you can set to "gdb" on the make command-line when
> >> you want to.
> >
> > I guess it would be OK, if $GDB could be set to "gdb --args".  Thanks.
> 
> I added $GDB at what seems like the right place.
> 
> I think this issue is done now.

Yes, thanks again.

bug#15991: 24.3.50; Wishlist: Make test suite user-friendlier

[Michael Albinus]

Glenn Morris <rgm@gnu.org> writes:

> I changed it so that `make tramp-tests': i) always runs the tests, and
> ii) does no logging at all (ie, writes to stdout).
>
> Whereas `make tramp-tests.log' writes to the .log file, only if that is
> out of date.
>
> I think that makes sense?
>
> Almost anything is possible of course, but I don't want to add too many
> different options for running things, in case it all gets too confusing.

That works perfectly, thanks a lot!

Best regards, Michael.