bug#26337: 26.0.50; Command to run tests with latest source

bug#26337: 26.0.50; Command to run tests with latest source

[Tino Calancha]

Severity: wishlist

I often while debugging follow this workflow:

1) Make some changes in one branch.
2) Compile Emacs.
3) Run one test file with:
M-& emacs -batch -l ert -l ? -f ert-run-tests-batch-and-exit RET

I never remember the exact command in 3), so i always first check the manual.
We have `ert-run-tests-interactively', but that command won't use
the just compiled sources: you need first to reload the new .elc.

I just wrote one command to do 1-3 above.
Interactively prompts for the file with the tests.
With a prefix argument, also prompts for the selector.  Default to run
all the tests.

Do you think could be worth to have such a command in ert.el?
--8<-----------------------------cut here---------------start------------->8---
commit 14602c92c6e456000b6c2e649b68976f87cd2a4d
Author: Tino Calancha <tino.calancha@gmail.com>
Date:   Sun Apr 2 14:05:51 2017 +0900

    * lisp/emacs-lisp/ert.el (ert-run-tests-batch-in-file): New command.

diff --git a/lisp/emacs-lisp/ert.el b/lisp/emacs-lisp/ert.el
index e7387e463c..c80d8ee5ae 100644
--- a/lisp/emacs-lisp/ert.el
+++ b/lisp/emacs-lisp/ert.el
@@ -1449,6 +1449,41 @@ ert-run-tests-batch
                      (ert-test-name test)))))))
    nil))
 
+(declare-function dired-get-filename "dired" (&optional localp no-error-if-not-filep))
+;;;###autoload
+(defun ert-run-tests-batch-in-file (file &optional selector)
+  "Run the tests in FILE specified by SELECTOR.
+Interactively prompt for FILE.
+Called with a prefix arg prompt for SELECTOR."
+  (interactive
+   (let* ((prefix current-prefix-arg)
+          (def (and (derived-mode-p 'dired-mode) (dired-get-filename)))
+          (file (read-file-name "Run test: " nil def 'mustmatch nil
+                                (lambda (f) (equal "el" (file-name-extension f)))))
+          (test (if (not prefix) "t"
+                  (with-temp-buffer
+                    (insert-file-contents file)
+                    (let (all-tests)
+                      (while (re-search-forward "^\\s-*(ert-deftest \\([^[:blank:]]+\\)" nil t)
+                        (push (match-string-no-properties 1) all-tests))
+                      (unless all-tests
+                        (error "File '%s' doesn't contain any test" file))
+                      (completing-read "Select a test: " all-tests nil 'mustmatch nil nil "t"))))))
+     (list file test)))
+  (if (and (stringp selector) (not (string= selector "t")))
+      (setq selector (concat selector "\\\\'")) "t")
+  (let ((buf (get-buffer-create "*ert*"))
+        (program (expand-file-name invocation-name invocation-directory)))
+    (with-current-buffer buf
+      (let ((inhibit-read-only t))
+        (erase-buffer)
+        (call-process program nil (current-buffer) t
+                      "-batch" "-l" "ert" "-l" file "-eval"
+                      (format
+                       "(ert-run-tests-batch-and-exit\ \"%s\")"
+                       selector))))
+    (display-buffer buf)))
+
 ;;;###autoload
 (defun ert-run-tests-batch-and-exit (&optional selector)
   "Like `ert-run-tests-batch', but exits Emacs when done.
--8<-----------------------------cut here---------------end--------------->8---
In GNU Emacs 26.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.22.9)
 of 2017-04-02
Repository revision: a184a7edc58e1e053aa317a0f162df7e225597e1

bug#26337: 26.0.50; Command to run tests with latest source

[Glenn Morris]

Tino Calancha wrote:

> I often while debugging follow this workflow:
>
> 1) Make some changes in one branch.
> 2) Compile Emacs.
> 3) Run one test file with:
> M-& emacs -batch -l ert -l ? -f ert-run-tests-batch-and-exit RET

I use make for step 3. It seems natural after using it in step 2. Eg

cd test
make lisp/vc/ediff-ptch-tests.log

(till yesterday, it used to work without ".log" as well - see emacs-devel)

The test file names are completable using whatever your shell uses for
file name completion (eg alt-/ in bash). In emacs-25 they are completable
with bash TAB completion (from bash Makefile completion), which was nicer.

So personally I'm not sure I see the need to add an autoloaded ert
command to do this.

bug#26337: 26.0.50; Command to run tests with latest source

[Tino Calancha]

On Sun, 2 Apr 2017, Glenn Morris wrote:

> Tino Calancha wrote:
>
>> I often while debugging follow this workflow:
>>
>> 1) Make some changes in one branch.
>> 2) Compile Emacs.
>> 3) Run one test file with:
>> M-& emacs -batch -l ert -l ? -f ert-run-tests-batch-and-exit RET
> The test file names are completable using whatever your shell uses for
> file name completion (eg alt-/ in bash). In emacs-25 they are completable
> with bash TAB completion (from bash Makefile completion), which was nicer.
>
> So personally I'm not sure I see the need to add an autoloaded ert
> command to do this.
The autocomplete is not the main point of this addition.  The point is
let you run a test file (not necessarily from the official test suite),
in batch using the latest compiled source with easy.

> I use make for step 3. It seems natural after using it in step 2. Eg
>
> cd test
> make lisp/vc/ediff-ptch-tests.log
>
> (till yesterday, it used to work without ".log" as well - see emacs-devel)
You are right, that sounds pretty good.  Note, it doesn't work if the test 
you want to run it doesn't exit in the current branch or it's not as you
expect.. 
I use to write the test file in /tmp.  Then i can run the test after
compile the sources for whatever branch.

For example, last week i copied
test/lisp/emacs-lisp/cl-lib-tests.el
from commit 1f5b4ed628
into /tmp.  Then, i checkout several earlier commits to see when
cl-lib-symbol-macrolet test fail.  It started failing with 0d112c00ba,
but in that commit all tests in
/lisp/emacs-lisp/cl-lib-tests.el pass because
'cl-lib-symbol-macrolet' test was added later.  In that branch i must run
the file copied in /tmp.  In that case is handy to use the porposed 
command.

You can think of it as a manual bisect, but it's not exactly that.
Suppose now you are in a branch fixing this issue.  You change things,
compile and then you run the test in /tmp, which contains the 
'cl-lib-symbol-macrolet' test.

bug#26337: 26.0.50; Command to run tests with latest source

[Glenn Morris]

I still think it's a bit OOT to add something just to save typing:

emacs -batch -l sometests.el -f ert-run-tests-batch

Maybe you could define "ert-batch" as an alias for ert-run-tests-batch
(or ert-run-tests-batch-and-exit)? Then it really is very little typing.

bug#26337: 26.0.50; Command to run tests with latest source

[Tino Calancha]

On Mon, 3 Apr 2017, Glenn Morris wrote:

>
> I still think it's a bit OOT to add something just to save typing:
>
> emacs -batch -l sometests.el -f ert-run-tests-batch
>
> Maybe you could define "ert-batch" as an alias for ert-run-tests-batch
> (or ert-run-tests-batch-and-exit)? Then it really is very little typing.
Thanks, it sounds good idea.  I will follow it and close the report.