[patch] Add test-run capability to Makefile

[patch] Add test-run capability to Makefile

[Martyn Jago]

[-- Attachment #1: Type: text/plain, Size: 403 bytes --]


Since the Makefile is currently getting a refresh I thought it would be
very useful to allow running of the regression test suite directly from
make using `make test'.

The following patch achieves this and allows a user to run the
regression test suite on the their system following installation. This
makes it super-easy for a user to validate their latest org-mode
installation.

HTH

Best, Martyn


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: add-test-run-capability-to-makefile --]
[-- Type: text/x-patch, Size: 1296 bytes --]

From f025591e918d8921eaea0188da0b0da806c8b6f6 Mon Sep 17 00:00:00 2001
From: Martyn Jago <martyn.jago@btinternet.com>
Date: Wed, 4 Jan 2012 14:24:27 +0000
Subject: [PATCH] Add test command to Makefile
 * Makefile:

This patch enables `make test' to run the regression tests on the
user's system following installation. This makes it super-easy for a
user to validate their system.
---
 Makefile |    6 ++++++
 1 files changed, 6 insertions(+), 0 deletions(-)

diff --git a/Makefile b/Makefile
index 520b0fd..491d761 100644
--- a/Makefile
+++ b/Makefile
@@ -36,6 +36,9 @@ infodir = $(prefix)/share/info
 BATCH=$(EMACS) -batch -q -no-site-file -eval                             			\
   "(setq load-path (cons (expand-file-name \"./lisp/\") (cons \"$(lispdir)\" load-path)))" $(BATCH_EXTRA)
 
+BATCHTEST=$(EMACS) -Q -batch -l testing/org-test.el \
+  -eval "(setq org-confirm-babel-evaluate nil)" -f org-test-run-batch-tests
+
 # Specify the byte-compiler for compiling org-mode files
 ELC= $(BATCH) -f batch-byte-compile
 
@@ -496,6 +499,9 @@ pushreleasetag:
 	git tag -m "Adding release tag" -a release_$(TAG)
 	git push orgmode@orgmode.org:org-mode.git release_$(TAG)
 
+test:
+	$(BATCHTEST)
+ 
 # Dependencies
 
 lisp/org.elc:		lisp/org-macs.el lisp/org-compat.el lisp/org-faces.el
-- 
1.7.3.4

Re: [patch] Add test-run capability to Makefile

[Achim Gratz]

Martyn Jago <martyn.jago@btinternet.com> writes:
> Since the Makefile is currently getting a refresh I thought it would be
> very useful to allow running of the regression test suite directly from
> make using `make test'.

Good idea, I just added it to my Makefile fork as well.

> The following patch achieves this and allows a user to run the
> regression test suite on the their system following installation. This
> makes it super-easy for a user to validate their latest org-mode
> installation.

Four tests require htmlize, which is a non-standard package (and can't
be found on my system in batch mode unless I direct Emacs to load my
.emacs anyway).  It would be nice if the tests would expectedly fail if
htmlize can't be found, maybe with an additional message in the summary
that full coverage hasn't been reached due to missing libraries.


Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptation for Waldorf rackAttack V1.04R1:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

Re: [patch] Add test-run capability to Makefile

[Eric Schulte]

Achim Gratz <Stromeko@nexgo.de> writes:

> Martyn Jago <martyn.jago@btinternet.com> writes:
>> Since the Makefile is currently getting a refresh I thought it would be
>> very useful to allow running of the regression test suite directly from
>> make using `make test'.
>
> Good idea, I just added it to my Makefile fork as well.
>

Very cool, I anticipate using this feature with great frequency.

>
>> The following patch achieves this and allows a user to run the
>> regression test suite on the their system following installation. This
>> makes it super-easy for a user to validate their latest org-mode
>> installation.
>
> Four tests require htmlize, which is a non-standard package (and can't
> be found on my system in batch mode unless I direct Emacs to load my
> .emacs anyway).  It would be nice if the tests would expectedly fail if
> htmlize can't be found, maybe with an additional message in the summary
> that full coverage hasn't been reached due to missing libraries.
>

If the following simple macro is added to org-test.el,

#+begin_src emacs-lisp
  (defmacro depends-upon (feature deftest)
    (declare (indent 1))
    (if (featurep feature)
        `,deftest
      `(ert-deftest ,(cadr deftest) () :expected-result :failed (should nil))))
#+end_src

Then the offending tests can be wrapped in `depends-upon' forms, e.g.

(depends-upon htmlize
  (ert-deftest should-have-htmlize ()
    (should (featurep 'htmlize))))

If you have the names of the offending tests this should be a simple
fix.  I'm happy to apply this to the test suite, but I don't know which
four tests rely upon htmlize.

Cheers,

>
>
> Achim.

-- 
Eric Schulte
http://cs.unm.edu/~eschulte/

Re: [patch] Add test-run capability to Makefile

[Achim Gratz]

Eric Schulte <eric.schulte@gmx.com> writes:
> If you have the names of the offending tests this should be a simple
> fix.  I'm happy to apply this to the test suite, but I don't know which
> four tests rely upon htmlize.

Actually, it wasn't htmlize that made these four tests fail (only two of
them use it and while a warning is shown, the test itself passes).  What
was missing is that I need to define org-release since in batch mode I'm
not loading org-install.el.  This happens only in my fork since I
removed the version strings from org.el, I just corrected the default.mk
to take care of this.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptation for Waldorf Blofeld V1.15B11:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

Re: [patch] Add test-run capability to Makefile

[Bastien]

Martyn Jago <martyn.jago@btinternet.com> writes:

> Since the Makefile is currently getting a refresh I thought it would be
> very useful to allow running of the regression test suite directly from
> make using `make test'.

Nice -- I will test this through Achim's branch.

Thanks!

-- 
 Bastien

Re: [patch] Add test-run capability to Makefile

[Martyn Jago]

[-- Attachment #1: Type: text/plain, Size: 781 bytes --]

Bastien <bzg@altern.org> writes:

> Martyn Jago <martyn.jago@btinternet.com> writes:
>
>> Since the Makefile is currently getting a refresh I thought it would be
>> very useful to allow running of the regression test suite directly from
>> make using `make test'.
>
> Nice -- I will test this through Achim's branch.
>
> Thanks!

Thanks! 

However before you test it could you consider applying the following
patch. It is a fix to a relative character position offset change of 2
caused by the trailing whitespace cleanup (my bad for leaving dirty test
data)!

I may re-consider cleaning trailing whitespace in the post-save hook,
which is something I used to do, but which could generate a lot of
commit /noise/ where trailing whitespace removal isn't de-rigueur.

Best, Martyn 


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: fix-to-test-org-babel/org-babel-get-inline-src-block-matches --]
[-- Type: text/x-patch, Size: 1069 bytes --]

From 2327ec34fa8f29ea2aa03194a99ee535fefc3c94 Mon Sep 17 00:00:00 2001
From: Martyn Jago <martyn.jago@btinternet.com>
Date: Wed, 4 Jan 2012 21:09:46 +0000
Subject: [PATCH] Fix to test-org-babel/org-babel-get-inline-src-block-matches.
 * testing/lisp/test-ob.el:
 Character position offset -=2 to account for removal of trailing spaces

---
 testing/lisp/test-ob.el |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/testing/lisp/test-ob.el b/testing/lisp/test-ob.el
index 5f42dbb..dac6866 100644
--- a/testing/lisp/test-ob.el
+++ b/testing/lisp/test-ob.el
@@ -247,7 +247,7 @@ this is simple"
     (let ((test-point (point)))
       (should (fboundp 'org-babel-get-inline-src-block-matches))
       (should (re-search-forward "src_" nil t)) ;; 1
-      (should (= (+ test-point 140) (match-end 0)))
+      (should (= (+ test-point 138) (match-end 0)))
       (should (org-babel-get-inline-src-block-matches))
       (should (re-search-forward "}" nil (point-at-bol))) ;; 1
       (should-not (org-babel-get-inline-src-block-matches))
-- 
1.7.3.4

Re: [patch][babel] test fix - was: Add test-run capability to Makefile

[Martyn Jago]

[-- Attachment #1: Type: text/plain, Size: 459 bytes --]

Martyn Jago <martyn.jago@btinternet.com> writes:


[...]

>
> However before you test it could you consider applying the following
> patch. It is a fix to a relative character position offset change of 2
> caused by the trailing whitespace cleanup (my bad for leaving dirty test
> data)!
>

[...]

This patch seems to have been overlooked, and is still failing - my bad
for tagging it to the end of this thread.

Here it is again just in case.

Best, Martyn


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: inline-src-block-test-fix --]
[-- Type: text/x-patch, Size: 1069 bytes --]

From 2327ec34fa8f29ea2aa03194a99ee535fefc3c94 Mon Sep 17 00:00:00 2001
From: Martyn Jago <martyn.jago@btinternet.com>
Date: Wed, 4 Jan 2012 21:09:46 +0000
Subject: [PATCH] Fix to test-org-babel/org-babel-get-inline-src-block-matches.
 * testing/lisp/test-ob.el:
 Character position offset -=2 to account for removal of trailing spaces

---
 testing/lisp/test-ob.el |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/testing/lisp/test-ob.el b/testing/lisp/test-ob.el
index 5f42dbb..dac6866 100644
--- a/testing/lisp/test-ob.el
+++ b/testing/lisp/test-ob.el
@@ -247,7 +247,7 @@ this is simple"
     (let ((test-point (point)))
       (should (fboundp 'org-babel-get-inline-src-block-matches))
       (should (re-search-forward "src_" nil t)) ;; 1
-      (should (= (+ test-point 140) (match-end 0)))
+      (should (= (+ test-point 138) (match-end 0)))
       (should (org-babel-get-inline-src-block-matches))
       (should (re-search-forward "}" nil (point-at-bol))) ;; 1
       (should-not (org-babel-get-inline-src-block-matches))
-- 
1.7.3.4

Re: [patch][babel] test fix - was: Add test-run capability to Makefile

[Eric Schulte]

Martyn Jago <martyn.jago@btinternet.com> writes:

> Martyn Jago <martyn.jago@btinternet.com> writes:
>
>
> [...]
>
>>
>> However before you test it could you consider applying the following
>> patch. It is a fix to a relative character position offset change of 2
>> caused by the trailing whitespace cleanup (my bad for leaving dirty test
>> data)!
>>
>
> [...]
>
> This patch seems to have been overlooked, and is still failing - my bad
> for tagging it to the end of this thread.
>
> Here it is again just in case.
>

Applied, Thanks

>
> Best, Martyn
>
>

-- 
Eric Schulte
http://cs.unm.edu/~eschulte/