[Bug] Tests for experimental org-features should expect to fail if not activated by the user

[Bug] Tests for experimental org-features should expect to fail if not activated by the user

[Achim Gratz]

Tests for experimental org features (e.g. from contrib/ ) should expect
to fail when the user has not configured their inclusion into the
current setup.  In other words, things like "(require org-element)"
should not break the test run, but instead just note that this test has
failed expectedly and continue testing.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptations for Waldorf Q V3.00R3 and Q+ V3.54R2:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

Re: [Bug] Tests for experimental org-features should expect to fail if not activated by the user

[Eric Schulte]

Moving forward on this point, many of the existing tests explicitly
`require' new Org-mode functionality (mainly language support for
testing code blocks execution).  I do not think that tests should ever
be activating new packages changing a users global environment.

For this reason I have just pushed up a commit which changes all

  (require 'org-foo)

to

  (unless (featurep 'org-foo)
    (signal 'missing-test-dependency "Org support for doing foo."))

so that those tests simple aren't run on the users system.  Please let
me know if anyone thinks this is a mistake and we can discuss.  The only
drawback I see is that batch-mode scripts will have to explicitly
activate the features which they would like to test, by evaluating forms
like (require 'org-foo) before the call to the test suite -- and I would
argue that being explicit about such things is a benefit.

Cheers,

Achim Gratz <Stromeko@nexgo.de> writes:

> Tests for experimental org features (e.g. from contrib/ ) should expect
> to fail when the user has not configured their inclusion into the
> current setup.  In other words, things like "(require org-element)"
> should not break the test run, but instead just note that this test has
> failed expectedly and continue testing.
>
>
> Regards,
> Achim.

-- 
Eric Schulte
http://cs.unm.edu/~eschulte/

Re: [Bug] Tests for experimental org-features should expect to fail if not activated by the user

[Achim Gratz]

Eric Schulte <eric.schulte@gmx.com> writes:
> Moving forward on this point, many of the existing tests explicitly
> `require' new Org-mode functionality (mainly language support for
> testing code blocks execution).  I do not think that tests should ever
> be activating new packages changing a users global environment.

Thanks.

I've had to deactivate the inclusion of ./testing/lisp/contrib into the
test flow due to a recent removal of said directory by Nicolas (he's
probably doing that patch himself) and now get this:

lisp/org-mode> make test-dirty |& grep failed
   failed    5/109  ob-exp/export-from-a-temp-buffer
   failed   17/109  org-missing-dependency/test-ob-C
   failed   18/109  org-missing-dependency/test-ob-R
   failed   19/109  org-missing-dependency/test-ob-awk
   failed   20/109  org-missing-dependency/test-ob-fortran
   failed   21/109  org-missing-dependency/test-ob-lilypond
   failed   22/109  org-missing-dependency/test-ob-maxima
   failed   23/109  org-missing-dependency/test-ob-octave
   failed   24/109  org-missing-dependency/test-ob-python
   failed   25/109  org-missing-dependency/test-org-element
   failed   26/109  org-missing-dependency/test-org-export

This is expected, as my local configuration doesn't have any of these
packages activated.

lisp/org-mode> make test-dirty | & grep FAILED
   FAILED   27/109  test-ob-exp/org-babel-exp-src-blocks/w-no-file
   FAILED   62/109  test-org-babel/get-src-block-info-body
   FAILED   63/109  test-org-babel/get-src-block-info-language
   FAILED   64/109  test-org-babel/get-src-block-info-tangle

This is unexpected and I don't really see where they are coming from.
The backtrace always starts at (regexp-quote org-test-file-ob-anchor).

> For this reason I have just pushed up a commit which changes all
>
>   (require 'org-foo)
>
> to
>
>   (unless (featurep 'org-foo)
>     (signal 'missing-test-dependency "Org support for doing foo."))
>
> so that those tests simple aren't run on the users system.  Please let
> me know if anyone thinks this is a mistake and we can discuss.  The only
> drawback I see is that batch-mode scripts will have to explicitly
> activate the features which they would like to test, by evaluating forms
> like (require 'org-foo) before the call to the test suite -- and I would
> argue that being explicit about such things is a benefit.

Well, currently all testing runs under my user settings, which is not
ideal for testing anyway.  I suppose it would not be difficult to just
inject a different (minimal) startup file specifically for testing to
separate these issues.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptations for KORG EX-800 and Poly-800MkII V0.9:
http://Synth.Stromeko.net/Downloads.html#KorgSDada

Re: [Bug] Tests for experimental org-features should expect to fail if not activated by the user

[Eric Schulte]

Achim Gratz <Stromeko@nexgo.de> writes:

> Eric Schulte <eric.schulte@gmx.com> writes:
>> Moving forward on this point, many of the existing tests explicitly
>> `require' new Org-mode functionality (mainly language support for
>> testing code blocks execution).  I do not think that tests should ever
>> be activating new packages changing a users global environment.
>
> Thanks.
>
> I've had to deactivate the inclusion of ./testing/lisp/contrib into the
> test flow due to a recent removal of said directory by Nicolas (he's
> probably doing that patch himself) and now get this:
>

Yes,

I talked with Nicolas this morning, and at my urging he moved everything
from ./testing/lisp/contrib into ./testing/lisp and removed the contrib
directory.  I think moving forward we are best served by a single test
directory, especially given that we already have good methods of
conditionally loading only those tests appropriate for the users
environment.

>
> lisp/org-mode> make test-dirty |& grep failed
>    failed    5/109  ob-exp/export-from-a-temp-buffer
>    failed   17/109  org-missing-dependency/test-ob-C
>    failed   18/109  org-missing-dependency/test-ob-R
>    failed   19/109  org-missing-dependency/test-ob-awk
>    failed   20/109  org-missing-dependency/test-ob-fortran
>    failed   21/109  org-missing-dependency/test-ob-lilypond
>    failed   22/109  org-missing-dependency/test-ob-maxima
>    failed   23/109  org-missing-dependency/test-ob-octave
>    failed   24/109  org-missing-dependency/test-ob-python
>    failed   25/109  org-missing-dependency/test-org-element
>    failed   26/109  org-missing-dependency/test-org-export
>
> This is expected, as my local configuration doesn't have any of these
> packages activated.
>

Yes, I assume that the lower case "failed" means expected failure.

>
> lisp/org-mode> make test-dirty | & grep FAILED
>    FAILED   27/109  test-ob-exp/org-babel-exp-src-blocks/w-no-file
>    FAILED   62/109  test-org-babel/get-src-block-info-body
>    FAILED   63/109  test-org-babel/get-src-block-info-language
>    FAILED   64/109  test-org-babel/get-src-block-info-tangle
>
> This is unexpected and I don't really see where they are coming from.
> The backtrace always starts at (regexp-quote org-test-file-ob-anchor).
>

My guess is that when I removed require statements for org-html and
org-ascii from from test-org-html and test-org-exp, they removed some
functionality not loaded by default in the test scripts.  These should
probably be added into the Makefile.

>
>> For this reason I have just pushed up a commit which changes all
>>
>>   (require 'org-foo)
>>
>> to
>>
>>   (unless (featurep 'org-foo)
>>     (signal 'missing-test-dependency "Org support for doing foo."))
>>
>> so that those tests simple aren't run on the users system.  Please let
>> me know if anyone thinks this is a mistake and we can discuss.  The only
>> drawback I see is that batch-mode scripts will have to explicitly
>> activate the features which they would like to test, by evaluating forms
>> like (require 'org-foo) before the call to the test suite -- and I would
>> argue that being explicit about such things is a benefit.
>
> Well, currently all testing runs under my user settings, which is not
> ideal for testing anyway.  I suppose it would not be difficult to just
> inject a different (minimal) startup file specifically for testing to
> separate these issues.
>

Yea, I also run most tests from my normal Emacs session, which is part
of the motivation for removing these embedded require statements.

Cheers,

>
>
> Regards,
> Achim.

-- 
Eric Schulte
http://cs.unm.edu/~eschulte/

Re: [Bug] Tests for experimental org-features should expect to fail if not activated by the user

[Achim Gratz]

Eric Schulte <eric.schulte@gmx.com> writes:
> My guess is that when I removed require statements for org-html and
> org-ascii from from test-org-html and test-org-exp, they removed some
> functionality not loaded by default in the test scripts.  These should
> probably be added into the Makefile.

Unless someone can give me a hint, I'll have a look at this
later... it'll probably become weekend before I get to this.

> Yea, I also run most tests from my normal Emacs session, which is part
> of the motivation for removing these embedded require statements.

I'll see that I create a minimal config and use that just for testing
from the Makefile (in my Makefile fork of course).  If somebody has a
good template to work from, please post, otherwise I should be able to
scrape it together from the parts that have just been removed.  BTW,
where should that startup file go, directly under testing?


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Samples for the Waldorf Blofeld:
http://Synth.Stromeko.net/Downloads.html#BlofeldSamplesExtra

Re: [Bug] Tests for experimental org-features should expect to fail if not activated by the user

[Achim Gratz]

Eric Schulte <eric.schulte@gmx.com> writes:
[...]

Thanks for your latest round of patches.  It all works now when adding
"testing/" to the load path.  I'll add this to the default configuration
in my Makefile fork.  There's another patch by Nicolas that added a few
test, one of which seems broken:

OVERVIEW
Mark set
Test test-org-export/fuzzy-links backtrace:
  signal(error ("No link found"))
  error("No link found")
  org-open-at-point()
  (prog1 (goto-line 4) (org-open-at-point) (should (looking-at "<<Test
  (progn (org-mode) (progn (insert "#+TARGET: Test\n#+NAME: Test\n|a|b
  (unwind-protect (progn (org-mode) (progn (insert "#+TARGET: Test\n#+
  (save-current-buffer (set-buffer temp-buffer) (unwind-protect (progn
  (with-current-buffer temp-buffer (unwind-protect (progn (org-mode) (
  (let ((temp-buffer (generate-new-buffer " *temp*"))) (with-current-b
  (with-temp-buffer (org-mode) (progn (insert "#+TARGET: Test\n#+NAME:
  (org-test-with-temp-text "#+TARGET: Test\n#+NAME: Test\n|a|b|\n<<Tes
  (lambda nil (org-test-with-temp-text "#+TARGET: Test\n#+NAME: Test\n
  byte-code("\306\307!▒q\210\310\216\311 \312\216\313\314\315\316\3
  ert--run-test-internal([cl-struct-ert--test-execution-info [cl-struc
  byte-code("\306\307!\211▒r\310\311!q\210\312 d\313\223)L\210\314\216
  ert-run-test([cl-struct-ert-test test-org-export/fuzzy-links "Test f
  ert-run-or-rerun-test([cl-struct-ert--stats "\\(org\\|ob\\)" [[cl-st
  ert-run-tests("\\(org\\|ob\\)" #[(event-type &rest event-args) \30
  ert-run-tests-batch("\\(org\\|ob\\)")
  ert-run-tests-batch-and-exit("\\(org\\|ob\\)")
  (let ((org-id-track-globally t) (org-id-locations-file (convert-stan
  org-test-run-batch-tests()
  call-interactively(org-test-run-batch-tests nil nil)
  command-execute(org-test-run-batch-tests)
  command-line-1(("-L" "testing/" "-L" "lisp/" "--eval" "(defconst org
  command-line()
  normal-top-level()
Test test-org-export/fuzzy-links condition:
    (error "No link found")
   FAILED   87/112  test-org-export/fuzzy-links




Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Factory and User Sound Singles for Waldorf rackAttack:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds

Re: [Bug] Tests for experimental org-features should expect to fail if not activated by the user

[Nicolas Goaziou]

Hello,

Achim Gratz <Stromeko@nexgo.de> writes:

> There's another patch by Nicolas that added a few test, one of which
> seems broken:

This should be fixed now. Thank you for reporting it.


Regards,

-- 
Nicolas Goaziou

Re: [Bug] Tests for experimental org-features should expect to fail if not activated by the user

[Achim Gratz]

Nicolas Goaziou <n.goaziou@gmail.com> writes:
> This should be fixed now. Thank you for reporting it.

I confirm the fix, testing is clean again, thank you very much.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptation for Waldorf microQ V2.22R2:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

Re: [Bug] Tests for experimental org-features should expect to fail if not activated by the user

[Achim Gratz]

Achim Gratz <Stromeko@nexgo.de> writes:
>  It all works now when adding "testing/" to the load path.  I'll add
> this to the default configuration in my Makefile fork.

I've improved the test invocation in my Makefile fork.  It now starts
from a clean Emacs invocation withoutsite startup and loads the
configured Babel languages and possibly extra packages before running
the tests.  All Babel languages are activated by default except R, since
R seems to depend on ess (i.e. even if you add it to the list, the tests
still fail due to that missing dependency).  Extra packages, for
instance from contrib, can be required by adding their names to
BTEST_EXTRA in local.mk and possibly prepending or appending load-path
elements or other command line options via BTEST_PRE and BTEST_POST.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Factory and User Sound Singles for Waldorf Blofeld:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds

Re: [Bug] Tests for experimental org-features should expect to fail if not activated by the user

[Achim Gratz]

Two test failures in current HEAD:

--8<---------------cut here---------------start------------->8---
Test test-org-footnote/normalize-outside-org backtrace:
  org-footnote-in-valid-context-p()
  org-footnote-at-reference-p()
  byte-code(\20\304\202 \305    \n\306#\204\307\310\311\"\21\20
  org-footnote-get-next-reference()
  org-footnote-normalize()
  (let ((major-mode (quote message-mode))) (org-footnote-normalize))
  (progn (insert "Body[fn::def]\n-- \nFake signature\n-- \nSignature")
  (unwind-protect (progn (insert "Body[fn::def]\n-- \nFake signature\n
  (save-current-buffer (set-buffer temp-buffer) (unwind-protect (progn
  (with-current-buffer temp-buffer (unwind-protect (progn (insert "Bod
  (let ((temp-buffer (generate-new-buffer " *temp*"))) (with-current-b
  (with-temp-buffer (insert "Body[fn::def]\n-- \nFake signature\n-- \n
  (let ((org-footnote-tag-for-non-org-mode-files nil) (message-signatu
  (lambda nil (let ((org-footnote-tag-for-non-org-mode-files nil)) (wi
  byte-code("\306\307!▒q\210\310\216\311 \312\216\313\314\315\316\3
  ert--run-test-internal([cl-struct-ert--test-execution-info [cl-struc
  byte-code("\306\307!\211▒r\310\311!q\210\312 d\313\223)L\210\314\216
  ert-run-test([cl-struct-ert-test test-org-footnote/normalize-outside
  ert-run-or-rerun-test([cl-struct-ert--stats "\\(org\\|ob\\)" [[cl-st
  ert-run-tests("\\(org\\|ob\\)" #[(event-type &rest event-args) \30
  ert-run-tests-batch("\\(org\\|ob\\)")
  ert-run-tests-batch-and-exit("\\(org\\|ob\\)")
  (let ((org-id-track-globally t) (org-id-locations-file (convert-stan
  org-test-run-batch-tests()
  call-interactively(org-test-run-batch-tests nil nil)
  command-execute(org-test-run-batch-tests)
  command-line-1(("-L" "lisp/" "-L" "testing/" "--eval" "(defconst org
  command-line()
  normal-top-level()
Test test-org-footnote/normalize-outside-org condition:
    (void-variable message-cite-prefix-regexp)
   FAILED  166/192  test-org-footnote/normalize-outside-org
--8<---------------cut here---------------end--------------->8---

--8<---------------cut here---------------start------------->8---
Test test-org-export/export-scope backtrace:
  signal(ert-test-failed (((should (equal (org-export-as (quote test))
  ert-fail(((should (equal (org-export-as (quote test)) "text\n")) :fo
  (if (unwind-protect (setq value-2005 (apply fn-2003 args-2004)) (set
  (unless (unwind-protect (setq value-2005 (apply fn-2003 args-2004)) 
  (let (form-description-2007) (unless (unwind-protect (setq value-200
  (let ((value-2005 (quote ert-form-evaluation-aborted-2006))) (let (f
  (let ((fn-2003 (function equal)) (args-2004 (list (org-export-as (qu
  (should (equal (org-export-as (quote test)) "text\n"))
  (progn (fset (quote org-test-center-block) (function* (lambda (obj c
  (unwind-protect (progn (fset (quote org-test-center-block) (function
  (let* ((--cl-letf-bound-- (fboundp (quote org-test-center-block))) (
  (letf (((symbol-function (quote org-test-center-block)) (function* (
  (progn (fset (quote org-test-comment) (function* (lambda (obj conten
  (unwind-protect (progn (fset (quote org-test-comment) (function* (la
  (let* ((--cl-letf-bound-- (fboundp (quote org-test-comment))) (--cl-
  (letf (((symbol-function (quote org-test-comment)) (function* (lambd
  (progn (fset (quote org-test-comment-block) (function* (lambda (obj 
  (unwind-protect (progn (fset (quote org-test-comment-block) (functio
  (let* ((--cl-letf-bound-- (fboundp (quote org-test-comment-block))) 
  (letf (((symbol-function (quote org-test-comment-block)) (function* 
  (progn (fset (quote org-test-drawer) (function* (lambda (obj content
  (unwind-protect (progn (fset (quote org-test-drawer) (function* (lam
  (let* ((--cl-letf-bound-- (fboundp (quote org-test-drawer))) (--cl-l
  (letf (((symbol-function (quote org-test-drawer)) (function* (lambda
  (progn (fset (quote org-test-dynamic-block) (function* (lambda (obj 
  (unwind-protect (progn (fset (quote org-test-dynamic-block) (functio
  (let* ((--cl-letf-bound-- (fboundp (quote org-test-dynamic-block))) 
  (letf (((symbol-function (quote org-test-dynamic-block)) (function* 
  (progn (fset (quote org-test-example-block) (function* (lambda (obj 
  (unwind-protect (progn (fset (quote org-test-example-block) (functio
  (let* ((--cl-letf-bound-- (fboundp (quote org-test-example-block))) 
  (letf (((symbol-function (quote org-test-example-block)) (function* 
  (progn (fset (quote org-test-export-block) (function* (lambda (obj c
  (unwind-protect (progn (fset (quote org-test-export-block) (function
  (let* ((--cl-letf-bound-- (fboundp (quote org-test-export-block))) (
  (letf (((symbol-function (quote org-test-export-block)) (function* (
  (progn (fset (quote org-test-fixed-width) (function* (lambda (obj co
  (unwind-protect (progn (fset (quote org-test-fixed-width) (function*
  (let* ((--cl-letf-bound-- (fboundp (quote org-test-fixed-width))) (-
  (letf (((symbol-function (quote org-test-fixed-width)) (function* (l
  (progn (fset (quote org-test-footnote-definition) (function* (lambda
  (unwind-protect (progn (fset (quote org-test-footnote-definition) (f
  (let* ((--cl-letf-bound-- (fboundp (quote org-test-footnote-definiti
  (letf (((symbol-function (quote org-test-footnote-definition)) (func
  (progn (fset (quote org-test-headline) (function* (lambda (obj conte
  (unwind-protect (progn (fset (quote org-test-headline) (function* (l
  (let* ((--cl-letf-bound-- (fboundp (quote org-test-headline))) (--cl
  (letf (((symbol-function (quote org-test-headline)) (function* (lamb
  (progn (fset (quote org-test-horizontal-rule) (function* (lambda (ob
  (unwind-protect (progn (fset (quote org-test-horizontal-rule) (funct
  (let* ((--cl-letf-bound-- (fboundp (quote org-test-horizontal-rule))
  (letf (((symbol-function (quote org-test-horizontal-rule)) (function
  (progn (fset (quote org-test-inlinetask) (function* (lambda (obj con
  (unwind-protect (progn (fset (quote org-test-inlinetask) (function* 
  (let* ((--cl-letf-bound-- (fboundp (quote org-test-inlinetask))) (--
  (letf (((symbol-function (quote org-test-inlinetask)) (function* (la
  (progn (fset (quote org-test-item) (function* (lambda (obj contents 
  (unwind-protect (progn (fset (quote org-test-item) (function* (lambd
  (let* ((--cl-letf-bound-- (fboundp (quote org-test-item))) (--cl-let
  (letf (((symbol-function (quote org-test-item)) (function* (lambda (
  (progn (fset (quote org-test-keyword) (function* (lambda (obj conten
  (unwind-protect (progn (fset (quote org-test-keyword) (function* (la
  (let* ((--cl-letf-bound-- (fboundp (quote org-test-keyword))) (--cl-
  (letf (((symbol-function (quote org-test-keyword)) (function* (lambd
  (progn (fset (quote org-test-latex-environment) (function* (lambda (
  (unwind-protect (progn (fset (quote org-test-latex-environment) (fun
  (let* ((--cl-letf-bound-- (fboundp (quote org-test-latex-environment
  (letf (((symbol-function (quote org-test-latex-environment)) (functi
  (progn (fset (quote org-test-babel-call) (function* (lambda (obj con
  (unwind-protect (progn (fset (quote org-test-babel-call) (function* 
  (let* ((--cl-letf-bound-- (fboundp (quote org-test-babel-call))) (--
  (letf (((symbol-function (quote org-test-babel-call)) (function* (la
  (progn (fset (quote org-test-paragraph) (function* (lambda (obj cont
  (unwind-protect (progn (fset (quote org-test-paragraph) (function* (
  (let* ((--cl-letf-bound-- (fboundp (quote org-test-paragraph))) (--c
  (letf (((symbol-function (quote org-test-paragraph)) (function* (lam
  (progn (fset (quote org-test-plain-list) (function* (lambda (obj con
  (unwind-protect (progn (fset (quote org-test-plain-list) (function* 
  (let* ((--cl-letf-bound-- (fboundp (quote org-test-plain-list))) (--
  (letf (((symbol-function (quote org-test-plain-list)) (function* (la
  (progn (fset (quote org-test-property-drawer) (function* (lambda (ob
  (unwind-protect (progn (fset (quote org-test-property-drawer) (funct
  (let* ((--cl-letf-bound-- (fboundp (quote org-test-property-drawer))
  (letf (((symbol-function (quote org-test-property-drawer)) (function
  (progn (fset (quote org-test-quote-block) (function* (lambda (obj co
  (unwind-protect (progn (fset (quote org-test-quote-block) (function*
  (let* ((--cl-letf-bound-- (fboundp (quote org-test-quote-block))) (-
  (letf (((symbol-function (quote org-test-quote-block)) (function* (l
  (progn (fset (quote org-test-quote-section) (function* (lambda (obj 
  (unwind-protect (progn (fset (quote org-test-quote-section) (functio
  (let* ((--cl-letf-bound-- (fboundp (quote org-test-quote-section))) 
  (letf (((symbol-function (quote org-test-quote-section)) (function* 
  (progn (fset (quote org-test-section) (function* (lambda (obj conten
  (unwind-protect (progn (fset (quote org-test-section) (function* (la
  (let* ((--cl-letf-bound-- (fboundp (quote org-test-section))) (--cl-
  (letf (((symbol-function (quote org-test-section)) (function* (lambd
  (progn (fset (quote org-test-special-block) (function* (lambda (obj 
  (unwind-protect (progn (fset (quote org-test-special-block) (functio
  (let* ((--cl-letf-bound-- (fboundp (quote org-test-special-block))) 
  (letf (((symbol-function (quote org-test-special-block)) (function* 
  (progn (fset (quote org-test-src-block) (function* (lambda (obj cont
  (unwind-protect (progn (fset (quote org-test-src-block) (function* (
  (let* ((--cl-letf-bound-- (fboundp (quote org-test-src-block))) (--c
  (letf (((symbol-function (quote org-test-src-block)) (function* (lam
  (progn (fset (quote org-test-table) (function* (lambda (obj contents
  (unwind-protect (progn (fset (quote org-test-table) (function* (lamb
  (let* ((--cl-letf-bound-- (fboundp (quote org-test-table))) (--cl-le
  (letf (((symbol-function (quote org-test-table)) (function* (lambda 
  (progn (fset (quote org-test-verse-block) (function* (lambda (obj co
  (unwind-protect (progn (fset (quote org-test-verse-block) (function*
  (let* ((--cl-letf-bound-- (fboundp (quote org-test-verse-block))) (-
  (letf (((symbol-function (quote org-test-verse-block)) (function* (l
  (progn (fset (quote org-test-emphasis) (function* (lambda (obj conte
  (unwind-protect (progn (fset (quote org-test-emphasis) (function* (l
  (let* ((--cl-letf-bound-- (fboundp (quote org-test-emphasis))) (--cl
  (letf (((symbol-function (quote org-test-emphasis)) (function* (lamb
  (progn (fset (quote org-test-entity) (function* (lambda (obj content
  (unwind-protect (progn (fset (quote org-test-entity) (function* (lam
  (let* ((--cl-letf-bound-- (fboundp (quote org-test-entity))) (--cl-l
  (letf (((symbol-function (quote org-test-entity)) (function* (lambda
  (progn (fset (quote org-test-export-snippet) (function* (lambda (obj
  (unwind-protect (progn (fset (quote org-test-export-snippet) (functi
  (let* ((--cl-letf-bound-- (fboundp (quote org-test-export-snippet)))
  (letf (((symbol-function (quote org-test-export-snippet)) (function*
  (progn (fset (quote org-test-footnote-reference) (function* (lambda 
  (unwind-protect (progn (fset (quote org-test-footnote-reference) (fu
  (let* ((--cl-letf-bound-- (fboundp (quote org-test-footnote-referenc
  (letf (((symbol-function (quote org-test-footnote-reference)) (funct
  (progn (fset (quote org-test-inline-babel-call) (function* (lambda (
  (unwind-protect (progn (fset (quote org-test-inline-babel-call) (fun
  (let* ((--cl-letf-bound-- (fboundp (quote org-test-inline-babel-call
  (letf (((symbol-function (quote org-test-inline-babel-call)) (functi
  (progn (fset (quote org-test-inline-src-block) (function* (lambda (o
  (unwind-protect (progn (fset (quote org-test-inline-src-block) (func
  (let* ((--cl-letf-bound-- (fboundp (quote org-test-inline-src-block)
  (letf (((symbol-function (quote org-test-inline-src-block)) (functio
  (progn (fset (quote org-test-line-break) (function* (lambda (obj con
  (unwind-protect (progn (fset (quote org-test-line-break) (function* 
  (let* ((--cl-letf-bound-- (fboundp (quote org-test-line-break))) (--
  (letf (((symbol-function (quote org-test-line-break)) (function* (la
  (progn (fset (quote org-test-latex-fragment) (function* (lambda (obj
  (unwind-protect (progn (fset (quote org-test-latex-fragment) (functi
  (let* ((--cl-letf-bound-- (fboundp (quote org-test-latex-fragment)))
  (letf (((symbol-function (quote org-test-latex-fragment)) (function*
  (progn (fset (quote org-test-link) (function* (lambda (obj contents 
  (unwind-protect (progn (fset (quote org-test-link) (function* (lambd
  (let* ((--cl-letf-bound-- (fboundp (quote org-test-link))) (--cl-let
  (letf (((symbol-function (quote org-test-link)) (function* (lambda (
  (progn (fset (quote org-test-macro) (function* (lambda (obj contents
  (unwind-protect (progn (fset (quote org-test-macro) (function* (lamb
  (let* ((--cl-letf-bound-- (fboundp (quote org-test-macro))) (--cl-le
  (letf (((symbol-function (quote org-test-macro)) (function* (lambda 
  (progn (fset (quote org-test-radio-target) (function* (lambda (obj c
  (unwind-protect (progn (fset (quote org-test-radio-target) (function
  (let* ((--cl-letf-bound-- (fboundp (quote org-test-radio-target))) (
  (letf (((symbol-function (quote org-test-radio-target)) (function* (
  (progn (fset (quote org-test-statistics-cookie) (function* (lambda (
  (unwind-protect (progn (fset (quote org-test-statistics-cookie) (fun
  (let* ((--cl-letf-bound-- (fboundp (quote org-test-statistics-cookie
  (letf (((symbol-function (quote org-test-statistics-cookie)) (functi
  (progn (fset (quote org-test-subscript) (function* (lambda (obj cont
  (unwind-protect (progn (fset (quote org-test-subscript) (function* (
  (let* ((--cl-letf-bound-- (fboundp (quote org-test-subscript))) (--c
  (letf (((symbol-function (quote org-test-subscript)) (function* (lam
  (progn (fset (quote org-test-superscript) (function* (lambda (obj co
  (unwind-protect (progn (fset (quote org-test-superscript) (function*
  (let* ((--cl-letf-bound-- (fboundp (quote org-test-superscript))) (-
  (letf (((symbol-function (quote org-test-superscript)) (function* (l
  (progn (fset (quote org-test-target) (function* (lambda (obj content
  (unwind-protect (progn (fset (quote org-test-target) (function* (lam
  (let* ((--cl-letf-bound-- (fboundp (quote org-test-target))) (--cl-l
  (letf (((symbol-function (quote org-test-target)) (function* (lambda
  (progn (fset (quote org-test-time-stamp) (function* (lambda (obj con
  (unwind-protect (progn (fset (quote org-test-time-stamp) (function* 
  (let* ((--cl-letf-bound-- (fboundp (quote org-test-time-stamp))) (--
  (letf (((symbol-function (quote org-test-time-stamp)) (function* (la
  (progn (fset (quote org-test-verbatim) (function* (lambda (obj conte
  (unwind-protect (progn (fset (quote org-test-verbatim) (function* (l
  (let* ((--cl-letf-bound-- (fboundp (quote org-test-verbatim))) (--cl
  (letf (((symbol-function (quote org-test-verbatim)) (function* (lamb
  (letf* (((symbol-function (quote org-test-verbatim)) (function* (lam
  (flet ((org-test-verbatim (obj contents info) (org-element-verbatim-
  (org-test-with-backend "test" (forward-line 3) (should (equal (org-e
  (prog1 (org-test-with-backend "test" (forward-line 3) (should (equal
  (progn (org-mode) (progn (insert "\n* Head1\n** Head2\ntext\n*** Hea
  (unwind-protect (progn (org-mode) (progn (insert "\n* Head1\n** Head
  (save-current-buffer (set-buffer temp-buffer) (unwind-protect (progn
  (with-current-buffer temp-buffer (unwind-protect (progn (org-mode) (
  (let ((temp-buffer (generate-new-buffer " *temp*"))) (with-current-b
  (with-temp-buffer (org-mode) (progn (insert "\n* Head1\n** Head2\nte
  (org-test-with-temp-text "\n* Head1\n** Head2\ntext\n*** Head3" (org
  (lambda nil (org-test-with-temp-text "\n* Head1\n** Head2\ntext\n***
  byte-code("\306\307!▒q\210\310\216\311 \312\216\313\314\315\316\3
  ert--run-test-internal([cl-struct-ert--test-execution-info [cl-struc
  byte-code("\306\307!\211▒r\310\311!q\210\312 d\313\223)L\210\314\216
  ert-run-test([cl-struct-ert-test test-org-export/export-scope "Test 
  ert-run-or-rerun-test([cl-struct-ert--stats "\\(org\\|ob\\)" [[cl-st
  ert-run-tests("\\(org\\|ob\\)" #[(event-type &rest event-args) \30
  ert-run-tests-batch("\\(org\\|ob\\)")
  ert-run-tests-batch-and-exit("\\(org\\|ob\\)")
  (let ((org-id-track-globally t) (org-id-locations-file (convert-stan
  org-test-run-batch-tests()
  call-interactively(org-test-run-batch-tests nil nil)
  command-execute(org-test-run-batch-tests)
  command-line-1(("-L" "lisp/" "-L" "testing/" "-L" "contrib/lisp" "--
  command-line()
  normal-top-level()
Test test-org-export/export-scope condition:
    (ert-test-failed
     ((should
       (equal
        (org-export-as ...)
        "text\n"))
      :form
      (equal "* Head1\n** Head2\ntext\n*** Head3\n" "text\n")
      :value nil :explanation
      (arrays-of-different-length 32 5 "* Head1\n** Head2\ntext\n*** Head3\n" "text\n" first-mismatch-at 0)))
   FAILED   95/130  test-org-export/export-scope
--8<---------------cut here---------------end--------------->8---


Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Factory and User Sound Singles for Waldorf Q+, Q and microQ:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds

Re: [Bug] Tests for experimental org-features should expect to fail if not activated by the user

[Nicolas Goaziou]

Hello,

Achim Gratz <Stromeko@nexgo.de> writes:

> Two test failures in current HEAD:

I cannot reproduce any of them, interactively or in batch mode[1].

I'm not sure where they could come from.


Regards,

[1] Though, I have 8 unexpected failures there, but not related to
those described here.

-- 
Nicolas Goaziou

Re: [Bug] Tests for experimental org-features should expect to fail if not activated by the user

[Sebastien Vauban]

Hello Achim,

Achim Gratz wrote:
> Two test failures in current HEAD:
>
> Test test-org-footnote/normalize-outside-org backtrace:
>   org-footnote-in-valid-context-p()
>   [...]
>   normal-top-level()
> Test test-org-footnote/normalize-outside-org condition:
>     (void-variable message-cite-prefix-regexp)
>    FAILED  166/192  test-org-footnote/normalize-outside-org
>
> Test test-org-export/export-scope backtrace:
>   signal(ert-test-failed (((should (equal (org-export-as (quote test))
>   [...]
>   normal-top-level()
> Test test-org-export/export-scope condition:
>     (ert-test-failed
>      ((should
>        (equal
>         (org-export-as ...)
>         "text\n"))
>       :form
>       (equal "* Head1\n** Head2\ntext\n*** Head3\n" "text\n")
>       :value nil :explanation
>       (arrays-of-different-length 32 5 "* Head1\n** Head2\ntext\n*** Head3\n" "text\n" first-mismatch-at 0)))
>    FAILED   95/130  test-org-export/export-scope

I don't understand why once there are 192 tests, once 130. I thought that the
second figure was the total number of tests, hence should be stable over the
test runs?

Best regards,
  Seb

-- 
Sebastien Vauban

Re: [Bug] Tests for experimental org-features should expect to fail if not activated by the user

[Achim Gratz]

"Sebastien Vauban"
<wxhgmqzgwmuf@spammotel.com> writes:
> I don't understand why once there are 192 tests, once 130. I thought that the
> second figure was the total number of tests, hence should be stable over the
> test runs?

The number of available tests depend on the Babel languages that are
activated.  One of those failures was with a test run that had all
languages (except R) activated, the other one with no languages (except
emacs-lisp, which is always active), but org-element and org-export made
active.  Those two also seem to pull in ob-sh and maybe other Babel
languages, I would have to check.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

DIY Stuff:
http://Synth.Stromeko.net/DIY.html

Re: [Bug] Tests for experimental org-features should expect to fail if not activated by the user

[Achim Gratz]

Nicolas Goaziou <n.goaziou@gmail.com> writes:
> I cannot reproduce any of them, interactively or in batch mode[1].

This is the invocation (to be run in bash, like make would do):

TMPDIR=/tmp/tmp-orgtest /usr/local/bin/emacs -batch -Q \
-L lisp/ -L testing/ -L contrib/lisp \
--eval '(defconst org-release "7.8.03-Test")' -l testing/org-test.el \
--eval '(require '"'"'org-element)' --eval '(require '"'"'org-export)' \
--eval '(setq org-confirm-babel-evaluate nil)' -f org-test-run-batch-tests

[...]
2 unexpected results:
   FAILED  test-org-export/export-scope
   FAILED  test-org-footnote/normalize-outside-org

> I'm not sure where they could come from.

Since I run this with "-Q", I suspect that you implicitly rely on some
customization someplace.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Wavetables for the Waldorf Blofeld:
http://Synth.Stromeko.net/Downloads.html#BlofeldUserWavetables

Re: [Bug] Tests for experimental org-features should expect to fail if not activated by the user

[Nicolas Goaziou]

Hello,

Achim Gratz <Stromeko@nexgo.de> writes:

> Nicolas Goaziou <n.goaziou@gmail.com> writes:
>> I cannot reproduce any of them, interactively or in batch mode[1].
>
> This is the invocation (to be run in bash, like make would do):
>
> TMPDIR=/tmp/tmp-orgtest /usr/local/bin/emacs -batch -Q \
> -L lisp/ -L testing/ -L contrib/lisp \
> --eval '(defconst org-release "7.8.03-Test")' -l testing/org-test.el \
> --eval '(require '"'"'org-element)' --eval '(require '"'"'org-export)' \
> --eval '(setq org-confirm-babel-evaluate nil)' -f org-test-run-batch-tests
>
> [...]
> 2 unexpected results:
>    FAILED  test-org-export/export-scope
>    FAILED  test-org-footnote/normalize-outside-org

With your command, I was able to reproduce the first one (along with 24
unexpected results), for which I have pushed a fix. I still don't see
the second one.


Regards,

-- 
Nicolas Goaziou

Re: [Bug] Tests for experimental org-features should expect to fail if not activated by the user

[Achim Gratz]

Nicolas Goaziou <n.goaziou@gmail.com> writes:
>> [...]
>> 2 unexpected results:
>>    FAILED  test-org-export/export-scope
>>    FAILED  test-org-footnote/normalize-outside-org
>
> With your command, I was able to reproduce the first one (along with 24
> unexpected results), for which I have pushed a fix. I still don't see
> the second one.

Well, that is likely because you run the test on uncompiled source,
while I am running it on compiled orgmode... maybe one of these pesky
byte-compiler warnings shouldn't have been ignored.  I can't reproduce
the 24 unexpected results, but they probably have a similar origin
(partially compiled orgmode?).


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptations for KORG EX-800 and Poly-800MkII V0.9:
http://Synth.Stromeko.net/Downloads.html#KorgSDada

Re: [Bug] Tests for experimental org-features should expect to fail if not activated by the user

[Nicolas Goaziou]

Achim Gratz <Stromeko@nexgo.de> writes:

> Well, that is likely because you run the test on uncompiled source,
> while I am running it on compiled orgmode... maybe one of these pesky
> byte-compiler warnings shouldn't have been ignored.  I can't reproduce
> the 24 unexpected results, but they probably have a similar origin
> (partially compiled orgmode?).

It's possible. I have two different Org installations. One compiled for
my work, and the other, uncompiled for development. Both are latest git
version.

When batch running tests on my compiled Org, I get:

--8<---------------cut here---------------start------------->8---
Ran 130 tests, 105 results as expected, 25 unexpected (2012-03-04 15:42:25+0100)
9 expected failures

25 unexpected results:
   FAILED  ob-emacs-lisp/commented-last-block-line-no-var
   FAILED  ob-emacs-lisp/commented-last-block-line-with-var
   FAILED  ob-exp/exports-both
   FAILED  ob-exp/mixed-blocks-with-exports-both
   FAILED  ob-exp/noweb-no-export-and-exports-both
   FAILED  ob-exp/noweb-on-export
   FAILED  ob-exp/noweb-on-export-with-exports-results
   FAILED  ob-exp/noweb-strip-export-ensure-strips
   FAILED  test-ob-lob/do-not-eval-lob-lines-in-example-blocks-on-export
   FAILED  test-ob-sh/dont-error-on-empty-results
   FAILED  test-ob/commented-last-block-line-no-var
   FAILED  test-ob/commented-last-block-line-with-var
   FAILED  test-ob/org-babel-remove-result--results-default
   FAILED  test-org-babel/combining-scalar-and-raw-result-types
   FAILED  test-org-babel/elisp-in-header-arguments
   FAILED  test-org-babel/inline-src-blocks
   FAILED  test-org-babel/inline-src_blk-default-results-replace-line-1
   FAILED  test-org-babel/just-one-results-block
   FAILED  test-org-babel/multi-line-header-arguments
   FAILED  test-org-babel/parse-header-args
   FAILED  test-org-babel/parse-header-args2
   FAILED  test-org-babel/simple-named-code-block
   FAILED  test-org-babel/simple-variable-resolution
   FAILED  test-org-footnote/normalize-outside-org
   FAILED  test-org-table/simple-formula
--8<---------------cut here---------------end--------------->8---

So, indeed, the second failing test appears.  It should pass now (at
least it does even on my compiled Org installation).

I think the others come from a file-error ("Opening output file").

Thanks for your help.


Regards,

-- 
Nicolas Goaziou

Re: [Bug] Tests for experimental org-features should expect to fail if not activated by the user

[Achim Gratz]

Nicolas Goaziou <n.goaziou@gmail.com> writes:
> So, indeed, the second failing test appears.  It should pass now (at
> least it does even on my compiled Org installation).

I confirm the fix, thank you very much.

> I think the others come from a file-error ("Opening output file").

I can see now why this happens: the temporary directory for the test
should already exist — make usually takes care of that, but running it
by hand does not.  So these failures should go away if you "mkdir
/tmp/tmp-orgtest" before starting the tests and you should also get a
clean test run.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptation for Waldorf rackAttack V1.04R1:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada