Re: [PATCH] Patches for problems discovered with interactive testing

[gerard.vermeulen@posteo.net]

On 19.01.2024 13:50, Ihor Radchenko wrote:
> gerard.vermeulen@posteo.net writes:
> 
>> This patch addresses a number of glitches discovered during 
>> interactive
>> testing and contains possible fixes (sub-patch test-ob-maxima fixes
>> batch testing on my system, but not interactive testing).
> 
> Thanks for your interest in improving Org mode tests!
> 
>> The test-ob-maxima patch may be due to a different Maxima version, but
>> I do not think so.  In batch testing test-ob-maxima passes contrary to
>> interactive mode.
> 
> On my side, Maxima tests pass both interactively and non-interactively.
> I use Maxima 5.47.0
> What is your version?
The same: maxima --version
Maxima 5.47.0

In batch testing without the patch I get:

Test ob-maxima/batch+verbatim condition:
     (ert-test-failed
      ((should
        (equal (org-babel-execute-src-block)
               "(assume(z > 0),integrate(exp(-t)*t^z,t,0,inf))\n
                 gamma(z + 1)"))
       :form
       (equal
        "(linenum:0,\n(assume(z > 0),integrate(exp(-t)*t^z,t,0,inf))\n    
                              gamma(z + 1)"
        "(assume(z > 0),integrate(exp(-t)*t^z,t,0,inf))\n                 
                 gamma(z + 1)")
       :value nil :explanation
       (arrays-of-different-length 104 92
                                   "(linenum:0,\n(assume(z > 
0),integrate(exp(-t)*t^z,t,0,inf))\n                                 
gamma(z + 1)"
                                   "(assume(z > 
0),integrate(exp(-t)*t^z,t,0,inf))\n                                 
gamma(z + 1)"
                                   first-mismatch-at 1)))
    FAILED   162/1252  ob-maxima/batch+verbatim (0.694916 sec) at 
../lisp/test-ob
-maxima.el:71

I do not know where the difference comes from.  I am not a Maxima user
and not inclined to track this down unless you tell me it is important.
> 
>> The test-ob-python patch unsets org-src-preserve-indentation which I
>> have set.
>> 
>> The test-ob patch also unsets org-src-preserve-indentation.
> 
>> The test-org patch also unsets org-src-preserve-indentation.  This
>> reduces the number of failures to 2 while testing test-org
>> interactively.
> 
> It is generally not expected that Org tests will pass when you try to
> run them on Emacs instance with your personal config. This is because
> you can have arbitrary customizations, hooks, or advices that change 
> the
> Org mode defaults.
> 
> let-binding a couple of variables cannot fix this general problem for
> all scenarios. So, I do not see any reason to accept patches that make
> the tests run with personal config of a single user.
> 

Before I made the patch, I had reverted changes that worked 
interactively
in my personal config but caused failures with make test.

Because of the Maxima test failure (also in batch), I was not sure that 
all
tests pass on a reference system.

> What might be useful, however, is making sure that tests do not fail
> interactively with a clean Emacs config (make repro).
> Check out testing/README file.
> 

I have a super-weird system where make repro opens Emacs with
as default-directory my home directory.  So I have to make symbolic
links from my home to the testing and lisp directories.
But make test works.

Because of testing my 
0001-org-babel-demarcate-block-split-using-element-API.patch
interactively in my personal config (before make test which takes
quite some time), I figured out that it would guess to suppress false
failures.
> 
>>  (ert-deftest ob-maxima/string-input ()
>>    "Test of string input"
>>    (org-test-at-id "b5842ed4-8e8b-4b18-a1c9-cef006b6a6c8"
>>      (org-babel-next-src-block 2)
>> -    (should (equal "- sin(x)" (org-babel-execute-src-block)))))
>> +    (should (equal '(("(linenum:0," "") ("-" "sin(x)"))
>> +                   (org-babel-execute-src-block)))))
> 
> The test in question is testing the output of
> 
> #+begin_src maxima :var fun="sin(x)" :var q=2 :results silent
> print(diff(fun, x, q))$
> #+end_src
> 
> It is certainly not expected that such code block returns "linenum:0".

I would say that it is an unexplained difference, I still get (with 
"linenum:0") :

#+header: :wrap "src text -n"
#+name: lst:maxima-example
#+begin_src maxima -n :batch batch :cmdline --quiet :exports both 
:results raw
rat(1/(x+1) + x/(x-1));
#+end_src

#+name: lst:maxima-example-results
#+results: lst:maxima-example
#+begin_src text -n
(%i1) (linenum:0,
(%i1) rat(1/(x+1)+x/(x-1))
                                   2
                                  x  + 2 x - 1
(%o1)/R/                         ------------
                                      2
                                     x  - 1
#+end_src

> 
> Tests in Org mode are verifying that Org mode behaves correctly. It
> makes no sense to change the test when Org mode result is not correct -
> we should instead fix Org mode.
> 
> In the majority of cases, if the test is failing, it is not something
> wrong with the test, but something wrong with Org mode code that test 
> is
> verifying.
> 
>>  (ert-deftest test-ob/does-not-replace-a-block-with-the-results ()
>>    (org-test-with-temp-text "#+NAME: foo
>> diff --git a/testing/lisp/test-org-element.el 
>> b/testing/lisp/test-org-element.el
>> index ca7d77e28..f0958ff59 100644
>> --- a/testing/lisp/test-org-element.el
>> +++ b/testing/lisp/test-org-element.el
>> @@ -4457,7 +4457,7 @@ Text
>>       '(paragraph nil "  Two spaces\n\n \n  Two spaces"))
>>      '(paragraph nil "Two spaces\n\n\nTwo spaces")))
>>    (should
>> -   (equal
>> +   (equal ;; Fails in test. Why? (equal ...) form works in ielm.
> 
> It would help if you provided the error ERT displayed when the test 
> failed.

I cannot reproduce it.  Now it works in interactive testing (personal 
config
as when I first saw this).

If you ask me, I am still willing to collect the 2 sub-patches with the
let bindings which improve testing interactively in personal configs
somewhat.  If not, it is OK (I understand now that it is not important).

Regards -- Gerard