* testing macros and fixtures
@ 2015-10-12 21:13 Phillip Lord
2015-10-13 2:54 ` Kaushal Modi
2015-10-13 9:16 ` Artur Malabarba
0 siblings, 2 replies; 5+ messages in thread
From: Phillip Lord @ 2015-10-12 21:13 UTC (permalink / raw)
To: emacs-devel
I have been thinking about testing since it was raised in one of the
last gargantuan threads.
ERT is quite nice, but one of the things that I have found lacking is a
nice set of predicates, for use within should.
So, when I wrote test for my "lentic" package I needed some functions
like, so that I could do things like:
(should
(test-eq-after-this
"blah-before.txt"
"blah-after.txt"
(insert "hello")))
which opens "blah-before.txt" runs (insert "hello") then compares the
result with "blah-after.txt". My version of this also does a diff of the
results if the two are not equal.
I've noticed that "puppet-mode" has some thing similar. For instance:
(puppet-test-with-temp-buffer "# class
bar"
(should (eq (puppet-test-face-at 1) 'font-lock-comment-delimiter-face))
(should (eq (puppet-test-face-at 3) 'font-lock-comment-face))
(should (eq (puppet-test-face-at 7) 'font-lock-comment-face))
(should (eq (puppet-test-face-at 8) 'font-lock-comment-face))
(should-not (puppet-test-face-at 9))))
And julia-mode has indentation checking tests like so:
(julia--should-indent
"
if foo
bar
else
baz
end"
"
if foo
bar
else
baz
end"))
My own experience is that these are actually quite hard to right. The
ones in lentic have never worked quite right -- that is, when it all
works they are fine, but restoring state after a crash doesn't always
work. Similarly, checking that, for example, test files are not already
open before a test is run interactively.
So, the point of my question is this; are there any good libraries
providing this kind of fixture logic? A lot of this should surely be
re-usable between different packages.
Phil
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: testing macros and fixtures
2015-10-12 21:13 testing macros and fixtures Phillip Lord
@ 2015-10-13 2:54 ` Kaushal Modi
2015-10-13 9:23 ` Phillip Lord
2015-10-13 9:16 ` Artur Malabarba
1 sibling, 1 reply; 5+ messages in thread
From: Kaushal Modi @ 2015-10-13 2:54 UTC (permalink / raw)
To: Phillip Lord; +Cc: Emacs developers
[-- Attachment #1: Type: text/plain, Size: 2361 bytes --]
verilog-mode runs similar tests from outside emacs.
From the terminal, the verilog-mode functions are run in emacs batch mode
on files in the tests/ dir and and the results are compared against the
files in tests_ok/ dir (if I understand correctly).
Repo on github which is then commited to emacs core:
https://github.com/veripool/verilog-mode
Perl script to run batch tests:
https://github.com/veripool/verilog-mode/blob/master/batch_test.pl
--
Kaushal Modi
On Mon, Oct 12, 2015 at 5:13 PM, Phillip Lord <phillip.lord@russet.org.uk>
wrote:
>
> I have been thinking about testing since it was raised in one of the
> last gargantuan threads.
>
> ERT is quite nice, but one of the things that I have found lacking is a
> nice set of predicates, for use within should.
>
> So, when I wrote test for my "lentic" package I needed some functions
> like, so that I could do things like:
>
> (should
> (test-eq-after-this
> "blah-before.txt"
> "blah-after.txt"
> (insert "hello")))
>
> which opens "blah-before.txt" runs (insert "hello") then compares the
> result with "blah-after.txt". My version of this also does a diff of the
> results if the two are not equal.
>
> I've noticed that "puppet-mode" has some thing similar. For instance:
>
> (puppet-test-with-temp-buffer "# class
> bar"
> (should (eq (puppet-test-face-at 1) 'font-lock-comment-delimiter-face))
> (should (eq (puppet-test-face-at 3) 'font-lock-comment-face))
> (should (eq (puppet-test-face-at 7) 'font-lock-comment-face))
> (should (eq (puppet-test-face-at 8) 'font-lock-comment-face))
> (should-not (puppet-test-face-at 9))))
>
> And julia-mode has indentation checking tests like so:
>
> (julia--should-indent
> "
> if foo
> bar
> else
> baz
> end"
> "
> if foo
> bar
> else
> baz
> end"))
>
>
> My own experience is that these are actually quite hard to right. The
> ones in lentic have never worked quite right -- that is, when it all
> works they are fine, but restoring state after a crash doesn't always
> work. Similarly, checking that, for example, test files are not already
> open before a test is run interactively.
>
> So, the point of my question is this; are there any good libraries
> providing this kind of fixture logic? A lot of this should surely be
> re-usable between different packages.
>
> Phil
>
>
>
>
>
[-- Attachment #2: Type: text/html, Size: 4019 bytes --]
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: testing macros and fixtures
2015-10-12 21:13 testing macros and fixtures Phillip Lord
2015-10-13 2:54 ` Kaushal Modi
@ 2015-10-13 9:16 ` Artur Malabarba
2015-10-13 11:34 ` Phillip Lord
1 sibling, 1 reply; 5+ messages in thread
From: Artur Malabarba @ 2015-10-13 9:16 UTC (permalink / raw)
To: Phillip Lord; +Cc: emacs-devel
[-- Attachment #1: Type: text/plain, Size: 2286 bytes --]
2015-10-12 22:13 GMT+01:00 Phillip Lord
> ERT is quite nice, but one of the things that I have found lacking is a
> nice set of predicates, for use within should.
>
> So, when I wrote test for my "lentic" package I needed some functions
> like, so that I could do things like:
>
> (should
> (test-eq-after-this
> "blah-before.txt"
> "blah-after.txt"
> (insert "hello")))
>
> which opens "blah-before.txt" runs (insert "hello") then compares the
> result with "blah-after.txt".
I think that's a pretty specific use-case. So it's best to let people write
their own macros for this.
(defmacro should-eq-after-body (file-before file-after &rest body)
`(let ((bef
(with-temp-buffer
(insert-file-contents file-before)
,@body
(buffer-string))))
(should (string= bef (with-temp-buffer
(insert-file-contents file-after)
(buffer-string))))))
> My version of this also does a diff of the
> results if the two are not equal.
This certainly seems very useful. Maybe ert should do this on *all*
multi-line string comparisons instead of doing its default
“different-strings” report (which, most of the time, just says “strings are
of different length”).
> I've noticed that "puppet-mode" has some thing similar. For instance:
> ...
>
> And julia-mode has indentation checking tests like so:
>
> ...
>
> My own experience is that these are actually quite hard to right. The
> ones in lentic have never worked quite right -- that is, when it all
> works they are fine, but restoring state after a crash doesn't always
> work. Similarly, checking that, for example, test files are not already
> open before a test is run interactively.
Yes, there are many packages that use custom-deisgned temp-buffers for
testing. In fact, most non-trivial packages do. I'd really like to see ert
offer a common interface for this. The difficulty for that is that each
package has very different needs when it comes to testing in buffers, so I
have no idea what this common interface could be.
> So, the point of my question is this; are there any good libraries
> providing this kind of fixture logic?
Don't know.
[-- Attachment #2: Type: text/html, Size: 2823 bytes --]
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: testing macros and fixtures
2015-10-13 2:54 ` Kaushal Modi
@ 2015-10-13 9:23 ` Phillip Lord
0 siblings, 0 replies; 5+ messages in thread
From: Phillip Lord @ 2015-10-13 9:23 UTC (permalink / raw)
To: Kaushal Modi; +Cc: Emacs developers
Okay. This is definately something worth looking at, then. I think
writing tests is probably too hard at the moment!
Phil
Kaushal Modi <kaushal.modi@gmail.com> writes:
> verilog-mode runs similar tests from outside emacs.
>
>>From the terminal, the verilog-mode functions are run in emacs batch mode
> on files in the tests/ dir and and the results are compared against the
> files in tests_ok/ dir (if I understand correctly).
>
> Repo on github which is then commited to emacs core:
> https://github.com/veripool/verilog-mode
>
> Perl script to run batch tests:
> https://github.com/veripool/verilog-mode/blob/master/batch_test.pl
>
>
> --
> Kaushal Modi
>
> On Mon, Oct 12, 2015 at 5:13 PM, Phillip Lord <phillip.lord@russet.org.uk>
> wrote:
>
>>
>> I have been thinking about testing since it was raised in one of the
>> last gargantuan threads.
>>
>> ERT is quite nice, but one of the things that I have found lacking is a
>> nice set of predicates, for use within should.
>>
>> So, when I wrote test for my "lentic" package I needed some functions
>> like, so that I could do things like:
>>
>> (should
>> (test-eq-after-this
>> "blah-before.txt"
>> "blah-after.txt"
>> (insert "hello")))
>>
>> which opens "blah-before.txt" runs (insert "hello") then compares the
>> result with "blah-after.txt". My version of this also does a diff of the
>> results if the two are not equal.
>>
>> I've noticed that "puppet-mode" has some thing similar. For instance:
>>
>> (puppet-test-with-temp-buffer "# class
>> bar"
>> (should (eq (puppet-test-face-at 1) 'font-lock-comment-delimiter-face))
>> (should (eq (puppet-test-face-at 3) 'font-lock-comment-face))
>> (should (eq (puppet-test-face-at 7) 'font-lock-comment-face))
>> (should (eq (puppet-test-face-at 8) 'font-lock-comment-face))
>> (should-not (puppet-test-face-at 9))))
>>
>> And julia-mode has indentation checking tests like so:
>>
>> (julia--should-indent
>> "
>> if foo
>> bar
>> else
>> baz
>> end"
>> "
>> if foo
>> bar
>> else
>> baz
>> end"))
>>
>>
>> My own experience is that these are actually quite hard to right. The
>> ones in lentic have never worked quite right -- that is, when it all
>> works they are fine, but restoring state after a crash doesn't always
>> work. Similarly, checking that, for example, test files are not already
>> open before a test is run interactively.
>>
>> So, the point of my question is this; are there any good libraries
>> providing this kind of fixture logic? A lot of this should surely be
>> re-usable between different packages.
>>
>> Phil
>>
>>
>>
>>
>>
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: testing macros and fixtures
2015-10-13 9:16 ` Artur Malabarba
@ 2015-10-13 11:34 ` Phillip Lord
0 siblings, 0 replies; 5+ messages in thread
From: Phillip Lord @ 2015-10-13 11:34 UTC (permalink / raw)
To: Artur Malabarba; +Cc: emacs-devel
Artur Malabarba <bruce.connor.am@gmail.com> writes:
> 2015-10-12 22:13 GMT+01:00 Phillip Lord
>
>> ERT is quite nice, but one of the things that I have found lacking is a
>> nice set of predicates, for use within should.
>>
>> So, when I wrote test for my "lentic" package I needed some functions
>> like, so that I could do things like:
>>
>> (should
>> (test-eq-after-this
>> "blah-before.txt"
>> "blah-after.txt"
>> (insert "hello")))
>>
>> which opens "blah-before.txt" runs (insert "hello") then compares the
>> result with "blah-after.txt".
>
> I think that's a pretty specific use-case. So it's best to let people write
> their own macros for this.
>
> (defmacro should-eq-after-body (file-before file-after &rest body)
> `(let ((bef
> (with-temp-buffer
> (insert-file-contents file-before)
> ,@body
> (buffer-string))))
> (should (string= bef (with-temp-buffer
> (insert-file-contents file-after)
> (buffer-string))))))
I'd imagine it's pretty common -- assuming that (insert "hello") is
anything at all of course.
>> My version of this also does a diff of the
>> results if the two are not equal.
>
> This certainly seems very useful. Maybe ert should do this on *all*
> multi-line string comparisons instead of doing its default
> “different-strings” report (which, most of the time, just says “strings are
> of different length”).
Yep. I use a function to replace "string=" that looks like this.
(defun lentic-test-equal-loudly (a b)
"Actually, this just tests equality and shouts if not."
;; change this to t to disable noisy printout
(if lentic-test-quiet
(string= a b)
(if (string= a b)
t
(message "Results:\n%s\n:Complete\nShouldbe:\n%s\nComplete:" cloned-results cloned-file)
(let* ((a-buffer
(generate-new-buffer "a"))
(b-buffer
(generate-new-buffer "b"))
(a-file
(make-temp-file
(buffer-name a-buffer)))
(b-file
(make-temp-file
(buffer-name b-buffer))))
(with-current-buffer
a-buffer
(insert a)
(write-file a-file))
(with-current-buffer
b-buffer
(insert b)
(write-file b-file))
(message "diff:%senddiff:"
(with-temp-buffer
(call-process
"diff"
nil
(current-buffer)
nil
"-c"
a-file
b-file)
(buffer-string))))
nil)))
An egregious hack, of course, but it was quick and does what I need.
> Yes, there are many packages that use custom-deisgned temp-buffers for
> testing. In fact, most non-trivial packages do. I'd really like to see ert
> offer a common interface for this. The difficulty for that is that each
> package has very different needs when it comes to testing in buffers, so I
> have no idea what this common interface could be.
I think I'll put this next into my todo list. Maintaining the current
(dreadful) infrastructure that I have for lentic is painful to say the
least, and I think a lot of this should be reusuable.
As you say, there are lots of examples.
Phil
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2015-10-13 11:34 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2015-10-12 21:13 testing macros and fixtures Phillip Lord
2015-10-13 2:54 ` Kaushal Modi
2015-10-13 9:23 ` Phillip Lord
2015-10-13 9:16 ` Artur Malabarba
2015-10-13 11:34 ` Phillip Lord
Code repositories for project(s) associated with this external index
https://git.savannah.gnu.org/cgit/emacs.git
https://git.savannah.gnu.org/cgit/emacs/org-mode.git
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.