Python-style doctests in Guile (implemented, please comment)

Python-style doctests in Guile (implemented, please comment)

[Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 2774 bytes --]

Hi,


I implemented doctests in Guile which allow writing SRFI-64 tests
directly in the docstring. Here’s a minimal example:


    (import (examples doctests))
    
    (define (one)
      "(test 'foo
         (test-equal 1 (one)))"
      1)
    
    (doctests-testmod (current-module))


Writing simple tests directly in the procedure-documentation makes for
very natural test-driven-design which gets extended into basic
documentation automatically: Start the procedure, write a basic test in
the docstring, implement the body till the test passes and finally
create api-docs from the docstrings which then also show basic usage
examples in the tests.


My questions: This method here is the simplest implemenation of this
feature. It would be great if you could comment on it. Is this structure
good? Would you do it differently? Is "(test-equal 1 (one))" close
enough to an example in the REPL, or would it serve for better
usage-examples in another structure?


The output from running the minimal example is plain SRFI-64 output,
though with extended identifiers:


    $ ./doctests-testone.scm 
    %%%% Starting test ._-guile-user--foo--foo  (Writing full log to "._-guile-user--foo--foo.log")
    # of expected passes      1
    
    $ cat ._-guile-user--foo--foo.log 
    %%%% Starting test ._-guile-user--foo--foo
    Group begin: ._-guile-user--foo--foo
    Test begin:
      source-line: 2
      source-form: (test-equal "bar" (foo))
    Test end:
      result-kind: pass
      actual-value: "bar"
      expected-value: "bar"
    Group end: ._-guile-user--foo--foo
    # of expected passes      1


The syntax for the identifier is:


     ._-<filename-or-module>--<function name>--<testid>


I mainly know doctests from Python. There they use a special syntax
which looks like the interpreter.

For Guile I rather chose to start the tests with "(test 'testname" and
treat as test everything read by (read).

Avoiding (test-begin 'testname) and (test-end 'testname) makes for
easier parsing.  (they are added automatically using the 'testname)

Here’s a somewhat longer example with the test-execution tucked away in
the main file so it does not interfere with importing the module:
<https://bitbucket.org/ArneBab/wisp/src/1ce02fce232f429fc1cbd29c666e36d34ec22d76/examples/doctests-test.scm>

The full code is available in the wisp-repo:
<https://bitbucket.org/ArneBab/wisp/src/1ce02fce232f429fc1cbd29c666e36d34ec22d76/examples/doctests.scm>

(The code is originally written in wisp, but I decided to use
parenthizes syntax nontheless, because using s-exps makes for more
obvious parsing)


Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein
ohne es zu merken

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Python-style doctests in Guile (implemented, please comment)

[Chaos Eternal]

Great job!

I have a new idea using s-exps to define tests:
simple way:
(define-syntax define-examples
(syntax-rules () ((_ e ...) (quote (e ...)))))

then we can simply (read the-file)  then (match e ((define-examples e ...))
(do-test e))


On Mon, Jul 31, 2017 at 3:35 AM Arne Babenhauserheide <arne_bab@web.de>
wrote:

> Hi,
>
>
> I implemented doctests in Guile which allow writing SRFI-64 tests
> directly in the docstring. Here’s a minimal example:
>
>
>     (import (examples doctests))
>
>     (define (one)
>       "(test 'foo
>          (test-equal 1 (one)))"
>       1)
>
>     (doctests-testmod (current-module))
>
>
> Writing simple tests directly in the procedure-documentation makes for
> very natural test-driven-design which gets extended into basic
> documentation automatically: Start the procedure, write a basic test in
> the docstring, implement the body till the test passes and finally
> create api-docs from the docstrings which then also show basic usage
> examples in the tests.
>
>
> My questions: This method here is the simplest implemenation of this
> feature. It would be great if you could comment on it. Is this structure
> good? Would you do it differently? Is "(test-equal 1 (one))" close
> enough to an example in the REPL, or would it serve for better
> usage-examples in another structure?
>
>
> The output from running the minimal example is plain SRFI-64 output,
> though with extended identifiers:
>
>
>     $ ./doctests-testone.scm
>     %%%% Starting test ._-guile-user--foo--foo  (Writing full log to
> "._-guile-user--foo--foo.log")
>     # of expected passes      1
>
>     $ cat ._-guile-user--foo--foo.log
>     %%%% Starting test ._-guile-user--foo--foo
>     Group begin: ._-guile-user--foo--foo
>     Test begin:
>       source-line: 2
>       source-form: (test-equal "bar" (foo))
>     Test end:
>       result-kind: pass
>       actual-value: "bar"
>       expected-value: "bar"
>     Group end: ._-guile-user--foo--foo
>     # of expected passes      1
>
>
> The syntax for the identifier is:
>
>
>      ._-<filename-or-module>--<function name>--<testid>
>
>
> I mainly know doctests from Python. There they use a special syntax
> which looks like the interpreter.
>
> For Guile I rather chose to start the tests with "(test 'testname" and
> treat as test everything read by (read).
>
> Avoiding (test-begin 'testname) and (test-end 'testname) makes for
> easier parsing.  (they are added automatically using the 'testname)
>
> Here’s a somewhat longer example with the test-execution tucked away in
> the main file so it does not interfere with importing the module:
> <
> https://bitbucket.org/ArneBab/wisp/src/1ce02fce232f429fc1cbd29c666e36d34ec22d76/examples/doctests-test.scm
> >
>
> The full code is available in the wisp-repo:
> <
> https://bitbucket.org/ArneBab/wisp/src/1ce02fce232f429fc1cbd29c666e36d34ec22d76/examples/doctests.scm
> >
>
> (The code is originally written in wisp, but I decided to use
> parenthizes syntax nontheless, because using s-exps makes for more
> obvious parsing)
>
>
> Best wishes,
> Arne
> --
> Unpolitisch sein
> heißt politisch sein
> ohne es zu merken
>

Re: Python-style doctests in Guile (implemented, please comment)

[Mark H Weaver]

Hi Arne,

Arne Babenhauserheide <arne_bab@web.de> writes:

> I implemented doctests in Guile which allow writing SRFI-64 tests
> directly in the docstring. Here’s a minimal example:
>
>
>     (import (examples doctests))
>     
>     (define (one)
>       "(test 'foo
>          (test-equal 1 (one)))"
>       1)

I don't think that tests should go into doc strings, for a couple of
reasons.  First, doc strings already have a purpose, and that's for
documentation.  While it may sometimes be beneficial to include a few
examples in the documentation, a full test suite does not, IMO, belong
in the doc string.

The other issue is that this would involve putting code directly into a
quoted string literal, which causes several problems.  One has to do
with editor support for S-expressions.  For users of Emacs Scheme mode
and/or paredit, it would mean when writing the tests without the editor
support that we are accustomed to.  Additionally, if the test code
involve string constants, then the embedded quotes would need to be
escaped with backslashes, and any escaped characters within the strings
would need double-backslashes, etc.  Finally, putting code within a
string literal entails a loss of the hygiene that has been so carefully
developed in modern Scheme macro expanders.

For these reasons, I'm strongly opposed to this style of testing.

If you want to do something like this, I would suggest instead defining
some macros like 'define-with-tests' that allow you to put the unit
tests together with each definition.

What do you think?

    Regards,
      Mark

Re: Python-style doctests in Guile (implemented, please comment)

[Panicz Maciej Godek]

2017-07-31 14:51 GMT+02:00 Mark H Weaver <mhw@netris.org>:

> Hi Arne,
>
> Arne Babenhauserheide <arne_bab@web.de> writes:
>
> > I implemented doctests in Guile which allow writing SRFI-64 tests
> > directly in the docstring. Here’s a minimal example:
> >
> >
> >     (import (examples doctests))
> >
> >     (define (one)
> >       "(test 'foo
> >          (test-equal 1 (one)))"
> >       1)
>
> I don't think that tests should go into doc strings, for a couple of
> reasons.  First, doc strings already have a purpose, and that's for
> documentation.  While it may sometimes be beneficial to include a few
> examples in the documentation, a full test suite does not, IMO, belong
> in the doc string.
>
> The other issue is that this would involve putting code directly into a
> quoted string literal, which causes several problems.  One has to do
> with editor support for S-expressions.  For users of Emacs Scheme mode
> and/or paredit, it would mean when writing the tests without the editor
> support that we are accustomed to.  Additionally, if the test code
> involve string constants, then the embedded quotes would need to be
> escaped with backslashes, and any escaped characters within the strings
> would need double-backslashes, etc.  Finally, putting code within a
> string literal entails a loss of the hygiene that has been so carefully
> developed in modern Scheme macro expanders.
>
> For these reasons, I'm strongly opposed to this style of testing.
>
> If you want to do something like this, I would suggest instead defining
> some macros like 'define-with-tests' that allow you to put the unit
> tests together with each definition.
>
> What do you think?
>

I agree with Mark, that putting tests inside a string in Lisp is a terrible
idea,
because Lisp doesn't have Python's shortcommings, i.e. you can quote
symbolic structures in Lisp, whereas you can't do that in Python

There is no point in trading something better for something worse merely
because people from Python (or elsewhere) can't afford this "better".

I usually interleave definitions with tests using the "e.g." form, as you
can see
here, for example:
https://github.com/plande/grand-scheme/blob/master/grand/combinatorics.scm

(I've recently found that SRFI-78 does a very similar thing, as SRFI-64 is
in my view unnecessarily complex, especially if you want your tests to serve
as examples)

Re: Python-style doctests in Guile (implemented, please comment)

[Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 3166 bytes --]

Hi Mark,

String-literals are a problem I did hit, and I’d be happy to lose that
problem without losing the ease of starting a procedure with tests which
double as automatically verified documentation.

Mark H Weaver <mhw@netris.org> writes:
>>     (import (examples doctests))
>>     
>>     (define (one)
>>       "(test 'foo
>>          (test-equal 1 (one)))"
>>       1)
>
> While it may sometimes be beneficial to include a few
> examples in the documentation, a full test suite does not, IMO, belong
> in the doc string.

I think there’s a misconception here: These doctests are not intended to
replace a full test suite. They provide simple tests which double as
automatically verified documentation.

This is why I asked whether what I implemented is too complex (by
providing all of srfi-64 here). If you get clear benefits from
editor-support, the test is typically too complex for a doctest.
However editor-support could be provided as it is for org-mode: By
editing the region in a specialized sub-buffer.

The tests here are first-of-all intended for humans to read.

Why does code in string-literals bring a loss of hygiene? I’s read in
the module as if it had been written directly in a lambda and read
during parsing. Am I missing something or are you envisioning mutation
of the string prior to reading and evaluating it?

Panicz Maciej Godek <godek.maciek@gmail.com> writes:
> I agree with Mark, that putting tests inside a string in Lisp is a
> terrible idea, because Lisp doesn't have Python's shortcommings,
…
> There is no point in trading something better for something worse merely
> because people from Python (or elsewhere) can't afford this "better".

This doesn’t correctly represent the situation of Python. It is
perfectly possible in Python to write tests in literal code — for
example by using attributes of a function to hold functions which run
the tests.

What doctests provide is a way to write example usage first and foremost
for humans, directly at the top of the function definition, and have it
checked automatically to ensure that these examples in auto-generated
documentation actually work and keep working.

Using a define-with-tests (or define-with-examples) does not allow
writing for humans first, so it does not reach feature-parity. I could
use pretty-print to create an examples section of the documentation, but
I won’t know how it is going to be formatted while writing the code.
(though this need not be a pure drawback)

This is why I’m looking into doctests in the first place. If you have
something which provides feature parity, I’m all for using that
instead. Requirements:

- Can be verified automatically.
- Becomes part of auto-generated documentation.
- Is "physically" close to the definition of the procedure (same file,
  no other definitions between the tests/examples and the procedure).

Ideally it should look like what I’d run in the REPL to use the
procedure, but I don’t think that this must be a hard requirement.

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein
ohne es zu merken

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Python-style doctests in Guile (implemented, please comment)

[Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 684 bytes --]


Chaos Eternal <eternalchaos@shlug.org> writes:

> Great job!
>
> I have a new idea using s-exps to define tests:
> simple way:
> (define-syntax define-examples
> (syntax-rules () ((_ e ...) (quote (e ...)))))
>
> then we can simply (read the-file)  then (match e ((define-examples e ...))
> (do-test e))

So you’d write something like the following?

(define (foo)
        (define-examples
            ((foo) 'foo))
        'foo)

It has the same limitations as I see for define-with-tests (see my other
answer), do you see advantages over the define-with-tests approach?

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein
ohne es zu merken

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Python-style doctests in Guile (implemented, please comment)

[Chaos Eternal]

No, I don't think so.
First, s-exp itself can be very literal written just like the srfi-64:
#+begin_src
(test 'foo
         (test-equal 1
                            (one)))
#+end_src
the meanning of this code is very straightforward: do a test whether result
of (one) equals to 1.
and now with my `define-examples‘
#+begin_src
(define-examples
  (test 'foo
          (test-equal 1
                             (one))))
(define (one)
            1)
#+end_src
this code is also straightforward.
and even we can do this:
#+begin_src
(define-examples
  (doc
  "the purpose of one is just return 1")
  (test 'foo
          (test-equal 1
                             (one))))
(define (one)
            1)
#+end_src
IMHO
 the real problem of putting codes into literal string is that breaks the
consistency of lisp, that say, the semantics defined by the embed code can
not be extracted with just s-exp reader.
on the other hand, the s-exp is the ultimate way to represent structures,
and it shall be adopted whenever possible.

On Tue, Aug 1, 2017 at 1:46 AM Arne Babenhauserheide <arne_bab@web.de>
wrote:

>
> Chaos Eternal <eternalchaos@shlug.org> writes:
>
> > Great job!
> >
> > I have a new idea using s-exps to define tests:
> > simple way:
> > (define-syntax define-examples
> > (syntax-rules () ((_ e ...) (quote (e ...)))))
> >
> > then we can simply (read the-file)  then (match e ((define-examples e
> ...))
> > (do-test e))
>
> So you’d write something like the following?
>
> (define (foo)
>         (define-examples
>             ((foo) 'foo))
>         'foo)
>
> It has the same limitations as I see for define-with-tests (see my other
> answer), do you see advantages over the define-with-tests approach?
>
> Best wishes,
> Arne
> --
> Unpolitisch sein
> heißt politisch sein
> ohne es zu merken
>

Re: Python-style doctests in Guile (implemented, please comment)

[Vítor De Araújo]

Another possibility beside docstrings would be to add it as a property
to the function. I'm not sure if this is a documented feature, but if
the first form in a function is a literal vector rather than a string
(or in addition to a string, I've just found out!), it will be
interpreted as a sequence of (KEY . VAL) properties to the function. For
example:

(define (double x)
  "Returns twice the value of a given number."
  #((examples [(double 5) 10]
              [(double 0) 0]))
  (* 2 x))

scheme@(guile-user)> (procedure-properties double)
$2 = ((name . double) (documentation . "Returns twice the value of a
given number.") (examples ((double 5) 10) ((double 0) 0)))

scheme@(guile-user)> (procedure-property double 'examples)
$3 = (((double 5) 10) ((double 0) 0))


On 31/07/2017 14:23, Arne Babenhauserheide wrote:
> Hi Mark,
> 
> String-literals are a problem I did hit, and I’d be happy to lose that
> problem without losing the ease of starting a procedure with tests which
> double as automatically verified documentation.
> 
> Mark H Weaver <mhw@netris.org> writes:
>>>     (import (examples doctests))
>>>     
>>>     (define (one)
>>>       "(test 'foo
>>>          (test-equal 1 (one)))"
>>>       1)
>>
>> While it may sometimes be beneficial to include a few
>> examples in the documentation, a full test suite does not, IMO, belong
>> in the doc string.
> 
> I think there’s a misconception here: These doctests are not intended to
> replace a full test suite. They provide simple tests which double as
> automatically verified documentation.
> 
> This is why I asked whether what I implemented is too complex (by
> providing all of srfi-64 here). If you get clear benefits from
> editor-support, the test is typically too complex for a doctest.
> However editor-support could be provided as it is for org-mode: By
> editing the region in a specialized sub-buffer.
> 
> The tests here are first-of-all intended for humans to read.
> 
> Why does code in string-literals bring a loss of hygiene? I’s read in
> the module as if it had been written directly in a lambda and read
> during parsing. Am I missing something or are you envisioning mutation
> of the string prior to reading and evaluating it?
> 
> Panicz Maciej Godek <godek.maciek@gmail.com> writes:
>> I agree with Mark, that putting tests inside a string in Lisp is a
>> terrible idea, because Lisp doesn't have Python's shortcommings,
> …
>> There is no point in trading something better for something worse merely
>> because people from Python (or elsewhere) can't afford this "better".
> 
> This doesn’t correctly represent the situation of Python. It is
> perfectly possible in Python to write tests in literal code — for
> example by using attributes of a function to hold functions which run
> the tests.
> 
> What doctests provide is a way to write example usage first and foremost
> for humans, directly at the top of the function definition, and have it
> checked automatically to ensure that these examples in auto-generated
> documentation actually work and keep working.
> 
> Using a define-with-tests (or define-with-examples) does not allow
> writing for humans first, so it does not reach feature-parity. I could
> use pretty-print to create an examples section of the documentation, but
> I won’t know how it is going to be formatted while writing the code.
> (though this need not be a pure drawback)
> 
> This is why I’m looking into doctests in the first place. If you have
> something which provides feature parity, I’m all for using that
> instead. Requirements:
> 
> - Can be verified automatically.
> - Becomes part of auto-generated documentation.
> - Is "physically" close to the definition of the procedure (same file,
>   no other definitions between the tests/examples and the procedure).
> 
> Ideally it should look like what I’d run in the REPL to use the
> procedure, but I don’t think that this must be a hard requirement.
> 
> Best wishes,
> Arne
> 

-- 
Vítor De Araújo
https://elmord.org/

Re: Python-style doctests in Guile (implemented, please comment)

[Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 2317 bytes --]

Hi Vitor,

I’m sorry for answering so late; your suggestion stuck to my mind as a
clean and schemish solution for my usecase (thank you!) but it took me
quite a while to realize it.

Vítor De Araújo <vbuaraujo@sapo.pt> writes:
> (define (double x)
>   "Returns twice the value of a given number."
>   #((examples [(double 5) 10]
>               [(double 0) 0]))
>   (* 2 x))

I now implemented this for tests based on srfi-64:

https://bitbucket.org/ArneBab/wisp/src/299795dbb3fecea91dcdde480817b36fc45ccc5f/examples/doctests.scm

Here’s the documentation:

;;; doctests --- simple testing by adding procedure-properties with tests.

;;; Usage

;; Add a tests property to a procedure to have simple unit tests.

;; Simple tests:
;;
;; (define (A)
;;     #((tests (test-eqv 'A (A))
;;              (test-assert #t)))
;;     'A)
;;
;; Named tests:
;;
;; (define (A)
;;     #((tests ('test1 (test-eqv 'A (A))
;;                      (test-assert #t))
;;              ('test2 (test-assert #t))))
;;     'A)
;;
;; Allows for docstrings:
;;
;; (define (A)
;;     "returns 'A"
;;     #((tests (test-eqv 'A (A))
;;              (test-assert #t)))
;;     'A)

;; For writing the test before the implementation, start with the test and #f:

;; (define (A)
;;     #((tests (test-eqv 'A (A))))
;;     #f)

This is what was missing for me to get an elegance in test-driven design
which I had been missing till now. And it really feels much more elegant
than the stringly doctesting in Python.

The output looks like this:

%%%% Starting test examples-doctests.scm--doctests-testmod--mytest  (Writing full log to "examples-doctests.scm--doctests-testmod--mytest.log")
# of expected passes      3

%%%% Starting test examples-doctests.scm--doctests-testmod--mytest2  (Writing full log to "examples-doctests.scm--doctests-testmod--mytest2.log")
# of expected passes      4

%%%% Starting test examples-doctests.scm--subtract  (Writing full log to "examples-doctests.scm--subtract.log")
# of expected passes      5


Now there’s one first question: How should I call it? :-)

Currently the modules is (examples doctests). Do you have a better idea
than "doctests"?

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein
ohne es zu merken

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Python-style doctests in Guile (implemented, please comment)

[Vítor De Araújo]

On 10/10/2017 15:21, Arne Babenhauserheide wrote:
> Hi Vitor,
> 
> I’m sorry for answering so late; your suggestion stuck to my mind as a
> clean and schemish solution for my usecase (thank you!) but it took me
> quite a while to realize it.

I'm glad you liked it. :)

[snip]
> Now there’s one first question: How should I call it? :-)
> 
> Currently the modules is (examples doctests). Do you have a better idea
> than "doctests"?

Maybe "proptests", since they are stored in the function properties
rather than inside the docstring? Or maybe "vectests", since they are
written inside a vector? (Of these I think "proptest" sounds better,
because it sounds more like "doctest".) Or maybe just go along with
"doctests".

> Best wishes,
> Arne
> 

-- 
Vítor De Araújo
https://elmord.org/

Re: Python-style doctests in Guile (implemented, please comment)

[Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 1277 bytes --]

Vítor De Araújo <vbuaraujo@sapo.pt> writes:
> On 10/10/2017 15:21, Arne Babenhauserheide wrote:
>> I’m sorry for answering so late; your suggestion stuck to my mind as a
>> clean and schemish solution for my usecase (thank you!) but it took me
>> quite a while to realize it.
>
> I'm glad you liked it. :)

I do, very much so. It’s another case of Guile already providing
something I know from Python, but much cleaner — and (on the downside)
with less infrastructure built on top of it.

> [snip]
>> Now there’s one first question: How should I call it? :-)
>> 
>> Currently the modules is (examples doctests). Do you have a better idea
>> than "doctests"?
>
> Maybe "proptests", since they are stored in the function properties
> rather than inside the docstring? Or maybe "vectests", since they are
> written inside a vector? (Of these I think "proptest" sounds better,
> because it sounds more like "doctest".) Or maybe just go along with
> "doctests".

Hm, yes, though proptest seems hard to pronounce. Since the string is
called documentation string, the test could really just stick to being
called documentation test (doctest).

Thanks again!

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein
ohne es zu merken

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]