Test framework needed

Test framework needed

[Rainer M Krug]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi

I was bitten again from an unintended regression in org-mode, and that
the second time in two weeks.

I am probably not the right person to suggest this, but I think it is
time to introduce a test framework for org-mode, to ensure that the
(without doubt useful) approach to develop org-mode does not lead to
regressions.

org is, as I see it, complex and is getting broader and broader - time
management, task management, literate programming, recipes (I like the
idea), all have different (possibly contradicting?) requirements. And I
think it is getting more and more difficult for new contributors to have
the overview to make sure that something they are contributing actually
does not break something else.

As I think it would be *really bad* idea to split org, I think that it
needs to be assured that no regressions are introduced.

Unfortunately I have no idea about elisp and can therefore not
contribute to the framework itself, but I guess all of us could
contribute small org files as test cases and expected resulting outputs
(exports, tangling, agenda files, ...). If the framework could
accommodate those, everybody could help to bring together a (definitely
not perfect) test framework, that would bring org at least one step
closer to error-free growth.

I have to add I really love org and babel and have no idea what I would
use instead for literal programming in R.

Cheers,

Rainer

- -- 
Rainer M. Krug, PhD (Conservation Ecology, SUN), MSc (Conservation
Biology, UCT), Dipl. Phys. (Germany)

Centre of Excellence for Invasion Biology
Natural Sciences Building
Office Suite 2039
Stellenbosch University
Main Campus, Merriman Avenue
Stellenbosch
South Africa

Tel:        +33 - (0)9 53 10 27 44
Cell:       +27 - (0)8 39 47 90 42
Fax (SA):   +27 - (0)8 65 16 27 82
Fax (D) :   +49 - (0)3 21 21 25 22 44
Fax (FR):   +33 - (0)9 58 10 27 44
email:      Rainer@krugs.de

Skype:      RMkrug
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk2TKbwACgkQoYgNqgF2egpQJwCdG7C+3Ft+xGyLBiwY6aBwMLaC
TIIAn0RD3ypbvkfE1RAL4FK3ZwmnERdE
=GlJt
-----END PGP SIGNATURE-----

Re: Test framework needed

[Eric Abrahamsen]

On Wed, Mar 30 2011, Rainer M Krug wrote:

> Hi
>
> I was bitten again from an unintended regression in org-mode, and that
> the second time in two weeks.
>
> I am probably not the right person to suggest this, but I think it is
> time to introduce a test framework for org-mode, to ensure that the
> (without doubt useful) approach to develop org-mode does not lead to
> regressions.

This would be the page to start with, though the most likely candidate
(Elisp Regression Testing) is only available in Emacs trunk at the
moment…

http://www.emacswiki.org/emacs/UnitTesting

E

Re: Re: Test framework needed

[Rainer M Krug]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 30/03/11 15:46, Eric Abrahamsen wrote:
> On Wed, Mar 30 2011, Rainer M Krug wrote:
> 
>> Hi
>>
>> I was bitten again from an unintended regression in org-mode, and that
>> the second time in two weeks.
>>
>> I am probably not the right person to suggest this, but I think it is
>> time to introduce a test framework for org-mode, to ensure that the
>> (without doubt useful) approach to develop org-mode does not lead to
>> regressions.
> 
> This would be the page to start with, though the most likely candidate
> (Elisp Regression Testing) is only available in Emacs trunk at the
> moment…
> 
> http://www.emacswiki.org/emacs/UnitTesting

Am I right in assuming, that all of the possible test frameworks would
require org files and the expected output (tengle, export to ...,
agenda, ...)? In this case, would it make sense to start collecting
those, as they can easily be user contributed, consequently representing
a cross section of the use cases (even not intended use cases)?

Rainer


> 
> E
> 
> 


- -- 
Rainer M. Krug, PhD (Conservation Ecology, SUN), MSc (Conservation
Biology, UCT), Dipl. Phys. (Germany)

Centre of Excellence for Invasion Biology
Natural Sciences Building
Office Suite 2039
Stellenbosch University
Main Campus, Merriman Avenue
Stellenbosch
South Africa

Tel:        +33 - (0)9 53 10 27 44
Cell:       +27 - (0)8 39 47 90 42
Fax (SA):   +27 - (0)8 65 16 27 82
Fax (D) :   +49 - (0)3 21 21 25 22 44
Fax (FR):   +33 - (0)9 58 10 27 44
email:      Rainer@krugs.de

Skype:      RMkrug
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk2TNpwACgkQoYgNqgF2egpnWACeMFmP1p7mKsMXsELXcFSEqW99
mMkAnivsDT7zmguzphqziXzoKdx5fqz3
=AWd5
-----END PGP SIGNATURE-----

Re: Test framework needed

[Eric Abrahamsen]

On Wed, Mar 30 2011, Rainer M Krug wrote:

> On 30/03/11 15:46, Eric Abrahamsen wrote:
>> On Wed, Mar 30 2011, Rainer M Krug wrote:
>> 
>>> Hi
>>>
>>> I was bitten again from an unintended regression in org-mode, and that
>>> the second time in two weeks.
>>>
>>> I am probably not the right person to suggest this, but I think it is
>>> time to introduce a test framework for org-mode, to ensure that the
>>> (without doubt useful) approach to develop org-mode does not lead to
>>> regressions.
>> 
>> This would be the page to start with, though the most likely candidate
>> (Elisp Regression Testing) is only available in Emacs trunk at the
>> moment…
>> 
>> http://www.emacswiki.org/emacs/UnitTesting
>
> Am I right in assuming, that all of the possible test frameworks would
> require org files and the expected output (tengle, export to ...,
> agenda, ...)? In this case, would it make sense to start collecting
> those, as they can easily be user contributed, consequently representing
> a cross section of the use cases (even not intended use cases)?

Yup, what you would need is some org source files that exercise all of
the possible export options (for testing export, for example), including
weird edge cases, and then ERT (if that's what we ended up using) would
provide handy functions for making sure the export output matches
expectations. The excellent gentleman who created the ODT exporter,
whose name currently escapes me, has already created test files for his
exporter—that would be a perfect place to start.

Covering all of org's various functions would end up being a bit of a
PITA, though you're quite right that it's an excellent idea, and will
become more and more necessary.

E

Re: Test framework needed

[Christian Egli]

Rainer M Krug <r.m.krug@gmail.com> writes:

>> http://www.emacswiki.org/emacs/UnitTesting
>
> Am I right in assuming, that all of the possible test frameworks would
> require org files and the expected output (tengle, export to ...,
> agenda, ...)? In this case, would it make sense to start collecting
> those, as they can easily be user contributed, consequently representing
> a cross section of the use cases (even not intended use cases)?

Before you go too far with this; Orgmode already contains a unit test
suite. Look at the README in the testing directory
(http://orgmode.org/w/?p=org-mode.git;a=blob_plain;f=testing/README.org;hb=HEAD)

-- 
Christian Egli
Swiss Library for the Blind, Visually Impaired and Print Disabled
Grubenstrasse 12, CH-8045 Zürich, Switzerland

Re: Re: Test framework needed

[Rainer M Krug]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 30/03/11 16:11, Eric Abrahamsen wrote:
> On Wed, Mar 30 2011, Rainer M Krug wrote:
> 
>> On 30/03/11 15:46, Eric Abrahamsen wrote:
>>> On Wed, Mar 30 2011, Rainer M Krug wrote:
>>>
>>>> Hi
>>>>
>>>> I was bitten again from an unintended regression in org-mode, and that
>>>> the second time in two weeks.
>>>>
>>>> I am probably not the right person to suggest this, but I think it is
>>>> time to introduce a test framework for org-mode, to ensure that the
>>>> (without doubt useful) approach to develop org-mode does not lead to
>>>> regressions.
>>>
>>> This would be the page to start with, though the most likely candidate
>>> (Elisp Regression Testing) is only available in Emacs trunk at the
>>> moment…
>>>
>>> http://www.emacswiki.org/emacs/UnitTesting
>>
>> Am I right in assuming, that all of the possible test frameworks would
>> require org files and the expected output (tengle, export to ...,
>> agenda, ...)? In this case, would it make sense to start collecting
>> those, as they can easily be user contributed, consequently representing
>> a cross section of the use cases (even not intended use cases)?
> 
> Yup, what you would need is some org source files that exercise all of
> the possible export options (for testing export, for example), including
> weird edge cases, and then ERT (if that's what we ended up using) would
> provide handy functions for making sure the export output matches
> expectations. The excellent gentleman who created the ODT exporter,
> whose name currently escapes me, has already created test files for his
> exporter—that would be a perfect place to start.
> 
> Covering all of org's various functions would end up being a bit of a
> PITA, though you're quite right that it's an excellent idea, and will
> become more and more necessary.

So would there be a possibility of "normal org users" (if there is such
a thing ...) to contribute to this?

What would be needed? Any specific structure of the org files?

Rainer

> 
> E
> 
> 


- -- 
Rainer M. Krug, PhD (Conservation Ecology, SUN), MSc (Conservation
Biology, UCT), Dipl. Phys. (Germany)

Centre of Excellence for Invasion Biology
Natural Sciences Building
Office Suite 2039
Stellenbosch University
Main Campus, Merriman Avenue
Stellenbosch
South Africa

Tel:        +33 - (0)9 53 10 27 44
Cell:       +27 - (0)8 39 47 90 42
Fax (SA):   +27 - (0)8 65 16 27 82
Fax (D) :   +49 - (0)3 21 21 25 22 44
Fax (FR):   +33 - (0)9 58 10 27 44
email:      Rainer@krugs.de

Skype:      RMkrug
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk2TPJgACgkQoYgNqgF2egoWKwCeMjWgggD7JMhVTrQTHe3f7n6s
VhgAn2CG25hOa1Q4RPufarreQVYlezHm
=18s2
-----END PGP SIGNATURE-----

Re: Re: Test framework needed

[MidLifeXis at PerlMonks]

As a heavy Perl user, writing /automated/ tests is a large part of my dev work.

I would suggest / plea / encourage that whatever framework is used can be 
automated.  If it cannot be run as part of an automated process it is not going 
to be run.  Also consider a set of testing platforms (emacs version, supporting 
versions of other .el modules, OS version, external software).  There are many 
dependencies that org has - being able to automate this testing is a must.

Just my $0.02.

Brian / MidLifeXis



----- Original Message ----
From: Eric Abrahamsen <eric@ericabrahamsen.net>
To: emacs-orgmode@gnu.org
Sent: Wed, March 30, 2011 9:11:23 AM
Subject: [O] Re: Test framework needed

On Wed, Mar 30 2011, Rainer M Krug wrote:

> On 30/03/11 15:46, Eric Abrahamsen wrote:
>> On Wed, Mar 30 2011, Rainer M Krug wrote:
>> 
>>> Hi
>>>
>>> I was bitten again from an unintended regression in org-mode, and that
>>> the second time in two weeks.
>>>
>>> I am probably not the right person to suggest this, but I think it is
>>> time to introduce a test framework for org-mode, to ensure that the
>>> (without doubt useful) approach to develop org-mode does not lead to
>>> regressions.
>> 
>> This would be the page to start with, though the most likely candidate
>> (Elisp Regression Testing) is only available in Emacs trunk at the
>> moment…
>> 
>> http://www.emacswiki.org/emacs/UnitTesting
>
> Am I right in assuming, that all of the possible test frameworks would
> require org files and the expected output (tengle, export to ...,
> agenda, ...)? In this case, would it make sense to start collecting
> those, as they can easily be user contributed, consequently representing
> a cross section of the use cases (even not intended use cases)?

Yup, what you would need is some org source files that exercise all of
the possible export options (for testing export, for example), including
weird edge cases, and then ERT (if that's what we ended up using) would
provide handy functions for making sure the export output matches
expectations. The excellent gentleman who created the ODT exporter,
whose name currently escapes me, has already created test files for his
exporter—that would be a perfect place to start.

Covering all of org's various functions would end up being a bit of a
PITA, though you're quite right that it's an excellent idea, and will
become more and more necessary.

E

Re: Re: Test framework needed

[Rainer M Krug]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 30/03/11 16:18, Christian Egli wrote:
> Rainer M Krug <r.m.krug@gmail.com> writes:
> 
>>> http://www.emacswiki.org/emacs/UnitTesting
>>
>> Am I right in assuming, that all of the possible test frameworks would
>> require org files and the expected output (tengle, export to ...,
>> agenda, ...)? In this case, would it make sense to start collecting
>> those, as they can easily be user contributed, consequently representing
>> a cross section of the use cases (even not intended use cases)?
> 
> Before you go too far with this; Orgmode already contains a unit test
> suite. Look at the README in the testing directory
> (http://orgmode.org/w/?p=org-mode.git;a=blob_plain;f=testing/README.org;hb=HEAD)
> 

But it does not look as if it is used very often... There are not many
test org files, and I did not se anything which compares the resulting
exported / tangle  file with an expected output?

Please correct me if I am missing something.

This suite should actually be updated with effectively each patch which
introduces new features and run after each patch.

So is it only necessary to add meat to this framework?

Rainer

- -- 
Rainer M. Krug, PhD (Conservation Ecology, SUN), MSc (Conservation
Biology, UCT), Dipl. Phys. (Germany)

Centre of Excellence for Invasion Biology
Natural Sciences Building
Office Suite 2039
Stellenbosch University
Main Campus, Merriman Avenue
Stellenbosch
South Africa

Tel:        +33 - (0)9 53 10 27 44
Cell:       +27 - (0)8 39 47 90 42
Fax (SA):   +27 - (0)8 65 16 27 82
Fax (D) :   +49 - (0)3 21 21 25 22 44
Fax (FR):   +33 - (0)9 58 10 27 44
email:      Rainer@krugs.de

Skype:      RMkrug
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk2TPp4ACgkQoYgNqgF2egpBZQCfQN+g3J2+BiyKaWzh6o847mZS
PR4An0aW0Di6vfgzwsJgwfK4FajL65WP
=9/CW
-----END PGP SIGNATURE-----

Re: Re: Test framework needed

[Manuel Giraud]

Rainer M Krug <r.m.krug@gmail.com> writes:

> But it does not look as if it is used very often... There are not many
> test org files, and I did not se anything which compares the resulting
> exported / tangle  file with an expected output?

It could be useful to have this kind of framework for org but I think it
is difficult problem when it comes to org export. For example, is
removing the "validate xhtml" button from an html export make this
export invalid? I guess not.

> Please correct me if I am missing something.
>
> This suite should actually be updated with effectively each patch which
> introduces new features and run after each patch.

Which renders this framework far less automatic. I think that having a
set of org files against which one could try any export and *see* that
the results are almost correct would be enough.

-- 
Manuel Giraud

Re: Re: Test framework needed

[Aankhen]

On Wed, Mar 30, 2011 at 20:43, Manuel Giraud
<manuel.giraud@univ-nantes.fr> wrote:
> Rainer M Krug <r.m.krug@gmail.com> writes:
> [snip]
>
>> Please correct me if I am missing something.
>>
>> This suite should actually be updated with effectively each patch which
>> introduces new features and run after each patch.
>
> Which renders this framework far less automatic. I think that having a
> set of org files against which one could try any export and *see* that
> the results are almost correct would be enough.

I think the “automated” part refers to running the tests.  What you
suggest—having a set of files that you could manually export to verify
the results—wouldn’t be of much help, IMHO.  First, it’d require a lot
more time than executing a single command and checking the summary at
the end.  Second, it’d be very error-prone.

A comprehensive automated test suite gives people writing patches an
easier way to perform regression testing and catch any unintended
consequences.  On the other hand, it /does/ take a lot of effort to
keep it in sync with the codebase… maybe we need a Test Fairy. ;-)

Aankhen

Re: Re: Test framework needed

[Eric Schulte]

Aankhen <aankhen@gmail.com> writes:

> On Wed, Mar 30, 2011 at 20:43, Manuel Giraud
> <manuel.giraud@univ-nantes.fr> wrote:
>> Rainer M Krug <r.m.krug@gmail.com> writes:
>> [snip]
>>
>>> Please correct me if I am missing something.
>>>
>>> This suite should actually be updated with effectively each patch which
>>> introduces new features and run after each patch.
>>
>> Which renders this framework far less automatic. I think that having a
>> set of org files against which one could try any export and *see* that
>> the results are almost correct would be enough.
>

I think an "automatic" solution would involve running a batch Emacs
process which loads up Org-mode and then loads and runs the existing
test suite.  This would be both easily automatable and would allow
testing of various versions of Emacs with relative ease.

Best -- Eric

Re: Re: Test framework needed

[Eric Schulte]

Rainer M Krug <r.m.krug@gmail.com> writes:

> On 30/03/11 16:18, Christian Egli wrote:
>> Rainer M Krug <r.m.krug@gmail.com> writes:
>> 
>>>> http://www.emacswiki.org/emacs/UnitTesting
>>>
>>> Am I right in assuming, that all of the possible test frameworks would
>>> require org files and the expected output (tengle, export to ...,
>>> agenda, ...)? In this case, would it make sense to start collecting
>>> those, as they can easily be user contributed, consequently representing
>>> a cross section of the use cases (even not intended use cases)?
>> 
>> Before you go too far with this; Orgmode already contains a unit test
>> suite. Look at the README in the testing directory
>> (http://orgmode.org/w/?p=org-mode.git;a=blob_plain;f=testing/README.org;hb=HEAD)
>> 
>
> But it does not look as if it is used very often... There are not many
> test org files, and I did not se anything which compares the resulting
> exported / tangle  file with an expected output?
>
> Please correct me if I am missing something.
>

You are correct that the existing test suite needs some attention and
some more tests.  Just a general commitment to convert problem reports
from the mailing list into unit tests should be a step in the right
direction.

However the existing test suite (while under populated) is the result of
multiple previous discussion of this nature on the mailing list, and I
think that abandoning the existing infrastructure would be a step in the
wrong direction.

>
> This suite should actually be updated with effectively each patch which
> introduces new features and run after each patch.
>

Agreed, in a perfect world...

>
> So is it only necessary to add meat to this framework?
>

Yes, I believe the best way forward would be to add tests to the
existing framework.

Best -- Eric

>
> Rainer

Re: Re: Test framework needed

[Suvayu Ali]

On Wed, 30 Mar 2011 15:42:19 -0600
"Eric Schulte" <schulte.eric@gmail.com> wrote:

> >
> > This suite should actually be updated with effectively each patch
> > which introduces new features and run after each patch.
> >  
> 
> Agreed, in a perfect world...
> 
> >
> > So is it only necessary to add meat to this framework?
> >  
> 
> Yes, I believe the best way forward would be to add tests to the
> existing framework.

I have a possibly completely useless idea regarding "automatically"
checking regressions. As far I understand the problem now is its not
very feasible to do automated tests with what ever test suite we have
(or will have after the improvements) and see the exported results for
each patch, as at some step it involves human intervention (as in, was
the export good).

So maybe we can have a directory on the Worg website (not part of the
Worg git repo) where every week or so the test suites will publish with
what ever the org-mode head is at the time for all the supported
formats. Then us "puny lisp illiterate" users can check up on it over
the course of the week and report back to the list if there is a
problem.

Since this way people can look at the export formats they are
interested in, none of the formats get treated like a step child
either. Would that be feasible? Or did I completely misunderstand the
problem at hand?

-- 
Suvayu

Open source is the future. It sets us free.

Re: Re: Test framework needed

[Eric Schulte]

Suvayu Ali <fatkasuvayu+linux@gmail.com> writes:

> On Wed, 30 Mar 2011 15:42:19 -0600
> "Eric Schulte" <schulte.eric@gmail.com> wrote:
>
>> >
>> > This suite should actually be updated with effectively each patch
>> > which introduces new features and run after each patch.
>> >  
>> 
>> Agreed, in a perfect world...
>> 
>> >
>> > So is it only necessary to add meat to this framework?
>> >  
>> 
>> Yes, I believe the best way forward would be to add tests to the
>> existing framework.
>
> I have a possibly completely useless idea regarding "automatically"
> checking regressions. As far I understand the problem now is its not
> very feasible to do automated tests with what ever test suite we have
> (or will have after the improvements) and see the exported results for
> each patch, as at some step it involves human intervention (as in, was
> the export good).
>

I would disagree that we need user interaction in the test suite.  There
are already fully automated tests which (e.g., export to some backend
like html or tex and then programatically check for properties of the
exported results.

It is certainly likely that I am missing something, but I can't think of
a situation or a feature of Org-mode which could not be tested under the
current setup (mainly due to the fact that *every* user action in Emacs
reduces to a series of function calls which could be programatically
recreated).

>
> So maybe we can have a directory on the Worg website (not part of the
> Worg git repo) where every week or so the test suites will publish with
> what ever the org-mode head is at the time for all the supported
> formats. Then us "puny lisp illiterate" users can check up on it over
> the course of the week and report back to the list if there is a
> problem.
>
> Since this way people can look at the export formats they are
> interested in, none of the formats get treated like a step child
> either. Would that be feasible? Or did I completely misunderstand the
> problem at hand?

I'd think that a better way for contributing to the test suite in a
non-lisp manner would be to submit test cases, e.g. "this block of
Org-mode text should export to this but sometimes instead exports to
this", or "when I press this key sequence in this place in this org-mode
text I expect x to happen to the text".

We could even potentially leverage the existing Emacs macro system to
build a *record* method so that users could semi-automatically record
their actions allowing an interactive method of recording tests (or
submitting a re-creatable bug report).  Or at least recording enough
information so that someone with a little bit more elisp-fu could wrap
the recorded actions into a unit test.

Hope this is helpful -- Eric

Re: Re: Test framework needed

[Rainer M Krug]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 31/03/11 05:40, Eric Schulte wrote:
> Suvayu Ali <fatkasuvayu+linux@gmail.com> writes:
> 
>> On Wed, 30 Mar 2011 15:42:19 -0600
>> "Eric Schulte" <schulte.eric@gmail.com> wrote:
>>
>>>>
>>>> This suite should actually be updated with effectively each patch
>>>> which introduces new features and run after each patch.
>>>>  
>>>
>>> Agreed, in a perfect world...
>>>
>>>>
>>>> So is it only necessary to add meat to this framework?
>>>>  
>>>
>>> Yes, I believe the best way forward would be to add tests to the
>>> existing framework.
>>
>> I have a possibly completely useless idea regarding "automatically"
>> checking regressions. As far I understand the problem now is its not
>> very feasible to do automated tests with what ever test suite we have
>> (or will have after the improvements) and see the exported results for
>> each patch, as at some step it involves human intervention (as in, was
>> the export good).

Good or not good is subjective - but *consistent* is not - and
consistency is important. Even if I do not like the default LaTeX output
from org, I can tweak it, but there is a problem if there are unexpected
changes in the export, which break my customizations or make it
difficult to recreate old documents, especially if these changes are not
documented.

>>
> 
> I would disagree that we need user interaction in the test suite.  There
> are already fully automated tests which (e.g., export to some backend
> like html or tex and then programatically check for properties of the
> exported results.

Exactly - the tests is R work this way: you have some code which is
executed, and then the resulting *output* is redirected into a file.
these results are then compared to a reference output, and if they are
not *identical* an error is raised.

Similar could be done in org: export to LaTeX should always result in
the same output, unless a change is intended (e.g. additional headers,
improvements, ...). So one could compare the resulting .tex file with a
reference .tex file for this test automatically, without user intervention.

> 
> It is certainly likely that I am missing something, but I can't think of
> a situation or a feature of Org-mode which could not be tested under the
> current setup (mainly due to the fact that *every* user action in Emacs
> reduces to a series of function calls which could be programatically
> recreated).
> 
>>
>> So maybe we can have a directory on the Worg website (not part of the
>> Worg git repo) where every week or so the test suites will publish with
>> what ever the org-mode head is at the time for all the supported
>> formats. Then us "puny lisp illiterate" users can check up on it over
>> the course of the week and report back to the list if there is a
>> problem.
>>
>> Since this way people can look at the export formats they are
>> interested in, none of the formats get treated like a step child
>> either. Would that be feasible? Or did I completely misunderstand the
>> problem at hand?
> 
> I'd think that a better way for contributing to the test suite in a
> non-lisp manner would be to submit test cases, e.g. "this block of
> Org-mode text should export to this but sometimes instead exports to
> this", or "when I press this key sequence in this place in this org-mode
> text I expect x to happen to the text".

Correct - this is what we would, in addition to programmatic tests of
individual functions, need. I would actually say that the exports /
tangling / agendas / ... are the possibly the more important test cases,
as they 1) only show in a later stage of ones project, and 2) errors in
functions are easily detected by users and reported - and fizxed quite
quickly and finally 3) I guess an export / ... includes quite a lot of
functions, which are therefore tested as well (kind off...).

> 
> We could even potentially leverage the existing Emacs macro system to
> build a *record* method so that users could semi-automatically record
> their actions allowing an interactive method of recording tests (or
> submitting a re-creatable bug report).  Or at least recording enough
> information so that someone with a little bit more elisp-fu could wrap
> the recorded actions into a unit test.

That would be brilliant. Like the error reporting:

atach the current buffer, record what was done and *the individual
configuration of org / emacs* and finally email / upload it to an
address, where it is automatically added to other submitted test cases,
might bring us a long way closer to an very useful test base. I am
actually ot aware of any other test framework, which let's "normal"
users submit test cases via email / internet - I think that would be a
very useful addition.


> 
> Hope this is helpful -- Eric

Most definitely,

Rainer

> 


- -- 
Rainer M. Krug, PhD (Conservation Ecology, SUN), MSc (Conservation
Biology, UCT), Dipl. Phys. (Germany)

Centre of Excellence for Invasion Biology
Natural Sciences Building
Office Suite 2039
Stellenbosch University
Main Campus, Merriman Avenue
Stellenbosch
South Africa

Tel:        +33 - (0)9 53 10 27 44
Cell:       +27 - (0)8 39 47 90 42
Fax (SA):   +27 - (0)8 65 16 27 82
Fax (D) :   +49 - (0)3 21 21 25 22 44
Fax (FR):   +33 - (0)9 58 10 27 44
email:      Rainer@krugs.de

Skype:      RMkrug
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk2UKhoACgkQoYgNqgF2egr8LACdEUpds2WnA9LYRHugLlQ9jrK5
dvoAn3A9qu3F5eWB5OFgYJlrgHds8BHW
=6OYS
-----END PGP SIGNATURE-----