Merging Org tests

Merging Org tests

[Rasmus]

Hi,

I'm starting to prepare the merger of Org 9+ to Emacs master.  I have not
followed the entire recent Emacs+ELPA discussion in the other thread.

I understand that it would be desirable to include the Org tests in Emacs.

The readme/file-organization file in the "test" dir of emacs.git is a bit
confusing.  Why does it talk about an "automated" folder that isn't there?
I guess it's simply "lisp" folder.

Also, how strictly must the suggested file organization be followed?
Skimming "make-test-deps.emacs-lisp" it seems it might have to be followed
rather strictly.  In Org, tests are already organized into a different
file hierarchy and use a slightly different naming scheme.

Here's the Org "testing" folder:

    http://orgmode.org/cgit.cgi/org-mode.git/tree/testing

The main differences are:

- In Emacs "test" ressources are in a parallel folder alongside lisp, in
  the Org repo they are stored in a sibling folder to the one with the
  tests. 
- In the Org repo, tests are prefixed with "test-", in Emacs "test" they
  should have a "-tests" suffix.
- The Org tests depend on the library "org-test.el", though I guess that
  is not really an issue. 

What would be the best strategy for including the Org tests in the Emacs
repo?

1. Include Org tests in the "manual" folder and keep the current structure.
2. Start to reorganize the tests in the Org repo to follow the Emacs
   scheme.
3. Keep tests in the Org repo.

Thanks,
Rasmus

-- 
Send from my Emacs

Re: Merging Org tests

[Eli Zaretskii]

> From: Rasmus <rasmus@gmx.us>
> Date: Tue, 14 Feb 2017 16:10:30 +0100
> 
> I understand that it would be desirable to include the Org tests in Emacs.

Yes, I think so.

> The readme/file-organization file in the "test" dir of emacs.git is a bit
> confusing.  Why does it talk about an "automated" folder that isn't there?
> I guess it's simply "lisp" folder.

Yes, that was a left-over from the old organization.  I fixed the
text.

> Also, how strictly must the suggested file organization be followed?
> Skimming "make-test-deps.emacs-lisp" it seems it might have to be followed
> rather strictly.  In Org, tests are already organized into a different
> file hierarchy and use a slightly different naming scheme.

Is it a problem to change Org test files to follow the current
organization below test/?  IOW, put all the Org files in
test/lisp/org/?

> What would be the best strategy for including the Org tests in the Emacs
> repo?
> 
> 1. Include Org tests in the "manual" folder and keep the current structure.
> 2. Start to reorganize the tests in the Org repo to follow the Emacs
>    scheme.
> 3. Keep tests in the Org repo.

My preference is 2, if it's feasible.

Thanks.

Re: Merging Org tests

[Phillip Lord]

Rasmus <rasmus@gmx.us> writes:

> I'm starting to prepare the merger of Org 9+ to Emacs master.  I have not
> followed the entire recent Emacs+ELPA discussion in the other thread.
>
> I understand that it would be desirable to include the Org tests in Emacs.
>
> The readme/file-organization file in the "test" dir of emacs.git is a bit
> confusing.

Apologies, my fault.

> Why does it talk about an "automated" folder that isn't there?  I
> guess it's simply "lisp" folder.

Yeah, that was the original idea, but we ended up with src, lisp and manual

> Also, how strictly must the suggested file organization be followed?
> Skimming "make-test-deps.emacs-lisp" it seems it might have to be followed
> rather strictly.

It depends what you want. make-test-deps.emacs-lisp is there so that
"make check-maybe" can work sanely to run (some approximation of) just
those tests that need running. I thought this was a good thing to have,
because "make check" takes too long to run in a commit hook.

If you don't follow the file organisation, then make check will still
work.


> In Org, tests are already organized into a different file hierarchy
> and use a slightly different naming scheme.
>
> Here's the Org "testing" folder:
>
>     http://orgmode.org/cgit.cgi/org-mode.git/tree/testing
>
> The main differences are:
>
> - In Emacs "test" ressources are in a parallel folder alongside lisp, in
>   the Org repo they are stored in a sibling folder to the one with the
>   tests. 

This is a consequence of Emacs' monolithic layout. I think your layout is
better and core should adopt your layout, but that's a slightly
different issue.


> - In the Org repo, tests are prefixed with "test-", in Emacs "test" they
>   should have a "-tests" suffix.

I picked "tests" over "test" on the basis of popularity, IIRC.

> - The Org tests depend on the library "org-test.el", though I guess that
>   is not really an issue. 
>
> What would be the best strategy for including the Org tests in the Emacs
> repo?
>
> 1. Include Org tests in the "manual" folder and keep the current structure.
> 2. Start to reorganize the tests in the Org repo to follow the Emacs
>    scheme.
> 3. Keep tests in the Org repo.

Really be good to get tests in into main, so not 3. That's the main
thing.

Phil

Re: Merging Org tests

[Michael Albinus]

phillip.lord@russet.org.uk (Phillip Lord) writes:

>> Also, how strictly must the suggested file organization be followed?
>> Skimming "make-test-deps.emacs-lisp" it seems it might have to be followed
>> rather strictly.
>
> It depends what you want. make-test-deps.emacs-lisp is there so that
> "make check-maybe" can work sanely to run (some approximation of) just
> those tests that need running. I thought this was a good thing to have,
> because "make check" takes too long to run in a commit hook.

In general it is a good thing (tm). But there shall be
improvements. Currently, if you want to run foo-tests.el, it depends on
foo.el. Often, this is sufficient. But for larger packages, like Org or
Tramp, this is not sufficient. So it might be good to maintain a list of
foo*.el files, a test file foo-tests.el shall depend on.

>> What would be the best strategy for including the Org tests in the Emacs
>> repo?
>>
>> 1. Include Org tests in the "manual" folder and keep the current structure.
>> 2. Start to reorganize the tests in the Org repo to follow the Emacs
>>    scheme.
>> 3. Keep tests in the Org repo.
>
> Really be good to get tests in into main, so not 3. That's the main
> thing.

Like Eli, I'm in favor of 2.

And pls mark expensive tests with the tag `expensive'. We should teach
elpa's "make check" to skip expensive tests, as we do in emacs' "make
check".

> Phil

Best regards, Michael.

Re: Merging Org tests

[Phillip Lord]

Michael Albinus <michael.albinus@gmx.de> writes:

> phillip.lord@russet.org.uk (Phillip Lord) writes:
>
>>> Also, how strictly must the suggested file organization be followed?
>>> Skimming "make-test-deps.emacs-lisp" it seems it might have to be followed
>>> rather strictly.
>>
>> It depends what you want. make-test-deps.emacs-lisp is there so that
>> "make check-maybe" can work sanely to run (some approximation of) just
>> those tests that need running. I thought this was a good thing to have,
>> because "make check" takes too long to run in a commit hook.
>
> In general it is a good thing (tm). But there shall be
> improvements. Currently, if you want to run foo-tests.el, it depends on
> foo.el. Often, this is sufficient. But for larger packages, like Org or
> Tramp, this is not sufficient. So it might be good to maintain a list of
> foo*.el files, a test file foo-tests.el shall depend on.

Of course. make-test-deps.emacs-lisp just generates a make file for the
cases where I can infer dependencies based on the name. If there are
special cases, we could just stick the dependencies in to the main
Makefile by hand. Or, alternatively, rewrite make-test-deps.emacs-lisp
to be more intelligent and pull out require forms from the test files.

Phil

Re: Merging Org tests

[Michael Albinus]

phillip.lord@russet.org.uk (Phillip Lord) writes:

> Of course. make-test-deps.emacs-lisp just generates a make file for the
> cases where I can infer dependencies based on the name. If there are
> special cases, we could just stick the dependencies in to the main
> Makefile by hand.

Yep.

> Or, alternatively, rewrite make-test-deps.emacs-lisp
> to be more intelligent and pull out require forms from the test files.

Wouldn't work for Tramp. Most of the tramp-*.elc files are loaded via
autoload. And which one is loaded depends on the test data. For example,
tramp-adb.elc is loaded only if tramp-tests.elc tests a connection
"/adb::".

> Phil

Best regards, Michael.