Locations of Tests

Locations of Tests

[Phillip Lord]

I was thinking that a "jump to tests" feature would be nice for
emacs-lisp mode the other day. While I had a quick thing about how to
implement this, I looked at the emacs source, and noticed that the
location of automated tests is a little erratic.

So, we have thingatpt.el (as a test), file-notify-tests.el (for
filenotify). There is also no reflection of the organisation of the lisp
in the tests. So, elisp-mode-tests.el at top level, while elisp-mode.el
is in lisp/progmodes. Similar issue with gnus-test.el and pcase.el.

Is there an organisation for tests that I am unaware off? Would there be
interest in a policy based on file name?

Phil

Re: Locations of Tests

[John Wiegley]

>>>>> Phillip Lord <phillip.lord@russet.org.uk> writes:

> So, we have thingatpt.el (as a test), file-notify-tests.el (for filenotify).
> There is also no reflection of the organisation of the lisp in the tests.
> So, elisp-mode-tests.el at top level, while elisp-mode.el is in
> lisp/progmodes. Similar issue with gnus-test.el and pcase.el.
> 
> Is there an organisation for tests that I am unaware off? Would there be
> interest in a policy based on file name?

Huge interest. There should be a purely mechanical way of going from a source
file in core, to the set of tests for that file. Then we could add M-x
emacs-lisp-jump-to-test.

I'd happily accept a patch to move things into accordance with the naming
structure under lisp/.

John

Re: Locations of Tests

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 228 bytes --]

On 4 Nov 2015 5:53 pm, "John Wiegley" <jwiegley@gmail.com> wrote:
> I'd happily accept a patch to move things into accordance with the naming
> structure under lisp/.

Same here. Just make sure it doesn't mess up git blame. ;-)

[-- Attachment #2: Type: text/html, Size: 344 bytes --]

Re: Locations of Tests

[John Wiegley]

>>>>> Artur Malabarba <bruce.connor.am@gmail.com> writes:

> Same here. Just make sure it doesn't mess up git blame. ;-)

From *Man git-blame*:

       The origin of lines is automatically followed across whole-file renames (currently
       there is no option to turn the rename-following off). To follow lines moved from
       one file to another, or to follow lines that were copied and pasted from another
       file, etc., see the -C and -M options.

John

Re: Locations of Tests

[Michael Albinus]

John Wiegley <jwiegley@gmail.com> writes:

>> So, we have thingatpt.el (as a test), file-notify-tests.el (for filenotify).
>> There is also no reflection of the organisation of the lisp in the tests.
>> So, elisp-mode-tests.el at top level, while elisp-mode.el is in
>> lisp/progmodes. Similar issue with gnus-test.el and pcase.el.
>> 
>> Is there an organisation for tests that I am unaware off? Would there be
>> interest in a policy based on file name?
>
> Huge interest. There should be a purely mechanical way of going from a source
> file in core, to the set of tests for that file. Then we could add M-x
> emacs-lisp-jump-to-test.
>
> I'd happily accept a patch to move things into accordance with the naming
> structure under lisp/.

Before we rename everything, wouldn't a simple directive in the source
file do the job?

;; Package-Tests: test/automated/file-notify-tests.el

I don't know how much of the path name shall be included there. We have
also files outside Emacs core, for example in Elpa. Here the path name
could start at the package root directory:

;; Package-Tests: tests/stream-tests.el

> John

Best regards, Michael.

Re: Locations of Tests

[Steinar Bang]

>>>>> Artur Malabarba <bruce.connor.am@gmail.com>:
> On 4 Nov 2015 5:53 pm, "John Wiegley" <jwiegley@gmail.com> wrote:

>> I'd happily accept a patch to move things into accordance with the naming
>> structure under lisp/.

> Same here. Just make sure it doesn't mess up git blame. ;-)

...and file history.

What this means in practice, is that files should be moved verbatim.

You can change files once they are in place in the new location, but the
first commit in the new location should be of files with identical SHA1
hashes as the files in the old location.

git tracks history across moves/renames by matching the file content
SHA1 hashes of files in other locations.

Re: Locations of Tests

[John Wiegley]

>>>>> Michael Albinus <michael.albinus@gmx.de> writes:

> Before we rename everything, wouldn't a simple directive in the source file
> do the job?

> ;; Package-Tests: test/automated/file-notify-tests.el

That would also work, but having a naming structure also makes it easier for
contributors to know where new files they create should go. So I'd prefer to
have a convention, rather than make it freeform, with a set of textual links
to relate the two sets.

John

Re: Locations of Tests

[David Kastrup]

Steinar Bang <sb@dod.no> writes:

>>>>>> Artur Malabarba <bruce.connor.am@gmail.com>:
>> On 4 Nov 2015 5:53 pm, "John Wiegley" <jwiegley@gmail.com> wrote:
>
>>> I'd happily accept a patch to move things into accordance with the naming
>>> structure under lisp/.
>
>> Same here. Just make sure it doesn't mess up git blame. ;-)
>
> ...and file history.
>
> What this means in practice, is that files should be moved verbatim.
>
> You can change files once they are in place in the new location, but the
> first commit in the new location should be of files with identical SHA1
> hashes as the files in the old location.
>
> git tracks history across moves/renames by matching the file content
> SHA1 hashes of files in other locations.

That's what always works, but git blame's -C option will try getting
stuff matched even if you don't properly rename.  But -C is expensive.
And it's always a nuisance to supply the like of C-x v g with additional
options.

-- 
David Kastrup

Re: Locations of Tests

[Phillip Lord]

[-- Attachment #1: Type: text/plain, Size: 1229 bytes --]



John Wiegley <jwiegley@gmail.com> writes:

>>>>>> Michael Albinus <michael.albinus@gmx.de> writes:
>
>> Before we rename everything, wouldn't a simple directive in the source file
>> do the job?
>
>> ;; Package-Tests: test/automated/file-notify-tests.el
>
> That would also work, but having a naming structure also makes it easier for
> contributors to know where new files they create should go. So I'd prefer to
> have a convention, rather than make it freeform, with a set of textual links
> to relate the two sets.

I'd tend to agree.

I think I'd like to get a specification sorted first before making the
changes. I've included my starter-for-ten below. I'm willing to take any
feedback here -- or I could add it to the emacs-repo as an orphan
branch, then people could change directly.

I've given locations for test files, and resources (data files). This is
both for Emacs-lisp source files and for lisp tests for features
provided in C (inotify-test.el for example). There are also suggestions
for ELPA packages.

I've said nothing about C tests for the C core, because I know nothing
about C testing (and almost nothing about C coding!). Adding a
buffer-local as suggested above as a legacy might make sense.

Phil


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: emacs-tests.org --]
[-- Type: text/x-org, Size: 2276 bytes --]


#+TITLE: The Location of Emacs-Lisp Tests

* Introduction

In this document, we describe the relationship between Emacs-Lisp files and
their associated automated test files.


* In Emacs

The Emacs repository contains a very large number of Emacs-Lisp files, many of
which pre-date both formal package support for Emacs and automated unit
testing. The test layout is, therefore, somewhat different than for ELPA
packages.

All paths are relative to the Emacs root directory.

** Source

Lisp files are stored in the ~lisp~ directory or its sub-directories.
Sub-directories are in many cases themed after packages (~gnus~, ~org~,
~calc~), related functionality (~net~, ~emacs-lisp~, ~progmodes~) or status
(~obsolete~).

C source is stored in the ~src~ directory, which is flat.

** Test Files

Automated tests should be stored in the ~test/automated~ directory. Tests
should reflect the directory structure of the source tree; so tests for files
in the ~emacs-lisp~ source directory should reside in the
~test/automated/emacs-lisp~ directory.

Tests should normally reside in a file with an ~-test~ added to the name of
the tested source file; hence ~ert.el~ is tested in ~ert-test.el~, or
~pcase.el~ is tested in ~pcase-test.el~.

Where features of the C source are tested using Emacs-Lisp test files, these
should reside in ~/test/automated/c~ and be named after the C file.

** Resource Files

Resource files for tests (containing test data) should reside in a directory
named after the feature with a ~-resources~ suffix. Hence, tests files for
~flymake.el~ should reside in a directory called ~flymake-resources~.

No guidance is given for the organisation of resource files inside the
~-resource~ directory; files can be organised at the author's discretion.


* In ELPA

All paths are given relative to the package root.

** Source Files

ELPA lisp files should be stored at top-level within the package.

** Test Files

Test files should normally reside in the ~test~ directory, and be named after
the file being tested.

** Resources Files

Resource files for tests should reside in the ~dev-resources~ directory. No
guidance is given for the organisation of resource files inside the
~dev-resource~ directory; files can be organised at the author's discretion.

Re: Locations of Tests

[Phillip Lord]

David Kastrup <dak@gnu.org> writes:

> Steinar Bang <sb@dod.no> writes:
>
>>>>>>> Artur Malabarba <bruce.connor.am@gmail.com>:
>>> On 4 Nov 2015 5:53 pm, "John Wiegley" <jwiegley@gmail.com> wrote:
>>
>>>> I'd happily accept a patch to move things into accordance with the naming
>>>> structure under lisp/.
>>
>>> Same here. Just make sure it doesn't mess up git blame. ;-)
>>
>> ...and file history.
>>
>> What this means in practice, is that files should be moved verbatim.
>>
>
> That's what always works, but git blame's -C option will try getting
> stuff matched even if you don't properly rename.  But -C is expensive.
> And it's always a nuisance to supply the like of C-x v g with additional
> options.


Okay, so I hearby publically state that if I become responsible for
another git mega-thread, I will serve my penance by giving up Emacs and
moving to Ed for all future text manipulation needs.

Phil

Re: Locations of Tests

[Stephen Leake]

phillip.lord@russet.org.uk (Phillip Lord) writes:

> John Wiegley <jwiegley@gmail.com> writes:
>
>
> #+TITLE: The Location of Emacs-Lisp Tests

This looks good in general; a couple minor comments below.

> * Introduction
>
> In this document, we describe the relationship between Emacs-Lisp files and
> their associated automated test files.
>
>
> * In Emacs
>
> The Emacs repository contains a very large number of Emacs-Lisp files, many of
> which pre-date both formal package support for Emacs and automated unit
> testing. The test layout is, therefore, somewhat different than for ELPA
> packages.
>
> All paths are relative to the Emacs root directory.
>
> ** Source
>
> Lisp files are stored in the ~lisp~ directory or its sub-directories.
> Sub-directories are in many cases themed after packages (~gnus~, ~org~,
> ~calc~), related functionality (~net~, ~emacs-lisp~, ~progmodes~) or status
> (~obsolete~).
>
> C source is stored in the ~src~ directory, which is flat.
>
> ** Test Files
>
> Automated tests should be stored in the ~test/automated~ directory. Tests
> should reflect the directory structure of the source tree; so tests for files
> in the ~emacs-lisp~ source directory should reside in the
> ~test/automated/emacs-lisp~ directory.
>
> Tests should normally reside in a file with an ~-test~ added to the name of
> the tested source file; hence ~ert.el~ is tested in ~ert-test.el~, or
> ~pcase.el~ is tested in ~pcase-test.el~.
>
> Where features of the C source are tested using Emacs-Lisp test files, these
> should reside in ~/test/automated/c~ and be named after the C file.

Since we are trying to mimic the source layout, perhaps it would be
better to have:

tests/automated/src  ;; tests of files in the src dir

tests/automated/lisp ;; tests of file in the lisp dir

tests/automated/lisp/prog-modes ;; etc

> ** Resource Files
>
> Resource files for tests (containing test data) should reside in a directory
> named after the feature with a ~-resources~ suffix. Hence, tests files for
> ~flymake.el~ should reside in a directory called ~flymake-resources~.
>
> No guidance is given for the organisation of resource files inside the
> ~-resource~ directory; files can be organised at the author's discretion.
>
>
> * In ELPA
>
> All paths are given relative to the package root.
>
> ** Source Files
>
> ELPA lisp files should be stored at top-level within the package.
>
> ** Test Files
>
> Test files should normally reside in the ~test~ directory, and be named after
> the file being tested.

With the -test suffix; it helps to be very clear, 

> ** Resources Files
>
> Resource files for tests should reside in the ~dev-resources~ directory. No
> guidance is given for the organisation of resource files inside the
> ~dev-resource~ directory; files can be organised at the author's discretion.

-- 
-- Stephe

Re: Locations of Tests

[Artur Malabarba]

>> ** Test Files
>>
>> Automated tests should be stored in the ~test/automated~ directory. Tests
>> should reflect the directory structure of the source tree; so tests for files
>> in the ~emacs-lisp~ source directory should reside in the
>> ~test/automated/emacs-lisp~ directory.
>>
>> Tests should normally reside in a file with an ~-test~ added to the name of
>> the tested source file; hence ~ert.el~ is tested in ~ert-test.el~, or
>> ~pcase.el~ is tested in ~pcase-test.el~.

Then let's also remember to rename all the *-test.el out there to *-test.el

Re: Locations of Tests

[Nicolas Petton]

[-- Attachment #1: Type: text/plain, Size: 193 bytes --]

Artur Malabarba <bruce.connor.am@gmail.com> writes:

> Then let's also remember to rename all the *-test.el out there to
> *-test.el

hmmm... I'm not following.  Could you rephrase this?

Nico

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

Re: Locations of Tests

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 443 bytes --]

On 5 Nov 2015 3:02 pm, "Nicolas Petton" <nicolas@petton.fr> wrote:
>
> Artur Malabarba <bruce.connor.am@gmail.com> writes:
>
> > Then let's also remember to rename all the *-test.el out there to
> > *-test.el
>
> hmmm... I'm not following.  Could you rephrase this?
>
> Nico

Oops. :-) missed an s in there.
I meant  "rename all the *-tests.el to  *-test.el".
Or the other way around, I'm just saying to normalise the use of plural
/singular.

[-- Attachment #2: Type: text/html, Size: 708 bytes --]

Re: Locations of Tests

[Juanma Barranquero]

[-- Attachment #1: Type: text/plain, Size: 349 bytes --]

On Thu, Nov 5, 2015 at 2:41 PM, Stephen Leake <
stephen_leake@stephe-leake.org> wrote:

> Since we are trying to mimic the source layout, perhaps it would be
> better to have:
>
> tests/automated/src  ;; tests of files in the src dir
>
> tests/automated/lisp ;; tests of file in the lisp dir
>
> tests/automated/lisp/prog-modes ;; etc

Yes, please.

[-- Attachment #2: Type: text/html, Size: 523 bytes --]

Re: Locations of Tests

[Phillip Lord]

Stephen Leake <stephen_leake@stephe-leake.org> writes:

> phillip.lord@russet.org.uk (Phillip Lord) writes:
>
>> Where features of the C source are tested using Emacs-Lisp test files, these
>> should reside in ~/test/automated/c~ and be named after the C file.
>
> Since we are trying to mimic the source layout, perhaps it would be
> better to have:
>
> tests/automated/src  ;; tests of files in the src dir
>
> tests/automated/lisp ;; tests of file in the lisp dir
>

>> ** Test Files
>>
>> Test files should normally reside in the ~test~ directory, and be named after
>> the file being tested.
>
> With the -test suffix; it helps to be very clear, 
>


Both good calls, with incorporate.

Re: Locations of Tests

[Phillip Lord]

Artur Malabarba <bruce.connor.am@gmail.com> writes:

> On 5 Nov 2015 3:02 pm, "Nicolas Petton" <nicolas@petton.fr> wrote:
>>
>> Artur Malabarba <bruce.connor.am@gmail.com> writes:
>>
>> > Then let's also remember to rename all the *-test.el out there to
>> > *-test.el
>>
>> hmmm... I'm not following.  Could you rephrase this?
>>
>> Nico
>
> Oops. :-) missed an s in there.
> I meant  "rename all the *-tests.el to  *-test.el".
> Or the other way around, I'm just saying to normalise the use of plural
> /singular.


Yeah, I checked. "-tests.el" is more common in existing code, so easier
to go that way. Although, it would be good if this worked for other
packages (GNU ELPA and others). I'll check these out also.

Phil

Makefile-help (was Re: Locations of Tests)

[Phillip Lord]

Phillip Lord <phillip.lord@russet.org.uk> writes:

> Is there an organisation for tests that I am unaware off? Would there be
> interest in a policy based on file name?


I've started work on writing a script to move all the files (seems
better than doing it manually as it's more checkable), and I've updated
my specification based on comments; I'll push all of this to a scratch
branch tomorrow.

While doing this, I've realised that I'm going to need to update the
Makefile (which assumes a flat directory structure for tests). 

My make expertise is quite limited. Would anyone be free to help me get
this working.

Phil

Re: Makefile-help (was Re: Locations of Tests)

[Stephen Leake]

phillip.lord@russet.org.uk (Phillip Lord) writes:

> Phillip Lord <phillip.lord@russet.org.uk> writes:
>
>> Is there an organisation for tests that I am unaware off? Would there be
>> interest in a policy based on file name?
>
>
> I've started work on writing a script to move all the files (seems
> better than doing it manually as it's more checkable), and I've updated
> my specification based on comments; I'll push all of this to a scratch
> branch tomorrow.
>
> While doing this, I've realised that I'm going to need to update the
> Makefile (which assumes a flat directory structure for tests). 
>
> My make expertise is quite limited. Would anyone be free to help me get
> this working.

I'm fairly good with make, and have time to work on this.

Let me know the scratch branch name.

-- 
-- Stephe

Location of tests (again) (was Re: Makefile-help)

[Phillip Lord]

Stephen Leake <stephen_leake@stephe-leake.org> writes:

> phillip.lord@russet.org.uk (Phillip Lord) writes:
>
>> Phillip Lord <phillip.lord@russet.org.uk> writes:
>>
>>> Is there an organisation for tests that I am unaware off? Would there be
>>> interest in a policy based on file name?
>>
>>
>> I've started work on writing a script to move all the files (seems
>> better than doing it manually as it's more checkable), and I've updated
>> my specification based on comments; I'll push all of this to a scratch
>> branch tomorrow.
>>
>> While doing this, I've realised that I'm going to need to update the
>> Makefile (which assumes a flat directory structure for tests). 
>>
>> My make expertise is quite limited. Would anyone be free to help me get
>> this working.
>
> I'm fairly good with make, and have time to work on this.
>
> Let me know the scratch branch name.


Actually, it turned out not to be so difficult. I didn't know about
eval in make, but once I'd worked that out, the changes were quite
minimal.

It's on scratch/multi-level-test-makefile

My solution is currently limited to a fixed number of subdirs (i.e it's
not fully recursive), but then that isn't like to change quickly, if at
all. I'd welcome feedback, though.

The proposed directory layout is inside a file called emacs-tests.org at
top level (obviously not where I intend to leave it). I've accepted most
of the changes people suggested except for one. Some one suggested to
but tests of C core ~test/automated/src~ which sounded good. But
thinking about it, it probably makes more sense to leave this for C
level testing framework. So now we have

test/automated/lisp
test/automated/lisp/emacs-lisp
test/automated/lisp/progmodes (and so on....)
test/automated/lisp/c (lisp tests for the C core)

test/automated/lisp/legacy (ones which don't fit, to be fixed)

test/automated/src (empty but reserved for C tests of C core)


There's a shell-script embedded in emacs-tests.org to actually apply the
move. As we are in feature freeze, I'd plan to do this immediately after
the release branch is ready.

Comments welcome.

Phil

Re: Location of tests (again) (was Re: Makefile-help)

[Juanma Barranquero]

[-- Attachment #1: Type: text/plain, Size: 818 bytes --]

On Sat, Nov 7, 2015 at 11:55 AM, Phillip Lord <phillip.lord@russet.org.uk>
wrote:

> Some one suggested to
> but tests of C core ~test/automated/src~ which sounded good. But
> thinking about it, it probably makes more sense to leave this for C
> level testing framework. So now we have
>
> test/automated/lisp
> test/automated/lisp/emacs-lisp
> test/automated/lisp/progmodes (and so on....)
> test/automated/lisp/c (lisp tests for the C core)

This breaks the symmetry between our source layout and the test layout.
And, of course, if we some day add a C testing framework there's no reason
why lisp tests and C tests cannot share test/automated/c.

In fact, it seems to me like a nice property that if we have both C and
elisp tests for, say, keymap.c, then keymap-tests.c and keymap-tests.el are
in close proximity.

[-- Attachment #2: Type: text/html, Size: 1022 bytes --]

Re: Location of tests (again) (was Re: Makefile-help)

[Phillip Lord]

Juanma Barranquero <lekktu@gmail.com> writes:

> On Sat, Nov 7, 2015 at 11:55 AM, Phillip Lord <phillip.lord@russet.org.uk>
> wrote:
>
>> Some one suggested to
>> but tests of C core ~test/automated/src~ which sounded good. But
>> thinking about it, it probably makes more sense to leave this for C
>> level testing framework. So now we have
>>
>> test/automated/lisp
>> test/automated/lisp/emacs-lisp
>> test/automated/lisp/progmodes (and so on....)
>> test/automated/lisp/c (lisp tests for the C core)
>
> This breaks the symmetry between our source layout and the test layout.

It does, and I think this is unfortunate.

> And, of course, if we some day add a C testing framework there's no reason
> why lisp tests and C tests cannot share test/automated/c.
>
> In fact, it seems to me like a nice property that if we have both C and
> elisp tests for, say, keymap.c, then keymap-tests.c and keymap-tests.el are
> in close proximity.

I thought about that. Obviously it makes the Makefile harder (since you
need one that can cope with both the C framework and the Lisp
framework), which is why I though to keep things separate. This seemed
worth breaking the symmetry.

Very open to persuasion.

Phil

Re: Location of tests (again) (was Re: Makefile-help)

[Juanma Barranquero]

[-- Attachment #1: Type: text/plain, Size: 593 bytes --]

On Sat, Nov 7, 2015 at 12:22 PM, Phillip Lord <phillip.lord@russet.org.uk>
wrote:

> I thought about that. Obviously it makes the Makefile harder (since you
> need one that can cope with both the C framework and the Lisp
> framework), which is why I though to keep things separate.

Well, Emacs isn't a project known by the simplicity of its makefiles ;-)

> Very open to persuasion.

We don't have a C testing framework, and we don't know when, or if, we'll
have one. I'd say worrying about how complex the makefiles will be for that
case is a bit premature.

Just my 0.02€

[-- Attachment #2: Type: text/html, Size: 797 bytes --]

Re: Location of tests (again) (was Re: Makefile-help)

[Eli Zaretskii]

> From: phillip.lord@russet.org.uk (Phillip Lord)
> Date: Sat, 07 Nov 2015 11:22:04 +0000
> Cc: Stephen Leake <stephen_leake@stephe-leake.org>,
> 	Emacs developers <emacs-devel@gnu.org>
> 
> Juanma Barranquero <lekktu@gmail.com> writes:
> 
> > On Sat, Nov 7, 2015 at 11:55 AM, Phillip Lord <phillip.lord@russet.org.uk>
> > wrote:
> >
> >> Some one suggested to
> >> but tests of C core ~test/automated/src~ which sounded good. But
> >> thinking about it, it probably makes more sense to leave this for C
> >> level testing framework. So now we have
> >>
> >> test/automated/lisp
> >> test/automated/lisp/emacs-lisp
> >> test/automated/lisp/progmodes (and so on....)
> >> test/automated/lisp/c (lisp tests for the C core)
> >
> > This breaks the symmetry between our source layout and the test layout.
> 
> It does, and I think this is unfortunate.

Sorry, I don't understand why not use the simpler

 test/automated/src (tests for the C core)
 test/automated/lisp
 test/automated/lisp/emacs-lisp
 ...

> > In fact, it seems to me like a nice property that if we have both C and
> > elisp tests for, say, keymap.c, then keymap-tests.c and keymap-tests.el are
> > in close proximity.
> 
> I thought about that. Obviously it makes the Makefile harder (since you
> need one that can cope with both the C framework and the Lisp
> framework), which is why I though to keep things separate. This seemed
> worth breaking the symmetry.

We don't have any C framework, and I very much doubt we will any time
soon.  (I don't even think it's feasible, since Emacs is not a
library, and doesn't provide a way of exercising it from a C program.)
We can move files again if and when we do, just like we intend to move
files now.  OTOH, breaking the symmetry on behalf of a feature we
don't have doesn't sound right to me.

Re: Location of tests (again) (was Re: Makefile-help)

[Phillip Lord]

Juanma Barranquero <lekktu@gmail.com> writes:

> On Sat, Nov 7, 2015 at 12:22 PM, Phillip Lord <phillip.lord@russet.org.uk>
> wrote:
>
>> I thought about that. Obviously it makes the Makefile harder (since you
>> need one that can cope with both the C framework and the Lisp
>> framework), which is why I though to keep things separate.
>
> Well, Emacs isn't a project known by the simplicity of its makefiles ;-)
>
>> Very open to persuasion.
>
> We don't have a C testing framework, and we don't know when, or if, we'll
> have one. I'd say worrying about how complex the makefiles will be for that
> case is a bit premature.
>
> Just my 0.02€


Okay, well that's two voices, so I am persuaded. The changes have been
pushed.

Phil

Re: Location of tests (again) (was Re: Makefile-help)

[Stephen Leake]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: phillip.lord@russet.org.uk (Phillip Lord)
>> I thought about that. Obviously it makes the Makefile harder (since you
>> need one that can cope with both the C framework and the Lisp
>> framework), which is why I though to keep things separate. This seemed
>> worth breaking the symmetry.
>
> We don't have any C framework, and I very much doubt we will any time
> soon.  (I don't even think it's feasible, since Emacs is not a
> library, and doesn't provide a way of exercising it from a C program.)
> We can move files again if and when we do, just like we intend to move
> files now.  OTOH, breaking the symmetry on behalf of a feature we
> don't have doesn't sound right to me.

+1

-- 
-- Stephe

Re: Location of tests (again) (was Re: Makefile-help)

[Przemysław Wojnowski]

phillip.lord@russet.org.uk (Phillip Lord) writes:

> test/automated/lisp
> test/automated/lisp/emacs-lisp
> test/automated/lisp/progmodes (and so on....)
> test/automated/lisp/c (lisp tests for the C core)
> test/automated/lisp/legacy (ones which don't fit, to be fixed)
> test/automated/src (empty but reserved for C tests of C core)

Sorry to comment on this so late, but it wouldn't it make sense to move
automated tests to directly under "test/" ? "automated/" here looks a
bit artificial to me.

Sooner or later all tests will be automated anyway.

Re: Location of tests (again) (was Re: Makefile-help)

[Phillip Lord]

Przemysław Wojnowski <esperanto@cumego.com> writes:

> phillip.lord@russet.org.uk (Phillip Lord) writes:
>
>> test/automated/lisp
>> test/automated/lisp/emacs-lisp
>> test/automated/lisp/progmodes (and so on....)
>> test/automated/lisp/c (lisp tests for the C core)
>> test/automated/lisp/legacy (ones which don't fit, to be fixed)
>> test/automated/src (empty but reserved for C tests of C core)
>
> Sorry to comment on this so late,

No worries. I'm not going to make this move now, anyway, given that we
are approaching feature freeze (technically, I guess, tests are not a
feature but still...).

> but it wouldn't it make sense to move automated tests to directly
> under "test/" ? "automated/" here looks a bit artificial to me.
>
> Sooner or later all tests will be automated anyway.

I doubt the latter is true (simply because some display related tests
will always involve looking at emacs).

Having said that, I largely agree with you. We'd have something like?

test/lisp
test/src
test/non-ert

Other names which leap to mind:

test/non-automated
test/manual

Phil

Re: Location of tests (again) (was Re: Makefile-help)

[Przemysław Wojnowski]

phillip.lord@russet.org.uk (Phillip Lord) writes:
[...]
>> Sooner or later all tests will be automated anyway.
>
> I doubt the latter is true (simply because some display related tests
> will always involve looking at emacs).
Ok, "all" was exaggerated a bit, but most things can be tested
automatically.

> Having said that, I largely agree with you. We'd have something like?
>
> test/lisp
> test/src
[...]
> test/manual
I'm for "test/manual" here, but other names are fine too.

(I would also change "test/src/" to "test/c", because "lisp" are sources
too, after all. But I don't insist on that, especially that it may result
in a flood here. ;-) )

Re: Location of tests (again) (was Re: Makefile-help)

[John Wiegley]

>>>>> Przemysław Wojnowski <esperanto@cumego.com> writes:

>> test/lisp
>> test/src
> [...]
>> test/manual
> I'm for "test/manual" here, but other names are fine too.

I like this structure.

John

bikeshedding (was Re: Location of tests (again) (was Re: Makefile-help))

[Stephen Leake]

Przemysław Wojnowski <esperanto@cumego.com> writes:

> (I would also change "test/src/" to "test/c", because "lisp" are sources
> too, after all. But I don't insist on that, especially that it may result
> in a flood here. ;-) )

Only if you also change <root>/src to <root>/c.

Or to:

<root>
    src/
        src/lisp
        src/c

or just include lisp and C in one src/ directory; is that really a problem?

-- 
-- Stephe

Re: Location of tests (again) (was Re: Makefile-help)

[Phillip Lord]

Przemysław Wojnowski <esperanto@cumego.com> writes:

> phillip.lord@russet.org.uk (Phillip Lord) writes:
> [...]
>>> Sooner or later all tests will be automated anyway.
>>
>> I doubt the latter is true (simply because some display related tests
>> will always involve looking at emacs).
> Ok, "all" was exaggerated a bit, but most things can be tested
> automatically.
>
>> Having said that, I largely agree with you. We'd have something like?
>>
>> test/lisp
>> test/src
> [...]
>> test/manual
> I'm for "test/manual" here, but other names are fine too.
>
> (I would also change "test/src/" to "test/c", because "lisp" are sources
> too, after all. But I don't insist on that, especially that it may result
> in a flood here. ;-) )


I suggested that -- personally, I don't like having "src" in "test"
because "src" is, well, source, not test. But the general feeling was
that keeping it symmetrical makes more sense.

Either way, it answers my use-case of a deterministic link between
source and test.

Phil

UI tests

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Unlike the typical graphical applications, Emacs's UIs are mostly
controlled by events that have a very shallow translation to Lisp
data.  Even mouse clicks can be faked, and a convenient Lisp function
to make that totally easy would not be hard to write.

Perhaps the input side of testing UI features will be fairly easy.

The output side is not so straightforward, but if we set up something
to accumulate a sort of dribble record of X11 output operations, in
the form of Lisp-visible data, maybe tests could check that to see
if they have changed from before.


-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: UI tests

[John Wiegley]

>>>>> Richard Stallman <rms@gnu.org> writes:

> The output side is not so straightforward, but if we set up something to
> accumulate a sort of dribble record of X11 output operations, in the form of
> Lisp-visible data, maybe tests could check that to see if they have changed
> from before.

Ah, I hadn't considered using output deltas as a way of testing. Good idea!

John

Re: UI tests

[Phillip Lord]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
> Unlike the typical graphical applications, Emacs's UIs are mostly
> controlled by events that have a very shallow translation to Lisp
> data.  Even mouse clicks can be faked, and a convenient Lisp function
> to make that totally easy would not be hard to write.

I would agree.

I've been working on a library (called sisyphus) to support higher-level
operations for testing, and that would be a nice addition.


> Perhaps the input side of testing UI features will be fairly easy.
>
> The output side is not so straightforward, but if we set up something
> to accumulate a sort of dribble record of X11 output operations, in
> the form of Lisp-visible data, maybe tests could check that to see
> if they have changed from before.

If that would allow me to test things like before and after-string
overlay properties that would be excellent. Testing, for example,
completion engines is hard in Emacs since the buffer doesn't change,
only the visualisation of it.

One additional problem, though, is that Emacs doesn't really have a
concept of "headless". Things like syntax highlighting in buffers just
work different when running in batch, because Emacs assumes it's running
on a TTY. For example, HTML generated by htmlize comes out differently
in batch, than from a windowed Emacs, because Emacs in batch cannot
display colour.

Phil

Re: UI tests

[Alan Mackenzie]

Hello, Phillip.

On Thu, Nov 12, 2015 at 02:09:32PM +0000, Phillip Lord wrote:

[ .... ]

> One additional problem, though, is that Emacs doesn't really have a
> concept of "headless". Things like syntax highlighting in buffers just
> work different when running in batch, because Emacs assumes it's running
> on a TTY. For example, HTML generated by htmlize comes out differently
> in batch, than from a windowed Emacs, because Emacs in batch cannot
> display colour.

I'm not sure whether or not this is relevant, but Emacs can be duped
into doing font-locking in batch mode.  Please examine the following
function (out of CC Mode's 000tests.el), and see if there's anything
in it you could use:

(defun cc-test-force-font-lock-buffer ()
  ;; Try to forcibly font lock the current buffer, even in batch mode.
  ;; We're doing really dirty things to trick font-lock into action in
  ;; batch mode in the different emacsen.
  (let ((orig-noninteractive-function
         (and (fboundp 'noninteractive)
              (symbol-function 'noninteractive)))
        (orig-noninteractive-variable
         (and (boundp 'noninteractive)
              (symbol-value 'noninteractive)))
        ;; font-lock in XEmacs 19 looks at a variable named `noninteractive'.
        (noninteractive nil))
    (unwind-protect
        (progn
          (when orig-noninteractive-function
            ;; XEmacs (at least 21.4) calls `noninteractive' to check
            ;; for batch mode, so we let it lie.
            (fset 'noninteractive (lambda () nil)))
          (font-lock-mode 1)
          (unless (or (get-text-property (point-min) 'face)
                      (next-single-property-change (point-min) 'face))
            ;; Some emacsen have already fontified the buffer above,
            ;; but others need some more coercion..
            (let (;; Avoid getting some lazy fontification package that
                  ;; might decide that nothing should be done.
                  (font-lock-fontify-buffer-function
                   'font-lock-default-fontify-buffer))
              (font-lock-fontify-buffer))))
      (when orig-noninteractive-function
        (fset 'noninteractive orig-noninteractive-function)))))


> Phil

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: UI tests

[Eli Zaretskii]

> From: phillip.lord@russet.org.uk (Phillip Lord)
> Date: Thu, 12 Nov 2015 14:09:32 +0000
> Cc: stephen_leake@stephe-leake.org, esperanto@cumego.com, emacs-devel@gnu.org
> 
> One additional problem, though, is that Emacs doesn't really have a
> concept of "headless". Things like syntax highlighting in buffers just
> work different when running in batch, because Emacs assumes it's running
> on a TTY. For example, HTML generated by htmlize comes out differently
> in batch, than from a windowed Emacs, because Emacs in batch cannot
> display colour.

Most GUI APIs (like Xlib etc.) won't work without a real display and a
window (a.k.a. "frame"), so if we want a "headless" Emacs that will
work as with GUI frames, we will have to emulate all those calls.  Not
for a faint at heart, I'd say.

Re: UI tests

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > One additional problem, though, is that Emacs doesn't really have a
  > concept of "headless".

We could implement a fake terminal of any desired type and use
it to test display for real terminals of that type.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: UI tests

[Przemysław Wojnowski]

Richard Stallman <rms@gnu.org> writes:
> Unlike the typical graphical applications, Emacs's UIs are mostly
> controlled by events that have a very shallow translation to Lisp
> data.  Even mouse clicks can be faked, and a convenient Lisp function
> to make that totally easy would not be hard to write.
>
> Perhaps the input side of testing UI features will be fairly easy.
>
> The output side is not so straightforward, but if we set up something
> to accumulate a sort of dribble record of X11 output operations, in
> the form of Lisp-visible data, maybe tests could check that to see
> if they have changed from before.

I'm not sure what do you mean by "output side" here?

My understanding is that inputs and outputs are on system's boundaries
and isolate system's functionality using stable interfaces (common in
different architectural pattern - e.g. [1], [2]).

In Emacs, as you wrote, user actions can be faked and that can be used
in tests.

The other side - the outputs - is not that clear to me. Usually the
outputs are hidden under stable interfaces. The interfaces allow to
exchange implementations without affecting the system and the boundary
is used to test the output of the system. In case of display output,
this won't allow to test every single pixel (so no X11 output), but at
least it will allow to test that the system produces correct output for
display, no matter what that display (its underlying library) is.

Two things to note when introducing automated tests:
1. Tests pyramid [3] - indicates which tests are most important and
ratio between them. TL;DR Emacs should focus on unit tests.

2. Ice Cream Cone anti-pattern [3,4] is a common trap for existing
projects that start to add automated tests.

[1] Layers in "Pattern Oriented Software Architecture vol 1"
[2] http://alistair.cockburn.us/Hexagonal+architecture
[3] http://googletesting.blogspot.co.uk/2015/04/just-say-no-to-more-end-to-end-tests.html
(from "The True Value of Tests")
[4] http://watirmelon.com/2012/01/31/introducing-the-software-testing-ice-cream-cone/

Re: UI tests

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I'm not sure what do you mean by "output side" here?

I mean anything that shows on the screen.

When it comes to testing the user interface at a lower
level, we would have to test what it does to the screen.


-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Locations of Tests

[Phillip Lord]

John Wiegley <jwiegley@gmail.com> writes:

>>>>>> Artur Malabarba <bruce.connor.am@gmail.com> writes:
>
>> Same here. Just make sure it doesn't mess up git blame. ;-)
>
>>From *Man git-blame*:
>
>        The origin of lines is automatically followed across whole-file renames (currently
>        there is no option to turn the rename-following off). To follow lines moved from
>        one file to another, or to follow lines that were copied and pasted from another
>        file, etc., see the -C and -M options.
>
> John

Okay, so I have moved all the test files according to the scheme
discussed, and pushed to branch feature/standard-test-location.

I wasn't able to do this in a single commit -- the move was one, so
hopefully git should have tracked the file moves. Then, I updated the
makefiles, fixed the tests that depend on their location. Finally, I had
to restore a couple of manual (non automake) makefiles which I
accidentally removed (very underhand, didn't check the .gitignore).

I'd appreciate if you could have a look at this, then I'll merge to
master.

Phil

Re: Locations of Tests

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 257 bytes --]

On 25 Nov 2015 9:23 am, "Phillip Lord" <phillip.lord@russet.org.uk> wrote:
> I wasn't able to do this in a single commit -- the move was one, so
> hopefully git should have tracked the file moves.

Why not try git blame so we can lose that "hopefully"? :-)

[-- Attachment #2: Type: text/html, Size: 388 bytes --]

Re: Locations of Tests

[Phillip Lord]

Artur Malabarba <bruce.connor.am@gmail.com> writes:

> On 25 Nov 2015 9:23 am, "Phillip Lord" <phillip.lord@russet.org.uk> wrote:
>> I wasn't able to do this in a single commit -- the move was one, so
>> hopefully git should have tracked the file moves.
>
> Why not try git blame so we can lose that "hopefully"? :-)


Potentially...

I did try this already and it seems to be good


d7df36e7 test/automated/elisp-mode-tests.el      (Stephen Leake      2015-08-10 21:53:19 -0500 405)   (list
d7df36e7 test/automated/elisp-mode-tests.el      (Stephen Leake      2015-08-10 21:53:19 -0500 406)    (xref-make "(cl-defgeneric xref-location-marker)"
d7df36e7 test/automated/elisp-mode-tests.el      (Stephen Leake      2015-08-10 21:53:19 -0500 407)           (xref-make-elisp-location
0382fd42 test/automated/elisp-mode-tests.el      (Stephen Leake      2015-08-13 12:54:39 -0500 408)            'xref-location-marker 'cl-defgeneric
6fdd750c test/lisp/progmodes/elisp-mode-tests.el (Phillip Lord       2015-11-24 22:35:23 +0000 409)            (expand-file-name "../../../lisp/progmodes/xref.el" emacs-test-dir)))
d7df36e7 test/automated/elisp-mode-tests.el      (Stephen Leake      2015-08-10 21:53:19 -0500 410)    (xref-make "(cl-defmethod xref-location-marker ((l xref-elisp-location)))"
d7df36e7 test/automated/elisp-mode-tests.el      (Stephen Leake      2015-08-10 21:53:19 -0500 411)           (xref-make-elisp-location
d7df36e7 test/automated/elisp-mode-tests.el      (Stephen Leake      2015-08-10 21:53:19 -0500 412)            '(xref-location-marker xref-elisp-location) 'cl-defmethod
6fdd750c test/lisp/progmodes/elisp-mode-tests.el (Phillip Lord       2015-11-24 22:35:23 +0000 413)            (expand-file-name "../../../lisp/progmodes/elisp-mode.el" emacs-test-dir)))
d7df36e7 test/automated/elisp-mode-tests.el      (Stephen Leake      2015-08-10 21:53:19 -0500 414)    (xref-make "(cl-defmethod xref-location-marker ((l xref-file-location)))"
d7df36e7 test/automated/elisp-mode-tests.el      (Stephen Leake      2015-08-10 21:53:19 -0500 415)           (xref-make-elisp-location
d7df36e7 test/automated/elisp-mode-tests.el      (Stephen Leake      2015-08-10 21:53:19 -0500 416)            '(xref-location-marker xref-file-location) 'cl-defmethod


I fear this has failed for the four Makefile.

./lisp/progmodes/flymake-resources/Makefile
./manual/indent/Makefile
./manual/etags/Makefile
./manual/etags/make-src/Makefile

which all appear to have been written by me. The "real" makefile
(Makefile.in) is good.

Phil