all messages for Emacs-related lists mirrored at yhetil.org
 help / color / mirror / code / Atom feed
* Locations of Tests
@ 2015-11-04 17:36 Phillip Lord
  2015-11-04 17:53 ` John Wiegley
  2015-11-06 10:02 ` Makefile-help (was Re: Locations of Tests) Phillip Lord
  0 siblings, 2 replies; 43+ messages in thread
From: Phillip Lord @ 2015-11-04 17:36 UTC (permalink / raw)
  To: emacs-devel



I was thinking that a "jump to tests" feature would be nice for
emacs-lisp mode the other day. While I had a quick thing about how to
implement this, I looked at the emacs source, and noticed that the
location of automated tests is a little erratic.

So, we have thingatpt.el (as a test), file-notify-tests.el (for
filenotify). There is also no reflection of the organisation of the lisp
in the tests. So, elisp-mode-tests.el at top level, while elisp-mode.el
is in lisp/progmodes. Similar issue with gnus-test.el and pcase.el.

Is there an organisation for tests that I am unaware off? Would there be
interest in a policy based on file name?

Phil



^ permalink raw reply	[flat|nested] 43+ messages in thread

* Re: Locations of Tests
  2015-11-04 17:36 Locations of Tests Phillip Lord
@ 2015-11-04 17:53 ` John Wiegley
  2015-11-04 18:14   ` Artur Malabarba
  2015-11-04 19:07   ` Michael Albinus
  2015-11-06 10:02 ` Makefile-help (was Re: Locations of Tests) Phillip Lord
  1 sibling, 2 replies; 43+ messages in thread
From: John Wiegley @ 2015-11-04 17:53 UTC (permalink / raw)
  To: Phillip Lord; +Cc: emacs-devel

>>>>> Phillip Lord <phillip.lord@russet.org.uk> writes:

> So, we have thingatpt.el (as a test), file-notify-tests.el (for filenotify).
> There is also no reflection of the organisation of the lisp in the tests.
> So, elisp-mode-tests.el at top level, while elisp-mode.el is in
> lisp/progmodes. Similar issue with gnus-test.el and pcase.el.
> 
> Is there an organisation for tests that I am unaware off? Would there be
> interest in a policy based on file name?

Huge interest. There should be a purely mechanical way of going from a source
file in core, to the set of tests for that file. Then we could add M-x
emacs-lisp-jump-to-test.

I'd happily accept a patch to move things into accordance with the naming
structure under lisp/.

John



^ permalink raw reply	[flat|nested] 43+ messages in thread

* Re: Locations of Tests
  2015-11-04 17:53 ` John Wiegley
@ 2015-11-04 18:14   ` Artur Malabarba
  2015-11-04 18:21     ` John Wiegley
  2015-11-04 19:15     ` Steinar Bang
  2015-11-04 19:07   ` Michael Albinus
  1 sibling, 2 replies; 43+ messages in thread
From: Artur Malabarba @ 2015-11-04 18:14 UTC (permalink / raw)
  To: emacs-devel, Phillip Lord

[-- Attachment #1: Type: text/plain, Size: 228 bytes --]

On 4 Nov 2015 5:53 pm, "John Wiegley" <jwiegley@gmail.com> wrote:
> I'd happily accept a patch to move things into accordance with the naming
> structure under lisp/.

Same here. Just make sure it doesn't mess up git blame. ;-)

[-- Attachment #2: Type: text/html, Size: 344 bytes --]

^ permalink raw reply	[flat|nested] 43+ messages in thread

* Re: Locations of Tests
  2015-11-04 18:14   ` Artur Malabarba
@ 2015-11-04 18:21     ` John Wiegley
  2015-11-25  9:23       ` Phillip Lord
  2015-11-04 19:15     ` Steinar Bang
  1 sibling, 1 reply; 43+ messages in thread
From: John Wiegley @ 2015-11-04 18:21 UTC (permalink / raw)
  To: Artur Malabarba; +Cc: Phillip Lord, emacs-devel

>>>>> Artur Malabarba <bruce.connor.am@gmail.com> writes:

> Same here. Just make sure it doesn't mess up git blame. ;-)

From *Man git-blame*:

       The origin of lines is automatically followed across whole-file renames (currently
       there is no option to turn the rename-following off). To follow lines moved from
       one file to another, or to follow lines that were copied and pasted from another
       file, etc., see the -C and -M options.

John



^ permalink raw reply	[flat|nested] 43+ messages in thread

* Re: Locations of Tests
  2015-11-04 17:53 ` John Wiegley
  2015-11-04 18:14   ` Artur Malabarba
@ 2015-11-04 19:07   ` Michael Albinus
  2015-11-04 19:15     ` John Wiegley
  1 sibling, 1 reply; 43+ messages in thread
From: Michael Albinus @ 2015-11-04 19:07 UTC (permalink / raw)
  To: emacs-devel

John Wiegley <jwiegley@gmail.com> writes:

>> So, we have thingatpt.el (as a test), file-notify-tests.el (for filenotify).
>> There is also no reflection of the organisation of the lisp in the tests.
>> So, elisp-mode-tests.el at top level, while elisp-mode.el is in
>> lisp/progmodes. Similar issue with gnus-test.el and pcase.el.
>> 
>> Is there an organisation for tests that I am unaware off? Would there be
>> interest in a policy based on file name?
>
> Huge interest. There should be a purely mechanical way of going from a source
> file in core, to the set of tests for that file. Then we could add M-x
> emacs-lisp-jump-to-test.
>
> I'd happily accept a patch to move things into accordance with the naming
> structure under lisp/.

Before we rename everything, wouldn't a simple directive in the source
file do the job?

;; Package-Tests: test/automated/file-notify-tests.el

I don't know how much of the path name shall be included there. We have
also files outside Emacs core, for example in Elpa. Here the path name
could start at the package root directory:

;; Package-Tests: tests/stream-tests.el

> John

Best regards, Michael.



^ permalink raw reply	[flat|nested] 43+ messages in thread

* Re: Locations of Tests
  2015-11-04 18:14   ` Artur Malabarba
  2015-11-04 18:21     ` John Wiegley
@ 2015-11-04 19:15     ` Steinar Bang
  2015-11-04 19:23       ` David Kastrup
  1 sibling, 1 reply; 43+ messages in thread
From: Steinar Bang @ 2015-11-04 19:15 UTC (permalink / raw)
  To: emacs-devel

>>>>> Artur Malabarba <bruce.connor.am@gmail.com>:
> On 4 Nov 2015 5:53 pm, "John Wiegley" <jwiegley@gmail.com> wrote:

>> I'd happily accept a patch to move things into accordance with the naming
>> structure under lisp/.

> Same here. Just make sure it doesn't mess up git blame. ;-)

...and file history.

What this means in practice, is that files should be moved verbatim.

You can change files once they are in place in the new location, but the
first commit in the new location should be of files with identical SHA1
hashes as the files in the old location.

git tracks history across moves/renames by matching the file content
SHA1 hashes of files in other locations.




^ permalink raw reply	[flat|nested] 43+ messages in thread

* Re: Locations of Tests
  2015-11-04 19:07   ` Michael Albinus
@ 2015-11-04 19:15     ` John Wiegley
  2015-11-04 21:26       ` Phillip Lord
  0 siblings, 1 reply; 43+ messages in thread
From: John Wiegley @ 2015-11-04 19:15 UTC (permalink / raw)
  To: Michael Albinus; +Cc: emacs-devel

>>>>> Michael Albinus <michael.albinus@gmx.de> writes:

> Before we rename everything, wouldn't a simple directive in the source file
> do the job?

> ;; Package-Tests: test/automated/file-notify-tests.el

That would also work, but having a naming structure also makes it easier for
contributors to know where new files they create should go. So I'd prefer to
have a convention, rather than make it freeform, with a set of textual links
to relate the two sets.

John



^ permalink raw reply	[flat|nested] 43+ messages in thread

* Re: Locations of Tests
  2015-11-04 19:15     ` Steinar Bang
@ 2015-11-04 19:23       ` David Kastrup
  2015-11-04 21:33         ` Phillip Lord
  0 siblings, 1 reply; 43+ messages in thread
From: David Kastrup @ 2015-11-04 19:23 UTC (permalink / raw)
  To: emacs-devel

Steinar Bang <sb@dod.no> writes:

>>>>>> Artur Malabarba <bruce.connor.am@gmail.com>:
>> On 4 Nov 2015 5:53 pm, "John Wiegley" <jwiegley@gmail.com> wrote:
>
>>> I'd happily accept a patch to move things into accordance with the naming
>>> structure under lisp/.
>
>> Same here. Just make sure it doesn't mess up git blame. ;-)
>
> ...and file history.
>
> What this means in practice, is that files should be moved verbatim.
>
> You can change files once they are in place in the new location, but the
> first commit in the new location should be of files with identical SHA1
> hashes as the files in the old location.
>
> git tracks history across moves/renames by matching the file content
> SHA1 hashes of files in other locations.

That's what always works, but git blame's -C option will try getting
stuff matched even if you don't properly rename.  But -C is expensive.
And it's always a nuisance to supply the like of C-x v g with additional
options.

-- 
David Kastrup



^ permalink raw reply	[flat|nested] 43+ messages in thread

* Re: Locations of Tests
  2015-11-04 19:15     ` John Wiegley
@ 2015-11-04 21:26       ` Phillip Lord
  2015-11-05 13:41         ` Stephen Leake
  0 siblings, 1 reply; 43+ messages in thread
From: Phillip Lord @ 2015-11-04 21:26 UTC (permalink / raw)
  To: emacs-devel; +Cc: johnw, Michael Albinus

[-- Attachment #1: Type: text/plain, Size: 1229 bytes --]



John Wiegley <jwiegley@gmail.com> writes:

>>>>>> Michael Albinus <michael.albinus@gmx.de> writes:
>
>> Before we rename everything, wouldn't a simple directive in the source file
>> do the job?
>
>> ;; Package-Tests: test/automated/file-notify-tests.el
>
> That would also work, but having a naming structure also makes it easier for
> contributors to know where new files they create should go. So I'd prefer to
> have a convention, rather than make it freeform, with a set of textual links
> to relate the two sets.

I'd tend to agree.

I think I'd like to get a specification sorted first before making the
changes. I've included my starter-for-ten below. I'm willing to take any
feedback here -- or I could add it to the emacs-repo as an orphan
branch, then people could change directly.

I've given locations for test files, and resources (data files). This is
both for Emacs-lisp source files and for lisp tests for features
provided in C (inotify-test.el for example). There are also suggestions
for ELPA packages.

I've said nothing about C tests for the C core, because I know nothing
about C testing (and almost nothing about C coding!). Adding a
buffer-local as suggested above as a legacy might make sense.

Phil


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: emacs-tests.org --]
[-- Type: text/x-org, Size: 2276 bytes --]


#+TITLE: The Location of Emacs-Lisp Tests

* Introduction

In this document, we describe the relationship between Emacs-Lisp files and
their associated automated test files.


* In Emacs

The Emacs repository contains a very large number of Emacs-Lisp files, many of
which pre-date both formal package support for Emacs and automated unit
testing. The test layout is, therefore, somewhat different than for ELPA
packages.

All paths are relative to the Emacs root directory.

** Source

Lisp files are stored in the ~lisp~ directory or its sub-directories.
Sub-directories are in many cases themed after packages (~gnus~, ~org~,
~calc~), related functionality (~net~, ~emacs-lisp~, ~progmodes~) or status
(~obsolete~).

C source is stored in the ~src~ directory, which is flat.

** Test Files

Automated tests should be stored in the ~test/automated~ directory. Tests
should reflect the directory structure of the source tree; so tests for files
in the ~emacs-lisp~ source directory should reside in the
~test/automated/emacs-lisp~ directory.

Tests should normally reside in a file with an ~-test~ added to the name of
the tested source file; hence ~ert.el~ is tested in ~ert-test.el~, or
~pcase.el~ is tested in ~pcase-test.el~.

Where features of the C source are tested using Emacs-Lisp test files, these
should reside in ~/test/automated/c~ and be named after the C file.

** Resource Files

Resource files for tests (containing test data) should reside in a directory
named after the feature with a ~-resources~ suffix. Hence, tests files for
~flymake.el~ should reside in a directory called ~flymake-resources~.

No guidance is given for the organisation of resource files inside the
~-resource~ directory; files can be organised at the author's discretion.


* In ELPA

All paths are given relative to the package root.

** Source Files

ELPA lisp files should be stored at top-level within the package.

** Test Files

Test files should normally reside in the ~test~ directory, and be named after
the file being tested.

** Resources Files

Resource files for tests should reside in the ~dev-resources~ directory. No
guidance is given for the organisation of resource files inside the
~dev-resource~ directory; files can be organised at the author's discretion.





^ permalink raw reply	[flat|nested] 43+ messages in thread

* Re: Locations of Tests
  2015-11-04 19:23       ` David Kastrup
@ 2015-11-04 21:33         ` Phillip Lord
  0 siblings, 0 replies; 43+ messages in thread
From: Phillip Lord @ 2015-11-04 21:33 UTC (permalink / raw)
  To: emacs-devel

David Kastrup <dak@gnu.org> writes:

> Steinar Bang <sb@dod.no> writes:
>
>>>>>>> Artur Malabarba <bruce.connor.am@gmail.com>:
>>> On 4 Nov 2015 5:53 pm, "John Wiegley" <jwiegley@gmail.com> wrote:
>>
>>>> I'd happily accept a patch to move things into accordance with the naming
>>>> structure under lisp/.
>>
>>> Same here. Just make sure it doesn't mess up git blame. ;-)
>>
>> ...and file history.
>>
>> What this means in practice, is that files should be moved verbatim.
>>
>
> That's what always works, but git blame's -C option will try getting
> stuff matched even if you don't properly rename.  But -C is expensive.
> And it's always a nuisance to supply the like of C-x v g with additional
> options.


Okay, so I hearby publically state that if I become responsible for
another git mega-thread, I will serve my penance by giving up Emacs and
moving to Ed for all future text manipulation needs.

Phil




^ permalink raw reply	[flat|nested] 43+ messages in thread

* Re: Locations of Tests
  2015-11-04 21:26       ` Phillip Lord
@ 2015-11-05 13:41         ` Stephen Leake
  2015-11-05 13:59           ` Artur Malabarba
                             ` (2 more replies)
  0 siblings, 3 replies; 43+ messages in thread
From: Stephen Leake @ 2015-11-05 13:41 UTC (permalink / raw)
  To: Phillip Lord; +Cc: johnw, Michael Albinus, emacs-devel

phillip.lord@russet.org.uk (Phillip Lord) writes:

> John Wiegley <jwiegley@gmail.com> writes:
>
>
> #+TITLE: The Location of Emacs-Lisp Tests

This looks good in general; a couple minor comments below.

> * Introduction
>
> In this document, we describe the relationship between Emacs-Lisp files and
> their associated automated test files.
>
>
> * In Emacs
>
> The Emacs repository contains a very large number of Emacs-Lisp files, many of
> which pre-date both formal package support for Emacs and automated unit
> testing. The test layout is, therefore, somewhat different than for ELPA
> packages.
>
> All paths are relative to the Emacs root directory.
>
> ** Source
>
> Lisp files are stored in the ~lisp~ directory or its sub-directories.
> Sub-directories are in many cases themed after packages (~gnus~, ~org~,
> ~calc~), related functionality (~net~, ~emacs-lisp~, ~progmodes~) or status
> (~obsolete~).
>
> C source is stored in the ~src~ directory, which is flat.
>
> ** Test Files
>
> Automated tests should be stored in the ~test/automated~ directory. Tests
> should reflect the directory structure of the source tree; so tests for files
> in the ~emacs-lisp~ source directory should reside in the
> ~test/automated/emacs-lisp~ directory.
>
> Tests should normally reside in a file with an ~-test~ added to the name of
> the tested source file; hence ~ert.el~ is tested in ~ert-test.el~, or
> ~pcase.el~ is tested in ~pcase-test.el~.
>
> Where features of the C source are tested using Emacs-Lisp test files, these
> should reside in ~/test/automated/c~ and be named after the C file.

Since we are trying to mimic the source layout, perhaps it would be
better to have:

tests/automated/src  ;; tests of files in the src dir

tests/automated/lisp ;; tests of file in the lisp dir

tests/automated/lisp/prog-modes ;; etc

> ** Resource Files
>
> Resource files for tests (containing test data) should reside in a directory
> named after the feature with a ~-resources~ suffix. Hence, tests files for
> ~flymake.el~ should reside in a directory called ~flymake-resources~.
>
> No guidance is given for the organisation of resource files inside the
> ~-resource~ directory; files can be organised at the author's discretion.
>
>
> * In ELPA
>
> All paths are given relative to the package root.
>
> ** Source Files
>
> ELPA lisp files should be stored at top-level within the package.
>
> ** Test Files
>
> Test files should normally reside in the ~test~ directory, and be named after
> the file being tested.

With the -test suffix; it helps to be very clear, 

> ** Resources Files
>
> Resource files for tests should reside in the ~dev-resources~ directory. No
> guidance is given for the organisation of resource files inside the
> ~dev-resource~ directory; files can be organised at the author's discretion.

-- 
-- Stephe



^ permalink raw reply	[flat|nested] 43+ messages in thread

* Re: Locations of Tests
  2015-11-05 13:41         ` Stephen Leake
@ 2015-11-05 13:59           ` Artur Malabarba
  2015-11-05 15:02             ` Nicolas Petton
  2015-11-05 15:27           ` Juanma Barranquero
  2015-11-06  9:55           ` Phillip Lord
  2 siblings, 1 reply; 43+ messages in thread
From: Artur Malabarba @ 2015-11-05 13:59 UTC (permalink / raw)
  To: Stephen Leake; +Cc: emacs-devel, johnw, Michael Albinus, Phillip Lord

>> ** Test Files
>>
>> Automated tests should be stored in the ~test/automated~ directory. Tests
>> should reflect the directory structure of the source tree; so tests for files
>> in the ~emacs-lisp~ source directory should reside in the
>> ~test/automated/emacs-lisp~ directory.
>>
>> Tests should normally reside in a file with an ~-test~ added to the name of
>> the tested source file; hence ~ert.el~ is tested in ~ert-test.el~, or
>> ~pcase.el~ is tested in ~pcase-test.el~.

Then let's also remember to rename all the *-test.el out there to *-test.el



^ permalink raw reply	[flat|nested] 43+ messages in thread

* Re: Locations of Tests
  2015-11-05 13:59           ` Artur Malabarba
@ 2015-11-05 15:02             ` Nicolas Petton
  2015-11-05 15:08               ` Artur Malabarba
  0 siblings, 1 reply; 43+ messages in thread
From: Nicolas Petton @ 2015-11-05 15:02 UTC (permalink / raw)
  To: bruce.connor.am, Stephen Leake
  Cc: johnw, Michael Albinus, Phillip Lord, emacs-devel

[-- Attachment #1: Type: text/plain, Size: 193 bytes --]

Artur Malabarba <bruce.connor.am@gmail.com> writes:

> Then let's also remember to rename all the *-test.el out there to
> *-test.el

hmmm... I'm not following.  Could you rephrase this?

Nico

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

^ permalink raw reply	[flat|nested] 43+ messages in thread

* Re: Locations of Tests
  2015-11-05 15:02             ` Nicolas Petton
@ 2015-11-05 15:08               ` Artur Malabarba
  2015-11-06  9:56                 ` Phillip Lord
  0 siblings, 1 reply; 43+ messages in thread
From: Artur Malabarba @ 2015-11-05 15:08 UTC (permalink / raw)
  To: Nicolas Petton
  Cc: Stephen Leake, johnw, Michael Albinus, Phillip Lord, emacs-devel

[-- Attachment #1: Type: text/plain, Size: 443 bytes --]

On 5 Nov 2015 3:02 pm, "Nicolas Petton" <nicolas@petton.fr> wrote:
>
> Artur Malabarba <bruce.connor.am@gmail.com> writes:
>
> > Then let's also remember to rename all the *-test.el out there to
> > *-test.el
>
> hmmm... I'm not following.  Could you rephrase this?
>
> Nico

Oops. :-) missed an s in there.
I meant  "rename all the *-tests.el to  *-test.el".
Or the other way around, I'm just saying to normalise the use of plural
/singular.

[-- Attachment #2: Type: text/html, Size: 708 bytes --]

^ permalink raw reply	[flat|nested] 43+ messages in thread

* Re: Locations of Tests
  2015-11-05 13:41         ` Stephen Leake
  2015-11-05 13:59           ` Artur Malabarba
@ 2015-11-05 15:27           ` Juanma Barranquero
  2015-11-06  9:55           ` Phillip Lord
  2 siblings, 0 replies; 43+ messages in thread
From: Juanma Barranquero @ 2015-11-05 15:27 UTC (permalink / raw)
  To: Stephen Leake; +Cc: Emacs developers, johnw, Michael Albinus, Phillip Lord

[-- Attachment #1: Type: text/plain, Size: 349 bytes --]

On Thu, Nov 5, 2015 at 2:41 PM, Stephen Leake <
stephen_leake@stephe-leake.org> wrote:

> Since we are trying to mimic the source layout, perhaps it would be
> better to have:
>
> tests/automated/src  ;; tests of files in the src dir
>
> tests/automated/lisp ;; tests of file in the lisp dir
>
> tests/automated/lisp/prog-modes ;; etc

Yes, please.

[-- Attachment #2: Type: text/html, Size: 523 bytes --]

^ permalink raw reply	[flat|nested] 43+ messages in thread

* Re: Locations of Tests
  2015-11-05 13:41         ` Stephen Leake
  2015-11-05 13:59           ` Artur Malabarba
  2015-11-05 15:27           ` Juanma Barranquero
@ 2015-11-06  9:55           ` Phillip Lord
  2 siblings, 0 replies; 43+ messages in thread
From: Phillip Lord @ 2015-11-06  9:55 UTC (permalink / raw)
  To: Stephen Leake; +Cc: johnw, Michael Albinus, emacs-devel

Stephen Leake <stephen_leake@stephe-leake.org> writes:

> phillip.lord@russet.org.uk (Phillip Lord) writes:
>
>> Where features of the C source are tested using Emacs-Lisp test files, these
>> should reside in ~/test/automated/c~ and be named after the C file.
>
> Since we are trying to mimic the source layout, perhaps it would be
> better to have:
>
> tests/automated/src  ;; tests of files in the src dir
>
> tests/automated/lisp ;; tests of file in the lisp dir
>

>> ** Test Files
>>
>> Test files should normally reside in the ~test~ directory, and be named after
>> the file being tested.
>
> With the -test suffix; it helps to be very clear, 
>


Both good calls, with incorporate.



^ permalink raw reply	[flat|nested] 43+ messages in thread

* Re: Locations of Tests
  2015-11-05 15:08               ` Artur Malabarba
@ 2015-11-06  9:56                 ` Phillip Lord
  0 siblings, 0 replies; 43+ messages in thread
From: Phillip Lord @ 2015-11-06  9:56 UTC (permalink / raw)
  To: Artur Malabarba
  Cc: Stephen Leake, Nicolas Petton, johnw, Michael Albinus,
	emacs-devel

Artur Malabarba <bruce.connor.am@gmail.com> writes:

> On 5 Nov 2015 3:02 pm, "Nicolas Petton" <nicolas@petton.fr> wrote:
>>
>> Artur Malabarba <bruce.connor.am@gmail.com> writes:
>>
>> > Then let's also remember to rename all the *-test.el out there to
>> > *-test.el
>>
>> hmmm... I'm not following.  Could you rephrase this?
>>
>> Nico
>
> Oops. :-) missed an s in there.
> I meant  "rename all the *-tests.el to  *-test.el".
> Or the other way around, I'm just saying to normalise the use of plural
> /singular.


Yeah, I checked. "-tests.el" is more common in existing code, so easier
to go that way. Although, it would be good if this worked for other
packages (GNU ELPA and others). I'll check these out also.

Phil



^ permalink raw reply	[flat|nested] 43+ messages in thread

* Makefile-help (was Re: Locations of Tests)
  2015-11-04 17:36 Locations of Tests Phillip Lord
  2015-11-04 17:53 ` John Wiegley
@ 2015-11-06 10:02 ` Phillip Lord
  2015-11-07  6:48   ` Stephen Leake
  1 sibling, 1 reply; 43+ messages in thread
From: Phillip Lord @ 2015-11-06 10:02 UTC (permalink / raw)
  To: emacs-devel

Phillip Lord <phillip.lord@russet.org.uk> writes:

> Is there an organisation for tests that I am unaware off? Would there be
> interest in a policy based on file name?


I've started work on writing a script to move all the files (seems
better than doing it manually as it's more checkable), and I've updated
my specification based on comments; I'll push all of this to a scratch
branch tomorrow.

While doing this, I've realised that I'm going to need to update the
Makefile (which assumes a flat directory structure for tests). 

My make expertise is quite limited. Would anyone be free to help me get
this working.

Phil



^ permalink raw reply	[flat|nested] 43+ messages in thread

* Re: Makefile-help (was Re: Locations of Tests)
  2015-11-06 10:02 ` Makefile-help (was Re: Locations of Tests) Phillip Lord
@ 2015-11-07  6:48   ` Stephen Leake
  2015-11-07 10:55     ` Location of tests (again) (was Re: Makefile-help) Phillip Lord
  0 siblings, 1 reply; 43+ messages in thread
From: Stephen Leake @ 2015-11-07  6:48 UTC (permalink / raw)
  To: Phillip Lord; +Cc: emacs-devel

phillip.lord@russet.org.uk (Phillip Lord) writes:

> Phillip Lord <phillip.lord@russet.org.uk> writes:
>
>> Is there an organisation for tests that I am unaware off? Would there be
>> interest in a policy based on file name?
>
>
> I've started work on writing a script to move all the files (seems
> better than doing it manually as it's more checkable), and I've updated
> my specification based on comments; I'll push all of this to a scratch
> branch tomorrow.
>
> While doing this, I've realised that I'm going to need to update the
> Makefile (which assumes a flat directory structure for tests). 
>
> My make expertise is quite limited. Would anyone be free to help me get
> this working.

I'm fairly good with make, and have time to work on this.

Let me know the scratch branch name.

-- 
-- Stephe



^ permalink raw reply	[flat|nested] 43+ messages in thread

* Location of tests (again) (was Re: Makefile-help)
  2015-11-07  6:48   ` Stephen Leake
@ 2015-11-07 10:55     ` Phillip Lord
  2015-11-07 11:06       ` Juanma Barranquero
  2015-11-10 20:34       ` Przemysław Wojnowski
  0 siblings, 2 replies; 43+ messages in thread
From: Phillip Lord @ 2015-11-07 10:55 UTC (permalink / raw)
  To: Stephen Leake; +Cc: emacs-devel

Stephen Leake <stephen_leake@stephe-leake.org> writes:

> phillip.lord@russet.org.uk (Phillip Lord) writes:
>
>> Phillip Lord <phillip.lord@russet.org.uk> writes:
>>
>>> Is there an organisation for tests that I am unaware off? Would there be
>>> interest in a policy based on file name?
>>
>>
>> I've started work on writing a script to move all the files (seems
>> better than doing it manually as it's more checkable), and I've updated
>> my specification based on comments; I'll push all of this to a scratch
>> branch tomorrow.
>>
>> While doing this, I've realised that I'm going to need to update the
>> Makefile (which assumes a flat directory structure for tests). 
>>
>> My make expertise is quite limited. Would anyone be free to help me get
>> this working.
>
> I'm fairly good with make, and have time to work on this.
>
> Let me know the scratch branch name.


Actually, it turned out not to be so difficult. I didn't know about
eval in make, but once I'd worked that out, the changes were quite
minimal.

It's on scratch/multi-level-test-makefile

My solution is currently limited to a fixed number of subdirs (i.e it's
not fully recursive), but then that isn't like to change quickly, if at
all. I'd welcome feedback, though.

The proposed directory layout is inside a file called emacs-tests.org at
top level (obviously not where I intend to leave it). I've accepted most
of the changes people suggested except for one. Some one suggested to
but tests of C core ~test/automated/src~ which sounded good. But
thinking about it, it probably makes more sense to leave this for C
level testing framework. So now we have

test/automated/lisp
test/automated/lisp/emacs-lisp
test/automated/lisp/progmodes (and so on....)
test/automated/lisp/c (lisp tests for the C core)

test/automated/lisp/legacy (ones which don't fit, to be fixed)

test/automated/src (empty but reserved for C tests of C core)


There's a shell-script embedded in emacs-tests.org to actually apply the
move. As we are in feature freeze, I'd plan to do this immediately after
the release branch is ready.

Comments welcome.

Phil










^ permalink raw reply	[flat|nested] 43+ messages in thread

* Re: Location of tests (again) (was Re: Makefile-help)
  2015-11-07 10:55     ` Location of tests (again) (was Re: Makefile-help) Phillip Lord
@ 2015-11-07 11:06       ` Juanma Barranquero
  2015-11-07 11:22         ` Phillip Lord
  2015-11-10 20:34       ` Przemysław Wojnowski
  1 sibling, 1 reply; 43+ messages in thread
From: Juanma Barranquero @ 2015-11-07 11:06 UTC (permalink / raw)
  To: Phillip Lord; +Cc: Stephen Leake, Emacs developers

[-- Attachment #1: Type: text/plain, Size: 818 bytes --]

On Sat, Nov 7, 2015 at 11:55 AM, Phillip Lord <phillip.lord@russet.org.uk>
wrote:

> Some one suggested to
> but tests of C core ~test/automated/src~ which sounded good. But
> thinking about it, it probably makes more sense to leave this for C
> level testing framework. So now we have
>
> test/automated/lisp
> test/automated/lisp/emacs-lisp
> test/automated/lisp/progmodes (and so on....)
> test/automated/lisp/c (lisp tests for the C core)

This breaks the symmetry between our source layout and the test layout.
And, of course, if we some day add a C testing framework there's no reason
why lisp tests and C tests cannot share test/automated/c.

In fact, it seems to me like a nice property that if we have both C and
elisp tests for, say, keymap.c, then keymap-tests.c and keymap-tests.el are
in close proximity.

[-- Attachment #2: Type: text/html, Size: 1022 bytes --]

^ permalink raw reply	[flat|nested] 43+ messages in thread

* Re: Location of tests (again) (was Re: Makefile-help)
  2015-11-07 11:06       ` Juanma Barranquero
@ 2015-11-07 11:22         ` Phillip Lord
  2015-11-07 11:36           ` Juanma Barranquero
  2015-11-07 11:45           ` Eli Zaretskii
  0 siblings, 2 replies; 43+ messages in thread
From: Phillip Lord @ 2015-11-07 11:22 UTC (permalink / raw)
  To: Juanma Barranquero; +Cc: Stephen Leake, Emacs developers

Juanma Barranquero <lekktu@gmail.com> writes:

> On Sat, Nov 7, 2015 at 11:55 AM, Phillip Lord <phillip.lord@russet.org.uk>
> wrote:
>
>> Some one suggested to
>> but tests of C core ~test/automated/src~ which sounded good. But
>> thinking about it, it probably makes more sense to leave this for C
>> level testing framework. So now we have
>>
>> test/automated/lisp
>> test/automated/lisp/emacs-lisp
>> test/automated/lisp/progmodes (and so on....)
>> test/automated/lisp/c (lisp tests for the C core)
>
> This breaks the symmetry between our source layout and the test layout.

It does, and I think this is unfortunate.

> And, of course, if we some day add a C testing framework there's no reason
> why lisp tests and C tests cannot share test/automated/c.
>
> In fact, it seems to me like a nice property that if we have both C and
> elisp tests for, say, keymap.c, then keymap-tests.c and keymap-tests.el are
> in close proximity.

I thought about that. Obviously it makes the Makefile harder (since you
need one that can cope with both the C framework and the Lisp
framework), which is why I though to keep things separate. This seemed
worth breaking the symmetry.

Very open to persuasion.

Phil



^ permalink raw reply	[flat|nested] 43+ messages in thread

* Re: Location of tests (again) (was Re: Makefile-help)
  2015-11-07 11:22         ` Phillip Lord
@ 2015-11-07 11:36           ` Juanma Barranquero
  2015-11-07 17:46             ` Phillip Lord
  2015-11-07 11:45           ` Eli Zaretskii
  1 sibling, 1 reply; 43+ messages in thread
From: Juanma Barranquero @ 2015-11-07 11:36 UTC (permalink / raw)
  To: Phillip Lord; +Cc: Stephen Leake, Emacs developers

[-- Attachment #1: Type: text/plain, Size: 593 bytes --]

On Sat, Nov 7, 2015 at 12:22 PM, Phillip Lord <phillip.lord@russet.org.uk>
wrote:

> I thought about that. Obviously it makes the Makefile harder (since you
> need one that can cope with both the C framework and the Lisp
> framework), which is why I though to keep things separate.

Well, Emacs isn't a project known by the simplicity of its makefiles ;-)

> Very open to persuasion.

We don't have a C testing framework, and we don't know when, or if, we'll
have one. I'd say worrying about how complex the makefiles will be for that
case is a bit premature.

Just my 0.02€

[-- Attachment #2: Type: text/html, Size: 797 bytes --]

^ permalink raw reply	[flat|nested] 43+ messages in thread

* Re: Location of tests (again) (was Re: Makefile-help)
  2015-11-07 11:22         ` Phillip Lord
  2015-11-07 11:36           ` Juanma Barranquero
@ 2015-11-07 11:45           ` Eli Zaretskii
  2015-11-07 18:09             ` Stephen Leake
  1 sibling, 1 reply; 43+ messages in thread
From: Eli Zaretskii @ 2015-11-07 11:45 UTC (permalink / raw)
  To: Phillip Lord; +Cc: lekktu, stephen_leake, emacs-devel

> From: phillip.lord@russet.org.uk (Phillip Lord)
> Date: Sat, 07 Nov 2015 11:22:04 +0000
> Cc: Stephen Leake <stephen_leake@stephe-leake.org>,
> 	Emacs developers <emacs-devel@gnu.org>
> 
> Juanma Barranquero <lekktu@gmail.com> writes:
> 
> > On Sat, Nov 7, 2015 at 11:55 AM, Phillip Lord <phillip.lord@russet.org.uk>
> > wrote:
> >
> >> Some one suggested to
> >> but tests of C core ~test/automated/src~ which sounded good. But
> >> thinking about it, it probably makes more sense to leave this for C
> >> level testing framework. So now we have
> >>
> >> test/automated/lisp
> >> test/automated/lisp/emacs-lisp
> >> test/automated/lisp/progmodes (and so on....)
> >> test/automated/lisp/c (lisp tests for the C core)
> >
> > This breaks the symmetry between our source layout and the test layout.
> 
> It does, and I think this is unfortunate.

Sorry, I don't understand why not use the simpler

 test/automated/src (tests for the C core)
 test/automated/lisp
 test/automated/lisp/emacs-lisp
 ...

> > In fact, it seems to me like a nice property that if we have both C and
> > elisp tests for, say, keymap.c, then keymap-tests.c and keymap-tests.el are
> > in close proximity.
> 
> I thought about that. Obviously it makes the Makefile harder (since you
> need one that can cope with both the C framework and the Lisp
> framework), which is why I though to keep things separate. This seemed
> worth breaking the symmetry.

We don't have any C framework, and I very much doubt we will any time
soon.  (I don't even think it's feasible, since Emacs is not a
library, and doesn't provide a way of exercising it from a C program.)
We can move files again if and when we do, just like we intend to move
files now.  OTOH, breaking the symmetry on behalf of a feature we
don't have doesn't sound right to me.



^ permalink raw reply	[flat|nested] 43+ messages in thread

* Re: Location of tests (again) (was Re: Makefile-help)
  2015-11-07 11:36           ` Juanma Barranquero
@ 2015-11-07 17:46             ` Phillip Lord
  0 siblings, 0 replies; 43+ messages in thread
From: Phillip Lord @ 2015-11-07 17:46 UTC (permalink / raw)
  To: Juanma Barranquero; +Cc: Stephen Leake, Emacs developers

Juanma Barranquero <lekktu@gmail.com> writes:

> On Sat, Nov 7, 2015 at 12:22 PM, Phillip Lord <phillip.lord@russet.org.uk>
> wrote:
>
>> I thought about that. Obviously it makes the Makefile harder (since you
>> need one that can cope with both the C framework and the Lisp
>> framework), which is why I though to keep things separate.
>
> Well, Emacs isn't a project known by the simplicity of its makefiles ;-)
>
>> Very open to persuasion.
>
> We don't have a C testing framework, and we don't know when, or if, we'll
> have one. I'd say worrying about how complex the makefiles will be for that
> case is a bit premature.
>
> Just my 0.02€


Okay, well that's two voices, so I am persuaded. The changes have been
pushed.

Phil



^ permalink raw reply	[flat|nested] 43+ messages in thread

* Re: Location of tests (again) (was Re: Makefile-help)
  2015-11-07 11:45           ` Eli Zaretskii
@ 2015-11-07 18:09             ` Stephen Leake
  0 siblings, 0 replies; 43+ messages in thread
From: Stephen Leake @ 2015-11-07 18:09 UTC (permalink / raw)
  To: Eli Zaretskii; +Cc: lekktu, emacs-devel, Phillip Lord

Eli Zaretskii <eliz@gnu.org> writes:

>> From: phillip.lord@russet.org.uk (Phillip Lord)
>> I thought about that. Obviously it makes the Makefile harder (since you
>> need one that can cope with both the C framework and the Lisp
>> framework), which is why I though to keep things separate. This seemed
>> worth breaking the symmetry.
>
> We don't have any C framework, and I very much doubt we will any time
> soon.  (I don't even think it's feasible, since Emacs is not a
> library, and doesn't provide a way of exercising it from a C program.)
> We can move files again if and when we do, just like we intend to move
> files now.  OTOH, breaking the symmetry on behalf of a feature we
> don't have doesn't sound right to me.

+1

-- 
-- Stephe



^ permalink raw reply	[flat|nested] 43+ messages in thread

* Re: Location of tests (again) (was Re: Makefile-help)
  2015-11-07 10:55     ` Location of tests (again) (was Re: Makefile-help) Phillip Lord
  2015-11-07 11:06       ` Juanma Barranquero
@ 2015-11-10 20:34       ` Przemysław Wojnowski
  2015-11-10 20:45         ` Phillip Lord
  1 sibling, 1 reply; 43+ messages in thread
From: Przemysław Wojnowski @ 2015-11-10 20:34 UTC (permalink / raw)
  To: Phillip Lord; +Cc: Stephen Leake, emacs-devel

phillip.lord@russet.org.uk (Phillip Lord) writes:

> test/automated/lisp
> test/automated/lisp/emacs-lisp
> test/automated/lisp/progmodes (and so on....)
> test/automated/lisp/c (lisp tests for the C core)
> test/automated/lisp/legacy (ones which don't fit, to be fixed)
> test/automated/src (empty but reserved for C tests of C core)

Sorry to comment on this so late, but it wouldn't it make sense to move
automated tests to directly under "test/" ? "automated/" here looks a
bit artificial to me.

Sooner or later all tests will be automated anyway.



^ permalink raw reply	[flat|nested] 43+ messages in thread

* Re: Location of tests (again) (was Re: Makefile-help)
  2015-11-10 20:34       ` Przemysław Wojnowski
@ 2015-11-10 20:45         ` Phillip Lord
  2015-11-10 21:41           ` Przemysław Wojnowski
  0 siblings, 1 reply; 43+ messages in thread
From: Phillip Lord @ 2015-11-10 20:45 UTC (permalink / raw)
  To: Przemysław Wojnowski; +Cc: Stephen Leake, emacs-devel

Przemysław Wojnowski <esperanto@cumego.com> writes:

> phillip.lord@russet.org.uk (Phillip Lord) writes:
>
>> test/automated/lisp
>> test/automated/lisp/emacs-lisp
>> test/automated/lisp/progmodes (and so on....)
>> test/automated/lisp/c (lisp tests for the C core)
>> test/automated/lisp/legacy (ones which don't fit, to be fixed)
>> test/automated/src (empty but reserved for C tests of C core)
>
> Sorry to comment on this so late,

No worries. I'm not going to make this move now, anyway, given that we
are approaching feature freeze (technically, I guess, tests are not a
feature but still...).

> but it wouldn't it make sense to move automated tests to directly
> under "test/" ? "automated/" here looks a bit artificial to me.
>
> Sooner or later all tests will be automated anyway.

I doubt the latter is true (simply because some display related tests
will always involve looking at emacs).

Having said that, I largely agree with you. We'd have something like?

test/lisp
test/src
test/non-ert

Other names which leap to mind:

test/non-automated
test/manual

Phil



^ permalink raw reply	[flat|nested] 43+ messages in thread

* Re: Location of tests (again) (was Re: Makefile-help)
  2015-11-10 20:45         ` Phillip Lord
@ 2015-11-10 21:41           ` Przemysław Wojnowski
  2015-11-10 22:27             ` John Wiegley
                               ` (2 more replies)
  0 siblings, 3 replies; 43+ messages in thread
From: Przemysław Wojnowski @ 2015-11-10 21:41 UTC (permalink / raw)
  To: Phillip Lord; +Cc: Stephen Leake, emacs-devel

phillip.lord@russet.org.uk (Phillip Lord) writes:
[...]
>> Sooner or later all tests will be automated anyway.
>
> I doubt the latter is true (simply because some display related tests
> will always involve looking at emacs).
Ok, "all" was exaggerated a bit, but most things can be tested
automatically.

> Having said that, I largely agree with you. We'd have something like?
>
> test/lisp
> test/src
[...]
> test/manual
I'm for "test/manual" here, but other names are fine too.

(I would also change "test/src/" to "test/c", because "lisp" are sources
too, after all. But I don't insist on that, especially that it may result
in a flood here. ;-) )



^ permalink raw reply	[flat|nested] 43+ messages in thread

* Re: Location of tests (again) (was Re: Makefile-help)
  2015-11-10 21:41           ` Przemysław Wojnowski
@ 2015-11-10 22:27             ` John Wiegley
  2015-11-11  0:28             ` bikeshedding (was Re: Location of tests (again) (was Re: Makefile-help)) Stephen Leake
  2015-11-11 10:31             ` Location of tests (again) (was Re: Makefile-help) Phillip Lord
  2 siblings, 0 replies; 43+ messages in thread
From: John Wiegley @ 2015-11-10 22:27 UTC (permalink / raw)
  To: Przemysław Wojnowski; +Cc: emacs-devel, Stephen Leake, Phillip Lord

>>>>> Przemysław Wojnowski <esperanto@cumego.com> writes:

>> test/lisp
>> test/src
> [...]
>> test/manual
> I'm for "test/manual" here, but other names are fine too.

I like this structure.

John



^ permalink raw reply	[flat|nested] 43+ messages in thread

* bikeshedding (was Re: Location of tests (again) (was Re: Makefile-help))
  2015-11-10 21:41           ` Przemysław Wojnowski
  2015-11-10 22:27             ` John Wiegley
@ 2015-11-11  0:28             ` Stephen Leake
  2015-11-11 10:31             ` Location of tests (again) (was Re: Makefile-help) Phillip Lord
  2 siblings, 0 replies; 43+ messages in thread
From: Stephen Leake @ 2015-11-11  0:28 UTC (permalink / raw)
  To: Przemysław Wojnowski; +Cc: emacs-devel, Phillip Lord

Przemysław Wojnowski <esperanto@cumego.com> writes:

> (I would also change "test/src/" to "test/c", because "lisp" are sources
> too, after all. But I don't insist on that, especially that it may result
> in a flood here. ;-) )

Only if you also change <root>/src to <root>/c.

Or to:

<root>
    src/
        src/lisp
        src/c

or just include lisp and C in one src/ directory; is that really a problem?

-- 
-- Stephe



^ permalink raw reply	[flat|nested] 43+ messages in thread

* Re: Location of tests (again) (was Re: Makefile-help)
  2015-11-10 21:41           ` Przemysław Wojnowski
  2015-11-10 22:27             ` John Wiegley
  2015-11-11  0:28             ` bikeshedding (was Re: Location of tests (again) (was Re: Makefile-help)) Stephen Leake
@ 2015-11-11 10:31             ` Phillip Lord
  2015-11-11 17:02               ` UI tests Richard Stallman
  2 siblings, 1 reply; 43+ messages in thread
From: Phillip Lord @ 2015-11-11 10:31 UTC (permalink / raw)
  To: Przemysław Wojnowski; +Cc: Stephen Leake, emacs-devel

Przemysław Wojnowski <esperanto@cumego.com> writes:

> phillip.lord@russet.org.uk (Phillip Lord) writes:
> [...]
>>> Sooner or later all tests will be automated anyway.
>>
>> I doubt the latter is true (simply because some display related tests
>> will always involve looking at emacs).
> Ok, "all" was exaggerated a bit, but most things can be tested
> automatically.
>
>> Having said that, I largely agree with you. We'd have something like?
>>
>> test/lisp
>> test/src
> [...]
>> test/manual
> I'm for "test/manual" here, but other names are fine too.
>
> (I would also change "test/src/" to "test/c", because "lisp" are sources
> too, after all. But I don't insist on that, especially that it may result
> in a flood here. ;-) )


I suggested that -- personally, I don't like having "src" in "test"
because "src" is, well, source, not test. But the general feeling was
that keeping it symmetrical makes more sense.

Either way, it answers my use-case of a deterministic link between
source and test.

Phil



^ permalink raw reply	[flat|nested] 43+ messages in thread

* UI tests
  2015-11-11 10:31             ` Location of tests (again) (was Re: Makefile-help) Phillip Lord
@ 2015-11-11 17:02               ` Richard Stallman
  2015-11-11 17:25                 ` John Wiegley
                                   ` (2 more replies)
  0 siblings, 3 replies; 43+ messages in thread
From: Richard Stallman @ 2015-11-11 17:02 UTC (permalink / raw)
  To: Phillip Lord; +Cc: stephen_leake, esperanto, emacs-devel

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Unlike the typical graphical applications, Emacs's UIs are mostly
controlled by events that have a very shallow translation to Lisp
data.  Even mouse clicks can be faked, and a convenient Lisp function
to make that totally easy would not be hard to write.

Perhaps the input side of testing UI features will be fairly easy.

The output side is not so straightforward, but if we set up something
to accumulate a sort of dribble record of X11 output operations, in
the form of Lisp-visible data, maybe tests could check that to see
if they have changed from before.


-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.




^ permalink raw reply	[flat|nested] 43+ messages in thread

* Re: UI tests
  2015-11-11 17:02               ` UI tests Richard Stallman
@ 2015-11-11 17:25                 ` John Wiegley
  2015-11-12 14:09                 ` Phillip Lord
  2015-11-15 12:14                 ` Przemysław Wojnowski
  2 siblings, 0 replies; 43+ messages in thread
From: John Wiegley @ 2015-11-11 17:25 UTC (permalink / raw)
  To: Richard Stallman; +Cc: esperanto, emacs-devel, stephen_leake, Phillip Lord

>>>>> Richard Stallman <rms@gnu.org> writes:

> The output side is not so straightforward, but if we set up something to
> accumulate a sort of dribble record of X11 output operations, in the form of
> Lisp-visible data, maybe tests could check that to see if they have changed
> from before.

Ah, I hadn't considered using output deltas as a way of testing. Good idea!

John



^ permalink raw reply	[flat|nested] 43+ messages in thread

* Re: UI tests
  2015-11-11 17:02               ` UI tests Richard Stallman
  2015-11-11 17:25                 ` John Wiegley
@ 2015-11-12 14:09                 ` Phillip Lord
  2015-11-12 14:37                   ` Alan Mackenzie
                                     ` (2 more replies)
  2015-11-15 12:14                 ` Przemysław Wojnowski
  2 siblings, 3 replies; 43+ messages in thread
From: Phillip Lord @ 2015-11-12 14:09 UTC (permalink / raw)
  To: Richard Stallman; +Cc: stephen_leake, esperanto, emacs-devel

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
> Unlike the typical graphical applications, Emacs's UIs are mostly
> controlled by events that have a very shallow translation to Lisp
> data.  Even mouse clicks can be faked, and a convenient Lisp function
> to make that totally easy would not be hard to write.

I would agree.

I've been working on a library (called sisyphus) to support higher-level
operations for testing, and that would be a nice addition.


> Perhaps the input side of testing UI features will be fairly easy.
>
> The output side is not so straightforward, but if we set up something
> to accumulate a sort of dribble record of X11 output operations, in
> the form of Lisp-visible data, maybe tests could check that to see
> if they have changed from before.

If that would allow me to test things like before and after-string
overlay properties that would be excellent. Testing, for example,
completion engines is hard in Emacs since the buffer doesn't change,
only the visualisation of it.

One additional problem, though, is that Emacs doesn't really have a
concept of "headless". Things like syntax highlighting in buffers just
work different when running in batch, because Emacs assumes it's running
on a TTY. For example, HTML generated by htmlize comes out differently
in batch, than from a windowed Emacs, because Emacs in batch cannot
display colour.

Phil



^ permalink raw reply	[flat|nested] 43+ messages in thread

* Re: UI tests
  2015-11-12 14:09                 ` Phillip Lord
@ 2015-11-12 14:37                   ` Alan Mackenzie
  2015-11-12 16:33                   ` Eli Zaretskii
  2015-11-13 21:58                   ` Richard Stallman
  2 siblings, 0 replies; 43+ messages in thread
From: Alan Mackenzie @ 2015-11-12 14:37 UTC (permalink / raw)
  To: Phillip Lord; +Cc: esperanto, stephen_leake, Richard Stallman, emacs-devel

Hello, Phillip.

On Thu, Nov 12, 2015 at 02:09:32PM +0000, Phillip Lord wrote:

[ .... ]

> One additional problem, though, is that Emacs doesn't really have a
> concept of "headless". Things like syntax highlighting in buffers just
> work different when running in batch, because Emacs assumes it's running
> on a TTY. For example, HTML generated by htmlize comes out differently
> in batch, than from a windowed Emacs, because Emacs in batch cannot
> display colour.

I'm not sure whether or not this is relevant, but Emacs can be duped
into doing font-locking in batch mode.  Please examine the following
function (out of CC Mode's 000tests.el), and see if there's anything
in it you could use:

(defun cc-test-force-font-lock-buffer ()
  ;; Try to forcibly font lock the current buffer, even in batch mode.
  ;; We're doing really dirty things to trick font-lock into action in
  ;; batch mode in the different emacsen.
  (let ((orig-noninteractive-function
         (and (fboundp 'noninteractive)
              (symbol-function 'noninteractive)))
        (orig-noninteractive-variable
         (and (boundp 'noninteractive)
              (symbol-value 'noninteractive)))
        ;; font-lock in XEmacs 19 looks at a variable named `noninteractive'.
        (noninteractive nil))
    (unwind-protect
        (progn
          (when orig-noninteractive-function
            ;; XEmacs (at least 21.4) calls `noninteractive' to check
            ;; for batch mode, so we let it lie.
            (fset 'noninteractive (lambda () nil)))
          (font-lock-mode 1)
          (unless (or (get-text-property (point-min) 'face)
                      (next-single-property-change (point-min) 'face))
            ;; Some emacsen have already fontified the buffer above,
            ;; but others need some more coercion..
            (let (;; Avoid getting some lazy fontification package that
                  ;; might decide that nothing should be done.
                  (font-lock-fontify-buffer-function
                   'font-lock-default-fontify-buffer))
              (font-lock-fontify-buffer))))
      (when orig-noninteractive-function
        (fset 'noninteractive orig-noninteractive-function)))))


> Phil

-- 
Alan Mackenzie (Nuremberg, Germany).



^ permalink raw reply	[flat|nested] 43+ messages in thread

* Re: UI tests
  2015-11-12 14:09                 ` Phillip Lord
  2015-11-12 14:37                   ` Alan Mackenzie
@ 2015-11-12 16:33                   ` Eli Zaretskii
  2015-11-13 21:58                   ` Richard Stallman
  2 siblings, 0 replies; 43+ messages in thread
From: Eli Zaretskii @ 2015-11-12 16:33 UTC (permalink / raw)
  To: Phillip Lord; +Cc: esperanto, stephen_leake, rms, emacs-devel

> From: phillip.lord@russet.org.uk (Phillip Lord)
> Date: Thu, 12 Nov 2015 14:09:32 +0000
> Cc: stephen_leake@stephe-leake.org, esperanto@cumego.com, emacs-devel@gnu.org
> 
> One additional problem, though, is that Emacs doesn't really have a
> concept of "headless". Things like syntax highlighting in buffers just
> work different when running in batch, because Emacs assumes it's running
> on a TTY. For example, HTML generated by htmlize comes out differently
> in batch, than from a windowed Emacs, because Emacs in batch cannot
> display colour.

Most GUI APIs (like Xlib etc.) won't work without a real display and a
window (a.k.a. "frame"), so if we want a "headless" Emacs that will
work as with GUI frames, we will have to emulate all those calls.  Not
for a faint at heart, I'd say.



^ permalink raw reply	[flat|nested] 43+ messages in thread

* Re: UI tests
  2015-11-12 14:09                 ` Phillip Lord
  2015-11-12 14:37                   ` Alan Mackenzie
  2015-11-12 16:33                   ` Eli Zaretskii
@ 2015-11-13 21:58                   ` Richard Stallman
  2 siblings, 0 replies; 43+ messages in thread
From: Richard Stallman @ 2015-11-13 21:58 UTC (permalink / raw)
  To: Phillip Lord; +Cc: stephen_leake, esperanto, emacs-devel

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > One additional problem, though, is that Emacs doesn't really have a
  > concept of "headless".

We could implement a fake terminal of any desired type and use
it to test display for real terminals of that type.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.




^ permalink raw reply	[flat|nested] 43+ messages in thread

* Re: UI tests
  2015-11-11 17:02               ` UI tests Richard Stallman
  2015-11-11 17:25                 ` John Wiegley
  2015-11-12 14:09                 ` Phillip Lord
@ 2015-11-15 12:14                 ` Przemysław Wojnowski
  2015-11-16 19:32                   ` Richard Stallman
  2 siblings, 1 reply; 43+ messages in thread
From: Przemysław Wojnowski @ 2015-11-15 12:14 UTC (permalink / raw)
  To: Richard Stallman; +Cc: emacs-devel, stephen_leake, Phillip Lord

Richard Stallman <rms@gnu.org> writes:
> Unlike the typical graphical applications, Emacs's UIs are mostly
> controlled by events that have a very shallow translation to Lisp
> data.  Even mouse clicks can be faked, and a convenient Lisp function
> to make that totally easy would not be hard to write.
>
> Perhaps the input side of testing UI features will be fairly easy.
>
> The output side is not so straightforward, but if we set up something
> to accumulate a sort of dribble record of X11 output operations, in
> the form of Lisp-visible data, maybe tests could check that to see
> if they have changed from before.

I'm not sure what do you mean by "output side" here?

My understanding is that inputs and outputs are on system's boundaries
and isolate system's functionality using stable interfaces (common in
different architectural pattern - e.g. [1], [2]).

In Emacs, as you wrote, user actions can be faked and that can be used
in tests.

The other side - the outputs - is not that clear to me. Usually the
outputs are hidden under stable interfaces. The interfaces allow to
exchange implementations without affecting the system and the boundary
is used to test the output of the system. In case of display output,
this won't allow to test every single pixel (so no X11 output), but at
least it will allow to test that the system produces correct output for
display, no matter what that display (its underlying library) is.

Two things to note when introducing automated tests:
1. Tests pyramid [3] - indicates which tests are most important and
ratio between them. TL;DR Emacs should focus on unit tests.

2. Ice Cream Cone anti-pattern [3,4] is a common trap for existing
projects that start to add automated tests.

[1] Layers in "Pattern Oriented Software Architecture vol 1"
[2] http://alistair.cockburn.us/Hexagonal+architecture
[3] http://googletesting.blogspot.co.uk/2015/04/just-say-no-to-more-end-to-end-tests.html
(from "The True Value of Tests")
[4] http://watirmelon.com/2012/01/31/introducing-the-software-testing-ice-cream-cone/



^ permalink raw reply	[flat|nested] 43+ messages in thread

* Re: UI tests
  2015-11-15 12:14                 ` Przemysław Wojnowski
@ 2015-11-16 19:32                   ` Richard Stallman
  0 siblings, 0 replies; 43+ messages in thread
From: Richard Stallman @ 2015-11-16 19:32 UTC (permalink / raw)
  To: Przemysław Wojnowski; +Cc: emacs-devel, stephen_leake, phillip.lord

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I'm not sure what do you mean by "output side" here?

I mean anything that shows on the screen.

When it comes to testing the user interface at a lower
level, we would have to test what it does to the screen.


-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.




^ permalink raw reply	[flat|nested] 43+ messages in thread

* Re: Locations of Tests
  2015-11-04 18:21     ` John Wiegley
@ 2015-11-25  9:23       ` Phillip Lord
  2015-11-25  9:43         ` Artur Malabarba
  0 siblings, 1 reply; 43+ messages in thread
From: Phillip Lord @ 2015-11-25  9:23 UTC (permalink / raw)
  To: Artur Malabarba, jwiegley; +Cc: emacs-devel

John Wiegley <jwiegley@gmail.com> writes:

>>>>>> Artur Malabarba <bruce.connor.am@gmail.com> writes:
>
>> Same here. Just make sure it doesn't mess up git blame. ;-)
>
>>From *Man git-blame*:
>
>        The origin of lines is automatically followed across whole-file renames (currently
>        there is no option to turn the rename-following off). To follow lines moved from
>        one file to another, or to follow lines that were copied and pasted from another
>        file, etc., see the -C and -M options.
>
> John

Okay, so I have moved all the test files according to the scheme
discussed, and pushed to branch feature/standard-test-location.

I wasn't able to do this in a single commit -- the move was one, so
hopefully git should have tracked the file moves. Then, I updated the
makefiles, fixed the tests that depend on their location. Finally, I had
to restore a couple of manual (non automake) makefiles which I
accidentally removed (very underhand, didn't check the .gitignore).

I'd appreciate if you could have a look at this, then I'll merge to
master.

Phil



^ permalink raw reply	[flat|nested] 43+ messages in thread

* Re: Locations of Tests
  2015-11-25  9:23       ` Phillip Lord
@ 2015-11-25  9:43         ` Artur Malabarba
  2015-11-25 14:17           ` Phillip Lord
  0 siblings, 1 reply; 43+ messages in thread
From: Artur Malabarba @ 2015-11-25  9:43 UTC (permalink / raw)
  To: Phillip Lord; +Cc: John Wiegley, emacs-devel

[-- Attachment #1: Type: text/plain, Size: 257 bytes --]

On 25 Nov 2015 9:23 am, "Phillip Lord" <phillip.lord@russet.org.uk> wrote:
> I wasn't able to do this in a single commit -- the move was one, so
> hopefully git should have tracked the file moves.

Why not try git blame so we can lose that "hopefully"? :-)

[-- Attachment #2: Type: text/html, Size: 388 bytes --]

^ permalink raw reply	[flat|nested] 43+ messages in thread

* Re: Locations of Tests
  2015-11-25  9:43         ` Artur Malabarba
@ 2015-11-25 14:17           ` Phillip Lord
  0 siblings, 0 replies; 43+ messages in thread
From: Phillip Lord @ 2015-11-25 14:17 UTC (permalink / raw)
  To: Artur Malabarba; +Cc: John Wiegley, emacs-devel

Artur Malabarba <bruce.connor.am@gmail.com> writes:

> On 25 Nov 2015 9:23 am, "Phillip Lord" <phillip.lord@russet.org.uk> wrote:
>> I wasn't able to do this in a single commit -- the move was one, so
>> hopefully git should have tracked the file moves.
>
> Why not try git blame so we can lose that "hopefully"? :-)


Potentially...

I did try this already and it seems to be good


d7df36e7 test/automated/elisp-mode-tests.el      (Stephen Leake      2015-08-10 21:53:19 -0500 405)   (list
d7df36e7 test/automated/elisp-mode-tests.el      (Stephen Leake      2015-08-10 21:53:19 -0500 406)    (xref-make "(cl-defgeneric xref-location-marker)"
d7df36e7 test/automated/elisp-mode-tests.el      (Stephen Leake      2015-08-10 21:53:19 -0500 407)           (xref-make-elisp-location
0382fd42 test/automated/elisp-mode-tests.el      (Stephen Leake      2015-08-13 12:54:39 -0500 408)            'xref-location-marker 'cl-defgeneric
6fdd750c test/lisp/progmodes/elisp-mode-tests.el (Phillip Lord       2015-11-24 22:35:23 +0000 409)            (expand-file-name "../../../lisp/progmodes/xref.el" emacs-test-dir)))
d7df36e7 test/automated/elisp-mode-tests.el      (Stephen Leake      2015-08-10 21:53:19 -0500 410)    (xref-make "(cl-defmethod xref-location-marker ((l xref-elisp-location)))"
d7df36e7 test/automated/elisp-mode-tests.el      (Stephen Leake      2015-08-10 21:53:19 -0500 411)           (xref-make-elisp-location
d7df36e7 test/automated/elisp-mode-tests.el      (Stephen Leake      2015-08-10 21:53:19 -0500 412)            '(xref-location-marker xref-elisp-location) 'cl-defmethod
6fdd750c test/lisp/progmodes/elisp-mode-tests.el (Phillip Lord       2015-11-24 22:35:23 +0000 413)            (expand-file-name "../../../lisp/progmodes/elisp-mode.el" emacs-test-dir)))
d7df36e7 test/automated/elisp-mode-tests.el      (Stephen Leake      2015-08-10 21:53:19 -0500 414)    (xref-make "(cl-defmethod xref-location-marker ((l xref-file-location)))"
d7df36e7 test/automated/elisp-mode-tests.el      (Stephen Leake      2015-08-10 21:53:19 -0500 415)           (xref-make-elisp-location
d7df36e7 test/automated/elisp-mode-tests.el      (Stephen Leake      2015-08-10 21:53:19 -0500 416)            '(xref-location-marker xref-file-location) 'cl-defmethod


I fear this has failed for the four Makefile.

./lisp/progmodes/flymake-resources/Makefile
./manual/indent/Makefile
./manual/etags/Makefile
./manual/etags/make-src/Makefile

which all appear to have been written by me. The "real" makefile
(Makefile.in) is good.

Phil



^ permalink raw reply	[flat|nested] 43+ messages in thread

end of thread, other threads:[~2015-11-25 14:17 UTC | newest]

Thread overview: 43+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2015-11-04 17:36 Locations of Tests Phillip Lord
2015-11-04 17:53 ` John Wiegley
2015-11-04 18:14   ` Artur Malabarba
2015-11-04 18:21     ` John Wiegley
2015-11-25  9:23       ` Phillip Lord
2015-11-25  9:43         ` Artur Malabarba
2015-11-25 14:17           ` Phillip Lord
2015-11-04 19:15     ` Steinar Bang
2015-11-04 19:23       ` David Kastrup
2015-11-04 21:33         ` Phillip Lord
2015-11-04 19:07   ` Michael Albinus
2015-11-04 19:15     ` John Wiegley
2015-11-04 21:26       ` Phillip Lord
2015-11-05 13:41         ` Stephen Leake
2015-11-05 13:59           ` Artur Malabarba
2015-11-05 15:02             ` Nicolas Petton
2015-11-05 15:08               ` Artur Malabarba
2015-11-06  9:56                 ` Phillip Lord
2015-11-05 15:27           ` Juanma Barranquero
2015-11-06  9:55           ` Phillip Lord
2015-11-06 10:02 ` Makefile-help (was Re: Locations of Tests) Phillip Lord
2015-11-07  6:48   ` Stephen Leake
2015-11-07 10:55     ` Location of tests (again) (was Re: Makefile-help) Phillip Lord
2015-11-07 11:06       ` Juanma Barranquero
2015-11-07 11:22         ` Phillip Lord
2015-11-07 11:36           ` Juanma Barranquero
2015-11-07 17:46             ` Phillip Lord
2015-11-07 11:45           ` Eli Zaretskii
2015-11-07 18:09             ` Stephen Leake
2015-11-10 20:34       ` Przemysław Wojnowski
2015-11-10 20:45         ` Phillip Lord
2015-11-10 21:41           ` Przemysław Wojnowski
2015-11-10 22:27             ` John Wiegley
2015-11-11  0:28             ` bikeshedding (was Re: Location of tests (again) (was Re: Makefile-help)) Stephen Leake
2015-11-11 10:31             ` Location of tests (again) (was Re: Makefile-help) Phillip Lord
2015-11-11 17:02               ` UI tests Richard Stallman
2015-11-11 17:25                 ` John Wiegley
2015-11-12 14:09                 ` Phillip Lord
2015-11-12 14:37                   ` Alan Mackenzie
2015-11-12 16:33                   ` Eli Zaretskii
2015-11-13 21:58                   ` Richard Stallman
2015-11-15 12:14                 ` Przemysław Wojnowski
2015-11-16 19:32                   ` Richard Stallman

Code repositories for project(s) associated with this external index

	https://git.savannah.gnu.org/cgit/emacs.git
	https://git.savannah.gnu.org/cgit/emacs/org-mode.git

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.