Tests, Emacs-25 and Conditional Features

Tests, Emacs-25 and Conditional Features

[Phillip Lord]

I have two questions about tests.

First, does the "no changes on Emacs-25" rule apply to changes in the
test directory -- i.e. can next tests be added. Arguments for -- tests
are a good thing to add and should not break a release. Arguments
against -- we are in pre-test, and it's easier to have a simple rule.
Also, tests are currently painful to merge.

Second, should tests assume that all the conditional features (gnutls,
libxml) are available? If the answer is no, then how can we test that
conditional features are present when they should be; if the answer is
yes, then do we want tests that will fail some of the time.

Phil

Re: Tests, Emacs-25 and Conditional Features

[Michael Albinus]

phillip.lord@russet.org.uk (Phillip Lord) writes:

> Second, should tests assume that all the conditional features (gnutls,
> libxml) are available? If the answer is no, then how can we test that
> conditional features are present when they should be; if the answer is
> yes, then do we want tests that will fail some of the time.

In your tests, you could check the presence of a feature, and skip if it
isn't present. Like this:

(skip-unless (gnutls-available-p))
(skip-unless (functionp 'libxml-parse-xml-region))

> Phil

Best regards, Michael.

Re: Tests, Emacs-25 and Conditional Features

[Phillip Lord]

Michael Albinus <michael.albinus@gmx.de> writes:

> phillip.lord@russet.org.uk (Phillip Lord) writes:
>
>> Second, should tests assume that all the conditional features (gnutls,
>> libxml) are available? If the answer is no, then how can we test that
>> conditional features are present when they should be; if the answer is
>> yes, then do we want tests that will fail some of the time.
>
> In your tests, you could check the presence of a feature, and skip if it
> isn't present. Like this:
>
> (skip-unless (gnutls-available-p))
> (skip-unless (functionp 'libxml-parse-xml-region))


Yeah, thought about that. But then the test is skipped if
gnutls-available-p returns nil. What happens if I believe that I have
configured and build Emacs to include gnutls, but, for some reason it
isn't. The test will be skipped when it should succeed.

The same is true for all the other conditional features.

I wonder if we need a "conditional feature" test selector.

Phil

Re: Tests, Emacs-25 and Conditional Features

[Eli Zaretskii]

> From: phillip.lord@russet.org.uk (Phillip Lord)
> Date: Thu, 17 Mar 2016 10:14:44 +0000
> Cc: emacs-devel@gnu.org
> 
> > (skip-unless (gnutls-available-p))
> > (skip-unless (functionp 'libxml-parse-xml-region))
> 
> Yeah, thought about that. But then the test is skipped if
> gnutls-available-p returns nil. What happens if I believe that I have
> configured and build Emacs to include gnutls, but, for some reason it
> isn't. The test will be skipped when it should succeed.

Tests that are skipped are indicated in the test log file, so if you
see there a feature being skipped that you thought should be
available, you can investigate.

More generally, I don't see how Emacs could know what you believe.  We
need to define the meaning of that first, and then find a way to
communicate that to Emacs.

For example, one possible interpretation of "believe" is the report
displayed by 'configure' near its end.  A similar interpretation is
the value of system-configuration-features.  However, it might happen
that the user expects her build to support feature X, but that feature
is not in any of these, in which case the only way I see is to ask the
user.

Then there's the issue of how do you test this.  An API that was not
compiled into Emacs cannot be invoked, so what do we want to show the
user? a "void function" error? ert error messages are not so easy to
interpret.

IOW, I think you are talking about a kind of testing we don't yet
have, and it should be designed first.  The current test suite aims at
testing the code, using the available Emacs binary as best it can.  It
does NOT test user assumptions about the build -- that is a separate
and different issue, which requires rather different approach, IMO.

Re: Tests, Emacs-25 and Conditional Features

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: phillip.lord@russet.org.uk (Phillip Lord)
>> Date: Thu, 17 Mar 2016 10:14:44 +0000
>> Cc: emacs-devel@gnu.org
>> 
>> > (skip-unless (gnutls-available-p))
>> > (skip-unless (functionp 'libxml-parse-xml-region))
>> 
>> Yeah, thought about that. But then the test is skipped if
>> gnutls-available-p returns nil. What happens if I believe that I have
>> configured and build Emacs to include gnutls, but, for some reason it
>> isn't. The test will be skipped when it should succeed.
>
> Tests that are skipped are indicated in the test log file, so if you
> see there a feature being skipped that you thought should be
> available, you can investigate.
>
> More generally, I don't see how Emacs could know what you believe.  We
> need to define the meaning of that first, and then find a way to
> communicate that to Emacs.


Something like

(skip-unless (or (gnutls-available-p) conditional-feature-force))

So, we pick have a standard test selector which means "all the
conditional features should be expected to be on". Probably, you would
want "all the conditional features that are supposed to be on by
default, should be expected to be on"


> Then there's the issue of how do you test this.  An API that was not
> compiled into Emacs cannot be invoked, so what do we want to show the
> user? a "void function" error? ert error messages are not so easy to
> interpret.

It would be nice if there were a standard way of detecting this. As
Michael's example shows, we have a gnutls-available-p function which
is defined if gnutls is not available, but for libxml, we check for
non-definition of functions.

Phil

Re: Tests, Emacs-25 and Conditional Features

[Michael Albinus]

phillip.lord@russet.org.uk (Phillip Lord) writes:

Hi Philip,

> Something like
>
> (skip-unless (or (gnutls-available-p) conditional-feature-force))
>
> So, we pick have a standard test selector which means "all the
> conditional features should be expected to be on". Probably, you would
> want "all the conditional features that are supposed to be on by
> default, should be expected to be on"

That does not exist for everything. For example, in the file
notification case there are 5 different possible "conditional features",
all of them valid:

- inotify enabled
- kqueue enabled
- gfilenotify enabled
- w32notify enable
- file notification disabled.

How does a test library shall know what is intended by the user?

In all my tests packages, I have one test which checks all these
alternatives, and maybe checks additionally whether they make sense. See
file-notify-test00-availability or, more sophistiacted,
tramp-test00-availability. For all other test cases, I'll start with the
(skip-unless ...) form.

>> Then there's the issue of how do you test this.  An API that was not
>> compiled into Emacs cannot be invoked, so what do we want to show the
>> user? a "void function" error? ert error messages are not so easy to
>> interpret.
>
> It would be nice if there were a standard way of detecting this. As
> Michael's example shows, we have a gnutls-available-p function which
> is defined if gnutls is not available, but for libxml, we check for
> non-definition of functions.

This would require for all optional packages to provide such a
standardized form. We don't have it.

One way would be to test (skip-unless (featurep 'package)). But even
this is not possible in general, packages like xml.c miss

  Fprovide (intern_c_string ("libxml"), Qnil);

Maybe this is an error in xml.c, which should be fixed?

> Phil

Best regards, Michael.

Re: Tests, Emacs-25 and Conditional Features

[Phillip Lord]

Michael Albinus <michael.albinus@gmx.de> writes:

> phillip.lord@russet.org.uk (Phillip Lord) writes:
>
> Hi Philip,
>
>> Something like
>>
>> (skip-unless (or (gnutls-available-p) conditional-feature-force))
>>
>> So, we pick have a standard test selector which means "all the
>> conditional features should be expected to be on". Probably, you would
>> want "all the conditional features that are supposed to be on by
>> default, should be expected to be on"
>
> That does not exist for everything. For example, in the file
> notification case there are 5 different possible "conditional features",
> all of them valid:
>
> - inotify enabled
> - kqueue enabled
> - gfilenotify enabled
> - w32notify enable
> - file notification disabled.
>
> How does a test library shall know what is intended by the user?

Good point, although, if I understand the notification correctly, you
have an abstraction layer over these?

>>
>> It would be nice if there were a standard way of detecting this. As
>> Michael's example shows, we have a gnutls-available-p function which
>> is defined if gnutls is not available, but for libxml, we check for
>> non-definition of functions.
>
> This would require for all optional packages to provide such a
> standardized form. We don't have it.
>
> One way would be to test (skip-unless (featurep 'package)). But even
> this is not possible in general, packages like xml.c miss
>
>   Fprovide (intern_c_string ("libxml"), Qnil);
>
> Maybe this is an error in xml.c, which should be fixed?


Having a standard list of compile conditional features would be nice.

Phil

Re: Tests, Emacs-25 and Conditional Features

[Eli Zaretskii]

> From: phillip.lord@russet.org.uk (Phillip Lord)
> Date: Fri, 18 Mar 2016 17:19:22 +0000
> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org
> 
> Michael Albinus <michael.albinus@gmx.de> writes:
> 
> > phillip.lord@russet.org.uk (Phillip Lord) writes:
> >
> > Hi Philip,
> >
> >> Something like
> >>
> >> (skip-unless (or (gnutls-available-p) conditional-feature-force))
> >>
> >> So, we pick have a standard test selector which means "all the
> >> conditional features should be expected to be on". Probably, you would
> >> want "all the conditional features that are supposed to be on by
> >> default, should be expected to be on"
> >
> > That does not exist for everything. For example, in the file
> > notification case there are 5 different possible "conditional features",
> > all of them valid:
> >
> > - inotify enabled
> > - kqueue enabled
> > - gfilenotify enabled
> > - w32notify enable
> > - file notification disabled.
> >
> > How does a test library shall know what is intended by the user?
> 
> Good point, although, if I understand the notification correctly, you
> have an abstraction layer over these?

The abstraction layer is not relevant if you want to test the
low-level functionality.

Re: Tests, Emacs-25 and Conditional Features

[Eli Zaretskii]

> From: phillip.lord@russet.org.uk (Phillip Lord)
> Cc: michael.albinus@gmx.de,  emacs-devel@gnu.org
> Date: Fri, 18 Mar 2016 10:41:28 +0000
> 
> > More generally, I don't see how Emacs could know what you believe.  We
> > need to define the meaning of that first, and then find a way to
> > communicate that to Emacs.
> 
> Something like
> 
> (skip-unless (or (gnutls-available-p) conditional-feature-force))
> 
> So, we pick have a standard test selector which means "all the
> conditional features should be expected to be on". Probably, you would
> want "all the conditional features that are supposed to be on by
> default, should be expected to be on"

That won't be useful at all: many such features are
platform-dependent, for example.  It makes very little sense to try
invoking w32notify or w32-shell-execute on GNU/Linux.  Likewise with
features that depend on specific GUI toolkits.

> > Then there's the issue of how do you test this.  An API that was not
> > compiled into Emacs cannot be invoked, so what do we want to show the
> > user? a "void function" error? ert error messages are not so easy to
> > interpret.
> 
> It would be nice if there were a standard way of detecting this. As
> Michael's example shows, we have a gnutls-available-p function which
> is defined if gnutls is not available, but for libxml, we check for
> non-definition of functions.

It would be trivial to add a libxml2-available-p function, but I fail
to see how is this different from using fboundp.  And AFAIU, in the
context of this discussion, you don't want to trust the likes of
gnutls-available-p, either.

Re: Tests, Emacs-25 and Conditional Features

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:
>> Michael's example shows, we have a gnutls-available-p function which
>> is defined if gnutls is not available, but for libxml, we check for
>> non-definition of functions.
>
> It would be trivial to add a libxml2-available-p function, but I fail
> to see how is this different from using fboundp.  And AFAIU, in the
> context of this discussion, you don't want to trust the likes of
> gnutls-available-p, either.

One is one way, the other is the other way.

Principle of least surprise and all that stuff.

Phil

Re: Tests, Emacs-25 and Conditional Features

[Eli Zaretskii]

> From: phillip.lord@russet.org.uk (Phillip Lord)
> Cc: michael.albinus@gmx.de,  emacs-devel@gnu.org
> Date: Fri, 18 Mar 2016 17:20:30 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> >> Michael's example shows, we have a gnutls-available-p function which
> >> is defined if gnutls is not available, but for libxml, we check for
> >> non-definition of functions.
> >
> > It would be trivial to add a libxml2-available-p function, but I fail
> > to see how is this different from using fboundp.  And AFAIU, in the
> > context of this discussion, you don't want to trust the likes of
> > gnutls-available-p, either.
> 
> One is one way, the other is the other way.
> 
> Principle of least surprise and all that stuff.

Are we still talking about the feature test you wanted to have?

Re: Tests, Emacs-25 and Conditional Features

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: phillip.lord@russet.org.uk (Phillip Lord)
>> Cc: michael.albinus@gmx.de,  emacs-devel@gnu.org
>> Date: Fri, 18 Mar 2016 17:20:30 +0000
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> >> Michael's example shows, we have a gnutls-available-p function which
>> >> is defined if gnutls is not available, but for libxml, we check for
>> >> non-definition of functions.
>> >
>> > It would be trivial to add a libxml2-available-p function, but I fail
>> > to see how is this different from using fboundp.  And AFAIU, in the
>> > context of this discussion, you don't want to trust the likes of
>> > gnutls-available-p, either.
>> 
>> One is one way, the other is the other way.
>> 
>> Principle of least surprise and all that stuff.
>
> Are we still talking about the feature test you wanted to have?


Yeah. A consistent mechanism for identifying the presence of absence of
build conditional features would be no bad thing.

Phil

Re: Tests, Emacs-25 and Conditional Features

[Stefan Monnier]

> Yeah, thought about that. But then the test is skipped if
> gnutls-available-p returns nil. What happens if I believe that I have
> configured and build Emacs to include gnutls, but, for some reason it
> isn't. The test will be skipped when it should succeed.

This probably calls for a different kind of test.
E.g. one that takes the system-configuration-options and then confirms
that the build has the corresponding features.


        Stefan

Re: Tests, Emacs-25 and Conditional Features

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Fri, 18 Mar 2016 11:15:59 -0400
> 
> > Yeah, thought about that. But then the test is skipped if
> > gnutls-available-p returns nil. What happens if I believe that I have
> > configured and build Emacs to include gnutls, but, for some reason it
> > isn't. The test will be skipped when it should succeed.
> 
> This probably calls for a different kind of test.
> E.g. one that takes the system-configuration-options and then confirms
> that the build has the corresponding features.

That's one possibility, but it isn't the only one.  Some of the
features in system-configuration-options are detected automatically,
so the user might not always be aware that a feature she assumed she
has failed to auto-detect and is disabled.

Maybe a test that produced a report for which features are actually
available and working would be a possible solution.

Re: Tests, Emacs-25 and Conditional Features

[Stefan Monnier]

> That's one possibility, but it isn't the only one.  Some of the
> features in system-configuration-options are detected automatically,
> so the user might not always be aware that a feature she assumed she
> has failed to auto-detect and is disabled.

Well, we need some concrete data from which we can guess what the user
might assume.  `system-configuration-options' is one of them, but indeed
it doesn't cover all cases.

> Maybe a test that produced a report for which features are actually
> available and working would be a possible solution.

Not sure what you're thinking of.  But maybe we could stash the summary
we emit at the end of ./configure and use that additionally to
`system-configuration-options'.


        Stefan

Re: Tests, Emacs-25 and Conditional Features

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: emacs-devel@gnu.org
> Date: Fri, 18 Mar 2016 11:54:55 -0400
> 
> > Maybe a test that produced a report for which features are actually
> > available and working would be a possible solution.
> 
> Not sure what you're thinking of.  But maybe we could stash the summary
> we emit at the end of ./configure and use that additionally to
> `system-configuration-options'.

Something like that.  Except that the summary reported by configure
doesn't mention all the features, only the main ones.

Re: Tests, Emacs-25 and Conditional Features

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Stefan Monnier <monnier@iro.umontreal.ca>
>> Date: Fri, 18 Mar 2016 11:15:59 -0400
>> 
>> > Yeah, thought about that. But then the test is skipped if
>> > gnutls-available-p returns nil. What happens if I believe that I have
>> > configured and build Emacs to include gnutls, but, for some reason it
>> > isn't. The test will be skipped when it should succeed.
>> 
>> This probably calls for a different kind of test.
>> E.g. one that takes the system-configuration-options and then confirms
>> that the build has the corresponding features.
>
> That's one possibility, but it isn't the only one.  Some of the
> features in system-configuration-options are detected automatically,
> so the user might not always be aware that a feature she assumed she
> has failed to auto-detect and is disabled.
>
> Maybe a test that produced a report for which features are actually
> available and working would be a possible solution.

Failing a test, even a GUI which says "I have these features" would be
nice. I can add that.

Phil

Re: Tests, Emacs-25 and Conditional Features

[Dmitry Gutov]

On 03/16/2016 03:36 PM, Phillip Lord wrote:

> First, does the "no changes on Emacs-25" rule apply to changes in the
> test directory -- i.e. can next tests be added. Arguments for -- tests
> are a good thing to add and should not break a release. Arguments
> against -- we are in pre-test, and it's easier to have a simple rule.
> Also, tests are currently painful to merge.

I think it's fine (I'm guilty of this myself), but you should be 
prepared to help merging, or to at least verify the result of the merge 
to master.

Re: Tests, Emacs-25 and Conditional Features

[Eli Zaretskii]

> From: phillip.lord@russet.org.uk (Phillip Lord)
> Date: Wed, 16 Mar 2016 13:36:37 +0000
> 
> First, does the "no changes on Emacs-25" rule apply to changes in the
> test directory -- i.e. can next tests be added.

Yes, you can add new tests.

In general, anything that doesn't modify the code of Emacs proper and
of the auxiliary utilities (etags, ebrowse, etc.) can be added to the
release branch.

Re: Tests, Emacs-25 and Conditional Features

[John Wiegley]

[-- Attachment #1: Type: text/plain, Size: 844 bytes --]

>>>>> Eli Zaretskii <eliz@gnu.org> writes:

> Yes, you can add new tests.

> In general, anything that doesn't modify the code of Emacs proper and of the
> auxiliary utilities (etags, ebrowse, etc.) can be added to the release
> branch.

Yes, as Eli mentions, you can add new tests, and certain changes outside of
the code and utilities. When in doubt, please ask here.

The exception to this rule is that we cannot add such changes after the
release candidates start being made, since we run the risk of including a test
in the final release that does not work on all platforms. Since we're still in
the phase of creating pretests, it's OK (and helpful) to add more tests.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 629 bytes --]