Re: Tests, Emacs-25 and Conditional Features

[Eli Zaretskii <eliz@gnu.org>]

> From: phillip.lord@russet.org.uk (Phillip Lord)
> Date: Thu, 17 Mar 2016 10:14:44 +0000
> Cc: emacs-devel@gnu.org
> 
> > (skip-unless (gnutls-available-p))
> > (skip-unless (functionp 'libxml-parse-xml-region))
> 
> Yeah, thought about that. But then the test is skipped if
> gnutls-available-p returns nil. What happens if I believe that I have
> configured and build Emacs to include gnutls, but, for some reason it
> isn't. The test will be skipped when it should succeed.

Tests that are skipped are indicated in the test log file, so if you
see there a feature being skipped that you thought should be
available, you can investigate.

More generally, I don't see how Emacs could know what you believe.  We
need to define the meaning of that first, and then find a way to
communicate that to Emacs.

For example, one possible interpretation of "believe" is the report
displayed by 'configure' near its end.  A similar interpretation is
the value of system-configuration-features.  However, it might happen
that the user expects her build to support feature X, but that feature
is not in any of these, in which case the only way I see is to ask the
user.

Then there's the issue of how do you test this.  An API that was not
compiled into Emacs cannot be invoked, so what do we want to show the
user? a "void function" error? ert error messages are not so easy to
interpret.

IOW, I think you are talking about a kind of testing we don't yet
have, and it should be designed first.  The current test suite aims at
testing the code, using the available Emacs binary as best it can.  It
does NOT test user assumptions about the build -- that is a separate
and different issue, which requires rather different approach, IMO.