make check fails due to missing test directory

make check fails due to missing test directory

[Jeffrey Walton]

Hi Everyone,

I'm building 26.2 from sources. My typical workflow is:

  1. unpack tarball
  2. configure
  3. make
  4. make check
  5. sudo make install

The workflow works for about 60 GNU packages I build. However, Emacs fails with:

gmake[2]: Entering directory '/home/build/emacs-26.2/lisp'
gmake[2]: Nothing to be done for 'compile-targets'.
gmake[2]: Leaving directory '/home/build/emacs-26.2/lisp'
gmake[1]: Leaving directory '/home/build/emacs-26.2/lisp'
You do not seem to have the test/ directory.
Maybe you are using a release tarfile, rather than a repository checkout.
gmake: *** [Makefile:943: have-tests] Error 1

How does one tun 'make check' before an install? What special steps
need to be performed?

(I feel like this is an engineering bug in Emacs. I'm baffled 'make
check' does not work out of the box. Only Emacs has this bug).

Thanks in advance.

Re: make check fails due to missing test directory

[Eli Zaretskii]

> From: Jeffrey Walton <noloader@gmail.com>
> Date: Fri, 3 May 2019 14:41:48 -0400
> 
> I'm building 26.2 from sources. My typical workflow is:
> 
>   1. unpack tarball
>   2. configure
>   3. make
>   4. make check
>   5. sudo make install
> 
> The workflow works for about 60 GNU packages I build. However, Emacs fails with:
> 
> gmake[2]: Entering directory '/home/build/emacs-26.2/lisp'
> gmake[2]: Nothing to be done for 'compile-targets'.
> gmake[2]: Leaving directory '/home/build/emacs-26.2/lisp'
> gmake[1]: Leaving directory '/home/build/emacs-26.2/lisp'
> You do not seem to have the test/ directory.
> Maybe you are using a release tarfile, rather than a repository checkout.
> gmake: *** [Makefile:943: have-tests] Error 1
> 
> How does one tun 'make check' before an install? What special steps
> need to be performed?

You need to download the test suite.  It is not part of the release
tarball.  You can find it in the Emacs development repository.

Re: make check fails due to missing test directory

[Paul Eggert]

[-- Attachment #1: Type: text/plain, Size: 947 bytes --]

On 5/3/19 12:13 PM, Eli Zaretskii wrote:
> You need to download the test suite.  It is not part of the release
> tarball.  You can find it in the Emacs development repository.

Sure, but as his email suggests, it's better for 'make check' to succeed
when the set of tests is empty. That way, standardized build procedures
(like his) are more likely to do the right thing. I installed the patch
0001 (attached) into the master branch to do that.

We've talked before about distributing tests in the Emacs tarball -
which is what the GNU Codings Standards recommend and what pretty much
every other GNU package does - and this email suggests that now's a good
time to do that. Distributing tests adds about 2.5% to the size of the
compressed tarball, and nowadays that is a small price to pay for making
tests available to people building from a release tarball rather than a
Git clone. Proposed patch attached as patch 0002, and comments welcome.


[-- Attachment #2: 0001-Skip-tests-if-test-subdir-is-missing.patch --]
[-- Type: text/x-patch, Size: 1464 bytes --]

From 965372cd55c9a9ad5a9e17614dc98502c79ab1e3 Mon Sep 17 00:00:00 2001
From: Paul Eggert <eggert@cs.ucla.edu>
Date: Fri, 3 May 2019 14:19:26 -0700
Subject: [PATCH 1/2] Skip tests if test subdir is missing

Problem reported by Jeffrey Walton in:
https://lists.gnu.org/r/emacs-devel/2019-05/msg00041.html
* Makefile.in (CHECK_TARGETS): New macro; use it
to simplify 'check' and similar rules.
($(CHECK_TARGETS)): If tests are missing, do not fail
after issuing a diagnostic.  Just skip the tests.
---
 Makefile.in | 18 ++++++++----------
 1 file changed, 8 insertions(+), 10 deletions(-)

diff --git a/Makefile.in b/Makefile.in
index 06da415a4a..21362a9196 100644
--- a/Makefile.in
+++ b/Makefile.in
@@ -950,17 +950,15 @@ .PHONY:
 TAGS tags: lib lib-src # src
 	$(MAKE) -C src tags
 
-.PHONY: have-tests
-have-tests:
-	@if test ! -d test; then \
-	  echo "You do not seem to have the test/ directory."; \
-	  echo "Maybe you are using a release tarfile, rather than a repository checkout."; \
-	 exit 1; \
-	fi
-
-.PHONY: check check-maybe check-expensive check-all
-check check-maybe check-expensive check-all: have-tests all
+CHECK_TARGETS = check check-maybe check-expensive check-all
+.PHONY: $(CHECK_TARGETS)
+$(CHECK_TARGETS): all
+ifeq ($(wildcard test),test)
 	$(MAKE) -C test $@
+else
+	@echo "You do not seem to have the test/ directory."
+	@echo "Maybe you used a release tarfile that lacks tests."
+endif
 
 dist:
 	cd ${srcdir}; ./make-dist
-- 
2.21.0


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: 0002-Distribute-test-cases-in-tarballs-by-default.patch --]
[-- Type: text/x-patch; name="0002-Distribute-test-cases-in-tarballs-by-default.patch", Size: 4699 bytes --]

From c0f5ddc7b376976634fb4708519a12bc147ba312 Mon Sep 17 00:00:00 2001
From: Paul Eggert <eggert@cs.ucla.edu>
Date: Fri, 3 May 2019 14:53:58 -0700
Subject: [PATCH 2/2] Distribute test cases in tarballs by default
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

* INSTALL, INSTALL.REPO, admin/make-tarball.txt:
Mention ‘make check’.
* configure.ac: Update comment.
* etc/NEWS: Say that tarballs have a test directory.
* make-dist (with_tests): Default to "yes".
Add an option --no-tests to make it "no".
---
 INSTALL                |  4 ++++
 INSTALL.REPO           |  4 ++--
 admin/make-tarball.txt |  2 +-
 configure.ac           |  3 +--
 etc/NEWS               |  3 +++
 make-dist              | 11 ++++++++---
 6 files changed, 19 insertions(+), 8 deletions(-)

diff --git a/INSTALL b/INSTALL
index e38635c60b..0d5d8aa1cb 100644
--- a/INSTALL
+++ b/INSTALL
@@ -82,6 +82,10 @@ sections if you need to.
 
 		 src/emacs -Q
 
+     To test Emacs further:
+
+		 make check
+
   6. Assuming that the program 'src/emacs' starts and displays its
      opening screen, you can install the program and its auxiliary
      files into their installation directories:
diff --git a/INSTALL.REPO b/INSTALL.REPO
index 6dca9dd714..b7433856c6 100644
--- a/INSTALL.REPO
+++ b/INSTALL.REPO
@@ -32,8 +32,8 @@ can invoke './configure -C'.  After configuring, build Emacs as follows:
 
   $ make
 
-If you want to install Emacs, type 'make install' instead of 'make' in
-the last command.
+You can also type 'make check' to test and 'make install' to install
+Emacs.
 
 Occasionally the file 'lisp/loaddefs.el' (and similar automatically
 generated files, such as 'esh-groups.el', and '*-loaddefs.el' in some
diff --git a/admin/make-tarball.txt b/admin/make-tarball.txt
index 47b60173f8..43992a0bb2 100644
--- a/admin/make-tarball.txt
+++ b/admin/make-tarball.txt
@@ -115,7 +115,7 @@ General steps (for each step, check for possible errors):
     results against the new tar contents.
 
 7.   tar -xf emacs-NEW.tar; cd emacs-NEW
-     ./configure --prefix=/tmp/emacs && make && make install
+     ./configure --prefix=/tmp/emacs && make check && make install
     Use 'script' or M-x compile to save the compilation log in
     compile-NEW.log and compare it against an old one.  The easiest way
     to do that is to visit the old log in Emacs, change the version
diff --git a/configure.ac b/configure.ac
index 79fe0c98c6..ecce0223e0 100644
--- a/configure.ac
+++ b/configure.ac
@@ -5726,9 +5726,8 @@ m4_define
 SUBDIR_MAKEFILES="subdir_makefiles"
 AC_CONFIG_FILES(subdir_makefiles)
 
-dnl test/ is not present in release tarfiles.
+dnl The test/ directory is missing if './make-dist --no-tests' was used.
 opt_makefile=test/Makefile
-
 if test -f "$srcdir/$opt_makefile.in"; then
   SUBDIR_MAKEFILES="$SUBDIR_MAKEFILES $opt_makefile"
   dnl Again, it's best not to use a variable.  Though you can add
diff --git a/etc/NEWS b/etc/NEWS
index 9e3559d27e..f76000b956 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -97,6 +97,9 @@ option was useful with modern debugging tools such as AddressSanitizer.
 (See etc/DEBUG for the details of using the modern replacements of the
 removed configure options.)
 
++++
+** The distribution tarball now has test cases; 'make check' runs them.
+
 ---
 ** Emacs now requires GTK 2.24 and GTK 3.10 for the GTK 2 and GTK 3
 builds respectively.
diff --git a/make-dist b/make-dist
index 74660032e9..e4e6d40d98 100755
--- a/make-dist
+++ b/make-dist
@@ -52,7 +52,7 @@ make_tar=
 default_gzip=gzip
 newer=""
 with_info=yes
-with_tests=no
+with_tests=yes
 changelog=yes
 verbose=no
 
@@ -110,11 +110,16 @@ verbose=
      ;;
 
     ## Include the test/ directory.
-    ## This option is mainly for the hydra build server.
+    ## This is for backward compability to when --no-tests was the default.
     "--tests")
       with_tests=yes
      ;;
 
+    ## Exclude the test/ directory.
+    "--no-tests")
+      with_tests=no
+     ;;
+
     "--verbose")
       verbose=yes
      ;;
@@ -131,9 +136,9 @@ verbose=
       echo "  --no-update	don't recompile or do analogous things"
       echo "  --no-changelog	don't generate the top-level ChangeLog"
       echo "  --no-info		don't include info files"
+      echo "  --no-tests	don't include the test/ directory"
       echo "  --snapshot	same as --clean-up --no-update --tar --no-check"
       echo "  --tar		make a tar file"
-      echo "  --tests	include the test/ directory"
       echo "  --verbose		noisier output"
       echo ""
       exit 0
-- 
2.21.0

Re: make check fails due to missing test directory

[Alan Mackenzie]

Hello, Paul

On Fri, May 03, 2019 at 15:01:58 -0700, Paul Eggert wrote:
> On 5/3/19 12:13 PM, Eli Zaretskii wrote:
> > You need to download the test suite.  It is not part of the release
> > tarball.  You can find it in the Emacs development repository.

> Sure, but as his email suggests, it's better for 'make check' to succeed
> when the set of tests is empty. That way, standardized build procedures
> (like his) are more likely to do the right thing. I installed the patch
> 0001 (attached) into the master branch to do that.

> We've talked before about distributing tests in the Emacs tarball -
> which is what the GNU Codings Standards recommend and what pretty much
> every other GNU package does - and this email suggests that now's a good
> time to do that. Distributing tests adds about 2.5% to the size of the
> compressed tarball, and nowadays that is a small price to pay for making
> tests available to people building from a release tarball rather than a
> Git clone. Proposed patch attached as patch 0002, and comments welcome.

The tests in make check are development tests.  They're not build tests.
In fact, they're fairly useless for anybody who isn't developing Emacs.
Somebody who is developing Emacs will have the git repository.

Therefore, we shouldn't be encouraging users to run these tests, just as
we don't encourage users to byte compile the .el files.  Both these
things have been done as part of the release process, and both are
wastes of time for users.  I, personally, would certainly not run these
tests after downloading an Emacs release.

So I think, if we're going to include these tests in releases, there
should be a strong recommendation in the docs NOT to run them, that
they're only there for people with special needs.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: make check fails due to missing test directory

[Paul Eggert]

Alan Mackenzie wrote:
> The tests in make check are development tests.  They're not build tests.
> In fact, they're fairly useless for anybody who isn't developing Emacs.
> Somebody who is developing Emacs will have the git repository.

In what sense does Emacs differ from other GNU packages, like Coreutils?

Coreutils has a series of tests that are similarly "useless" for anybody who 
isn't developing Coreutils. However, builders of Coreutils can run 'make check' 
to run these tests. If tests fail, builders can decide themselves whether to 
install the build, or to stick with an older version. And they can report 
problems to developers, often problems that the developers themselves couldn't 
easily find. The developers and builders can then collaborate to fix the 
problem. Yes, the process can be a bit awkward, but it beats leaving the bugs 
unfixed. And no part of this process requires builders to understand the code 
well enough to fix it.

Why shouldn't Emacs do that too?

Re: make check fails due to missing test directory

[Michael Albinus]

Alan Mackenzie <acm@muc.de> writes:

> Hello, Paul

Hi Alan,

> The tests in make check are development tests.  They're not build tests.
> In fact, they're fairly useless for anybody who isn't developing Emacs.
> Somebody who is developing Emacs will have the git repository.

If people report a Tramp problem I'm not able to reproduce, I encourage
them to run tramp-tests. Of course, with a proper remote target directory,
controlled by environment variables.

Best regards, Michael.

Emacs distribution tarball now supports 'make check'

[Paul Eggert]

On 5/3/19 3:01 PM, Paul Eggert wrote:
> We've talked before about distributing tests in the Emacs tarball -
> which is what the GNU Codings Standards recommend and what pretty much
> every other GNU package does - and this email suggests that now's a good
> time to do that. Distributing tests adds about 2.5% to the size of the
> compressed tarball, and nowadays that is a small price to pay for making
> tests available to people building from a release tarball rather than a
> Git clone. Proposed patch attached as patch 0002, and comments welcome.

No comments other than Alan's, so I updated that patch with Alan's
comments in mind (the updated version says that 'make check' is intended
mostly to help developers), and installed it into the master branch.