[PATCH 0/8] notmuch on Windows (Cygwin) and AppVeyor

[PATCH 0/8] notmuch on Windows (Cygwin) and AppVeyor

[Vladimir Panteleev]

Hi,

I've spent a bit of time over the past few days trying to get notmuch
to build and run tests on CygWin and AppVeyor CI's systems (AppVeyor
being a gratis continuous integration provider for Windows). Not from
some personal need, but simply because I generally enjoy and have some
experience with tinkering with CI systems.

I've gotten as far as to get everything to build and the test suite to
run (but not pass). A few ugly hacks were needed (luckily, mostly
outside notmuch's existing codebase), mainly related to talloc, as it
seems that it (or Waf, which it uses) does not speak Cygwin well. I
will probably draw the line here, as it looks like actually fixing
some of the failing tests would take some non-trivial work (e.g. one
test failure I saw seems to point towards a bug in Xapian/Cygwin). I'm
also not convinced that, practically speaking, the effort would be
well spent, as probably the best way to run notmuch under Windows
would be using Windows Subsystem for Linux instead of an userspace
POSIX environment like Cygwin or MSYS.

Some notes:

- 64-bit Cygwin seems to work much better than the 32-bit version,
  which ran into issues apparently caused by the limited address space
  in some scenarios I've seen.

- Partial updates of a Cygwin installation can predictably lead to a
  broken installation. --upgrade-also should always be used for
  unattended invocations of Cygwin's setup.exe.

- MSYS also packages Xapian and gmime, however an MSYS port is likely
  to be even more challenging, as it provides less of the POSIX
  environment than Cygwin.

You can see the current AppVeyor results here (warning, large page):

https://ci.appveyor.com/project/CyberShadow/notmuch/build/1.0.26

Development drafts of patch series is here:

https://github.com/CyberShadow/notmuch/commits/appveyor

Vladimir Panteleev (8):
  lib: Fix talloc not being defined on Cygwin
  Split CONFIGURE_LDFLAGS out of FINAL_LIBNOTMUCH_LDFLAGS
  lib: Fix libnotmuch.so link under Cygwin
  configure: Refactor uname parsing into a case statement
  configure: Detect Cygwin, and set PLATFORM accordingly
  Fix linking shared library on Cygwin
  test: Don't attempt to build symbol-test on Cygwin
  Add AppVeyor CI configuration and test script

 Makefile.global        |  12 +++++-
 Makefile.local         |   2 +-
 appveyor.yml           |   7 ++++
 configure              | 106 +++++++++++++++++++++++++++----------------------
 devel/appveyor.sh      |  39 ++++++++++++++++++
 lib/Makefile.local     |   2 +-
 lib/database-private.h |   4 +-
 test/Makefile.local    |   5 ++-
 8 files changed, 123 insertions(+), 54 deletions(-)
 create mode 100644 appveyor.yml
 create mode 100644 devel/appveyor.sh

-- 
2.13.3

[PATCH 1/8] lib: Fix talloc not being defined on Cygwin

[Vladimir Panteleev]

From: Vladimir Panteleev <vladimir@thecybershadow.net>

With GNU libc, talloc will not be declared unless _GNU_SOURCE is
defined before including stdio.h. Although this is done in
notmuch-private.h, that file is included after including inttypes.h in
database-private.h. On Cygwin, this apparently leads to including the
header file containing the talloc declaration (before _GNU_SOURCE is
defined), thus causing the notmuch compilation to fail.

* database-private.h: Include notmuch-private.h before including
  inttypes.h.
---
 lib/database-private.h | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/lib/database-private.h b/lib/database-private.h
index 5555554b..117211c7 100644
--- a/lib/database-private.h
+++ b/lib/database-private.h
@@ -21,6 +21,8 @@
 #ifndef NOTMUCH_DATABASE_PRIVATE_H
 #define NOTMUCH_DATABASE_PRIVATE_H
 
+#include "notmuch-private.h"
+
 /* According to WG14/N1124, a C++ implementation won't provide us a
  * macro like PRIx64 (which gives a printf format string for
  * formatting a uint64_t as hexadecimal) unless we define
@@ -30,8 +32,6 @@
 #define __STDC_FORMAT_MACROS
 #include <inttypes.h>
 
-#include "notmuch-private.h"
-
 #ifdef SILENCE_XAPIAN_DEPRECATION_WARNINGS
 #define XAPIAN_DEPRECATED(D) D
 #endif
-- 
2.13.3

[PATCH 2/8] Split CONFIGURE_LDFLAGS out of FINAL_LIBNOTMUCH_LDFLAGS

[Vladimir Panteleev]

Previously, FINAL_LIBNOTMUCH_LDFLAGS contained two different kinds of
flags:
- Linker configuration flags (LDFLAGS and AS_NEEDED_FLAGS)
- List of items to link (CONFIGURE_LDFLAGS)

As it can be necessary to prepend to the list of items to link, while
still allow them to be affected by the position-dependent linker
configuration flags, split CONFIGURE_LDFLAGS out of
FINAL_LIBNOTMUCH_LDFLAGS and mention it explicitly when necessary.
---
 Makefile.global    | 2 +-
 Makefile.local     | 2 +-
 lib/Makefile.local | 2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/Makefile.global b/Makefile.global
index cae4c7d1..5d10bff7 100644
--- a/Makefile.global
+++ b/Makefile.global
@@ -62,4 +62,4 @@ ifneq ($(LINKER_RESOLVES_LIBRARY_DEPENDENCIES),1)
 FINAL_NOTMUCH_LDFLAGS += $(CONFIGURE_LDFLAGS)
 FINAL_NOTMUCH_LINKER = CXX
 endif
-FINAL_LIBNOTMUCH_LDFLAGS = $(LDFLAGS) $(AS_NEEDED_LDFLAGS) $(CONFIGURE_LDFLAGS)
+FINAL_LIBNOTMUCH_LDFLAGS = $(LDFLAGS) $(AS_NEEDED_LDFLAGS)
diff --git a/Makefile.local b/Makefile.local
index af12ca7f..9b0f7d30 100644
--- a/Makefile.local
+++ b/Makefile.local
@@ -246,7 +246,7 @@ notmuch_client_modules = $(notmuch_client_srcs:.c=.o)
 notmuch.o: version.stamp
 
 notmuch: $(notmuch_client_modules) lib/libnotmuch.a util/libnotmuch_util.a parse-time-string/libparse-time-string.a
-	$(call quiet,CXX $(CFLAGS)) $^ $(FINAL_LIBNOTMUCH_LDFLAGS) -o $@
+	$(call quiet,CXX $(CFLAGS)) $^ $(FINAL_LIBNOTMUCH_LDFLAGS) $(CONFIGURE_LDFLAGS) -o $@
 
 notmuch-shared: $(notmuch_client_modules) lib/$(LINKER_NAME)
 	$(call quiet,$(FINAL_NOTMUCH_LINKER) $(CFLAGS)) $(notmuch_client_modules) $(FINAL_NOTMUCH_LDFLAGS) -o $@
diff --git a/lib/Makefile.local b/lib/Makefile.local
index 0b5c4b08..fbcdbda8 100644
--- a/lib/Makefile.local
+++ b/lib/Makefile.local
@@ -65,7 +65,7 @@ $(dir)/libnotmuch.a: $(libnotmuch_modules)
 	$(call quiet,AR) rcs $@ $^
 
 $(dir)/$(LIBNAME): $(libnotmuch_modules) util/libnotmuch_util.a parse-time-string/libparse-time-string.a
-	$(call quiet,CXX $(CXXFLAGS)) $(libnotmuch_modules) $(FINAL_LIBNOTMUCH_LDFLAGS) $(LIBRARY_LINK_FLAG) -o $@ util/libnotmuch_util.a parse-time-string/libparse-time-string.a
+	$(call quiet,CXX $(CXXFLAGS)) $(libnotmuch_modules) $(FINAL_LIBNOTMUCH_LDFLAGS) $(CONFIGURE_LDFLAGS) $(LIBRARY_LINK_FLAG) -o $@ util/libnotmuch_util.a parse-time-string/libparse-time-string.a
 
 $(dir)/$(SONAME): $(dir)/$(LIBNAME)
 	ln -sf $(LIBNAME) $@
-- 
2.13.3

[PATCH 3/8] lib: Fix libnotmuch.so link under Cygwin

[Vladimir Panteleev]

libnotmuch_util.a was specified before -lgmime-2.6 on the linker
command line, even though it uses symbols from it. On Cygwin, this
lead to a link error.

* Makefile.local: Fix the order of the arguments so that dependencies
  always occur after their dependents.
---
 lib/Makefile.local | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/Makefile.local b/lib/Makefile.local
index fbcdbda8..ff1f6363 100644
--- a/lib/Makefile.local
+++ b/lib/Makefile.local
@@ -65,7 +65,7 @@ $(dir)/libnotmuch.a: $(libnotmuch_modules)
 	$(call quiet,AR) rcs $@ $^
 
 $(dir)/$(LIBNAME): $(libnotmuch_modules) util/libnotmuch_util.a parse-time-string/libparse-time-string.a
-	$(call quiet,CXX $(CXXFLAGS)) $(libnotmuch_modules) $(FINAL_LIBNOTMUCH_LDFLAGS) $(CONFIGURE_LDFLAGS) $(LIBRARY_LINK_FLAG) -o $@ util/libnotmuch_util.a parse-time-string/libparse-time-string.a
+	$(call quiet,CXX $(CXXFLAGS)) $(libnotmuch_modules) $(FINAL_LIBNOTMUCH_LDFLAGS) util/libnotmuch_util.a parse-time-string/libparse-time-string.a $(CONFIGURE_LDFLAGS) $(LIBRARY_LINK_FLAG) -o $@
 
 $(dir)/$(SONAME): $(dir)/$(LIBNAME)
 	ln -sf $(LIBNAME) $@
-- 
2.13.3

[PATCH 4/8] configure: Refactor uname parsing into a case statement

[Vladimir Panteleev]

From: Vladimir Panteleev <vladimir@thecybershadow.net>

This will allow matching unames using patterns.
---
 configure | 101 +++++++++++++++++++++++++++++++++-----------------------------
 1 file changed, 54 insertions(+), 47 deletions(-)

diff --git a/configure b/configure
index c5e2ffed..17ac67bb 100755
--- a/configure
+++ b/configure
@@ -650,59 +650,66 @@ libdir_in_ldconfig=0
 
 printf "Checking which platform we are on... "
 uname=$(uname)
-if [ $uname = "Darwin" ] ; then
-    printf "Mac OS X.\n"
-    platform=MACOSX
-    linker_resolves_library_dependencies=0
-elif [ $uname = "SunOS" ] ; then
-    printf "Solaris.\n"
-    platform=SOLARIS
-    linker_resolves_library_dependencies=0
-elif [ $uname = "FreeBSD" ] ; then
-    printf "FreeBSD.\n"
-    platform=FREEBSD
-    linker_resolves_library_dependencies=0
-elif [ $uname = "OpenBSD" ] ; then
-    printf "OpenBSD.\n"
-    platform=OPENBSD
-    linker_resolves_library_dependencies=0
-elif [ $uname = "Linux" ] || [ $uname = "GNU" ] ; then
-    printf "%s\n" "$uname"
-    platform="$uname"
-    linker_resolves_library_dependencies=1
-
-    printf "Checking for %s in ldconfig... " "$libdir_expanded"
-    ldconfig_paths=$(/sbin/ldconfig -N -X -v 2>/dev/null | sed -n -e 's,^\(/.*\):\( (.*)\)\?$,\1,p')
-    # Separate ldconfig_paths only on newline (not on any potential
-    # embedded space characters in any filenames). Note, we use a
-    # literal newline in the source here rather than something like:
-    #
-    #	IFS=$(printf '\n')
-    #
-    # because the shell's command substitution deletes any trailing newlines.
-    IFS="
+case "$uname" in
+    Darwin)
+	printf "Mac OS X.\n"
+	platform=MACOSX
+	linker_resolves_library_dependencies=0
+	;;
+    SunOS)
+	printf "Solaris.\n"
+	platform=SOLARIS
+	linker_resolves_library_dependencies=0
+	;;
+    FreeBSD)
+	printf "FreeBSD.\n"
+	platform=FREEBSD
+	linker_resolves_library_dependencies=0
+	;;
+    OpenBSD)
+	printf "OpenBSD.\n"
+	platform=OPENBSD
+	linker_resolves_library_dependencies=0
+	;;
+    Linux|GNU)
+	printf "%s\n" "$uname"
+	platform="$uname"
+	linker_resolves_library_dependencies=1
+
+	printf "Checking for %s in ldconfig... " "$libdir_expanded"
+	ldconfig_paths=$(/sbin/ldconfig -N -X -v 2>/dev/null | sed -n -e 's,^\(/.*\):\( (.*)\)\?$,\1,p')
+	# Separate ldconfig_paths only on newline (not on any potential
+	# embedded space characters in any filenames). Note, we use a
+	# literal newline in the source here rather than something like:
+	#
+	#	IFS=$(printf '\n')
+	#
+	# because the shell's command substitution deletes any trailing newlines.
+	IFS="
 "
-    for path in $ldconfig_paths; do
-	if [ "$path" = "$libdir_expanded" ]; then
-	    libdir_in_ldconfig=1
+	for path in $ldconfig_paths; do
+	    if [ "$path" = "$libdir_expanded" ]; then
+		libdir_in_ldconfig=1
+	    fi
+	done
+	IFS=$DEFAULT_IFS
+	if [ "$libdir_in_ldconfig" = '0' ]; then
+	    printf "No (will set RPATH)\n"
+	else
+	    printf "Yes\n"
 	fi
-    done
-    IFS=$DEFAULT_IFS
-    if [ "$libdir_in_ldconfig" = '0' ]; then
-	printf "No (will set RPATH)\n"
-    else
-	printf "Yes\n"
-    fi
-else
-    printf "Unknown.\n"
-    platform="$uname"
-    linker_resolves_library_dependencies=0
-    cat <<EOF
+	;;
+    *)
+	printf "Unknown.\n"
+	platform="$uname"
+	linker_resolves_library_dependencies=0
+	cat <<EOF
 
 *** Warning: Unknown platform. Notmuch might or might not build correctly.
 
 EOF
-fi
+	;;
+esac
 
 if [ $errors -gt 0 ]; then
     cat <<EOF
-- 
2.13.3

[PATCH 5/8] configure: Detect Cygwin, and set PLATFORM accordingly

[Vladimir Panteleev]

From: Vladimir Panteleev <vladimir@thecybershadow.net>

---
 configure | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/configure b/configure
index 17ac67bb..f7a54dec 100755
--- a/configure
+++ b/configure
@@ -699,6 +699,11 @@ case "$uname" in
 	    printf "Yes\n"
 	fi
 	;;
+    CYGWIN_NT-*)
+	printf "Cygwin.\n"
+	platform=CYGWIN
+	linker_resolves_library_dependencies=0
+	;;
     *)
 	printf "Unknown.\n"
 	platform="$uname"
-- 
2.13.3

[PATCH 6/8] Fix linking shared library on Cygwin

[Vladimir Panteleev]

From: Vladimir Panteleev <vladimir@thecybershadow.net>

Apparently due to historical reasons, when given -lfoo, Cygwin's ld
will prefer to link against libfoo.a over libfoo.so (or even
libfoo.dll). As such, we need to explicitly specify the library file
name when on Cygwin to avoid creating a broken notmuch-shared binary.

* Makefile.global: Use -l:libnotmuch.so instead of -lnotmuch on
  Cygwin.
---
 Makefile.global | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/Makefile.global b/Makefile.global
index 5d10bff7..0b536302 100644
--- a/Makefile.global
+++ b/Makefile.global
@@ -48,11 +48,19 @@ GPG_FILE=$(SHA256_FILE).asc
 
 PV_FILE=bindings/python/notmuch/version.py
 
+# Cygwin's linker prefers .a files over shared objects, so indicate
+# that we want to link against the shared library explicitly.
+ifeq ($(PLATFORM),CYGWIN)
+NOTMUCH_LIBRARY_FLAG=-l:libnotmuch.so
+else
+NOTMUCH_LIBRARY_FLAG=-lnotmuch
+endif
+
 # Smash together user's values with our extra values
 STD_CFLAGS := -std=gnu99
 FINAL_CFLAGS = -DNOTMUCH_VERSION=$(VERSION) $(CPPFLAGS) $(STD_CFLAGS) $(CFLAGS) $(WARN_CFLAGS) $(extra_cflags) $(CONFIGURE_CFLAGS)
 FINAL_CXXFLAGS = $(CPPFLAGS) $(CXXFLAGS) $(WARN_CXXFLAGS) $(extra_cflags) $(extra_cxxflags) $(CONFIGURE_CXXFLAGS)
-FINAL_NOTMUCH_LDFLAGS = $(LDFLAGS) -Lutil -lnotmuch_util -Llib -lnotmuch
+FINAL_NOTMUCH_LDFLAGS = $(LDFLAGS) -Lutil -lnotmuch_util -Llib $(NOTMUCH_LIBRARY_FLAG)
 ifeq ($(LIBDIR_IN_LDCONFIG),0)
 FINAL_NOTMUCH_LDFLAGS += $(RPATH_LDFLAGS)
 endif
-- 
2.13.3

[PATCH 7/8] test: Don't attempt to build symbol-test on Cygwin

[Vladimir Panteleev]

From: Vladimir Panteleev <vladimir@thecybershadow.net>

The test fails to build on Cygwin; since Windows PE exports are
already namespaced (by module name), the test is likely unnecessary on
Windows anyway.
---
 test/Makefile.local | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/test/Makefile.local b/test/Makefile.local
index 0df72c92..05fcdd6f 100644
--- a/test/Makefile.local
+++ b/test/Makefile.local
@@ -48,10 +48,13 @@ test_main_srcs=$(dir)/arg-test.c \
 	      $(dir)/random-corpus.c \
 	      $(dir)/parse-time.c \
 	      $(dir)/smtp-dummy.c \
-	      $(dir)/symbol-test.cc \
 	      $(dir)/make-db-version.cc \
 	      $(dir)/ghost-report.cc
 
+ifneq ($(PLATFORM),CYGWIN)
+test_main_srcs+=$(dir)/symbol-test.cc
+endif
+
 test_srcs=$(test_main_srcs) $(dir)/database-test.c
 
 TEST_BINARIES := $(test_main_srcs:.c=)
-- 
2.13.3

[PATCH 8/8] Add AppVeyor CI configuration and test script

[Vladimir Panteleev]

From: Vladimir Panteleev <vladimir@thecybershadow.net>

At the time of writing, this successfully builds notmuch and runs the
test suite, with 550/793 tests passing.

Whether or not actually testing on AppVeyor is desirable, the test
script illustrates a verifiably working method to build notmuch on
Windows/Cygwin.
---
 appveyor.yml      |  7 +++++++
 devel/appveyor.sh | 39 +++++++++++++++++++++++++++++++++++++++
 2 files changed, 46 insertions(+)
 create mode 100644 appveyor.yml
 create mode 100644 devel/appveyor.sh

diff --git a/appveyor.yml b/appveyor.yml
new file mode 100644
index 00000000..b4ab6262
--- /dev/null
+++ b/appveyor.yml
@@ -0,0 +1,7 @@
+build_script:
+  # Update the CygWin installer
+  - C:\cygwin64\bin\wget.exe -O /cygdrive/c/cygwin64/setup-x86_64.exe https://cygwin.com/setup-x86_64.exe
+  # Update all packages and install dependencies
+  - C:\cygwin64\setup-x86_64.exe --quiet-mode --upgrade-also --packages gcc-core,gcc-g++,make,python2,python2-devel,libxapian-devel,libgmime2.6-devel,libiconv-devel,docbook-xsl,libxslt,libxml2,zlib-devel
+  # Run our script
+  - C:\cygwin64\bin\bash.exe -l %CD%/devel/appveyor.sh
diff --git a/devel/appveyor.sh b/devel/appveyor.sh
new file mode 100644
index 00000000..651f91f2
--- /dev/null
+++ b/devel/appveyor.sh
@@ -0,0 +1,39 @@
+#!/bin/bash
+set -xeuo pipefail
+shopt -s lastpipe
+
+# Test script for the AppVeyor CI service.
+# Ran from ../appveyor.yml.
+# Runs under CygWin bash.
+
+# Reset PATH variable inherited from Windows environment
+export PATH=/usr/local/bin:/usr/bin:/bin
+
+# Set correct working directory
+cd "$(dirname "$0")"/..
+
+# Download talloc
+test -f talloc-2.1.10.tar.gz || wget https://www.samba.org/ftp/talloc/talloc-2.1.10.tar.gz
+rm -rf talloc-2.1.10
+tar zxvf talloc-2.1.10.tar.gz
+
+# Build/install talloc
+(
+	cd talloc-2.1.10
+	./configure --prefix=/usr --disable-python
+	make || true # fails at linking - fix problem and resume build
+	ln -s cygtalloc-2.dll bin/default/talloc.dll
+	ln -s cygtalloc-2.dll bin/default/cygtalloc.dll
+	make
+	make install
+)
+
+./configure
+
+# Windows PE loader doesn't understand Cygwin symlinks
+cp -L /usr/lib/cygtalloc.dll cygtalloc-2.dll
+
+make
+make test
+
+"$@"
-- 
2.13.3

Re: [PATCH 0/8] notmuch on Windows (Cygwin) and AppVeyor

[David Bremner]

Vladimir Panteleev <notmuch@thecybershadow.net> writes:

> I will probably draw the line here, as it looks like actually fixing
> some of the failing tests would take some non-trivial work (e.g. one
> test failure I saw seems to point towards a bug in Xapian/Cygwin). I'm
> also not convinced that, practically speaking, the effort would be
> well spent, as probably the best way to run notmuch under Windows
> would be using Windows Subsystem for Linux instead of an userspace
> POSIX environment like Cygwin or MSYS.

If I understand correctly, then the series as a whole is mainly of
potential interest to developers wishing to take the notmuch-on-cygwin
project farther, rather than to end users. Are there bits proposed for
inclusion in notmuch now, or is the primary goal discussion and
archiving of the work?

d

Re: [PATCH 0/8] notmuch on Windows (Cygwin) and AppVeyor

[Vladimir Panteleev]

On 2017-08-25 11:35, David Bremner wrote:
> If I understand correctly, then the series as a whole is mainly of
> potential interest to developers wishing to take the notmuch-on-cygwin
> project farther, rather than to end users.

That's correct. It's possible that despite the test suite failures, a 
sufficiently large subset of notmuch's functionality may be working 
sufficiently correctly to use notmuch to some extent as it is now, 
though I haven't actually tried doing so.

> Are there bits proposed for
> inclusion in notmuch now, or is the primary goal discussion and
> archiving of the work?

If there is interest, patches 1-7 could be merged for the sole reason to 
prevent them from going stale, and patch 8 could serve as a base to 
ensure the partial Cygwin support does not regress from its current 
state. Otherwise, archiving is fine too.

If there's interest I can send a v2 with the nits pointed out in IRC.

-- 
Best regards,
  Vladimir