Re: notmuch, OpenBSD issues

Re: notmuch, OpenBSD issues

[David Bremner]

Allan Streib <astreib@indiana.edu> writes:

> Hi David,
>
> I've been using notmuch on OpenBSD for a while, and I really appreciate
> the project.

I hope you don't mind my replying to the list. Notmuch is not a one man project...

>
> In recent days I've had a couple problems building. First issue was a
> complaint about "cannonicalize_file_name" being an unknown symbol. I
> fixed that with the following:

Yeah, that's actually a bug in my fix other fix for openbsd.
There's actually a patch queued for merging on the mailing list, see

id:1397042643-10975-1-git-send-email-david@tethera.net

> The other issue is with the support for gzipped dump and restore;
> OpenBSD currently provides zlib version 1.2.3 which apparently is too
> old for what you are doing. For now I am just building from commit
> 69867c33 which pre-dates the inclusion of the zlib version checks. Is
> there a convenient way to make the gzipped dump and restore optional, so
> that I can continue to keep up with other changes?
>
> If not I can look into building a newer version of zlib for use with
> notmuch.

Unfortunately I can't think of a simple change to make gzipped dump and
restore optional; somewhat ironically we actually need the later zlib
for dumping uncompressed backups. This is clearly not a feature we want
to take away.  I guess it would be possible to write a compat shim on
top of zlib that implemented the passthrough that we need. Tomi Ollila,
who runs notmuch on various old systems, might a have a comment about
that.  Meanwhile, unlike libc, it should be possible to compile a newer 
zlib.

d

Re: notmuch, OpenBSD issues

[David Mazieres]

David Bremner <david@tethera.net> writes:

> Allan Streib <astreib@indiana.edu> writes:
>
>> I've been using notmuch on OpenBSD for a while, and I really appreciate
>> the project.

Since there is a thread on this...

I'm using notmuch 0.17 on openbsd (from the ports tree).  My problem is
that notmuch new is just unbearably slow.  I don't know if it's because
I'm running the 32-bit (i386) mode or what, but it takes over one second
per mail message.  E.g., this is typical of what I see when checking for
new mail:

        Processed 18 total files in 20s (0 files/sec.).
        Added 5 new messages to the database.

Linux is 10 times faster.  Have you seen any similar performance issues?

David

[SUPPORTIVE PATCH 0.17.200] build and use statically linked zlib-1.2.8 with notmuch

[Tomi Ollila]

¡¡ NOT TO BE MERGED TO NOTMUCH REPOSITORY !!

(Due to its intrusive nature this patch is not suitable for merging.)

Recent notmuch(*) requires zlib 1.2.5.2 or newer; systems that have older
versions of zlib installed can use this patch to use zlib 1.2.8 with
notmuch but keep the current one to be used by default.

Note that zlib 1.2.8 will be downloaded (if not already there) and built
during ./configure time.

(since ~ 0.17-194-g3921d23 -- Sat 2014-04-12)
---

HTH :)

Tomi

 Makefile.local       |  7 ++--
 compat/build-zlib.py | 96 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 configure            |  7 ++++
 3 files changed, 108 insertions(+), 2 deletions(-)
 create mode 100755 compat/build-zlib.py

diff --git a/Makefile.local b/Makefile.local
index 877a979..1780cee 100644
--- a/Makefile.local
+++ b/Makefile.local
@@ -292,11 +292,14 @@ notmuch_client_modules = $(notmuch_client_srcs:.c=.o)
 
 notmuch.o: version.stamp
 
+# XXX ZA used in unofficial supportive patch which uses self-built zlib XXX
+ZA=./compat/zlib-1.2.8/libz.a
+
 notmuch: $(notmuch_client_modules) lib/libnotmuch.a util/libutil.a parse-time-string/libparse-time-string.a
-	$(call quiet,CXX $(CFLAGS)) $^ $(FINAL_LIBNOTMUCH_LDFLAGS) -o $@
+	$(call quiet,CXX $(CFLAGS)) $^ $(ZA) $(FINAL_LIBNOTMUCH_LDFLAGS) -o $@
 
 notmuch-shared: $(notmuch_client_modules) lib/$(LINKER_NAME)
-	$(call quiet,$(FINAL_NOTMUCH_LINKER) $(CFLAGS)) $(notmuch_client_modules) $(FINAL_NOTMUCH_LDFLAGS) -o $@
+	$(call quiet,$(FINAL_NOTMUCH_LINKER) $(CFLAGS)) $(notmuch_client_modules) $(ZA) $(FINAL_NOTMUCH_LDFLAGS) -o $@
 
 .PHONY: install
 install: all install-man
diff --git a/compat/build-zlib.py b/compat/build-zlib.py
new file mode 100755
index 0000000..25e21d9
--- /dev/null
+++ b/compat/build-zlib.py
@@ -0,0 +1,96 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+# $ build-zlib.py $
+#
+# Authors: Tomi Ollila,
+# License: Public Domain
+#
+# Created: Sat 12 Apr 2014 17:51:42 EEST too
+# Last modified: Sat 12 Apr 2014 18:52:06 +0300 too
+
+# Not using string.format() for python 2.5 compatibility.
+# Not using print due to python version 2 & 3 differences.
+
+# The seemlingly extra 'pass'es work with python mode provided
+# with Emacs 24.1 and earlier (indentation!)...
+
+import sys
+import os
+try:  # Python 3
+    from urllib.request import urlopen
+except ImportError:  # Python 2
+    from urllib import urlopen
+import shutil
+import hashlib
+
+ZLIBurl='http://prdownloads.sourceforge.net/libpng/zlib-1.2.8.tar.gz?download'
+ZLIBtgz='zlib-1.2.8.tar.gz'
+ZLIBsum='36658cb768a54c1d4dec43c3116c27ed893e88b02ecfcb44f2166f9c0b7f2a0d'
+ZLIBdir='zlib-1.2.8'
+
+
+class Die(Exception):
+    def __init__(self, *_list):
+        sys.stderr.write("\n")
+        sys.stderr.write(_list[0] % _list[1:])
+        sys.stderr.write("\n\n")
+        raise SystemExit(1)
+    pass
+
+
+def dl_zlibtgz():
+    sys.stdout.write("Downloading %s...\n" % ZLIBtgz)
+    response = urlopen(ZLIBurl)
+    # python3 throws urllib.error.HTTPError, python2 does not.
+    if response.getcode() != 200:
+        sys.stderr.write(response.read(4096))  # max 4096 octets
+        exit(1)
+    outfile = open(ZLIBtgz, 'wb')
+    shutil.copyfileobj(response, outfile)
+    pass
+
+
+def main():
+    if not os.path.isfile(ZLIBtgz):
+        dl_zlibtgz()
+        pass
+
+    sys.stdout.write("Checksumming %s...\n" % ZLIBtgz)
+    infile = open(ZLIBtgz, 'rb')
+    sha256 = hashlib.sha256()
+    while True:
+        data = infile.read(65536)
+        if not data:
+            break
+        sha256.update(data)
+        pass
+    sys.stdout.write(" Expt'd sha256 %s\n" % ZLIBsum)
+    hexdigest = sha256.hexdigest()
+    sys.stdout.write(" Actual sha256 %s\n" % hexdigest)
+    if hexdigest != ZLIBsum:
+        raise Die("File '%s' checksum mismatch", ZLIBtgz)
+    pass
+
+    target = ZLIBdir + '/libz.a'
+    if os.path.isfile(target):
+        sys.stdout.write("'%s' exists. OK.\n" % target)
+        raise SystemExit(0)
+
+    def x(*args):
+        sys.stdout.write('x ' + ' '.join(args) + '\n')
+        os.spawnvp(os.P_WAIT, args[0], args)
+        pass
+
+    sys.stdout.write('Building %s\n' % ZLIBdir)
+    x ('rm', '-rf', ZLIBdir)
+    x ('tar', '-zxf', ZLIBtgz)
+    os.chdir(ZLIBdir)
+    x ('./configure')
+    x ('make')
+    os.chdir('..')
+    if not os.path.isfile(target):
+        raise Die("Could not build '%s' for some reason!", target)
+
+if __name__ == '__main__':
+    main()
+    pass  # pylint: disable=W0107
diff --git a/configure b/configure
index f447581..4ba0403 100755
--- a/configure
+++ b/configure
@@ -342,6 +342,13 @@ fi
 
 printf "Checking for zlib (>= 1.2.5.2)... "
 have_zlib=0
+# XXX unofficial supportive patch to use self-built zlib XXX
+printf "Checking/building zlib 1.2.8:\n"
+(cd compat; exec ../"$srcdir"/compat/build-zlib.py)
+have_zlib=1
+zlib_cflags="-I ./compat/zlib-1.2.8"
+zlib_ldflags=
+: ||
 if pkg-config --atleast-version=1.2.5.2 zlib; then
     printf "Yes.\n"
     have_zlib=1
-- 
1.8.0

Re: [SUPPORTIVE PATCH 0.17.200] build and use statically linked zlib-1.2.8 with notmuch

[Tomi Ollila]

On Fri, Apr 18 2014, Tomi Ollila <tomi.ollila@iki.fi> wrote:

> ¡¡ NOT TO BE MERGED TO NOTMUCH REPOSITORY !!
>
> (Due to its intrusive nature this patch is not suitable for merging.)

...

> --- a/configure
> +++ b/configure
> @@ -342,6 +342,13 @@ fi
>  
>  printf "Checking for zlib (>= 1.2.5.2)... "
>  have_zlib=0
> +# XXX unofficial supportive patch to use self-built zlib XXX
> +printf "Checking/building zlib 1.2.8:\n"
> +(cd compat; exec ../"$srcdir"/compat/build-zlib.py)

... and next version will check whether "$srcdir" starts with '/'
--- fix it by hand if you have problems there...

Tomi

> +have_zlib=1
> +zlib_cflags="-I ./compat/zlib-1.2.8"
> +zlib_ldflags=
> +: ||
>  if pkg-config --atleast-version=1.2.5.2 zlib; then
>      printf "Yes.\n"
>      have_zlib=1
> -- 
> 1.8.0

Re: notmuch, OpenBSD issues

[Allan Streib]

David Mazieres <dm-list-email-notmuch@scs.stanford.edu> writes:

> I'm using notmuch 0.17 on openbsd (from the ports tree).  My problem is
> that notmuch new is just unbearably slow.  I don't know if it's because
> I'm running the 32-bit (i386) mode or what, but it takes over one second
> per mail message.  E.g., this is typical of what I see when checking for
> new mail:
>
>         Processed 18 total files in 20s (0 files/sec.).
>         Added 5 new messages to the database.
>
> Linux is 10 times faster.  Have you seen any similar performance issues?

I didn't realize there was a port. I don't see it in my tree (amd64,
recent 5.5-current). I've been building from the git repo.

Anyway, yes I have seen similar on an older Mac OS X install I have at
home on a G4 PowerBook. I haven't had "unbearably" slow performance on
my amd64 OpenBSD installation, but there is sometimes a noticable lag.

I wonder if it would help to enable softupdates on the partition where
your database lives? http://www.openbsd.org/faq/faq14.html#SoftUpdates

Allan