[PATCH 0/4] Notmuch and Xapian update.

[PATCH 0/4] Notmuch and Xapian update.

[Marius Bakke]

This brings notmuch and xapian to the latest upstream releases.

Unfortunately some Xapian tests had to be excluded. If anyone knows
how to make getaddrinfo() work for 127.0.0.1 in the build chroot, I'm
all ears.

NB notmuch users: You may have to adjust 'notmuch-fcc-dirs' after this.
See release announcement:
https://notmuchmail.org/pipermail/notmuch/2016/023216.html

$ git diff --stat master
 gnu/local.mk                                          |  1 +
 gnu/packages/mail.scm                                 | 42 +++++++++++++++++-----
 .../patches/notmuch-emacs-25-compatibility-fix.patch  | 43 +++++++++++++++++++++++
 gnu/packages/search.scm                               | 28 +++++++++------
 4 files changed, 95 insertions(+), 19 deletions(-)

[PATCH 1/4] gnu: notmuch: Update to 0.23.

[Marius Bakke]

* gnu/packages/mail.scm (notmuch, python-notmuch): Update to 0.23.
[home-page]: Use https URL.
---
 gnu/packages/mail.scm | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/gnu/packages/mail.scm b/gnu/packages/mail.scm
index 47c1549..1cb3ef4 100644
--- a/gnu/packages/mail.scm
+++ b/gnu/packages/mail.scm
@@ -494,18 +494,18 @@ invoking @command{notifymuch} from the post-new hook.")
 (define-public notmuch
   (package
     (name "notmuch")
-    (version "0.22.1")
+    (version "0.23")
     (source (origin
               (method url-fetch)
               (uri (string-append "https://notmuchmail.org/releases/notmuch-"
                                   version ".tar.gz"))
               (sha256
                (base32
-                "0jwpda3q023dn3sp41n8648951i7iagfv8zzpriv7hpkjivlafg7"))))
+                "1f51l34rdhjf8lvafrwybkxdsdwx8k9397m7qxd8rdg2irjmpry5"))))
     (build-system gnu-build-system)
     (arguments
-     '(#:tests? #f ; FIXME: 694 tests; 170 fail and 100 are skipped
-                   ; with perl input: 50 fail and 100 are skipped
+     '(#:tests? #f ; FIXME: 723 tests; 187 fail and 100 are skipped
+                   ; with perl input: 67 fail and 100 are skipped
        #:phases (modify-phases %standard-phases
                   (replace 'configure
                     (lambda* (#:key outputs #:allow-other-keys)
@@ -528,7 +528,7 @@ invoking @command{notifymuch} from the post-new hook.")
        ("talloc" ,talloc)
        ("xapian" ,xapian)
        ("zlib" ,zlib)))
-    (home-page "http://notmuchmail.org/")
+    (home-page "https://notmuchmail.org/")
     (synopsis "Thread-based email index, search, and tagging")
     (description
      "Notmuch is a command-line based program for indexing, searching, read-
@@ -579,7 +579,7 @@ useful for email address completion.")
 (define-public python-notmuch
   (package
     (name "python-notmuch")
-    (version "0.22.1")
+    (version "0.23")
     ;; Notmuch python bindings are now unavailable on pypi.  The
     ;; bindings are distributed via the notmuch release tarball.
     (source (package-source notmuch))
-- 
2.10.1

Re: [PATCH 1/4] gnu: notmuch: Update to 0.23.

[Leo Famulari]

On Mon, Oct 17, 2016 at 05:55:03PM +0100, Marius Bakke wrote:
> * gnu/packages/mail.scm (notmuch, python-notmuch): Update to 0.23.
> [home-page]: Use https URL.

LGTM. Is it possible to use (package-version notmuch) as
python-notmuch's version?

Re: [PATCH 1/4] gnu: notmuch: Update to 0.23.

[Marius Bakke]

Leo Famulari <leo@famulari.name> writes:

> On Mon, Oct 17, 2016 at 05:55:03PM +0100, Marius Bakke wrote:
>> * gnu/packages/mail.scm (notmuch, python-notmuch): Update to 0.23.
>> [home-page]: Use https URL.
>
> LGTM. Is it possible to use (package-version notmuch) as
> python-notmuch's version?

That worked fine, thanks.

[PATCH 2/4] gnu: notmuch: Enable tests.

[Marius Bakke]

* gnu/packages/patches/notmuch-emacs-25-compatibility-fix.patch: New file.
* gnu/local.mk (dist_patch_DATA): Register it.
* gnu/packages/mail.scm (notmuch)[arguments]: Add verbose make flags.
  Add phase 'patch-notmuch-lib.el' to fix hard-coded /bin/sh.
  Add 'prepare-test-environment' phase to patch test shebangs and set
  environment.
[native-inputs]: Change from 'emacs-minimal' to 'emacs-no-x'. Add dtach,
  gnupg, man, perl and which.
---
 gnu/local.mk                                       |  1 +
 gnu/packages/mail.scm                              | 36 +++++++++++++++---
 .../notmuch-emacs-25-compatibility-fix.patch       | 43 ++++++++++++++++++++++
 3 files changed, 74 insertions(+), 6 deletions(-)
 create mode 100644 gnu/packages/patches/notmuch-emacs-25-compatibility-fix.patch

diff --git a/gnu/local.mk b/gnu/local.mk
index 9079b42..9d7e14b 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -728,6 +728,7 @@ dist_patch_DATA =						\
   %D%/packages/patches/ninja-tests.patch			\
   %D%/packages/patches/ninja-zero-mtime.patch			\
   %D%/packages/patches/node-9077.patch				\
+  %D%/packages/patches/notmuch-emacs-25-compatibility-fix.patch	\
   %D%/packages/patches/nss-pkgconfig.patch			\
   %D%/packages/patches/nvi-assume-preserve-path.patch		\
   %D%/packages/patches/nvi-dbpagesize-binpower.patch		\
diff --git a/gnu/packages/mail.scm b/gnu/packages/mail.scm
index 1cb3ef4..6ab4035 100644
--- a/gnu/packages/mail.scm
+++ b/gnu/packages/mail.scm
@@ -72,7 +72,9 @@
   #:use-module (gnu packages pkg-config)
   #:use-module (gnu packages flex)
   #:use-module (gnu packages gdb)
+  #:use-module (gnu packages man)
   #:use-module (gnu packages samba)
+  #:use-module (gnu packages screen)
   #:use-module (gnu packages tls)
   #:use-module (gnu packages networking)
   #:use-module (gnu packages web)
@@ -501,12 +503,20 @@ invoking @command{notifymuch} from the post-new hook.")
                                   version ".tar.gz"))
               (sha256
                (base32
-                "1f51l34rdhjf8lvafrwybkxdsdwx8k9397m7qxd8rdg2irjmpry5"))))
+                "1f51l34rdhjf8lvafrwybkxdsdwx8k9397m7qxd8rdg2irjmpry5"))
+              (patches
+               ;; This is included in the next release.
+               ;; https://notmuchmail.org/pipermail/notmuch/2016/023309.html
+               (search-patches "notmuch-emacs-25-compatibility-fix.patch"))))
     (build-system gnu-build-system)
     (arguments
-     '(#:tests? #f ; FIXME: 723 tests; 187 fail and 100 are skipped
-                   ; with perl input: 67 fail and 100 are skipped
+     '(#:make-flags (list "V=1") ; Verbose test output.
        #:phases (modify-phases %standard-phases
+                  (add-after 'unpack 'patch-notmuch-lib.el
+                    (lambda _
+                      (substitute* "emacs/notmuch-lib.el"
+                        (("/bin/sh") (which "sh")))
+                      #t))
                   (replace 'configure
                     (lambda* (#:key outputs #:allow-other-keys)
                       (setenv "CC" "gcc")
@@ -514,14 +524,28 @@ invoking @command{notifymuch} from the post-new hook.")
 
                       (let ((out (assoc-ref outputs "out")))
                         (zero? (system* "./configure"
-                                        (string-append "--prefix=" out)))))))))
+                                        (string-append "--prefix=" out))))))
+                  (add-before 'check 'prepare-test-environment
+                    (lambda _
+                      (setenv "TEST_CC" "gcc")
+                      ;; Patch various inline shell invocations.
+                      (substitute* (find-files "test" "\\.sh$")
+                        (("/bin/sh") (which "sh")))
+                      #t)))))
     (native-inputs
      `(("bash-completion" ,bash-completion)
-       ("emacs" ,emacs-minimal)
+       ("emacs" ,emacs-no-x) ; Minimal lacks libxml, needed for some tests.
        ("pkg-config" ,pkg-config)
        ("python" ,python-2)
        ("python-docutils" ,python2-docutils)
-       ("python-sphinx" ,python2-sphinx)))
+       ("python-sphinx" ,python2-sphinx)
+
+       ;; The following are required for tests only.
+       ("which" ,which)
+       ("dtach" ,dtach)
+       ("gnupg" ,gnupg)
+       ("man" ,man-db)
+       ("perl" ,perl)))
     (inputs
      `(("glib" ,glib)
        ("gmime" ,gmime)
diff --git a/gnu/packages/patches/notmuch-emacs-25-compatibility-fix.patch b/gnu/packages/patches/notmuch-emacs-25-compatibility-fix.patch
new file mode 100644
index 0000000..456b345
--- /dev/null
+++ b/gnu/packages/patches/notmuch-emacs-25-compatibility-fix.patch
@@ -0,0 +1,43 @@
+From f575a346df09c82691bb9e7c462836d982fe31f7 Mon Sep 17 00:00:00 2001
+From: David Bremner <david@tethera.net>
+Date: Sun, 9 Oct 2016 19:30:44 -0300
+Subject: [PATCH] emacs/show: force notmuch-show-buttonise-links to act on
+ lines
+
+This seems to fix a problem with emacs 25 creating partial buttons by
+calling n-s-b-l with a region that does not include the whole button.
+I'm not 100% sure it's legit to act outside the region passed by
+jit-lock, but goto-address-fontify-region (where I borrowed the code
+from) already does this, so this patch to not make things worse.
+---
+ emacs/notmuch-show.el | 12 +++++++-----
+ 1 file changed, 7 insertions(+), 5 deletions(-)
+
+diff --git a/emacs/notmuch-show.el b/emacs/notmuch-show.el
+index 641398d..e7d16f8 100644
+--- a/emacs/notmuch-show.el
++++ b/emacs/notmuch-show.el
+@@ -1174,13 +1174,15 @@ This also turns id:\"<message id>\"-parts and mid: links into
+ buttons for a corresponding notmuch search."
+   (goto-address-fontify-region start end)
+   (save-excursion
+-    (let (links)
+-      (goto-char start)
+-      (while (re-search-forward notmuch-id-regexp end t)
++    (let (links
++	  (beg-line (progn (goto-char start) (line-beginning-position)))
++	  (end-line (progn (goto-char end) (line-end-position))))
++      (goto-char beg-line)
++      (while (re-search-forward notmuch-id-regexp end-line t)
+ 	(push (list (match-beginning 0) (match-end 0)
+ 		    (match-string-no-properties 0)) links))
+-      (goto-char start)
+-      (while (re-search-forward notmuch-mid-regexp end t)
++      (goto-char beg-line)
++      (while (re-search-forward notmuch-mid-regexp end-line t)
+ 	(let* ((mid-cid (match-string-no-properties 1))
+ 	       (mid (save-match-data
+ 		      (string-match "^[^/]*" mid-cid)
+-- 
+2.10.1
+
-- 
2.10.1

Re: [PATCH 2/4] gnu: notmuch: Enable tests.

[Leo Famulari]

On Mon, Oct 17, 2016 at 05:55:04PM +0100, Marius Bakke wrote:
> * gnu/packages/patches/notmuch-emacs-25-compatibility-fix.patch: New file.
> * gnu/local.mk (dist_patch_DATA): Register it.
> * gnu/packages/mail.scm (notmuch)[arguments]: Add verbose make flags.
>   Add phase 'patch-notmuch-lib.el' to fix hard-coded /bin/sh.
>   Add 'prepare-test-environment' phase to patch test shebangs and set
>   environment.
> [native-inputs]: Change from 'emacs-minimal' to 'emacs-no-x'. Add dtach,
>   gnupg, man, perl and which.

A true nit-pick, but all those lines should start at column 0 rather
than being indented.

> +              (patches
> +               ;; This is included in the next release.
> +               ;; https://notmuchmail.org/pipermail/notmuch/2016/023309.html
> +               (search-patches "notmuch-emacs-25-compatibility-fix.patch"))))

Please mention the use of this patch in the commit changelog.

> +       ;; The following are required for tests only.
> +       ("which" ,which)
> +       ("dtach" ,dtach)
> +       ("gnupg" ,gnupg)
> +       ("man" ,man-db)
> +       ("perl" ,perl)))

Can you check if these inputs are referenced by the output? Ideally
not...

$ guix gc --references $(./pre-inst-env guix build notmuch)

> diff --git a/gnu/packages/patches/notmuch-emacs-25-compatibility-fix.patch b/gnu/packages/patches/notmuch-emacs-25-compatibility-fix.patch
> new file mode 100644
> index 0000000..456b345
> --- /dev/null
> +++ b/gnu/packages/patches/notmuch-emacs-25-compatibility-fix.patch
> @@ -0,0 +1,43 @@
> +From f575a346df09c82691bb9e7c462836d982fe31f7 Mon Sep 17 00:00:00 2001
> +From: David Bremner <david@tethera.net>
> +Date: Sun, 9 Oct 2016 19:30:44 -0300
> +Subject: [PATCH] emacs/show: force notmuch-show-buttonise-links to act on
> + lines

If possible, please add a hyperlink to the source of the patch. Ideally,
to a Git commit in a web-based Git repo interface.

Not knowing Emacs Lisp, I can't comment on the patch itself.

Re: [PATCH 2/4] gnu: notmuch: Enable tests.

[Marius Bakke]

Leo Famulari <leo@famulari.name> writes:

> On Mon, Oct 17, 2016 at 05:55:04PM +0100, Marius Bakke wrote:
>> * gnu/packages/patches/notmuch-emacs-25-compatibility-fix.patch: New file.
>> * gnu/local.mk (dist_patch_DATA): Register it.
>> * gnu/packages/mail.scm (notmuch)[arguments]: Add verbose make flags.
>>   Add phase 'patch-notmuch-lib.el' to fix hard-coded /bin/sh.
>>   Add 'prepare-test-environment' phase to patch test shebangs and set
>>   environment.
>> [native-inputs]: Change from 'emacs-minimal' to 'emacs-no-x'. Add dtach,
>>   gnupg, man, perl and which.
>
> A true nit-pick, but all those lines should start at column 0 rather
> than being indented.

Noted.

>
>> +              (patches
>> +               ;; This is included in the next release.
>> +               ;; https://notmuchmail.org/pipermail/notmuch/2016/023309.html
>> +               (search-patches "notmuch-emacs-25-compatibility-fix.patch"))))
>
> Please mention the use of this patch in the commit changelog.

Oops, thanks!

>
>> +       ;; The following are required for tests only.
>> +       ("which" ,which)
>> +       ("dtach" ,dtach)
>> +       ("gnupg" ,gnupg)
>> +       ("man" ,man-db)
>> +       ("perl" ,perl)))
>
> Can you check if these inputs are referenced by the output? Ideally
> not...
>
> $ guix gc --references $(./pre-inst-env guix build notmuch)

Good point, I did not verify this until now. They are not referenced by
the output.

>
>> diff --git a/gnu/packages/patches/notmuch-emacs-25-compatibility-fix.patch b/gnu/packages/patches/notmuch-emacs-25-compatibility-fix.patch
>> new file mode 100644
>> index 0000000..456b345
>> --- /dev/null
>> +++ b/gnu/packages/patches/notmuch-emacs-25-compatibility-fix.patch
>> @@ -0,0 +1,43 @@
>> +From f575a346df09c82691bb9e7c462836d982fe31f7 Mon Sep 17 00:00:00 2001
>> +From: David Bremner <david@tethera.net>
>> +Date: Sun, 9 Oct 2016 19:30:44 -0300
>> +Subject: [PATCH] emacs/show: force notmuch-show-buttonise-links to act on
>> + lines
>
> If possible, please add a hyperlink to the source of the patch. Ideally,
> to a Git commit in a web-based Git repo interface.
>
> Not knowing Emacs Lisp, I can't comment on the patch itself.

The bug was exposed by enabling tests against emacs 25 and will be
included in the next release. See this thread:

https://notmuchmail.org/pipermail/notmuch/2016/023227.html

I will add a gitweb link to the patch header.

[PATCH 3/4] gnu: xapian: Use 'modify-phases'.

[Marius Bakke]

* gnu/packages/search.scm (xapian)[arguments]: Use 'modify-phases'.
---
 gnu/packages/search.scm | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/gnu/packages/search.scm b/gnu/packages/search.scm
index 9eb4039..a43abde 100644
--- a/gnu/packages/search.scm
+++ b/gnu/packages/search.scm
@@ -46,12 +46,12 @@
     (inputs `(("zlib" ,zlib)
               ("util-linux" ,util-linux)))
     (arguments
-     `(#:phases (alist-cons-after
-                 'unpack 'patch-remotetcp-harness
-                 (lambda _
-                   (substitute* "tests/harness/backendmanager_remotetcp.cc"
-                     (("/bin/sh") (which "bash"))))
-                 %standard-phases)))
+     `(#:phases
+       (modify-phases %standard-phases
+         (add-after 'unpack 'patch-remotetcp-harness
+           (lambda _
+             (substitute* "tests/harness/backendmanager_remotetcp.cc"
+               (("/bin/sh") (which "bash"))))))))
     (synopsis "Search Engine Library")
     (description
      "Xapian is a highly adaptable toolkit which allows developers to easily
-- 
2.10.1

Re: [PATCH 3/4] gnu: xapian: Use 'modify-phases'.

[Leo Famulari]

On Mon, Oct 17, 2016 at 05:55:05PM +0100, Marius Bakke wrote:
> * gnu/packages/search.scm (xapian)[arguments]: Use 'modify-phases'.

> +     `(#:phases
> +       (modify-phases %standard-phases
> +         (add-after 'unpack 'patch-remotetcp-harness
> +           (lambda _
> +             (substitute* "tests/harness/backendmanager_remotetcp.cc"
> +               (("/bin/sh") (which "bash"))))))))

Please change it to use (which "sh") instead. Bash changes its behavior
based on whether it is called as `bash` or `sh` (see Invocation in
bash(1)).

Otherwise LGTM.

Re: [PATCH 3/4] gnu: xapian: Use 'modify-phases'.

[Marius Bakke]

Leo Famulari <leo@famulari.name> writes:

> On Mon, Oct 17, 2016 at 05:55:05PM +0100, Marius Bakke wrote:
>> * gnu/packages/search.scm (xapian)[arguments]: Use 'modify-phases'.
>
>> +     `(#:phases
>> +       (modify-phases %standard-phases
>> +         (add-after 'unpack 'patch-remotetcp-harness
>> +           (lambda _
>> +             (substitute* "tests/harness/backendmanager_remotetcp.cc"
>> +               (("/bin/sh") (which "bash"))))))))
>
> Please change it to use (which "sh") instead. Bash changes its behavior
> based on whether it is called as `bash` or `sh` (see Invocation in
> bash(1)).

I'm aware, but decided to remove it in the subsequent patch since it was
"buggy" anyway, and is not needed when remote TCP tests are disabled.

It was left to avoid justifying the fix when changing to modify-phases.
The phase also does not end on a #t.

But I agree, might as well just fix it and leave it even if the tests
don't use it.

[PATCH 4/4] gnu: xapian: Update to 1.4.0.

[Marius Bakke]

* gnu/packages/search.scm (xapian): Update to 1.4.0.
[source]: Use https URL.
[home-page]: Use https URL.
[arguments]: Remove 'patch-remotetcp-harness' phase. Replace 'check'
phase with a list of custom test targets.
---
 gnu/packages/search.scm | 22 +++++++++++++++-------
 1 file changed, 15 insertions(+), 7 deletions(-)

diff --git a/gnu/packages/search.scm b/gnu/packages/search.scm
index a43abde..5d7def5 100644
--- a/gnu/packages/search.scm
+++ b/gnu/packages/search.scm
@@ -35,30 +35,38 @@
 (define-public xapian
   (package
     (name "xapian")
-    (version "1.2.21")
+    (version "1.4.0")
     (source (origin
               (method url-fetch)
-              (uri (string-append "http://oligarchy.co.uk/xapian/" version
+              (uri (string-append "https://oligarchy.co.uk/xapian/" version
                                   "/xapian-core-" version ".tar.xz"))
               (sha256
-               (base32 "0grd2s6gf8yzqwdaa50g57j9d81mxkrrpkyldm2shgyizdc8gx33"))))
+               (base32 "0xv4da5rmqqzkkkzx2v3jwh5hz5zxhd2b7m8x30fk99a25blyn0h"))))
     (build-system gnu-build-system)
     (inputs `(("zlib" ,zlib)
               ("util-linux" ,util-linux)))
     (arguments
      `(#:phases
        (modify-phases %standard-phases
-         (add-after 'unpack 'patch-remotetcp-harness
+         (replace 'check
+           ;; As of Xapian 1.3.3, the TCP server implementation uses
+           ;; getaddrinfo(). This does not work in the build environment,
+           ;; so exclude those tests. See HACKING for the list of targets.
            (lambda _
-             (substitute* "tests/harness/backendmanager_remotetcp.cc"
-               (("/bin/sh") (which "bash"))))))))
+             (zero? (system* "make"
+                             "check-inmemory"
+                             "check-remoteprog"
+                             ;"check-remotetcp"
+                             "check-multi"
+                             "check-glass"
+                             "check-chert")))))))
     (synopsis "Search Engine Library")
     (description
      "Xapian is a highly adaptable toolkit which allows developers to easily
 add advanced indexing and search facilities to their own applications.  It
 supports the Probabilistic Information Retrieval model and also supports a
 rich set of boolean query operators.")
-    (home-page "http://xapian.org/")
+    (home-page "https://xapian.org/")
     (license (list gpl2+ bsd-3 x11))))
 
 (define-public libtocc
-- 
2.10.1

Re: [PATCH 4/4] gnu: xapian: Update to 1.4.0.

[Leo Famulari]

On Mon, Oct 17, 2016 at 05:55:06PM +0100, Marius Bakke wrote:
> * gnu/packages/search.scm (xapian): Update to 1.4.0.
> [source]: Use https URL.
> [home-page]: Use https URL.
> [arguments]: Remove 'patch-remotetcp-harness' phase. Replace 'check'
> phase with a list of custom test targets.

Works for me with mu.

Re: [PATCH 4/4] gnu: xapian: Update to 1.4.0.

[Marius Bakke]

Leo Famulari <leo@famulari.name> writes:

> On Mon, Oct 17, 2016 at 05:55:06PM +0100, Marius Bakke wrote:
>> * gnu/packages/search.scm (xapian): Update to 1.4.0.
>> [source]: Use https URL.
>> [home-page]: Use https URL.
>> [arguments]: Remove 'patch-remotetcp-harness' phase. Replace 'check'
>> phase with a list of custom test targets.
>
> Works for me with mu.

Thanks a lot for reviewing, Leo. I'll go ahead and push these tomorrow
with the mentioned changes unless there are other comments. Not happy
about excluding those Xapian tests, but don't know what to do about it.