WIP: promote nmbug to user sync tool

WIP: promote nmbug to user sync tool

[David Bremner]

One of the things that new (and old) users of notmuch often miss is a
way to sync tags between hosts. muchsync exists, but it's a third
party tool, and (if I understand correctly) it directly modifies the
notmuch database. Meanwhile we ship a moderately complex python script
called 'nmbug' which can be used to sync arbitrary tags between hosts.
This series is my (first pass at an) attempt to promote nmbug to a
notmuch subcommand "notmuch git". My plan is to replace my own
homebrew "commit notmuch dump output to git" script with this new
command; I think it may appeal to other users as well. As a bonus, it
provides a simple (for the user) way to do incremental backups of the
notmuch database.

I had to add a couple of caching tricks to make the script usable for
my database, but it is now reasonably fast (say 20s) to commit a days
worth of changes on my machine.

There are still some rough edges, mainly due to the heritage of the
script. Some I am aware of include:

- the (new) man page is inadequate
- notmuch git doesn't understand the common arguments to notmuch (--config and friends)
- the defaults for prefix and git repo are wrong for most users. I was thinking about either a
  --nmbug argument to set the old defaults, or having the script recognize if it is invoked as nmbug.
- There are a few too many global variables
- I just remembered, MAX_LASTMOD is unused, leftover from a previous design for querying tags.

One thing that not everyone will like is that this requires sexp
queries in your version of notmuch. It's exactly the kind of
application I had in mind when I built the sexp query parser (script
generated queries), and the sexp queries actually allow fixing an
escaping bug in the current nmbug

[PATCH 01/16] nmbug: promote to user tool "notmuch-git"

[David Bremner]

So far this is just a rename. Documentation and tests to follow.
---
 Makefile.local                      | 7 +++++--
 devel/nmbug/nmbug => notmuch-git.in | 2 +-
 2 files changed, 6 insertions(+), 3 deletions(-)
 rename devel/nmbug/nmbug => notmuch-git.in (99%)

diff --git a/Makefile.local b/Makefile.local
index d8bbf3e1..ca2310f4 100644
--- a/Makefile.local
+++ b/Makefile.local
@@ -1,7 +1,7 @@
 # -*- makefile-gmake -*-
 
 .PHONY: all
-all: notmuch notmuch-shared build-man build-info ruby-bindings python-cffi-bindings
+all: notmuch notmuch-shared build-man build-info ruby-bindings python-cffi-bindings notmuch-git
 ifeq ($(MAKECMDGOALS),)
 ifeq ($(shell cat .first-build-message 2>/dev/null),)
 	@NOTMUCH_FIRST_BUILD=1 $(MAKE) --no-print-directory all
@@ -294,7 +294,7 @@ endif
 SRCS  := $(SRCS) $(notmuch_client_srcs)
 CLEAN := $(CLEAN) notmuch notmuch-shared $(notmuch_client_modules)
 CLEAN := $(CLEAN) version.stamp notmuch-*.tar.gz.tmp
-CLEAN := $(CLEAN) .deps
+CLEAN := $(CLEAN) .deps notmuch-git
 
 DISTCLEAN := $(DISTCLEAN) .first-build-message Makefile.config sh.config sphinx.config
 
@@ -307,6 +307,9 @@ cppcheck:
 	@echo "No cppcheck found during configure; skipping static checking"
 endif
 
+notmuch-git: notmuch-git.in
+	sed s/@NOTMUCH_VERSION@/${VERSION}/ < notmuch-git.in > notmuch-git
+	chmod ugo+rx notmuch-git
 
 DEPS := $(SRCS:%.c=.deps/%.d)
 DEPS := $(DEPS:%.cc=.deps/%.d)
diff --git a/devel/nmbug/nmbug b/notmuch-git.in
similarity index 99%
rename from devel/nmbug/nmbug
rename to notmuch-git.in
index 043c1863..dd30b5be 100755
--- a/devel/nmbug/nmbug
+++ b/notmuch-git.in
@@ -51,7 +51,7 @@ except ImportError:  # Python 2
     from urllib import unquote as _unquote
 
 
-__version__ = '0.3'
+__version__ = '@NOTMUCH_VERSION@'
 
 _LOG = _logging.getLogger('nmbug')
 _LOG.setLevel(_logging.WARNING)
-- 
2.35.2

[PATCH 02/16] notmuch-git: add --git-dir argument

[David Bremner]

The intent is to change the names of the environment variables before
releasing this code, so avoid relying on environment variables in the
test framework.
---
 notmuch-git.in | 18 +++++++++++++-----
 1 file changed, 13 insertions(+), 5 deletions(-)

diff --git a/notmuch-git.in b/notmuch-git.in
index dd30b5be..62889303 100755
--- a/notmuch-git.in
+++ b/notmuch-git.in
@@ -57,11 +57,7 @@ _LOG = _logging.getLogger('nmbug')
 _LOG.setLevel(_logging.WARNING)
 _LOG.addHandler(_logging.StreamHandler())
 
-NMBGIT = _os.path.expanduser(
-    _os.getenv('NMBGIT', _os.path.join('~', '.nmbug')))
-_NMBGIT = _os.path.join(NMBGIT, '.git')
-if _os.path.isdir(_NMBGIT):
-    NMBGIT = _NMBGIT
+NMBGIT = None
 
 TAG_PREFIX = _os.getenv('NMBPREFIX', 'notmuch::')
 _HEX_ESCAPE_REGEX = _re.compile('%[0-9A-F]{2}')
@@ -718,6 +714,9 @@ if __name__ == '__main__':
     parser = argparse.ArgumentParser(
         description=__doc__.strip(),
         formatter_class=argparse.RawDescriptionHelpFormatter)
+    parser.add_argument(
+        '-C', '--git-dir', metavar='REPO',
+        help='Git repository to operate on.')
     parser.add_argument(
         '-v', '--version', action='version',
         version='%(prog)s {}'.format(__version__))
@@ -830,6 +829,15 @@ if __name__ == '__main__':
 
     args = parser.parse_args()
 
+    if args.git_dir:
+        NMBGIT = args.git_dir
+    else:
+        NMBGIT = _os.path.expanduser(
+        _os.getenv('NMBGIT', _os.path.join('~', '.nmbug')))
+        _NMBGIT = _os.path.join(NMBGIT, '.git')
+        if _os.path.isdir(_NMBGIT):
+            NMBGIT = _NMBGIT
+
     if args.log_level:
         level = getattr(_logging, args.log_level.upper())
         _LOG.setLevel(level)
-- 
2.35.2

[PATCH 03/16] notmuch-git: add --tag-prefix argument

[David Bremner]

The test suite will require setting the tag prefix (at least
initially), and this commit will enable doing that without relying on
environment variables (whose names are planned to change).
---
 notmuch-git.in | 13 ++++++++-----
 1 file changed, 8 insertions(+), 5 deletions(-)

diff --git a/notmuch-git.in b/notmuch-git.in
index 62889303..81d604bb 100755
--- a/notmuch-git.in
+++ b/notmuch-git.in
@@ -58,8 +58,8 @@ _LOG.setLevel(_logging.WARNING)
 _LOG.addHandler(_logging.StreamHandler())
 
 NMBGIT = None
+TAG_PREFIX = None
 
-TAG_PREFIX = _os.getenv('NMBPREFIX', 'notmuch::')
 _HEX_ESCAPE_REGEX = _re.compile('%[0-9A-F]{2}')
 _TAG_DIRECTORY = 'tags/'
 _TAG_FILE_REGEX = _re.compile(_TAG_DIRECTORY + '(?P<id>[^/]*)/(?P<tag>[^/]*)')
@@ -105,10 +105,6 @@ def _hex_quote(string, safe='+@=:,'):
         lambda match: match.group(0).lower(),
         uppercase_escapes)
 
-
-_ENCODED_TAG_PREFIX = _hex_quote(TAG_PREFIX, safe='+@=,')  # quote ':'
-
-
 def _xapian_quote(string):
     """
     Quote a string for Xapian's QueryParser.
@@ -717,6 +713,10 @@ if __name__ == '__main__':
     parser.add_argument(
         '-C', '--git-dir', metavar='REPO',
         help='Git repository to operate on.')
+    parser.add_argument(
+        '-p', '--tag-prefix', metavar='PREFIX',
+        default = _os.getenv('NMBPREFIX', 'notmuch::'),
+        help='Prefix of tags to operate on.')
     parser.add_argument(
         '-v', '--version', action='version',
         version='%(prog)s {}'.format(__version__))
@@ -838,6 +838,9 @@ if __name__ == '__main__':
         if _os.path.isdir(_NMBGIT):
             NMBGIT = _NMBGIT
 
+    TAG_PREFIX = args.tag_prefix
+    _ENCODED_TAG_PREFIX = _hex_quote(TAG_PREFIX, safe='+@=,')  # quote ':'
+
     if args.log_level:
         level = getattr(_logging, args.log_level.upper())
         _LOG.setLevel(level)
-- 
2.35.2

[PATCH 04/16] test: initial tests for notmuch-git

[David Bremner]

Exercise the main functionality of notmuch-git.  add_git_repos() will
hopefully be simplifed when an init subcommand is added.
---
 test/T850-git.sh | 95 ++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 95 insertions(+)
 create mode 100755 test/T850-git.sh

diff --git a/test/T850-git.sh b/test/T850-git.sh
new file mode 100755
index 00000000..f52dd60d
--- /dev/null
+++ b/test/T850-git.sh
@@ -0,0 +1,95 @@
+#!/usr/bin/env bash
+test_description='"notmuch git" to save and restore tags'
+. $(dirname "$0")/test-lib.sh || exit 1
+
+add_git_repos () {
+    # first the equivalent of "notmuch git init" (which doesn't exist yet)
+    git init --bare --initial-branch=master --quiet remote.git
+    git -C remote.git read-tree --empty
+    tree=$(git -C remote.git write-tree)
+    git -C remote.git hash-object -w /dev/null > /dev/null
+    commit=$(echo 'root commit' | git -C remote.git commit-tree $tree)
+    git -C remote.git update-ref refs/heads/master $commit
+
+    # now make a "local" repo to work on
+    git clone --bare --quiet remote.git tags.git
+    git -C tags.git config --add remote.origin.fetch "+refs/heads/*:refs/remotes/origin/*"
+    git -C tags.git fetch origin
+    git -C tags.git branch -u origin/master master
+}
+
+add_email_corpus
+add_git_repos
+
+test_begin_subtest "clone"
+# currently broken because of hard-wired requirement for config branch
+test_subtest_known_broken
+test_expect_success "notmuch git -C clone.git clone tags.git"
+
+test_begin_subtest "commit"
+notmuch git -C tags.git -p '' commit
+git -C tags.git ls-tree -r --name-only HEAD | xargs dirname | sort -u | sed s,tags/,id:, > OUTPUT
+notmuch search --output=messages '*' | sort > EXPECTED
+test_expect_equal_file_nonempty EXPECTED OUTPUT
+
+test_begin_subtest "checkout"
+notmuch dump > BEFORE
+notmuch tag -inbox '*'
+notmuch git -C tags.git -p '' checkout
+notmuch dump > AFTER
+test_expect_equal_file_nonempty BEFORE AFTER
+
+test_begin_subtest "archive"
+notmuch git -C tags.git -p '' archive | tar tf - | \
+    grep 20091117190054.GU3165@dottiness.seas.harvard.edu | sort > OUTPUT
+cat <<EOF > EXPECTED
+tags/20091117190054.GU3165@dottiness.seas.harvard.edu/
+tags/20091117190054.GU3165@dottiness.seas.harvard.edu/inbox
+tags/20091117190054.GU3165@dottiness.seas.harvard.edu/signed
+tags/20091117190054.GU3165@dottiness.seas.harvard.edu/unread
+EOF
+notmuch git -C tags.git -p '' checkout
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "status"
+notmuch tag +test id:20091117190054.GU3165@dottiness.seas.harvard.edu
+notmuch git -C tags.git -p '' status > OUTPUT
+cat <<EOF > EXPECTED
+A	20091117190054.GU3165@dottiness.seas.harvard.edu	test
+EOF
+notmuch git -C tags.git -p '' checkout
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "fetch"
+notmuch tag +test2 id:20091117190054.GU3165@dottiness.seas.harvard.edu
+notmuch git -C remote.git -p '' commit
+notmuch tag -test2 id:20091117190054.GU3165@dottiness.seas.harvard.edu
+notmuch git -C tags.git -p '' fetch
+notmuch git -C tags.git -p '' status > OUTPUT
+cat <<EOF > EXPECTED
+ a	20091117190054.GU3165@dottiness.seas.harvard.edu	test2
+EOF
+notmuch git -C tags.git -p '' checkout
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "merge"
+notmuch git -C tags.git -p '' merge
+notmuch dump id:20091117190054.GU3165@dottiness.seas.harvard.edu | grep -v '^#' > OUTPUT
+cat <<EOF > EXPECTED
++inbox +signed +test2 +unread -- id:20091117190054.GU3165@dottiness.seas.harvard.edu
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "push"
+notmuch tag +test3 id:20091117190054.GU3165@dottiness.seas.harvard.edu
+notmuch git -C tags.git -p '' commit
+notmuch tag -test3 id:20091117190054.GU3165@dottiness.seas.harvard.edu
+notmuch git -C tags.git -p '' push
+notmuch git -C remote.git -p '' checkout
+notmuch dump id:20091117190054.GU3165@dottiness.seas.harvard.edu | grep -v '^#' > OUTPUT
+cat <<EOF > EXPECTED
++inbox +signed +test2 +test3 +unread -- id:20091117190054.GU3165@dottiness.seas.harvard.edu
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+
+test_done
-- 
2.35.2

[PATCH 05/16] nmbug: Add an 'init' command

[David Bremner]

From: "W. Trevor King" <wking@tremily.us>

For folks that want to start versioning a new tag-space, instead of
cloning one that someone else has already started.

The empty-blob hash-object call avoids errors like:

  $ nmbug commit
  error: invalid object 100644 e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 for
'tags/...'
  fatal: git-write-tree: error building trees
  'git HASH(0x9ef3eb8) write-tree' exited with nonzero value

David Bremner suggested [1]:

  $ git hash-object -w /dev/null

instead of my Python version of:

  $ git hash-object -w --stdin <&-

but I expect that closing stdin is more portable than the /dev/null
path (which doesn't exist on Windows, for example).

The --bare init and use of NMBGIT as the work tree (what could go
wrong with an empty commit?) are suggestions from Michal Sojka [2].

[1]: id:87y4vu6uvf.fsf@maritornes.cs.unb.ca
     http://thread.gmane.org/gmane.mail.notmuch.general/18626/focus=18720
[2]: id:87a93a5or2.fsf@resox.2x.cz
     http://thread.gmane.org/gmane.mail.notmuch.general/19495/focus=19767
---
 notmuch-git.in | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/notmuch-git.in b/notmuch-git.in
index 81d604bb..d574f19e 100755
--- a/notmuch-git.in
+++ b/notmuch-git.in
@@ -377,6 +377,25 @@ def fetch(remote=None):
     _git(args=args, wait=True)
 
 
+def init(remote=None):
+    """
+    Create an empty nmbug repository.
+
+    This wraps 'git init' with a few extra steps to support subsequent
+    status and commit commands.
+    """
+    _spawn(args=['git', '--git-dir', NMBGIT, 'init', '--bare'], wait=True)
+    _git(args=['config', 'core.logallrefupdates', 'true'], wait=True)
+    # create an empty blob (e69de29bb2d1d6434b8b29ae775ad8c2e48c5391)
+    _git(args=['hash-object', '-w', '--stdin'], input='', wait=True)
+    _git(
+        args=[
+            'commit', '--allow-empty', '-m', 'Start a new nmbug repository'
+        ],
+        additional_env={'GIT_WORK_TREE': NMBGIT},
+        wait=True)
+
+
 def checkout():
     """
     Update the notmuch database from Git.
@@ -740,6 +759,7 @@ if __name__ == '__main__':
             'commit',
             'fetch',
             'help',
+            'init',
             'log',
             'merge',
             'pull',
-- 
2.35.2

[PATCH 06/16] CLI/git: suppress warnings about initial branch name

[David Bremner]

The canonical nmbug repository still uses "master" as the main branch
name, so defer any potential switch away from that name.
---
 notmuch-git.in | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/notmuch-git.in b/notmuch-git.in
index d574f19e..3a58fd28 100755
--- a/notmuch-git.in
+++ b/notmuch-git.in
@@ -384,7 +384,8 @@ def init(remote=None):
     This wraps 'git init' with a few extra steps to support subsequent
     status and commit commands.
     """
-    _spawn(args=['git', '--git-dir', NMBGIT, 'init', '--bare'], wait=True)
+    _spawn(args=['git', '--git-dir', NMBGIT, 'init',
+                 '--initial-branch=master', '--quiet', '--bare'], wait=True)
     _git(args=['config', 'core.logallrefupdates', 'true'], wait=True)
     # create an empty blob (e69de29bb2d1d6434b8b29ae775ad8c2e48c5391)
     _git(args=['hash-object', '-w', '--stdin'], input='', wait=True)
-- 
2.35.2

[PATCH 07/16] test: use "notmuch git init" for tests.

[David Bremner]

Stick with the test-suite specific clone script for now, until clone
no longer insists on the config branch (or init creates a config
branch).
---
 test/T850-git.sh | 8 +-------
 1 file changed, 1 insertion(+), 7 deletions(-)

diff --git a/test/T850-git.sh b/test/T850-git.sh
index f52dd60d..d682141f 100755
--- a/test/T850-git.sh
+++ b/test/T850-git.sh
@@ -3,13 +3,7 @@ test_description='"notmuch git" to save and restore tags'
 . $(dirname "$0")/test-lib.sh || exit 1
 
 add_git_repos () {
-    # first the equivalent of "notmuch git init" (which doesn't exist yet)
-    git init --bare --initial-branch=master --quiet remote.git
-    git -C remote.git read-tree --empty
-    tree=$(git -C remote.git write-tree)
-    git -C remote.git hash-object -w /dev/null > /dev/null
-    commit=$(echo 'root commit' | git -C remote.git commit-tree $tree)
-    git -C remote.git update-ref refs/heads/master $commit
+    notmuch git -C remote.git -p '' init
 
     # now make a "local" repo to work on
     git clone --bare --quiet remote.git tags.git
-- 
2.35.2

[PATCH 08/16] CLI/git: make existance of config branch optional on clone

[David Bremner]

This branch is actually only used by an associated
utility (notmuch-report), and notmuch-git works fine without it.

With this change we can use "notmuch git clone" in the setup for
the tests in T850-git.sh
---
 notmuch-git.in   | 8 +++++++-
 test/T850-git.sh | 9 +--------
 2 files changed, 8 insertions(+), 9 deletions(-)

diff --git a/notmuch-git.in b/notmuch-git.in
index 3a58fd28..8b397080 100755
--- a/notmuch-git.in
+++ b/notmuch-git.in
@@ -303,7 +303,13 @@ def clone(repository):
             wait=True)
     _git(args=['config', '--unset', 'core.worktree'], wait=True, expect=(0, 5))
     _git(args=['config', 'core.bare', 'true'], wait=True)
-    _git(args=['branch', 'config', 'origin/config'], wait=True)
+    (status, stdout, stderr) = _git(args=['show-ref', '--verify',
+                                          '--quiet',
+                                          'refs/remotes/origin/config'],
+                                    expect=(0,1),
+                                    wait=True)
+    if status == 0:
+        _git(args=['branch', 'config', 'origin/config'], wait=True)
     existing_tags = get_tags()
     if existing_tags:
         _LOG.warning(
diff --git a/test/T850-git.sh b/test/T850-git.sh
index d682141f..417692d4 100755
--- a/test/T850-git.sh
+++ b/test/T850-git.sh
@@ -4,20 +4,13 @@ test_description='"notmuch git" to save and restore tags'
 
 add_git_repos () {
     notmuch git -C remote.git -p '' init
-
-    # now make a "local" repo to work on
-    git clone --bare --quiet remote.git tags.git
-    git -C tags.git config --add remote.origin.fetch "+refs/heads/*:refs/remotes/origin/*"
-    git -C tags.git fetch origin
-    git -C tags.git branch -u origin/master master
+    notmuch git -C tags.git -p '' clone remote.git
 }
 
 add_email_corpus
 add_git_repos
 
 test_begin_subtest "clone"
-# currently broken because of hard-wired requirement for config branch
-test_subtest_known_broken
 test_expect_success "notmuch git -C clone.git clone tags.git"
 
 test_begin_subtest "commit"
-- 
2.35.2

[PATCH 09/16] test/git: add known broken test for tag with quotes.

[David Bremner]

There is current insufficient sanitization and/or escaping of tag names
internally in notmuch-git.
---
 test/T850-git.sh | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/test/T850-git.sh b/test/T850-git.sh
index 417692d4..408a6337 100755
--- a/test/T850-git.sh
+++ b/test/T850-git.sh
@@ -19,6 +19,16 @@ git -C tags.git ls-tree -r --name-only HEAD | xargs dirname | sort -u | sed s,ta
 notmuch search --output=messages '*' | sort > EXPECTED
 test_expect_equal_file_nonempty EXPECTED OUTPUT
 
+test_begin_subtest "commit, with quoted tag"
+test_subtest_known_broken
+notmuch git -C clone2.git -p '' clone tags.git
+git -C clone2.git ls-tree -r --name-only HEAD | grep /inbox > BEFORE
+notmuch tag '+"quoted tag"' '*'
+notmuch git -C clone2.git -p '' commit
+notmuch tag '-"quoted tag"' '*'
+git -C clone2.git ls-tree -r --name-only HEAD | grep /inbox > AFTER
+test_expect_equal_file_nonempty BEFORE AFTER
+
 test_begin_subtest "checkout"
 notmuch dump > BEFORE
 notmuch tag -inbox '*'
-- 
2.35.2

[PATCH 10/16] CLI/git: replace enumeration of tags with sexp query.

[David Bremner]

Unlike the (current) infix query parser provided by Xapian, the
notmuch specific sexp query parser supports prefixed wildcard queries,
so use those. In addition to being somewhat faster, this avoids
needing to escape all of the user's tags to pass via the shell.
---
 notmuch-git.in   | 23 ++++++++++++++++-------
 test/T850-git.sh |  6 +++++-
 2 files changed, 21 insertions(+), 8 deletions(-)

diff --git a/notmuch-git.in b/notmuch-git.in
index 8b397080..d8b7e45b 100755
--- a/notmuch-git.in
+++ b/notmuch-git.in
@@ -258,16 +258,17 @@ def _get_remote():
         stdout=_subprocess.PIPE, wait=True)
     return remote.strip()
 
+def _tag_query(prefix=None):
+    if prefix is None:
+        prefix = TAG_PREFIX
+    return '(tag (starts-with "{:s}"))'.format(prefix.replace('"','\\\"'))
 
 def get_tags(prefix=None):
     "Get a list of tags with a given prefix."
-    if prefix is None:
-        prefix = TAG_PREFIX
     (status, stdout, stderr) = _spawn(
-        args=['notmuch', 'search', '--output=tags', '*'],
+        args=['notmuch', 'search', '--query=sexp', '--output=tags', _tag_query(prefix)],
         stdout=_subprocess.PIPE, wait=True)
-    return [tag for tag in stdout.splitlines() if tag.startswith(prefix)]
-
+    return [tag for tag in stdout.splitlines()]
 
 def archive(treeish='HEAD', args=()):
     """
@@ -623,13 +624,12 @@ def get_status():
 def _index_tags():
     "Write notmuch tags to the nmbug.index."
     path = _os.path.join(NMBGIT, 'nmbug.index')
-    query = ' '.join('tag:"{tag}"'.format(tag=tag) for tag in get_tags())
     prefix = '+{0}'.format(_ENCODED_TAG_PREFIX)
     _git(
         args=['read-tree', '--empty'],
         additional_env={'GIT_INDEX_FILE': path}, wait=True)
     with _spawn(
-            args=['notmuch', 'dump', '--format=batch-tag', '--', query],
+            args=['notmuch', 'dump', '--format=batch-tag', '--query=sexp', '--', _tag_query()],
             stdout=_subprocess.PIPE) as notmuch:
         with _git(
                 args=['update-index', '--index-info'],
@@ -872,6 +872,15 @@ if __name__ == '__main__':
         level = getattr(_logging, args.log_level.upper())
         _LOG.setLevel(level)
 
+    (status, stdout, stderr) = _spawn(
+        args=['notmuch', 'config', 'get', 'built_with.sexp_queries'],
+        stdout=_subprocess.PIPE, wait=True)
+    if status != 0:
+        _LOG.error("failed to run notmuch")
+        sys.exit(1)
+    if stdout != "true\n":
+        _LOG.error("notmuch git needs sexp query support")
+
     if not getattr(args, 'func', None):
         parser.print_usage()
         _sys.exit(1)
diff --git a/test/T850-git.sh b/test/T850-git.sh
index 408a6337..4bf29b20 100755
--- a/test/T850-git.sh
+++ b/test/T850-git.sh
@@ -7,6 +7,11 @@ add_git_repos () {
     notmuch git -C tags.git -p '' clone remote.git
 }
 
+if [ $NOTMUCH_HAVE_SFSEXP -ne 1 ]; then
+    printf "Skipping due to missing sfsexp library\n"
+    test_done
+fi
+
 add_email_corpus
 add_git_repos
 
@@ -20,7 +25,6 @@ notmuch search --output=messages '*' | sort > EXPECTED
 test_expect_equal_file_nonempty EXPECTED OUTPUT
 
 test_begin_subtest "commit, with quoted tag"
-test_subtest_known_broken
 notmuch git -C clone2.git -p '' clone tags.git
 git -C clone2.git ls-tree -r --name-only HEAD | grep /inbox > BEFORE
 notmuch tag '+"quoted tag"' '*'
-- 
2.35.2

[PATCH 11/16] CLI/git: add @timed decorator, time a few functions

[David Bremner]

Perf will show which binaries are using the CPU cycles, and standard
python profilers will show which python functions, but neither is
great at finding which call to an external binary is taking time, or
locating I/O hotspots.
---
 notmuch-git.in | 19 ++++++++++++++++++-
 1 file changed, 18 insertions(+), 1 deletion(-)

diff --git a/notmuch-git.in b/notmuch-git.in
index d8b7e45b..0d9f50a1 100755
--- a/notmuch-git.in
+++ b/notmuch-git.in
@@ -128,6 +128,20 @@ def _xapian_unquote(string):
     return string
 
 
+def timed(fn):
+    """Timer decorator"""
+    from time import perf_counter
+
+    def inner(*args, **kwargs):
+        start_time = perf_counter()
+        rval = fn(*args, **kwargs)
+        end_time = perf_counter()
+        _LOG.info('{0}: {1:.8f}s elapsed'.format(fn.__name__, end_time - start_time))
+        return rval
+
+    return inner
+
+
 class SubprocessError(RuntimeError):
     "A subprocess exited with a nonzero status"
     def __init__(self, args, status, stdout=None, stderr=None):
@@ -360,6 +374,7 @@ def commit(treeish='HEAD', message=None):
         _git(args=['read-tree', treeish], wait=True)
         raise
 
+@timed
 def _update_index(status):
     with _git(
             args=['update-index', '--index-info'],
@@ -600,6 +615,7 @@ def _is_unmerged(ref='@{upstream}'):
     return base != fetch_head
 
 
+@timed
 def get_status():
     status = {
         'deleted': {},
@@ -620,7 +636,7 @@ def get_status():
     _os.remove(index)
     return status
 
-
+@timed
 def _index_tags():
     "Write notmuch tags to the nmbug.index."
     path = _os.path.join(NMBGIT, 'nmbug.index')
@@ -669,6 +685,7 @@ def _index_tags_for_message(id, status, tags):
         yield '{mode} {hash}\t{path}\n'.format(mode=mode, hash=hash, path=path)
 
 
+@timed
 def _diff_index(index, filter):
     """
     Get an {id: {tag, ...}} dict for a given filter.
-- 
2.35.2

[PATCH 12/16] CLI/git: rename private index file.

[David Bremner]

All files created by notmuch-git should be in a private directory to
avoid collisions.
---
 notmuch-git.in | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)

diff --git a/notmuch-git.in b/notmuch-git.in
index 0d9f50a1..b69d57e7 100755
--- a/notmuch-git.in
+++ b/notmuch-git.in
@@ -638,8 +638,9 @@ def get_status():
 
 @timed
 def _index_tags():
-    "Write notmuch tags to the nmbug.index."
-    path = _os.path.join(NMBGIT, 'nmbug.index')
+    "Write notmuch tags to private git index."
+    ensure_private_directory(NMBGIT)
+    path = _os.path.join(NMBGIT, 'notmuch','index')
     prefix = '+{0}'.format(_ENCODED_TAG_PREFIX)
     _git(
         args=['read-tree', '--empty'],
@@ -747,6 +748,12 @@ def _help(parser, command=None):
         parser.parse_args(['--help'])
 
 
+def ensure_private_directory(repo):
+    try:
+        _os.makedirs(_os.path.join(repo, 'notmuch'))
+    except FileExistsError:
+        pass
+
 if __name__ == '__main__':
     import argparse
 
-- 
2.35.2

[PATCH 13/16] CLI/git: create PrivateIndex class

[David Bremner]

If the index file matches a previously known revision of the database,
we can update the index incrementally using the recorded lastmod
counter. This is typically much faster than a full update, although it
could be slower in the case of large changes to the database.
---
 notmuch-git.in   | 220 ++++++++++++++++++++++++++++++++---------------
 test/T850-git.sh |  41 +++++++++
 2 files changed, 194 insertions(+), 67 deletions(-)

diff --git a/notmuch-git.in b/notmuch-git.in
index b69d57e7..b3f71699 100755
--- a/notmuch-git.in
+++ b/notmuch-git.in
@@ -50,6 +50,10 @@ except ImportError:  # Python 2
     from urllib import quote as _quote
     from urllib import unquote as _unquote
 
+import json as _json
+
+# hopefully big enough, handle 32 bit hosts
+MAX_LASTMOD=2**32
 
 __version__ = '@NOTMUCH_VERSION@'
 
@@ -621,51 +625,159 @@ def get_status():
         'deleted': {},
         'missing': {},
         }
-    index = _index_tags()
-    maybe_deleted = _diff_index(index=index, filter='D')
-    for id, tags in maybe_deleted.items():
-        (_, stdout, stderr) = _spawn(
-            args=['notmuch', 'search', '--output=files', 'id:{0}'.format(id)],
-            stdout=_subprocess.PIPE,
-            wait=True)
-        if stdout:
-            status['deleted'][id] = tags
-        else:
-            status['missing'][id] = tags
-    status['added'] = _diff_index(index=index, filter='A')
-    _os.remove(index)
+    with PrivateIndex(repo=NMBGIT, prefix=TAG_PREFIX) as index:
+        maybe_deleted = index.diff(filter='D')
+        for id, tags in maybe_deleted.items():
+            (_, stdout, stderr) = _spawn(
+                args=['notmuch', 'search', '--output=files', 'id:{0}'.format(id)],
+                stdout=_subprocess.PIPE,
+                wait=True)
+            if stdout:
+                status['deleted'][id] = tags
+            else:
+                status['missing'][id] = tags
+        status['added'] = index.diff(filter='A')
+
     return status
 
-@timed
-def _index_tags():
-    "Write notmuch tags to private git index."
-    ensure_private_directory(NMBGIT)
-    path = _os.path.join(NMBGIT, 'notmuch','index')
-    prefix = '+{0}'.format(_ENCODED_TAG_PREFIX)
-    _git(
-        args=['read-tree', '--empty'],
-        additional_env={'GIT_INDEX_FILE': path}, wait=True)
-    with _spawn(
-            args=['notmuch', 'dump', '--format=batch-tag', '--query=sexp', '--', _tag_query()],
-            stdout=_subprocess.PIPE) as notmuch:
+class PrivateIndex:
+    def __init__(self, repo, prefix):
+        try:
+            _os.makedirs(_os.path.join(repo, 'notmuch'))
+        except FileExistsError:
+            pass
+
+        file_name = 'notmuch/index'
+        self.index_path = _os.path.join(repo, file_name)
+        self.cache_path = _os.path.join(repo, 'notmuch', '{:s}.json'.format(_hex_quote(file_name)))
+
+        self.current_prefix = prefix
+
+        self.prefix = None
+        self.uuid = None
+        self.lastmod = None
+        self.checksum = None
+        self._load_cache_file()
+        self._index_tags()
+
+    def __enter__(self):
+        return self
+
+    def __exit__(self, type, value, traceback):
+        checksum = self._read_index_checksum()
+        (count, uuid, lastmod) = _read_database_lastmod()
+        with open(self.cache_path, "w") as f:
+            _json.dump({'prefix': self.current_prefix, 'uuid': uuid, 'lastmod': lastmod,  'checksum': checksum }, f)
+
+    def _load_cache_file(self):
+        try:
+            with open(self.cache_path) as f:
+                data = _json.load(f)
+                self.prefix = data['prefix']
+                self.uuid = data['uuid']
+                self.lastmod = data['lastmod']
+                self.checksum = data['checksum']
+        except FileNotFoundError:
+            return None
+        except _json.JSONDecodeError:
+            _LOG.error("Error decoding cache")
+            _sys.exit(1)
+
+    def _read_index_checksum (self):
+        """Read the index checksum, as defined by index-format.txt in the git source
+        WARNING: assumes SHA1 repo"""
+        import binascii
+        try:
+            with open(self.index_path, 'rb') as f:
+                size=_os.path.getsize(self.index_path)
+                f.seek(size-20);
+                return binascii.hexlify(f.read(20)).decode('ascii')
+        except FileNotFoundError:
+            return None
+
+    @timed
+    def _index_tags(self):
+        "Write notmuch tags to private git index."
+        prefix = '+{0}'.format(_ENCODED_TAG_PREFIX)
+        current_checksum = self._read_index_checksum()
+        if (self.prefix == None or self.prefix != self.current_prefix
+            or self.checksum == None or self.checksum != current_checksum):
+            _git(
+                args=['read-tree', '--empty'],
+                additional_env={'GIT_INDEX_FILE': self.index_path}, wait=True)
+
+        query = _tag_query()
+        clear_tags = False
+        (count,uuid,lastmod) = _read_database_lastmod()
+        if self.prefix == self.current_prefix and self.uuid \
+           and self.uuid == uuid and self.checksum == current_checksum:
+            query = '(and (infix "lastmod:{:d}..")) {:s})'.format(self.lastmod+1, query)
+            clear_tags = True
+        with _spawn(
+                args=['notmuch', 'dump', '--format=batch-tag', '--query=sexp', '--', query],
+                stdout=_subprocess.PIPE) as notmuch:
+            with _git(
+                    args=['update-index', '--index-info'],
+                    stdin=_subprocess.PIPE,
+                    additional_env={'GIT_INDEX_FILE': self.index_path}) as git:
+                for line in notmuch.stdout:
+                    if line.strip().startswith('#'):
+                        continue
+                    (tags_string, id) = [_.strip() for _ in line.split(' -- id:')]
+                    tags = [
+                        _unquote(tag[len(prefix):])
+                        for tag in tags_string.split()
+                        if tag.startswith(prefix)]
+                    id = _xapian_unquote(string=id)
+                    if clear_tags:
+                        for line in _clear_tags_for_message(index=self.index_path, id=id):
+                            git.stdin.write(line)
+                    for line in _index_tags_for_message(
+                            id=id, status='A', tags=tags):
+                        git.stdin.write(line)
+
+    @timed
+    def diff(self, filter):
+        """
+        Get an {id: {tag, ...}} dict for a given filter.
+
+        For example, use 'A' to find added tags, and 'D' to find deleted tags.
+        """
+        s = _collections.defaultdict(set)
         with _git(
-                args=['update-index', '--index-info'],
-                stdin=_subprocess.PIPE,
-                additional_env={'GIT_INDEX_FILE': path}) as git:
-            for line in notmuch.stdout:
-                if line.strip().startswith('#'):
-                    continue
-                (tags_string, id) = [_.strip() for _ in line.split(' -- id:')]
-                tags = [
-                    _unquote(tag[len(prefix):])
-                    for tag in tags_string.split()
-                    if tag.startswith(prefix)]
-                id = _xapian_unquote(string=id)
-                for line in _index_tags_for_message(
-                        id=id, status='A', tags=tags):
-                    git.stdin.write(line)
-    return path
+                args=[
+                    'diff-index', '--cached', '--diff-filter', filter,
+                    '--name-only', 'HEAD'],
+                additional_env={'GIT_INDEX_FILE': self.index_path},
+                stdout=_subprocess.PIPE) as p:
+            # Once we drop Python < 3.3, we can use 'yield from' here
+            for id, tag in _unpack_diff_lines(stream=p.stdout):
+                s[id].add(tag)
+        return s
+
+def _clear_tags_for_message(index, id):
+    """
+    Clear any existing index entries for message 'id'
+
+    Neither 'id' nor the tags in 'tags' should be encoded/escaped.
+    """
+
+    dir = 'tags/{id}'.format(id=_hex_quote(string=id))
+
+    with _git(
+            args=['ls-files', dir],
+            additional_env={'GIT_INDEX_FILE': index},
+            stdout=_subprocess.PIPE) as git:
+        for file in git.stdout:
+            line = '0 0000000000000000000000000000000000000000\t{:s}\n'.format(file.strip())
+            yield line
 
+def _read_database_lastmod():
+    with _spawn(
+            args=['notmuch', 'count', '--lastmod', '*'],
+            stdout=_subprocess.PIPE) as notmuch:
+        (count,uuid,lastmod_str) = notmuch.stdout.readline().split()
+        return (count,uuid,int(lastmod_str))
 
 def _index_tags_for_message(id, status, tags):
     """
@@ -686,26 +798,6 @@ def _index_tags_for_message(id, status, tags):
         yield '{mode} {hash}\t{path}\n'.format(mode=mode, hash=hash, path=path)
 
 
-@timed
-def _diff_index(index, filter):
-    """
-    Get an {id: {tag, ...}} dict for a given filter.
-
-    For example, use 'A' to find added tags, and 'D' to find deleted tags.
-    """
-    s = _collections.defaultdict(set)
-    with _git(
-            args=[
-                'diff-index', '--cached', '--diff-filter', filter,
-                '--name-only', 'HEAD'],
-            additional_env={'GIT_INDEX_FILE': index},
-            stdout=_subprocess.PIPE) as p:
-        # Once we drop Python < 3.3, we can use 'yield from' here
-        for id, tag in _unpack_diff_lines(stream=p.stdout):
-            s[id].add(tag)
-    return s
-
-
 def _diff_refs(filter, a='HEAD', b='@{upstream}'):
     with _git(
             args=['diff', '--diff-filter', filter, '--name-only', a, b],
@@ -748,12 +840,6 @@ def _help(parser, command=None):
         parser.parse_args(['--help'])
 
 
-def ensure_private_directory(repo):
-    try:
-        _os.makedirs(_os.path.join(repo, 'notmuch'))
-    except FileExistsError:
-        pass
-
 if __name__ == '__main__':
     import argparse
 
diff --git a/test/T850-git.sh b/test/T850-git.sh
index 4bf29b20..2358690f 100755
--- a/test/T850-git.sh
+++ b/test/T850-git.sh
@@ -33,6 +33,47 @@ notmuch tag '-"quoted tag"' '*'
 git -C clone2.git ls-tree -r --name-only HEAD | grep /inbox > AFTER
 test_expect_equal_file_nonempty BEFORE AFTER
 
+test_begin_subtest "commit (incremental)"
+notmuch tag +test id:20091117190054.GU3165@dottiness.seas.harvard.edu
+notmuch git -C tags.git -p '' commit
+git -C tags.git ls-tree -r --name-only HEAD |
+    grep 20091117190054 | sort > OUTPUT
+echo "--------------------------------------------------" >> OUTPUT
+notmuch tag -test id:20091117190054.GU3165@dottiness.seas.harvard.edu
+notmuch git -C tags.git -p '' commit
+git -C tags.git ls-tree -r --name-only HEAD |
+    grep 20091117190054 | sort >> OUTPUT
+cat <<EOF > EXPECTED
+tags/20091117190054.GU3165@dottiness.seas.harvard.edu/inbox
+tags/20091117190054.GU3165@dottiness.seas.harvard.edu/signed
+tags/20091117190054.GU3165@dottiness.seas.harvard.edu/test
+tags/20091117190054.GU3165@dottiness.seas.harvard.edu/unread
+--------------------------------------------------
+tags/20091117190054.GU3165@dottiness.seas.harvard.edu/inbox
+tags/20091117190054.GU3165@dottiness.seas.harvard.edu/signed
+tags/20091117190054.GU3165@dottiness.seas.harvard.edu/unread
+EOF
+test_expect_equal_file_nonempty EXPECTED OUTPUT
+
+test_begin_subtest "commit (change prefix)"
+notmuch tag +test::one id:20091117190054.GU3165@dottiness.seas.harvard.edu
+notmuch git -C tags.git -p 'test::' commit
+git -C tags.git ls-tree -r --name-only HEAD |
+    grep 20091117190054 | sort > OUTPUT
+echo "--------------------------------------------------" >> OUTPUT
+notmuch tag -test::one id:20091117190054.GU3165@dottiness.seas.harvard.edu
+notmuch git -C tags.git -p '' commit
+git -C tags.git ls-tree -r --name-only HEAD |
+    grep 20091117190054 | sort >> OUTPUT
+cat <<EOF > EXPECTED
+tags/20091117190054.GU3165@dottiness.seas.harvard.edu/one
+--------------------------------------------------
+tags/20091117190054.GU3165@dottiness.seas.harvard.edu/inbox
+tags/20091117190054.GU3165@dottiness.seas.harvard.edu/signed
+tags/20091117190054.GU3165@dottiness.seas.harvard.edu/unread
+EOF
+test_expect_equal_file_nonempty EXPECTED OUTPUT
+
 test_begin_subtest "checkout"
 notmuch dump > BEFORE
 notmuch tag -inbox '*'
-- 
2.35.2

[PATCH 14/16] CLI/git: create CachedIndex class

[David Bremner]

The "git-read-tree HEAD" is a bottleneck, but unfortunately sometimes
is needed. Cache the index checksum and hash to reduce the number of
times the operation is run. The overall design is a simplified version
of the PrivateIndex class, which is partially refactored to support
the new class.
---
 notmuch-git.in | 136 +++++++++++++++++++++++++++++++++++--------------
 1 file changed, 97 insertions(+), 39 deletions(-)

diff --git a/notmuch-git.in b/notmuch-git.in
index b3f71699..261b3f85 100755
--- a/notmuch-git.in
+++ b/notmuch-git.in
@@ -342,41 +342,98 @@ def _is_committed(status):
     return len(status['added']) + len(status['deleted']) == 0
 
 
+class CachedIndex:
+    def __init__(self, repo, treeish):
+        self.cache_path = _os.path.join(repo, 'notmuch', 'index_cache.json')
+        self.index_path = _os.path.join(repo, 'index')
+        self.current_treeish = treeish
+        # cached values
+        self.treeish = None
+        self.hash = None
+        self.index_checksum = None
+
+        self._load_cache_file()
+
+    def _load_cache_file(self):
+        try:
+            with open(self.cache_path) as f:
+                data = _json.load(f)
+                self.treeish = data['treeish']
+                self.hash = data['hash']
+                self.index_checksum = data['index_checksum']
+        except FileNotFoundError:
+            pass
+        except _json.JSONDecodeError:
+            _LOG.error("Error decoding cache")
+            _sys.exit(1)
+
+    def __enter__(self):
+        self.read_tree()
+        return self
+
+    def __exit__(self, type, value, traceback):
+        checksum = _read_index_checksum(self.index_path)
+        (_, hash, _) = _git(
+            args=['rev-parse', self.current_treeish],
+            stdout=_subprocess.PIPE,
+            wait=True)
+
+        with open(self.cache_path, "w") as f:
+            _json.dump({'treeish': self.current_treeish,
+                        'hash': hash.rstrip(),  'index_checksum': checksum }, f)
+
+    @timed
+    def read_tree(self):
+        current_checksum = _read_index_checksum(self.index_path)
+        (_, hash, _) = _git(
+            args=['rev-parse', self.current_treeish],
+            stdout=_subprocess.PIPE,
+            wait=True)
+        current_hash = hash.rstrip()
+
+        if self.current_treeish == self.treeish and \
+           self.index_checksum and self.index_checksum == current_checksum and \
+           self.hash and self.hash == current_hash:
+            return
+
+        _git(args=['read-tree', self.current_treeish], wait=True)
+
+
 def commit(treeish='HEAD', message=None):
     """
     Commit prefix-matching tags from the notmuch database to Git.
     """
+
     status = get_status()
 
     if _is_committed(status=status):
         _LOG.warning('Nothing to commit')
         return
 
-    _git(args=['read-tree', '--empty'], wait=True)
-    _git(args=['read-tree', treeish], wait=True)
-    try:
-        _update_index(status=status)
-        (_, tree, _) = _git(
-            args=['write-tree'],
-            stdout=_subprocess.PIPE,
-            wait=True)
-        (_, parent, _) = _git(
-            args=['rev-parse', treeish],
-            stdout=_subprocess.PIPE,
-            wait=True)
-        (_, commit, _) = _git(
-            args=['commit-tree', tree.strip(), '-p', parent.strip()],
-            input=message,
-            stdout=_subprocess.PIPE,
-            wait=True)
-        _git(
-            args=['update-ref', treeish, commit.strip()],
-            stdout=_subprocess.PIPE,
-            wait=True)
-    except Exception as e:
-        _git(args=['read-tree', '--empty'], wait=True)
-        _git(args=['read-tree', treeish], wait=True)
-        raise
+    with CachedIndex(NMBGIT, treeish) as index:
+        try:
+            _update_index(status=status)
+            (_, tree, _) = _git(
+                args=['write-tree'],
+                stdout=_subprocess.PIPE,
+                wait=True)
+            (_, parent, _) = _git(
+                args=['rev-parse', treeish],
+                stdout=_subprocess.PIPE,
+                wait=True)
+            (_, commit, _) = _git(
+                args=['commit-tree', tree.strip(), '-p', parent.strip()],
+                input=message,
+                stdout=_subprocess.PIPE,
+                wait=True)
+            _git(
+                args=['update-ref', treeish, commit.strip()],
+                stdout=_subprocess.PIPE,
+                wait=True)
+        except Exception as e:
+            _git(args=['read-tree', '--empty'], wait=True)
+            _git(args=['read-tree', treeish], wait=True)
+            raise
 
 @timed
 def _update_index(status):
@@ -664,7 +721,7 @@ class PrivateIndex:
         return self
 
     def __exit__(self, type, value, traceback):
-        checksum = self._read_index_checksum()
+        checksum = _read_index_checksum(self.index_path)
         (count, uuid, lastmod) = _read_database_lastmod()
         with open(self.cache_path, "w") as f:
             _json.dump({'prefix': self.current_prefix, 'uuid': uuid, 'lastmod': lastmod,  'checksum': checksum }, f)
@@ -683,23 +740,11 @@ class PrivateIndex:
             _LOG.error("Error decoding cache")
             _sys.exit(1)
 
-    def _read_index_checksum (self):
-        """Read the index checksum, as defined by index-format.txt in the git source
-        WARNING: assumes SHA1 repo"""
-        import binascii
-        try:
-            with open(self.index_path, 'rb') as f:
-                size=_os.path.getsize(self.index_path)
-                f.seek(size-20);
-                return binascii.hexlify(f.read(20)).decode('ascii')
-        except FileNotFoundError:
-            return None
-
     @timed
     def _index_tags(self):
         "Write notmuch tags to private git index."
         prefix = '+{0}'.format(_ENCODED_TAG_PREFIX)
-        current_checksum = self._read_index_checksum()
+        current_checksum = _read_index_checksum(self.index_path)
         if (self.prefix == None or self.prefix != self.current_prefix
             or self.checksum == None or self.checksum != current_checksum):
             _git(
@@ -755,6 +800,19 @@ class PrivateIndex:
                 s[id].add(tag)
         return s
 
+def _read_index_checksum (index_path):
+    """Read the index checksum, as defined by index-format.txt in the git source
+    WARNING: assumes SHA1 repo"""
+    import binascii
+    try:
+        with open(index_path, 'rb') as f:
+            size=_os.path.getsize(index_path)
+            f.seek(size-20);
+            return binascii.hexlify(f.read(20)).decode('ascii')
+    except FileNotFoundError:
+        return None
+
+
 def _clear_tags_for_message(index, id):
     """
     Clear any existing index entries for message 'id'
-- 
2.35.2

[PATCH 15/16] debian: install notmuch-git

[David Bremner]

Use a separate binary package to avoid dragging in dependencies on
python and git for those that do not want them.
---
 debian/control             | 16 ++++++++++++++++
 debian/notmuch-git.install |  1 +
 2 files changed, 17 insertions(+)
 create mode 100644 debian/notmuch-git.install

diff --git a/debian/control b/debian/control
index a11d4130..9706b0f7 100644
--- a/debian/control
+++ b/debian/control
@@ -66,6 +66,22 @@ Description: thread-based email index, search and tagging
  .
  This package contains the notmuch command-line interface
 
+Package: notmuch-git
+Architecture: all
+Depends:
+ git,
+ notmuch,
+ python3,
+ ${misc:Depends}
+Description: thread-based email index, search and tagging
+ Notmuch is a system for indexing, searching, reading, and tagging
+ large collections of email messages in maildir or mh format. It uses
+ the Xapian library to provide fast, full-text search with a very
+ convenient search syntax.
+ .
+ This package contains a simple tool to save, restore, and synchronize
+ notmuch tags via git repositories.
+
 Package: notmuch-doc
 Architecture: all
 Depends:
diff --git a/debian/notmuch-git.install b/debian/notmuch-git.install
new file mode 100644
index 00000000..ca632418
--- /dev/null
+++ b/debian/notmuch-git.install
@@ -0,0 +1 @@
+notmuch-git /usr/bin
-- 
2.35.2

[PATCH 16/16] WIP: start manual page for notmuch-git

[David Bremner]

---
 doc/conf.py              |   4 ++
 doc/index.rst            |   1 +
 doc/man1/notmuch-git.rst | 106 +++++++++++++++++++++++++++++++++++++++
 3 files changed, 111 insertions(+)
 create mode 100644 doc/man1/notmuch-git.rst

diff --git a/doc/conf.py b/doc/conf.py
index e46e1d4e..da0635bb 100644
--- a/doc/conf.py
+++ b/doc/conf.py
@@ -123,6 +123,10 @@ man_pages = [
      u'send mail with notmuch and emacs',
      [notmuch_authors], 1),
 
+    ('man1/notmuch-git', 'notmuch-git',
+     u'manage notmuch tags with git',
+     [notmuch_authors], 1),
+
     ('man5/notmuch-hooks', 'notmuch-hooks',
      u'hooks for notmuch',
      [notmuch_authors], 5),
diff --git a/doc/index.rst b/doc/index.rst
index fbdcf779..c380ee1d 100644
--- a/doc/index.rst
+++ b/doc/index.rst
@@ -15,6 +15,7 @@ Contents:
    man1/notmuch-dump
    notmuch-emacs
    man1/notmuch-emacs-mua
+   man1/notmuch-git
    man5/notmuch-hooks
    man1/notmuch-insert
    man1/notmuch-new
diff --git a/doc/man1/notmuch-git.rst b/doc/man1/notmuch-git.rst
new file mode 100644
index 00000000..4877f22d
--- /dev/null
+++ b/doc/man1/notmuch-git.rst
@@ -0,0 +1,106 @@
+.. _notmuch-git(1):
+
+============
+notmuch-git
+============
+
+SYNOPSIS
+========
+
+**notmuch** **git** [-h] [-C REPO] [-p PREFIX] [-v] [-l *log level*] *subcommand*
+
+DESCRIPTION
+===========
+
+Manage notmuch tags with Git.
+
+Options
+-------
+
+Supported options for **notmuch-git** include
+
+.. program:: notmuch-git
+
+.. option::  -h, --help
+
+   show help message and exit
+
+.. option:: -C repo, --git-dir repo
+
+   Operate on git repository *repo*
+
+.. option::  -p prefix, --tag-prefix prefix
+
+   Operate only on tags with prefix *prefix*
+
+.. option::   -v, --version
+
+   show notmuch-git's version number and exit
+
+.. option::   -l *level*, --log-level *level* {critical,error,warning,info,debug}
+
+   Log verbosity. Defaults to 'warning'.
+
+Subcommands
+-----------
+
+For help on a particular subcommand, run: 'notmuch-git ... <command> --help'.
+
+.. option:: archive [TREE-ISH] [ARG ...]
+
+Dump a tar archive of the current nmbug tag set using 'git archive'.
+
+For each tag *tag* for message with Message-Id *id* an empty file
+
+  tags/encode(*id*)/encode(*tag*)
+
+is written to the output.
+
+The encoding preserves alphanumerics, and the characters
+"+-_@=.:," (not the quotes).  All other octets are replaced with
+'%%' followed by a two digit hex number.
+
+positional arguments:
+  TREE-ISH    The tree or commit to produce an archive for. Defaults to
+              'HEAD'.
+  ARG         Argument passed through to 'git archive'. Set anything before
+              <tree-ish>, see any:`git-archive(1)` for details.
+
+.. option:: checkout
+
+Update the notmuch database from Git.
+
+This is mainly useful to discard your changes in notmuch relative
+to Git.
+Create a local nmbug repository from a remote source.
+
+.. option:: clone repository
+
+This wraps 'git clone', adding some options to avoid creating a
+working tree while preserving remote-tracking branches and
+upstreams.
+
+positional arguments:
+  repository  The (possibly remote) repository to clone from. See the URLS section of git-clone(1) for more information on specifying repositories.
+
+    clone               Create a local nmbug repository from a remote source.
+    commit              Commit prefix-matching tags from the notmuch database to Git.
+    fetch               Fetch changes from the remote repository.
+    help                Show help for an nmbug command.
+    init                Create an empty nmbug repository.
+    log                 A simple wrapper for 'git log'.
+    merge               Merge changes from 'reference' into HEAD and load the result into notmuch.
+    pull                Pull (merge) remote repository changes to notmuch.
+    push                Push the local nmbug Git state to a remote repository.
+    status              Show pending updates in notmuch or git repo.
+
+
+
+
+SEE ALSO
+========
+
+:any:`notmuch(1)`,
+:any:`notmuch-dump(1)`,
+:any:`notmuch-restore(1)`,
+:any:`notmuch-tag(1)`
-- 
2.35.2

Re: WIP: promote nmbug to user sync tool

[David Bremner]

David Bremner <david@tethera.net> writes:

> - the (new) man page is inadequate
> - notmuch git doesn't understand the common arguments to notmuch (--config and friends)
> - the defaults for prefix and git repo are wrong for most users. I was thinking about either a
>   --nmbug argument to set the old defaults, or having the script recognize if it is invoked as nmbug.

Related, the current script does not understand NOTMUCH_PROFILE. That
would be a natural way to locate the default git repo.

Re: WIP: promote nmbug to user sync tool

[Sean Whitton]

[please keep me CCed]

Hello David,

On Sat 23 Apr 2022 at 10:38am -03, David Bremner wrote:

> One of the things that new (and old) users of notmuch often miss is a
> way to sync tags between hosts. muchsync exists, but it's a third
> party tool, and (if I understand correctly) it directly modifies the
> notmuch database. Meanwhile we ship a moderately complex python script
> called 'nmbug' which can be used to sync arbitrary tags between hosts.
> This series is my (first pass at an) attempt to promote nmbug to a
> notmuch subcommand "notmuch git". My plan is to replace my own
> homebrew "commit notmuch dump output to git" script with this new
> command; I think it may appeal to other users as well. As a bonus, it
> provides a simple (for the user) way to do incremental backups of the
> notmuch database.
>
> I had to add a couple of caching tricks to make the script usable for
> my database, but it is now reasonably fast (say 20s) to commit a days
> worth of changes on my machine.

I think I'm the person who's already been using nmbug for personal tags
for an appreciable length of time.  It is great to see these caching
improvements and the addition of a test suite.

> There are still some rough edges, mainly due to the heritage of the
> script. Some I am aware of include:
>
> - the (new) man page is inadequate
> - notmuch git doesn't understand the common arguments to notmuch (--config and friends)
> - the defaults for prefix and git repo are wrong for most users. I was thinking about either a
>   --nmbug argument to set the old defaults, or having the script recognize if it is invoked as nmbug.
> - There are a few too many global variables
> - I just remembered, MAX_LASTMOD is unused, leftover from a previous design for querying tags.

Just looking at my current usage, there are two cases where I've wrapped
nmbug in some additional myrepos scripting.  The first is a status
command:

    status =
        nmbug-spw status | grep -v "^U\s" || true
        # `nmbug status` does not catch committed but unpushed changes
        git --no-pager log --branches \
            --not --remotes \
            --simplify-by-decoration --decorate --oneline

Possibly notmuch-git could learn how to do this?

Secondly, I've got this auto-commit command:

    autoci =
        nmbug-spw status | perl -ne'/^[AD][ad]?\s/ and $i++ > 500 and exit 1' \
                  && nmbug-spw commit

The guard has two purposes.  Firstly, it avoids wiping out git as part
of a cronjob auto-commit because the nmbug repo has just been cloned but
'nmbug checkout' hasn't been run.  Secondly, it avoids doing any
committing if there are any known remote changes that haven't been
integrated.  The latter thing is probably just a personal preference.

As for the former thing, I wonder if instead there could be some
mechanism, connected with the new caching, to associate nmbug repos with
the notmuch database, and refuse to operate unless that association
already exists?  So, 'nmbug checkout' would mark it as safe to sync back
and forth between the database and that repo no matter the number of
changes.

On Sat 23 Apr 2022 at 03:49pm -03, David Bremner wrote:

> Related, the current script does not understand NOTMUCH_PROFILE. That
> would be a natural way to locate the default git repo.

It would, but it wouldn't help with configuring a default prefix.
Perhaps an entry in .notmuch-config for that?  Currently I use a tiny
wrapper script:

    #!/bin/sh

    NMBGIT="$HOME/lib/nmbug-spw" NMBPREFIX="spw::" nmbug "$@"

but it would be great to just be able to type 'notmuch git ...'.

-- 
Sean Whitton

Re: WIP: promote nmbug to user sync tool

[David Bremner]

Sean Whitton <spwhitton@spwhitton.name> writes:

> [please keep me CCed]
>
> Hello David,
>
> On Sat 23 Apr 2022 at 10:38am -03, David Bremner wrote:
>

> Just looking at my current usage, there are two cases where I've wrapped
> nmbug in some additional myrepos scripting.  The first is a status
> command:
>
>     status =
>         nmbug-spw status | grep -v "^U\s" || true
>         # `nmbug status` does not catch committed but unpushed changes
>         git --no-pager log --branches \
>             --not --remotes \
>             --simplify-by-decoration --decorate --oneline
>
> Possibly notmuch-git could learn how to do this?

Perhaps. I think I would prefer something a bit more concise like a
count of unpushed commits. Do you tend to actually have meaningful
commit messages?

> Secondly, I've got this auto-commit command:
>
>     autoci =
>         nmbug-spw status | perl -ne'/^[AD][ad]?\s/ and $i++ > 500 and exit 1' \
>                   && nmbug-spw commit
>

[snip]

> As for the former thing, I wonder if instead there could be some
> mechanism, connected with the new caching, to associate nmbug repos with
> the notmuch database, and refuse to operate unless that association
> already exists?  So, 'nmbug checkout' would mark it as safe to sync back
> and forth between the database and that repo no matter the number of
> changes.

Personally I would be more worried about checkout (e.g. after init)
wiping out my notmuch database, since an errant commit can always be
reverted. Both cases seem to be covered by your heuristic. Perhaps we
could just count the size of the update, and insist on a --force option
if it is too large.

>
> On Sat 23 Apr 2022 at 03:49pm -03, David Bremner wrote:
>
>> Related, the current script does not understand NOTMUCH_PROFILE. That
>> would be a natural way to locate the default git repo.
>
> It would, but it wouldn't help with configuring a default prefix.
> Perhaps an entry in .notmuch-config for that?  Currently I use a tiny
> wrapper script:
>
>     #!/bin/sh
>
>     NMBGIT="$HOME/lib/nmbug-spw" NMBPREFIX="spw::" nmbug "$@"
>
> but it would be great to just be able to type 'notmuch git ...'.
>

For what it's worth, you can already call

    notmuch git -C $HOME/lib/nmbug-spw -p spw:: ...

if that is more convenient.

The defaults have already changed in my latest working branch so the
default repo is under $XDG_DATA_HOME/notmuch/$NOTMUCH_PROFILE/git, and
the default prefix is ''.  But re-reading this, I see see we polled two
people and got two answers for what a reasonable default prefix is, so a
configuration item is definitely needed for prefix. Probably it is also
reasonable to have one for repo location.

Re: WIP: promote nmbug to user sync tool

[Sean Whitton]

Hello,

On Sat 07 May 2022 at 09:01pm -03, David Bremner wrote:

> Sean Whitton <spwhitton@spwhitton.name> writes:
>
>> Just looking at my current usage, there are two cases where I've wrapped
>> nmbug in some additional myrepos scripting.  The first is a status
>> command:
>>
>>     status =
>>         nmbug-spw status | grep -v "^U\s" || true
>>         # `nmbug status` does not catch committed but unpushed changes
>>         git --no-pager log --branches \
>>             --not --remotes \
>>             --simplify-by-decoration --decorate --oneline
>>
>> Possibly notmuch-git could learn how to do this?
>
> Perhaps. I think I would prefer something a bit more concise like a
> count of unpushed commits. Do you tend to actually have meaningful
> commit messages?

I don't.  I just want output if there are unpushed changes and no output
if not.  A count sounds good to me.

> Personally I would be more worried about checkout (e.g. after init)
> wiping out my notmuch database, since an errant commit can always be
> reverted. Both cases seem to be covered by your heuristic. Perhaps we
> could just count the size of the update, and insist on a --force option
> if it is too large.

I think you're right.  It makes sense to build in safety features only
for the case of accidentally wiping out the db.

Either instead or in addition to something size-based, how about
requiring --force if there do not exist any tags with the prefix in the
notmuch database already?  The size thing is brittle; in my scripting
attempts, I've encountered several annoying edge cases.

> For what it's worth, you can already call
>
>     notmuch git -C $HOME/lib/nmbug-spw -p spw:: ...
>
> if that is more convenient.
>
> The defaults have already changed in my latest working branch so the
> default repo is under $XDG_DATA_HOME/notmuch/$NOTMUCH_PROFILE/git, and
> the default prefix is ''.  But re-reading this, I see see we polled two
> people and got two answers for what a reasonable default prefix is, so a
> configuration item is definitely needed for prefix. Probably it is also
> reasonable to have one for repo location.

Coolio.

-- 
Sean Whitton

Re: WIP: promote nmbug to user sync tool

[David Bremner]

Sean Whitton <spwhitton@spwhitton.name> writes:

>
> Either instead or in addition to something size-based, how about
> requiring --force if there do not exist any tags with the prefix in the
> notmuch database already?  The size thing is brittle; in my scripting
> attempts, I've encountered several annoying edge cases.
>

Looking at the code, I should add add a check for no existing tags in
any case, just to avoid a divide by zero error in 'check_safe_fraction".