nmfirehose - a crude nmbug companion tool

nmfirehose - a crude nmbug companion tool

[Jani Nikula]

[-- Attachment #1: Type: text/plain, Size: 2040 bytes --]


Hi all -

There was some talk on IRC about maintaining a development branch
containing the "maybe ready" patches [1] to better expose them to
real-life use before merging to master.

I set out to automate this a little, and ended up with nmfirehose. It's
a shell script that can be used to grab a bunch of patches from the
notmuch mail store based on queries, and apply them to a repository. In
particular, it can grab the "maybe ready" patches and apply them against
notmuch master.

Inevitable conflicts are handled by applying patches in oldest first
order, and dropping any conflicting series. An email thread approximates
a patch series; more than one patch series in a thread are folded into
one series.

Patches are applied incrementally, on top of each other by default, but
there's also support for non-incremental per-series apply against master
to see if there are conflicts. This is useful for the "review"
query. Conflicting patches can be automatically tagged if so desired
(but I don't recommend pushing these to nmbug without human review).

How well any of this works very much depends on how accurately the patch
messages are tagged.

Some examples (run within a notmuch git repo, with up-to-date nmbug tags
in the database):

# take nmbug maybe-ready patches and apply them against origin/master
$ nmfirehose

# use local branch firehose and force use it even if existing
$ nmfirehose -l firehose -F

# keep going after conflicts without reverting whole series
$ nmfirehose -k

# check each series in review queue against master, and tag conflicting
$ nmfirehose -q review -n -t

# help
$ nmfirehose -h

I originally planned on putting this to cron and pushing the result
(supported out-of-the-box) to some public repo, but in the end I think
people can use this locally just as well, at the pace they want.

The script tries not to screw up anything in your notmuch repo, but be
careful nonetheless.

Script attached; comments welcome. Is this worth adding to contrib?


BR,
Jani.

[1] http://nmbug.tethera.net/status/

[-- Attachment #2: nmfirehose --]
[-- Type: application/octet-stream, Size: 5863 bytes --]

#!/bin/sh
# Copyright (c) 2012 Jani Nikula
# License: same as notmuch

# Predefined queries. See http://nmbug.tethera.net/status/
QUERY_MAYBE_READY="tag:notmuch::patch and not tag:notmuch::pushed and not tag:notmuch::obsolete and not tag:notmuch::wip and not tag:notmuch::stale and not tag:notmuch::contrib and not tag:notmuch::moreinfo and not tag:notmuch::python and not tag:notmuch::vim and not tag:notmuch::wontfix and not tag:notmuch::needs-review"

QUERY_REVIEW="tag:notmuch::patch and not tag:notmuch::pushed and not tag:notmuch::obsolete and not tag:notmuch::stale and not tag:notmuch::wontfix and (tag:notmuch::moreinfo or tag:notmuch::needs-review)"

# Defaults for command line options
# local
LOCAL_BRANCH="firehose-`date +%F.%H%M%S`"
FORCE_LOCAL=
GIT_CHECKOUT_MODE="-b"
QUERY="$QUERY_MAYBE_READY"
# upstream
DO_FETCH=
UPSTREAM_NAME="origin"
UPSTREAM_BRANCH="master"
# pushing
DO_PUSH=
GIT_PUSH_FORCE=
REMOTE_BRANCH="firehose"
REMOTE_NAME="firehose"
# conflict management
DO_NON_INCREMENTAL=
DO_KEEP_GOING=
DO_TAG=
CONFLICT_TAG="maybe-stale"

usage()
{
    cat <<EOF
nmfirehose [options ...]

Apply patches on top of an upstream repository from a notmuch mail store based
on notmuch queries.

Run in a git tree with suitable remote(s) set ($UPSTREAM_NAME is the default
upstream remote).

An email thread is expected to contain exactly one patch series. Multiple series
in a thread are considered one series. The patch order in a series is based on
notmuch search --sort=oldest-first.

Local:
  -l LOCAL_BRANCH     local branch to use (default: $LOCAL_BRANCH)
  -F                  force use local branch even if existing
  -q QUERY_NAME       predefined query to use, see below (default: maybe-ready)
  -Q QUERY            query to use (default: same as -q maybe-ready)

Upstream baseline to apply patches on:
  -f                  fetch upstream first
  -u UPSTREAM_BRANCH  upstream branch (default: $UPSTREAM_BRANCH)
  -U UPSTREAM_NAME    upstream remote name (default: $UPSTREAM_NAME)

Pushing the result:
  -p                  push the result afterwards
  -P                  force push the result afterwards
  -r REMOTE_BRANCH    remote branch to push to (default: $REMOTE_BRANCH)
  -R REMOTE_NAME      remote name to push to (default: $REMOTE_NAME)

Conflict management:
  -n                  non-incremental, apply each series on upstream
  -k                  keep going after conflicts without reverting series
  -t                  tag conflicting patches
  -T TAG              tag for conflicting patches (default: $CONFLICT_TAG)

Predefined queries:

maybe-ready = $QUERY_MAYBE_READY

review = $QUERY_REVIEW
EOF
    exit 0
}

error()
{
    echo "nmfirehose: $*" >&2
    exit 1
}

set -eu

while getopts l:Fq:Q:fu:U:pPr:R:nktT:h opt; do
    case "$opt" in
	# local
	l) LOCAL_BRANCH="$OPTARG";;
	F) FORCE_LOCAL=1; GIT_CHECKOUT_MODE="-B";;
	q)
	    if [ "$OPTARG" = "maybe-ready" ]; then
		QUERY="$QUERY_MAYBE_READY"
	    elif [ "$OPTARG" = "review" ]; then
		QUERY="$QUERY_REVIEW"
	    else
		error "unknown predefined query '$OPTARG'"
	    fi
	    ;;
	Q) QUERY="$OPTARG";;
	# upstream
	f) DO_FETCH=1;;
	u) UPSTREAM_BRANCH="$OPTARG";;
	U) UPSTREAM_NAME="$OPTARG";;
	# pushing
	p) DO_PUSH=1;;
	P) DO_PUSH=1; GIT_PUSH_FORCE="--force";;
	r) REMOTE_BRANCH="$OPTARG";;
	R) REMOTE_NAME="$OPTARG";;
	# conflict management
	n) DO_NON_INCREMENTAL=1;;
	k) DO_KEEP_GOING=1;;
	t) DO_TAG=1;;
	T) CONFLICT_TAG="$OPTARG";;
	# help
	h) usage;;
	\?) error "try -h for usage";;
    esac
done
shift `expr $OPTIND - 1`

if [ "$#" != "0" ]; then
    error "unsupported positional argument(s) '$*'; try -h for usage"
fi

if [ -n "$DO_PUSH" -a -n "$DO_NON_INCREMENTAL" ]; then
    error "non-incremental push is not supported"
fi

# NOTE: This does not catch all the ways users can shoot themselves in the feet.
if ! git diff --quiet || ! git diff --cached --quiet; then
    error "uncommitted local changes"
fi

# FIXME: Check that we're in a git-dir and work-dir and not bare. git rev-parse.

if [ -n "$DO_FETCH" ]; then
    git fetch $UPSTREAM_NAME
fi

RESET_HEAD=$UPSTREAM_NAME/$UPSTREAM_BRANCH

# NOTE: This does not catch all the ways users can shoot themselves in the feet.
if [ "`git name-rev --name-only HEAD`" = "$LOCAL_BRANCH" ]; then
    if [ -n "$FORCE_LOCAL" ]; then
	git reset --hard $RESET_HEAD
    else
	error "already on branch $LOCAL_BRANCH, try -F to force"
    fi
else
    git checkout $GIT_CHECKOUT_MODE $LOCAL_BRANCH $UPSTREAM_NAME/$UPSTREAM_BRANCH
fi

cleanup()
{
    rv=$?
    # FIXME: could do more clever things here
    git am --abort >/dev/null 2>&1
    exit $rv
}
trap cleanup 1 2 3 6 15

THREADS=`notmuch search --output=threads --sort=oldest-first $QUERY`

for thread in $THREADS; do
    MESSAGES=`notmuch search --output=messages --sort=oldest-first $QUERY AND $thread`
    subject=

    for message in $MESSAGES; do
	if [ -z "$subject" ]; then
	    subject=`notmuch show --format=raw $message | grep -m 1 "^Subject: " | sed 's/^Subject: //'`
	    echo "$thread: \"$subject\":"
	fi

	if notmuch show --format=mbox $message | git am - >/dev/null 2>&1; then
	    echo "+ $message applied."
	else
	    git am --abort

	    if [ -n "$DO_TAG" -a -n "$CONFLICT_TAG" ]; then
		notmuch tag +$CONFLICT_TAG -- $message
	    fi

	    if [ -n "$DO_KEEP_GOING" ]; then
		echo "$message conflicts, skipping."
	    else
		if [ -n "$DO_NON_INCREMENTAL" ]; then
		    echo "- $message conflicts."
		    # branch will be reset anyway
		else
		    echo "- $message conflicts, reverting thread."
		    git reset --hard $RESET_HEAD
		fi
		break
	    fi
	fi
    done

    if [ -n "$DO_NON_INCREMENTAL" ]; then
	echo "non-incremental mode, reverting to $RESET_HEAD."
	git reset --hard $RESET_HEAD
    else
	RESET_HEAD=`git rev-parse HEAD`
    fi
done

if [ -n "$DO_PUSH" ]; then
    git push $GIT_PUSH_FORCE $REMOTE_NAME $LOCAL_BRANCH:$REMOTE_BRANCH
fi

Re: nmfirehose - a crude nmbug companion tool

[David Bremner]

On Sun, 11 Mar 2012 02:01:34 +0200, Jani Nikula <jani@nikula.org> wrote:
> 
> Hi all -
> 
> There was some talk on IRC about maintaining a development branch
> containing the "maybe ready" patches [1] to better expose them to
> real-life use before merging to master.

Hi Jani;

Thanks for writing this. At least I find it useful, so maybe we should
include it (or some variation) in contrib. Probably make nmbug a
subdirectory, and put both in there.

d