[PATCH v2] nmbug: Add 'clone' and replace FETCH_HEAD with @{upstream}

[PATCH v2] nmbug: Add 'clone' and replace FETCH_HEAD with @{upstream}

[W. Trevor King]

With two branches getting fetched (master and config), the branch
referenced by FETCH_HEAD is ambiguous.  For example, I have:

  $ cat FETCH_HEAD
  41d7bfa7184cc93c9dac139d1674e9530799e3b0 \
    not-for-merge   branch 'config' of http://nmbug.tethera.net/git/nmbug-tags
  acd379ccb973c45713eee9db177efc530f921954 \
    not-for-merge   branch 'master' of http://nmbug.tethera.net/git/nmbug-tags

(where I wrapped the line by hand).  This means that FETCH_HEAD
references the config branch:

  $ git rev-parse FETCH_HEAD
  41d7bfa7184cc93c9dac139d1674e9530799e3b0

which breaks all of the FETCH_HEAD logic in nmbug (where FETCH_HEAD is
assumed to point to the master branch).

Instead of relying on FETCH_HEAD, use @{upstream} as the
remote-tracking branch that should be merged/diffed/integrated into
HEAD.  @{upstream} was added in Git v1.7.0 (2010-02-12) [1], so
relying on it should be fairly safe.  One tricky bit is that bare
repositories don't set upstream tracking branches by default:

  $ git clone --bare http://nmbug.tethera.net/git/nmbug-tags.git nmbug-bare
  $ cd nmbug-bare
  $ git remote show origin
  * remote origin
    Fetch URL: http://nmbug.tethera.net/git/nmbug-tags.git
    Push  URL: http://nmbug.tethera.net/git/nmbug-tags.git
    HEAD branch: master
    Local refs configured for 'git push':
      config pushes to config (up to date)
      master pushes to master (up to date)

While in a non-bare clone:

  $ git clone http://nmbug.tethera.net/git/nmbug-tags.git
  $ cd nmbug-tags
  $ git remote show origin
  * remote origin
    Fetch URL: http://nmbug.tethera.net/git/nmbug-tags.git
    Push  URL: http://nmbug.tethera.net/git/nmbug-tags.git
    HEAD branch: master
    Remote branches:
      config tracked
      master tracked
    Local branch configured for 'git pull':
      master merges with remote master
    Local ref configured for 'git push':
      master pushes to master (up to date)

From the clone docs [2]:

  --bare::
        Make a 'bare' Git repository…
        Also the branch heads at the remote are copied directly
        to corresponding local branch heads, without mapping
        them to `refs/remotes/origin/`.  When this option is
        used, neither remote-tracking branches nor the related
        configuration variables are created.

To use @{upstream}, we need to the local vs. remote-tracking
distinction, so this commit adds 'nmbug clone', replacing the
previously suggested --bare clone with a non-bare --no-checkout
--separate-git-dir clone into a temporary work directory.  After
which:

  $ git rev-parse @{upstream}
  acd379ccb973c45713eee9db177efc530f921954

gives us the master-branch commit.  Existing nmbug users will have to
run the configuration tweaks and re-fetch by hand.  If you don't have
any local commits, you could also blow away your NMBGIT repository and
re-clone from scratch:

  $ nmbug clone http://nmbug.tethera.net/git/nmbug-tags.git

Besides removing the ambiguity of FETCH_HEAD, this commit allows users
to configure which upstream branch they want nmbug to track via 'git
config', in case they want to change their upstream repository.

[1]: http://git.kernel.org/cgit/git/git.git/tree/Documentation/RelNotes/1.7.0.txt
[2]: http://git.kernel.org/cgit/git/git.git/tree/Documentation/git-clone.txt
---

Changes since v1 [1]:

* Replaced confusing post-clone fetch with a single --no-checkout
  --separate-git-dir clone [2].  This avoids a bare repository's lack of
  remote-tracking branches by not creating a bare repository in the
  first place, while v1 tried to “fix” the bare repo.

Cheers,
Trevor

[1]: id:80db13662a0313711ca3a91e338a63d35e64a83c.1391898002.git.wking@tremily.us
     http://article.gmane.org/gmane.mail.notmuch.general/17154
[2]: id:20140308162633.GM16433@odin.tremily.us
     http://article.gmane.org/gmane.mail.notmuch.general/17409

 devel/nmbug/nmbug | 45 ++++++++++++++++++++++++++++++++-------------
 1 file changed, 32 insertions(+), 13 deletions(-)

diff --git a/devel/nmbug/nmbug b/devel/nmbug/nmbug
index 90d98b6..d6f5213 100755
--- a/devel/nmbug/nmbug
+++ b/devel/nmbug/nmbug
@@ -26,6 +26,7 @@ my $ESCAPED_RX =	qr{$ESCAPE_CHAR([A-Fa-f0-9]{2})};
 my %command = (
 	     archive	=> \&do_archive,
 	     checkout	=> \&do_checkout,
+	     clone	=> \&do_clone,
 	     commit	=> \&do_commit,
 	     fetch	=> \&do_fetch,
 	     help	=> \&do_help,
@@ -125,6 +126,15 @@ sub do_archive {
   system ('git', "--git-dir=$NMBGIT", 'archive', 'HEAD');
 }
 
+sub do_clone {
+  my $repository = shift;
+
+  my $tempwork = tempdir ('/tmp/nmbug-clone.XXXXXX', CLEANUP => 1);
+  system ('git', 'clone', '--no-checkout', '--separate-git-dir', $NMBGIT,
+          $repository, $tempwork) == 0
+    or die "'git clone' exited with nonzero value\n";
+  git ('config', '--unset', 'core.worktree');
+}
 
 sub is_committed {
   my $status = shift;
@@ -332,21 +342,24 @@ To discard your changes,  run 'nmbug checkout'
 
 sub do_pull {
   my $remote = shift || 'origin';
+  my $branch = shift || 'master';
 
   git ( 'fetch', $remote);
 
-  do_merge ();
+  do_merge ("$remote/$branch");
 }
 
 
 sub do_merge {
+  my $commit = shift || '@{upstream}';
+
   insist_committed ();
 
   my $tempwork = tempdir ('/tmp/nmbug-merge.XXXXXX', CLEANUP => 1);
 
   git ( { GIT_WORK_TREE => $tempwork }, 'checkout', '-f', 'HEAD');
 
-  git ( { GIT_WORK_TREE => $tempwork }, 'merge', 'FETCH_HEAD');
+  git ( { GIT_WORK_TREE => $tempwork }, 'merge', $commit);
 
   do_checkout ();
 }
@@ -407,11 +420,10 @@ sub do_status {
 
 
 sub is_unmerged {
+  my $commit = shift || '@{upstream}';
 
-  return 0 if (! -f $NMBGIT.'/FETCH_HEAD');
-
-  my $fetch_head = git ('rev-parse', 'FETCH_HEAD');
-  my $base = git ( 'merge-base', 'HEAD', 'FETCH_HEAD');
+  my $fetch_head = git ('rev-parse', $commit);
+  my $base = git ( 'merge-base', 'HEAD', $commit);
 
   return ($base ne $fetch_head);
 
@@ -473,7 +485,7 @@ sub diff_index {
 sub diff_refs {
   my $filter = shift;
   my $ref1 = shift || 'HEAD';
-  my $ref2 = shift || 'FETCH_HEAD';
+  my $ref2 = shift || '@{upstream}';
 
   my $fh= git_pipe ( 'diff', "--diff-filter=$filter", '--name-only',
 		 $ref1, $ref2);
@@ -561,10 +573,11 @@ git. Any extra arguments are used (one per line) as a commit message.
 
 push local nmbug git state to remote repo
 
-=item  B<pull> [remote]
+=item  B<pull> [remote] [branch]
 
 pull (merge) remote repo changes to notmuch. B<pull> is equivalent to
-B<fetch> followed by B<merge>.
+B<fetch> followed by B<merge>.  The default remote is C<origin>, and
+the default branch is C<master>.
 
 =back
 
@@ -572,6 +585,12 @@ B<fetch> followed by B<merge>.
 
 =over 8
 
+=item B<clone> repository
+
+Create a local nmbug repository from a remote source.  This wraps
+C<git clone>, adding some options to avoid creating a working tree
+while preserving remote-tracking branches and upstreams.
+
 =item B<checkout>
 
 Update the notmuch database from git. This is mainly useful to discard
@@ -589,12 +608,12 @@ print help [for subcommand]
 =item B<log> [parameters]
 
 A simple wrapper for git log. After running C<nmbug fetch>, you can
-inspect the changes with C<nmbug log HEAD..FETCH_HEAD>
+inspect the changes with C<nmbug log HEAD..@{upstream}>
 
-=item B<merge>
+=item B<merge> [commit]
 
-Merge changes from FETCH_HEAD into HEAD, and load the result into
-notmuch.
+Merge changes from C<commit> into HEAD, and load the result into
+notmuch.  The default commit is C<@{upstream}>.
 
 =item  B<status>
 
-- 
1.8.5.2.8.g0f6c0d1

Re: [PATCH v2] nmbug: Add 'clone' and replace FETCH_HEAD with @{upstream}

[David Bremner]

"W. Trevor King" <wking@tremily.us> writes:

> With two branches getting fetched (master and config), the branch
> referenced by FETCH_HEAD is ambiguous.  For example, I have:
>
>   $ cat FETCH_HEAD
>   41d7bfa7184cc93c9dac139d1674e9530799e3b0 \
>     not-for-merge   branch 'config' of http://nmbug.tethera.net/git/nmbug-tags
>   acd379ccb973c45713eee9db177efc530f921954 \
>     not-for-merge   branch 'master' of http://nmbug.tethera.net/git/nmbug-tags

This patch seems OK.  I think it makes sense to have an "nmbug clone"
command; perhaps the implementation can be simplified in the future (or
perhaps not).

d

WARNING: backwards incompatible change to nmbug pushed.

[David Bremner]

I have pushed this bug fix.

See Trevor's extensive commit message for the details, but the short
version is that if you probably want to run

% mv .nmbug .nmbug.bak && nmbug clone nmbug@nmbug.tethera.net:nmbug-tags

Previous versions of nmbug will work with the resulting repo.

d

Re: [PATCH v2] nmbug: Add 'clone' and replace FETCH_HEAD with @{upstream}

[David Bremner]

BTW, can you send a short patch for NEWS ?

d

[PATCH] NEWS: Document the recent 'nmbug clone' and @{upstream} changes

[W. Trevor King]

The changes just landed with c200167 (nmbug: Add 'clone' and replace
FETCH_HEAD with @{upstream}, 2014-03-09).

The preferred markup language for NEWS seems to be Markdown, which is
parsed by devel/news2wiki.pl into Markdown chunks for rendering by
ikiwiki [1].

[1]: http://notmuchmail.org/news/
---
 NEWS | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/NEWS b/NEWS
index d4f4ea4..e26fa0a 100644
--- a/NEWS
+++ b/NEWS
@@ -20,6 +20,26 @@ Bug fix for saved searches with newlines in them.
   Split lines confuse `notmuch count --batch`, so we remove embedded
   newlines before calling notmuch count.
 
+nmbug
+-----
+
+nmbug adds a `clone` command for setting up the initial repository and
+uses `@{upstream}` instead of `FETCH_HEAD` to track upstream changes.
+
+  The `@{upstream}` change reduces ambiguity when fetching multiple
+  branches, but requires existing users update their bare `NMBGIT`
+  repository (usually `~/.nmbug`) to a non-bare repository.  The
+  easiest way to do this is:
+
+  1. Push any local commits to a remote repository.
+  2. Remove your `NMBGIT` repository (e.g. `mv .nmbug .nmbug.bak`).
+  3. Use the new `clone` command to create a fresh clone:
+
+        nmbug clone nmbug@nmbug.tethera.net:nmbug-tags
+
+  4. If you had local commits in step 1, add a remote for that
+     repository and fetch them into the new repository.
+
 Notmuch 0.17 (2013-12-30)
 =========================
 
-- 
1.9.1.353.gc66d89d

Re: [PATCH] NEWS: Document the recent 'nmbug clone' and @{upstream} changes

[David Bremner]

"W. Trevor King" <wking@tremily.us> writes:

> The changes just landed with c200167 (nmbug: Add 'clone' and replace
> FETCH_HEAD with @{upstream}, 2014-03-09).
>
> The preferred markup language for NEWS seems to be Markdown, which is
> parsed by devel/news2wiki.pl into Markdown chunks for rendering by
> ikiwiki [1].
>
> [1]: http://notmuchmail.org/news/
> ---
>  NEWS | 20 ++++++++++++++++++++
>  1 file changed, 20 insertions(+)
>
> diff --git a/NEWS b/NEWS
> index d4f4ea4..e26fa0a 100644
> --- a/NEWS
> +++ b/NEWS
> @@ -20,6 +20,26 @@ Bug fix for saved searches with newlines in them.
>    Split lines confuse `notmuch count --batch`, so we remove embedded
>    newlines before calling notmuch count.
>  
> +nmbug
> +-----
> +
> +nmbug adds a `clone` command for setting up the initial repository and
> +uses `@{upstream}` instead of `FETCH_HEAD` to track upstream changes.
> +
> +  The `@{upstream}` change reduces ambiguity when fetching multiple
> +  branches, but requires existing users update their bare `NMBGIT`
> +  repository (usually `~/.nmbug`) to a non-bare repository.  The
> +  easiest way to do this is:

That bit about non-bare seems to be untrue/misleading?

As a step 0, I guess commit any tag changes to nmbug?
> +
> +  1. Push any local commits to a remote repository.
> +  2. Remove your `NMBGIT` repository (e.g. `mv .nmbug .nmbug.bak`).
> +  3. Use the new `clone` command to create a fresh clone:
> +
> +        nmbug clone nmbug@nmbug.tethera.net:nmbug-tags
> +

Jani mentioned on IRC that some people might track nmbug in a read only
way via git://

> +  4. If you had local commits in step 1, add a remote for that
> +     repository and fetch them into the new repository.

Is the "remote repository" in step 1 meant to be the central repo? or
just a backup?

Re: [PATCH] NEWS: Document the recent 'nmbug clone' and @{upstream} changes

[W. Trevor King]

[-- Attachment #1: Type: text/plain, Size: 3713 bytes --]

On Wed, Apr 09, 2014 at 08:22:26AM -0300, David Bremner wrote:
> W. Trevor King writes:
> 
> > The changes just landed with c200167 (nmbug: Add 'clone' and replace
> > FETCH_HEAD with @{upstream}, 2014-03-09).
> >
> > The preferred markup language for NEWS seems to be Markdown, which is
> > parsed by devel/news2wiki.pl into Markdown chunks for rendering by
> > ikiwiki [1].
> >
> > [1]: http://notmuchmail.org/news/
> > ---
> >  NEWS | 20 ++++++++++++++++++++
> >  1 file changed, 20 insertions(+)
> >
> > diff --git a/NEWS b/NEWS
> > index d4f4ea4..e26fa0a 100644
> > --- a/NEWS
> > +++ b/NEWS
> > @@ -20,6 +20,26 @@ Bug fix for saved searches with newlines in them.
> >    Split lines confuse `notmuch count --batch`, so we remove embedded
> >    newlines before calling notmuch count.
> >  
> > +nmbug
> > +-----
> > +
> > +nmbug adds a `clone` command for setting up the initial repository and
> > +uses `@{upstream}` instead of `FETCH_HEAD` to track upstream changes.
> > +
> > +  The `@{upstream}` change reduces ambiguity when fetching multiple
> > +  branches, but requires existing users update their bare `NMBGIT`
> > +  repository (usually `~/.nmbug`) to a non-bare repository.  The
> > +  easiest way to do this is:
> 
> That bit about non-bare seems to be untrue/misleading?

The old repositories were bare [1]:

  git clone --bare http://nmbug.tethera.net/git/nmbug-tags.git $HOME/.nmbug

the new repositories are not [2]:

    system ('git', 'clone', '--no-checkout', '--separate-git-dir', $NMBGIT,
            $repository, $tempwork) == 0

We need non-bare repositories to have remote-tracking branches
(distinct from local branches) [3], and we need remote-tracking
branches to have working @{upstream}.

I think that's reasonable support for my claim (and most of it is in
the original c200167 commit message), but maybe not?

> As a step 0, I guess commit any tag changes to nmbug?
> > +
> > +  1. Push any local commits to a remote repository.
> > +  2. Remove your `NMBGIT` repository (e.g. `mv .nmbug .nmbug.bak`).
> > +  3. Use the new `clone` command to create a fresh clone:
> > +
> > +        nmbug clone nmbug@nmbug.tethera.net:nmbug-tags
> > +
> 
> Jani mentioned on IRC that some people might track nmbug in a read only
> way via git://

Sure.  The current docs suggest HTTP [1], so I'm fine changing this to
the HTTP URL (or the Git URL).  I think folks will get the idea
regardless.  I can resubmit v2 with this update, or you can just
squash it in if/when this patch lands in master, whatever is easiest.

> > +  4. If you had local commits in step 1, add a remote for that
> > +     repository and fetch them into the new repository.
> 
> Is the "remote repository" in step 1 meant to be the central repo? or
> just a backup?

The backup.  If you have nothing to backup, you already got everything
back after cloning the central repo.

Cheers,
Trevor

[1]: http://notmuchmail.org/nmbug/
[2]: http://git.notmuchmail.org/git/notmuch/blob/c20016742681e1ed48c83de32639e10507ffa14d:/devel/nmbug/nmbug#l133
[3]: http://git.kernel.org/cgit/git/git.git/tree/Documentation/git-clone.txt     
  --bare::
        Make a 'bare' Git repository…
        Also the branch heads at the remote are copied directly
        to corresponding local branch heads, without mapping
        them to `refs/remotes/origin/`.  When this option is
        used, neither remote-tracking branches nor the related
        configuration variables are created.


-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [PATCH] NEWS: Document the recent 'nmbug clone' and @{upstream} changes

[David Bremner]

"W. Trevor King" <wking@tremily.us> writes:

> We need non-bare repositories to have remote-tracking branches
> (distinct from local branches) [3], and we need remote-tracking
> branches to have working @{upstream}.

OK, I see what you mean, the repository has "bare = false". On the other
hand we immediately blow away the work tree, and after the initial clone
it seems to work fine to set bare = true. It might even make sense to
apply


diff --git a/devel/nmbug/nmbug b/devel/nmbug/nmbug
index d6f5213..b18ded7 100755
--- a/devel/nmbug/nmbug
+++ b/devel/nmbug/nmbug
@@ -134,6 +134,7 @@ sub do_clone {
           $repository, $tempwork) == 0
     or die "'git clone' exited with nonzero value\n";
   git ('config', '--unset', 'core.worktree');
+  git ('config', 'core.bare', 'true');
 }


> I think that's reasonable support for my claim (and most of it is in
> the original c200167 commit message), but maybe not?

In any case, I think I think it's mainly a technicality, and that we
want to keep the level of detail in the release notes down a bit.  If
you don't like the above mini-patch, then maybe a NOTES section in the
nmbug docs.

>> Is the "remote repository" in step 1 meant to be the central repo? or
>> just a backup?
>
> The backup.  If you have nothing to backup, you already got everything
> back after cloning the central repo.

It might be less confusing to explicitly use the word "backup" in step 1 then.

Re: [PATCH] NEWS: Document the recent 'nmbug clone' and @{upstream} changes

[W. Trevor King]

[-- Attachment #1: Type: text/plain, Size: 2251 bytes --]

On Wed, Apr 09, 2014 at 10:01:25PM -0300, David Bremner wrote:
> W. Trevor King writes:
> 
> > We need non-bare repositories to have remote-tracking branches
> > (distinct from local branches) [3], and we need remote-tracking
> > branches to have working @{upstream}.
> 
> OK, I see what you mean, the repository has "bare = false". On the
> other hand we immediately blow away the work tree,

But whenever we do anything that might involve a working directory
(just merging), we create a temporary one and set GIT_WORK_TREE.  So
the ~/.nmbug repo is a plain-vanilla, non-bare repo with an ephemeral
working directory.

> > I think that's reasonable support for my claim (and most of it is in
> > the original c200167 commit message), but maybe not?
> 
> In any case, I think I think it's mainly a technicality, and that we
> want to keep the level of detail in the release notes down a bit.
> If you don't like the above mini-patch, then maybe a NOTES section
> in the nmbug docs.

Sure.  I think my previous email was my shot at explaining the
situation ;).  If it wasn't clear, maybe someone else could take a
stab at a NOTES blurb.

> >> Is the "remote repository" in step 1 meant to be the central repo? or
> >> just a backup?
> >
> > The backup.  If you have nothing to backup, you already got everything
> > back after cloning the central repo.
> 
> It might be less confusing to explicitly use the word "backup" in
> step 1 then.

How about:

  1. If you have any purely local commits (i.e. they aren't in the
     nmbug repository on nmbug.tethera.net), push them to a remote
     repository.  We'll restore them from the backup in step 4.

?

We can also use the suggested ~/.nmbug.bak from step 2 as the backup
repository in step 4.  I chose the current broad-sketch approach for
steps 1 and 4, because I assumed folks with local commits would be
comfortable pushing Git branches around, and didn't want to imply that
I'd covered all the corner cases (e.g. folks with my-funky-tags
branches or other wonky repos).

Cheers,
Trevor

-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]