State of the repository conversion

State of the repository conversion

[Eric S. Raymond]

I have a complete lift script.  It satisfies all the requests that I
have received for attic-file fixups, name canonicalization,
preservation of fixes-bugs information, and the like.

I meant to push a test copy to Gitorious this weekend, but life - and
the fact that the cleanup runs take about eight hours each -
intervened.  I have a cleanup running now which may produce a
result I am ready to push around 6PM EST.

I have made almost all requested updates to the Emacs wiki.  There
are two exceptions.

From Eli Zaretskii:

     . I would suggest describing the setup of git-merge-changelog,
       because as long as we keep ChangeLog files in the repository,
       people might bump into conflicts in the logs, and it would be nice
       to avoid that.

If this is really important to have on the Emacs wiki, someone else will have
to do it. Search engines don't turn up documentation on the tool and I
don't have direct experience with it.

     . I think we should discuss some more how to work with the
       development trunk and the release branch in parallel, and reflect
       the results of the discussion in the Wiki.

       The issue here is that the release branch and the trunk diverge
       very quickly after the branch point, and the result is that after
       you checkout the other branch, you generally need a very long
       build, many times a full bootstrap.  While on a modern system, a
       full bootstrap should take a few minutes, it is still annoyingly
       long, and makes higher the risk of losing the race against other
       committers.  In addition, you only have a single executable at any
       given time, so comparison of behavior between the two branches is
       difficult.

       So perhaps we should find a way of having two separate branches,
       such that switching between them does not require a build if
       nothing changed.

This is not a documentation request, it's a research agenda.  I can't
write up a technique that doesn't exist yet.

The only other major pre-conversion to-do left on my list is "Better
cross-VCS integration of smerge in vc.el", and we have Rüdiger
Sonderfeld's patch for that.  It can be merged as soon as trunk is
open.

The lift script and conversion machinery continue to be available
fotr review at

     git@gitorious.org:emacs-transition/emacs-transition.git

Getting to this point required two full months of work, including
several major feature additions to reposurgeon.  I will be rather
surprised if I ever see another conversion as complex as this one.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Democracy is two wolves and a lamb voting on what to have for lunch. Liberty is
a well-armed lamb contesting the vote!
	-- Benjamin Franklin

Re: State of the repository conversion

[Paul Eggert]

On 03/19/2014 10:51 AM, Eric S. Raymond wrote:
> I will be rather surprised if I ever see another conversion as complex as this one.
Me too, though I've been surprised in the past.  Thanks for all your work!

Re: State of the repository conversion

[Eli Zaretskii]

> From: esr@thyrsus.com (Eric S. Raymond)
> Date: Wed, 19 Mar 2014 13:51:24 -0400 (EDT)
> 
> >From Eli Zaretskii:
> 
>      . I would suggest describing the setup of git-merge-changelog,
>        because as long as we keep ChangeLog files in the repository,
>        people might bump into conflicts in the logs, and it would be nice
>        to avoid that.
> 
> If this is really important to have on the Emacs wiki, someone else will have
> to do it. Search engines don't turn up documentation on the tool and I
> don't have direct experience with it.

The documentation for git-merge-changelog is available as a large
commentary at the beginning of the source.  Also, someone created a
man page out of that commentary and posted it to the gnulib mailing
list a week or two ago.

>      . I think we should discuss some more how to work with the
>        development trunk and the release branch in parallel, and reflect
>        the results of the discussion in the Wiki.
> 
>        The issue here is that the release branch and the trunk diverge
>        very quickly after the branch point, and the result is that after
>        you checkout the other branch, you generally need a very long
>        build, many times a full bootstrap.  While on a modern system, a
>        full bootstrap should take a few minutes, it is still annoyingly
>        long, and makes higher the risk of losing the race against other
>        committers.  In addition, you only have a single executable at any
>        given time, so comparison of behavior between the two branches is
>        difficult.
> 
>        So perhaps we should find a way of having two separate branches,
>        such that switching between them does not require a build if
>        nothing changed.
> 
> This is not a documentation request, it's a research agenda.

It was a request to discuss and decide upon the recommended workflow.
Documenting the decision is the easy part.  We did something very
similar when we switched from CVS to bzr, and the results are on the
wiki.  The current Git wiki page simply repeats what the bzr workflow
said, which is probably sub-optimal, since Git is not bzr.

Re: State of the repository conversion

[Eric S. Raymond]

Eli Zaretskii <eliz@gnu.org>:
> The documentation for git-merge-changelog is available as a large
> commentary at the beginning of the source.  Also, someone created a
> man page out of that commentary and posted it to the gnulib mailing
> list a week or two ago.

OK.  When it has a stable public location I'll add a link to it on the wiki.

> > This is not a documentation request, it's a research agenda.
> 
> It was a request to discuss and decide upon the recommended workflow.
> Documenting the decision is the easy part.  We did something very
> similar when we switched from CVS to bzr, and the results are on the
> wiki.  The current Git wiki page simply repeats what the bzr workflow
> said, which is probably sub-optimal, since Git is not bzr.

On the other hand, it is the logical place for people to start - the
bzr workflow maps over fairly well even though the individual commands
have to change.

I suspect that if change is required here it will have to evolve from
practice rather than beung invented.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Re: State of the repository conversion

[Eli Zaretskii]

> Date: Wed, 19 Mar 2014 14:54:17 -0400
> From: "Eric S. Raymond" <esr@thyrsus.com>
> Cc: emacs-devel@gnu.org
> 
> I suspect that if change is required here it will have to evolve from
> practice rather than beung invented.

I suspect that the change (which I think will indeed be required) will
express itself in unpleasant arguments, as people start using their
own workflows taken from other projects, and we discover just how
incompatible some of them are with Emacs development.

It is better to draw upon the knowhow and experience of veteran git
users among us to identify in advance the preferred workflows, discuss
them, and make decisions blessed by the Powers That Be.  We have ample
time for that, as v24.4 release is at least several months away.

I'm sorry that I cannot lead those discussions, due to lack of
knowledge.  I hope someone else will.

Re: State of the repository conversion

[Stephen J. Turnbull]

About the workflow, Eli Zaretskii writes:

 > It is better to draw upon the knowhow and experience of veteran git
 > users among us to identify in advance the preferred workflows,

And the resulting recommendations will be ignored in favor of the
convenience of the powers that be, as the recommendations for Bazaar
were.  (And mostly, so were such recommendations ignored when Python
converted from Subversion to Mercurial.)  I don't know about Eric's
experience (he's participated in a lot more of these than I have), but
I suspect at least subconsciously he's thinking the same thing.

Please remember what that convenience boiled down to: "don't change
anything that doesn't need to change, because the current workflow
works fine for us".  I admit I was wrong to suggest "the workflows
that work in other projects."  Experience showed that people adopted
those workflows as needed for their own work, but many people
continued to use Bazaar as if it were CVS.  It's only a tiny bit
harder to use git that way (no bound branches or lightweight
checkouts, so you do have to learn to push, and merge from upstream if
your push wouldn't be a fast-forward).  But people are used to the
up-to-date check failing, I suspect, and it's not that hard to learn
-- everybody did learn in the bad old days of CVS.  So it amounts to
learning that you need to "git push" after "git commit", if you were
using a checkout with Bazaar.

People who do want to use git as a free-branching distributed VCS
already know how to do that, and they may or may not follow
"recommended" workflows themselves.

So I think this is really a non-problem.  Just translate bzr commands
to the git equivalents, configure public repos to prevent "destructive"
commands (require fast-forward pushes, mostly), and let the git fans
teach those who want to improve their git skills as issues come up.
Emacs doesn't have much structure in its abstract workflow (it
basically comes down to "hack, save, push") that requires careful
consideration of VCS operations.

Re: State of the repository conversion

[Eric S. Raymond]

Stephen J. Turnbull <stephen@xemacs.org>:
> So I think this is really a non-problem.  Just translate bzr commands
> to the git equivalents, configure public repos to prevent "destructive"
> commands (require fast-forward pushes, mostly), and let the git fans
> teach those who want to improve their git skills as issues come up.
> Emacs doesn't have much structure in its abstract workflow (it
> basically comes down to "hack, save, push") that requires careful
> consideration of VCS operations.

I agree with this analysis.  It's why I wrote GitForEmacsDevs as
nearly a literal translation into git commands from bzr commands.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Re: State of the repository conversion

[Óscar Fuentes]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

[snip]

> So I think this is really a non-problem.  Just translate bzr commands
> to the git equivalents, configure public repos to prevent "destructive"
> commands (require fast-forward pushes, mostly), and let the git fans
> teach those who want to improve their git skills as issues come up.
> Emacs doesn't have much structure in its abstract workflow (it
> basically comes down to "hack, save, push") that requires careful
> consideration of VCS operations.

I would encourage git newcomers to just use Magit. It provides a nice UI
for git, requiring very basic knowledge of git's specific features.

Re: State of the repository conversion

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Thu, 20 Mar 2014 02:01:38 +0100
> 
> I would encourage git newcomers to just use Magit. It provides a nice UI
> for git, requiring very basic knowledge of git's specific features.

That's unrelated to the issue at hand, which is how we use git at the
project level.

Re: State of the repository conversion

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

>> I would encourage git newcomers to just use Magit. It provides a nice UI
>> for git, requiring very basic knowledge of git's specific features.
>
> That's unrelated to the issue at hand, which is how we use git at the
> project level.

Magit can determine how you use git at the project level. Since it makes
ordinary tasks quite more simple and less error-prone, git's barroque UI
is no longer a significant factor.

Re: State of the repository conversion

[David Engster]

Óscar Fuentes writes:
> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> I would encourage git newcomers to just use Magit. It provides a nice UI
>>> for git, requiring very basic knowledge of git's specific features.
>>
>> That's unrelated to the issue at hand, which is how we use git at the
>> project level.
>
> Magit can determine how you use git at the project level. Since it makes
> ordinary tasks quite more simple and less error-prone, git's barroque UI
> is no longer a significant factor.

Since Eli is working on Windows, I don't think Magit is a good solution
for him. It is pretty slow there since it is often doing an excessive
number of calls of the Git binary under the hood.

-David

Re: State of the repository conversion

[Steinar Bang]

>>>>> David Engster <deng@randomsample.de>:

> Since Eli is working on Windows, I don't think Magit is a good
> solution for him. It is pretty slow there since it is often doing an
> excessive number of calls of the Git binary under the hood.

Making magit use the git.exe, rather than the git.cmd, made it fast
enough for me on windows, even on a fairly large project.

Re: State of the repository conversion

[Eli Zaretskii]

> From: Steinar Bang <sb@dod.no>
> Date: Thu, 20 Mar 2014 08:49:44 +0100
> 
> >>>>> David Engster <deng@randomsample.de>:
> 
> > Since Eli is working on Windows, I don't think Magit is a good
> > solution for him. It is pretty slow there since it is often doing an
> > excessive number of calls of the Git binary under the hood.
> 
> Making magit use the git.exe, rather than the git.cmd, made it fast
> enough for me on windows, even on a fairly large project.

Please don't waste your breath on these issues.  With or without
magit, I use git in several projects, and see no reason why Emacs
cannot be one more.  And what I do on my machines is hardly the
important part of this discussion, anyway.

Re: State of the repository conversion

[Óscar Fuentes]

David Engster <deng@randomsample.de> writes:

> Óscar Fuentes writes:
>> Eli Zaretskii <eliz@gnu.org> writes:
>>
>>>> I would encourage git newcomers to just use Magit. It provides a nice UI
>>>> for git, requiring very basic knowledge of git's specific features.
>>>
>>> That's unrelated to the issue at hand, which is how we use git at the
>>> project level.
>>
>> Magit can determine how you use git at the project level. Since it makes
>> ordinary tasks quite more simple and less error-prone, git's barroque UI
>> is no longer a significant factor.
>
> Since Eli is working on Windows, I don't think Magit is a good solution
> for him. It is pretty slow there since it is often doing an excessive
> number of calls of the Git binary under the hood.
>
> -David

Magit is slow on Windows only if you compare it to how it performs on
GNU/Linux. I regularly use Magit on a Windows netbook and it is faster
than bzr ever was on that machine. Also, there is room for improvement:
the slowest part is where Magit refreshes the buffers (after switching
to another branch, for example) and that is Emacs's VC being slow, not
Magit. I experimented with advising the VC functions that update the VC
status and the speedup is dramatic (one or two git invocations per Magit
operation instead of one per visited file.)

Re: State of the repository conversion

[Stefan]

> Magit. I experimented with advising the VC functions that update the VC
> status and the speedup is dramatic (one or two git invocations per Magit
> operation instead of one per visited file.)

Do post your hack (as a bug-report requesting a new VC feature), tho
better would be a proper patch.


        Stefan

Re: State of the repository conversion

[Óscar Fuentes]

Stefan <monnier@iro.umontreal.ca> writes:

>> Magit. I experimented with advising the VC functions that update the VC
>> status and the speedup is dramatic (one or two git invocations per Magit
>> operation instead of one per visited file.)
>
> Do post your hack

Below is the patch against an old version of magit.el.

> (as a bug-report requesting a new VC feature), tho better would be a
> proper patch.

This could be generalized to a generic VC feature, but I'm not sure it
is worth the trouble outside of Magit on Windows. Do you know any other
context (VC backend/package/platform) where it would make a difference?

Author: Oscar Fuentes <ofv@wanadoo.es>
Date:   Mon Aug 15 00:15:30 2011 +0200

    Speed-up magit-revert-buffers
    
    Advice some vc-git-* functions and use a cache. 100x faster on Windows
    when there are more than 10 files visited under the working copy.

1 file changed, 65 insertions(+), 14 deletions(-)
 magit.el | 79 ++++++++++++++++++++++++++++++++++++++++++++++++++++------------

	Modified   magit.el
diff --git a/magit.el b/magit.el
index 0e6f2b6..8a938c9 100644
--- a/magit.el
+++ b/magit.el
@@ -2210,23 +2210,74 @@ Please see the manual for a complete description of Magit.
 (defun magit-string-has-prefix-p (string prefix)
   (eq (compare-strings string nil (length prefix) prefix nil nil) t))
 
+(defvar magit-refresh-needing-buffers nil)
+(defvar magit-refresh-pending nil)
+(defvar magit-cached-vc-state nil)
+(defvar magit-vc-state-cache-buffer-name " *magit-vc-state-cache*")
+
 (defun magit-revert-buffers (dir &optional ignore-modtime)
-  (dolist (buffer (buffer-list))
-    (when (and buffer
-	       (buffer-file-name buffer)
-	       (magit-string-has-prefix-p (buffer-file-name buffer) dir))
-      (with-current-buffer buffer
-	(or (ignore-errors
-	     (and (or ignore-modtime (not (verify-visited-file-modtime buffer)))
-		  (not (buffer-modified-p buffer))
-		  (revert-buffer t t nil)))
-	    ;; Update the VC modeline info when revert-buffer failed to execute:
-	    (ignore-errors
-	     (vc-find-file-hook))))))
+  (let ((magit-cached-vc-state
+	 (list (magit-get-current-branch) dir)))
+    (magit-create-vc-state-cache dir)
+    (dolist (buffer (buffer-list))
+      (when (and buffer
+		 (buffer-file-name buffer)
+		 (magit-string-has-prefix-p (buffer-file-name buffer) dir))
+	(with-current-buffer buffer
+	  (or (ignore-errors
+		(and (or ignore-modtime (not (verify-visited-file-modtime buffer)))
+		     (not (buffer-modified-p buffer))
+		     (revert-buffer t t nil)))
+	      ;; Update the VC modeline info when revert-buffer failed to execute:
+	      (ignore-errors
+		(vc-find-file-hook)))))))
+  (kill-buffer magit-vc-state-cache-buffer-name)
   (setq magit-revert-buffers-pending nil))
 
-(defvar magit-refresh-needing-buffers nil)
-(defvar magit-refresh-pending nil)
+(defun magit-create-vc-state-cache (dir)
+  (let ((sbuffer (get-buffer-create magit-vc-state-cache-buffer-name)))
+    (with-current-buffer sbuffer
+      (setq default-directory dir)
+      (buffer-disable-undo)
+      (erase-buffer)
+      (call-process vc-git-executable nil sbuffer nil "ls-files" "-tdcm"))))
+
+(defun magit-lookup-vc-state (file)
+  (when (magit-string-has-prefix-p file (nth 1 magit-cached-vc-state))
+    (setq file (substring file (length (nth 1 magit-cached-vc-state)))))
+  (with-current-buffer magit-vc-state-cache-buffer-name
+    (goto-char (point-max))
+    (if (re-search-backward (concat "^. " file "$") nil t)
+	(let ((line (match-string 0)))
+	  (if (string-match "^\\(.\\) .+$" line)
+	      (magit-vc-git--state-code (match-string 1 line))))
+      'unregistered)))
+
+(defun magit-vc-git--state-code (code)
+  "Convert from a string to a added/deleted/modified state."
+  (case (string-to-char code)
+    (?C 'edited)
+    (?H 'up-to-date)
+    (?R 'removed)
+    (?M 'conflict)
+    (?T 'edited)))
+
+(defadvice vc-git-state (around ad-magit-vc-git-state activate)
+  (if magit-cached-vc-state
+      (setq ad-return-value (magit-lookup-vc-state file))
+    ad-do-it))
+
+(defadvice vc-git-working-revision (around ad-magit-vc-git-working-revision
+					   activate)
+  (if magit-cached-vc-state
+      (setq ad-return-value (nth 0 magit-cached-vc-state))
+    ad-do-it))
+
+(defadvice vc-git-registered (around ad-magit-vc-git-registered activate)
+  (if magit-cached-vc-state
+      (setq ad-return-value
+	    (not (eq 'unregistered (magit-lookup-vc-state file))))
+    ad-do-it))
 
 (defun magit-refresh-wrapper (func)
   (if magit-refresh-pending

Re: State of the repository conversion

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Thu, 20 Mar 2014 05:09:58 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> I would encourage git newcomers to just use Magit. It provides a nice UI
> >> for git, requiring very basic knowledge of git's specific features.
> >
> > That's unrelated to the issue at hand, which is how we use git at the
> > project level.
> 
> Magit can determine how you use git at the project level.

I very much hope that it will be the other way around: first we decide
how we want to work, and only afterwards tweak the tools (if needed)
to match that.

> Since it makes ordinary tasks quite more simple and less
> error-prone, git's barroque UI is no longer a significant factor.

We've been there: I don't think you can entirely ignore the
command-line UI to git.  Too much of the useful functionality isn't
accessible through front ends.

Re: State of the repository conversion

[Steinar Bang]

>>>>> Eli Zaretskii <eliz@gnu.org>:

> We've been there: I don't think you can entirely ignore the
> command-line UI to git.  Too much of the useful functionality isn't
> accessible through front ends.

In particular the merge-and-stop-before-commit, and then manually
picking parts of the tree from one of the branches (a former colleague
referred to this as a "gonzo merge").

The use case is where a subtree comes completely from one branch, while
the rest comes from a different branch (and I've had that particular use
case (driven by other necessitiesm, btw.  Not driven by git. git
actually made that easy)).

Re: State of the repository conversion

[Eli Zaretskii]

> From: "Stephen J. Turnbull" <stephen@xemacs.org>
> Cc: esr@thyrsus.com,
>     emacs-devel@gnu.org
> Date: Thu, 20 Mar 2014 09:08:32 +0900
> 
> About the workflow, Eli Zaretskii writes:
> 
>  > It is better to draw upon the knowhow and experience of veteran git
>  > users among us to identify in advance the preferred workflows,
> 
> And the resulting recommendations will be ignored in favor of the
> convenience of the powers that be, as the recommendations for Bazaar
> were.

I think almost all of the Emacs developers use the workflow described
on the wiki.  I know I do.  So I'm not sure what your assessment is
based on.

> Please remember what that convenience boiled down to: "don't change
> anything that doesn't need to change, because the current workflow
> works fine for us".

The current workflow includes kludges and workarounds for bzr
deficiencies.  E.g., the way we merge from the release branch is due
to the fact that bzr doesn't track cherry-picks.  Git does.  I see no
reason to stick to the old workflow in these cases.

And that is just one example.  I could go on.  But since no one seems
to be interested, I'll probably shut up, before I once again am called
a bullshitter.

> I admit I was wrong to suggest "the workflows that work in other
> projects."  Experience showed that people adopted those workflows as
> needed for their own work, but many people continued to use Bazaar
> as if it were CVS.  It's only a tiny bit harder to use git that way
> (no bound branches or lightweight checkouts, so you do have to learn
> to push, and merge from upstream if your push wouldn't be a
> fast-forward).  But people are used to the up-to-date check failing,
> I suspect, and it's not that hard to learn -- everybody did learn in
> the bad old days of CVS.  So it amounts to learning that you need to
> "git push" after "git commit", if you were using a checkout with
> Bazaar.

I think you completely miss the point here.  These issues are
important (and I see nothing wrong with having them described on the
wiki), but they are not the important part.

Re: State of the repository conversion

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

> E.g., the way we merge from the release branch is due
> to the fact that bzr doesn't track cherry-picks.  Git does.

No, it doesn't.

Which reminds me of a comment I made long time ago: applying fixes on
the maintenance branch and then cherry-picking them into the development
branch lowers the quality of the history of the development branch,
which is the everlasting one, while the maintenance branch will
eventually freeze and become almost irrelevant.

Re: State of the repository conversion

[Stefan]

> Which reminds me of a comment I made long time ago: applying fixes on
> the maintenance branch and then cherry-picking them into the development

Not sure why this matters: we don't cherry pick from the
maintenance branch.  Instead we merge the development branch.  And we
sometimes cherry pick fixes from the development branch into the
maintenance branch (which makes the subsequent merge a bit more
delicate, since we want to avoid applying that change twice).


        Stefan

Re: State of the repository conversion

[Steinar Bang]

>>>>> Stefan <monnier@iro.umontreal.ca>:

> Not sure why this matters: we don't cherry pick from the maintenance
> branch.  Instead we merge the development branch.  And we sometimes
> cherry pick fixes from the development branch into the maintenance
> branch (which makes the subsequent merge a bit more delicate, since we
> want to avoid applying that change twice).

What I do, after cherry picking a fix from the master HEAD back into a
release branch (named "release-branch" in the below commands) is
immediately do a:
 git checkout master
 git merge --no-ff --no-commit release-branch

(The latter command will apply the changes from the merge, and then stop
before committing)

Then I look at the changes in magit, and ideally there should be no
diffs, because I have done the same thing earlier in the lifetime of the
release branch, and have told git to drop irrelevant changes from the
release branch.

The way to tell git to drop the merge results and instead pick the
version from master, is to insert the master branch's version of the
file before committing, eg.:
 git checkout origin/master version.txt

(this picks the master branch's version of the, er..., version file.
"origin/master" means the version of the master branch that has been
pushed to the remote git repository named "origin")
name of the first branch 

After I am satisfied that the merge results contains no unexpected
changes with master, I do a:
 git commit
to complete the merge (or just do a `c' in magit).

Whenever I am doing a bugfix, I _try_ to start a local branch in the
earliest branch where the fix is relevant, and do the fix there.  Then,
instead of cherry picking, I can just merge the fix into the newer
branches that should have the same fix.

And most of the time the merge will be conflict free and correct.

But I always do
 git commit --no-ff --no-commit
in these cases, and look at the diffs resulting from the merge in magit.

Re: State of the repository conversion

[Florian Weimer]

* Steinar Bang:

> What I do, after cherry picking a fix from the master HEAD back into a
> release branch (named "release-branch" in the below commands) is
> immediately do a:
>  git checkout master
>  git merge --no-ff --no-commit release-branch

> After I am satisfied that the merge results contains no unexpected
> changes with master, I do a:
>  git commit
> to complete the merge (or just do a `c' in magit).

This complicates the history of the master branch.  It is not clear if
it adds any useful information, especially if there is no expectation
of ever doing development on release branches and merge this
development back to the master branch later.

> Whenever I am doing a bugfix, I _try_ to start a local branch in the
> earliest branch where the fix is relevant, and do the fix there.  Then,
> instead of cherry picking, I can just merge the fix into the newer
> branches that should have the same fix.

Tool support is generally better if you make the change in the oldest
supported release branch and merge that forward, but in practice, this
doesn't work that well.  Often, you don't know for sure what the
oldest release branch that needs the change is.  And from a general
engineering point of view, it is better to develop the best fix
possible on the master branch, test it, and then start backporting it,
taking shortcuts as necessary.  If you do it in the other order, you
tend to end up with a suboptimal change in the master branch.  And the
development history becomes really, really complicated, too.

Re: State of the repository conversion

[Steinar Bang]

>>>>> Florian Weimer <fw@deneb.enyo.de>:

> This complicates the history of the master branch.  It is not clear if
> it adds any useful information,

It makes later merges more likely to only contain the relevant changes.

And the merge happens at a time where I can remember what was done and
what the changes should be.

That is useful information the way I see it.

> especially if there is no expectation of ever doing development on
> release branches and merge this development back to the master branch
> later.

Like I said, the pattern that works well for me, and gives a nice
history on the file level, and the "git blame" level, is to do the fix
on the earliest release branch were it is relevant, and then merge it
forward. 

But that requires that the master is kept in sync with cherry picks
applied to the release branches.  If it isn't, the merge results can be
strange, and it can be challenging to see what has happened (not
impossible, and as long as nothing is committed or pushed, recovery is
easy).

> Tool support is generally better if you make the change in the oldest
> supported release branch and merge that forward, but in practice, this
> doesn't work that well.  Often, you don't know for sure what the
> oldest release branch that needs the change is.

True.  So one always end up with some cherry picks. 

> And from a general engineering point of view, it is better to develop
> the best fix possible on the master branch, test it, and then start
> backporting it, taking shortcuts as necessary.

Er... no.  It's not possible to generalize like that. For one particular
fix this may have been the right approach.  For another it may not.

Basically, the way important release fixes have come to me, they have
been reported as bugs against the released version, and then I naturally
start with the release branch for that version, and figure out the
problem, and do the fix.

Once that fix is complete and pushed (released as a patch, or as a new
version or whatever, and sent to the customer), I look at if the fix is
relevant in master and later releases.  If the code has changed little
enough that a merge seems to be possible, I merge the older release
branches forward.

In many fixes, this is the case.

If the code has changed so much that a merge isn't possible I try to
figure out if the bug is still present in the master branch, and if it
is, how to best fix it (it may be a completely different fix to what was
done in the release branch).

> If you do it in the other order, you tend to end up with a suboptimal
> change in the master branch.

Only if you merge blindly and assume that the fix will be correct for
all versions.

> And the development history becomes really, really complicated, too.

In what way? Could you be specific?

Re: State of the repository conversion

[Eli Zaretskii]

> From: Stefan <monnier@iro.umontreal.ca>
> Date: Thu, 20 Mar 2014 00:59:57 -0400
> Cc: emacs-devel@gnu.org
> 
> > Which reminds me of a comment I made long time ago: applying fixes on
> > the maintenance branch and then cherry-picking them into the development
> 
> Not sure why this matters: we don't cherry pick from the
> maintenance branch.  Instead we merge the development branch.  And we
> sometimes cherry pick fixes from the development branch into the
> maintenance branch (which makes the subsequent merge a bit more
> delicate, since we want to avoid applying that change twice).

See my other mail: why not install changes on the trunk, and then
cherry-pick some of them to the release branch?

Re: State of the repository conversion

[Stefan]

> See my other mail: why not install changes on the trunk, and then
> cherry-pick some of them to the release branch?

Because I want to be sure that changes applied to the release branch are
included in the trunk.  A "merge" from the release to the trunk is
a good way to be sure.


        Stefan

Re: State of the repository conversion

[David Kastrup]

Stefan <monnier@iro.umontreal.ca> writes:

>> See my other mail: why not install changes on the trunk, and then
>> cherry-pick some of them to the release branch?
>
> Because I want to be sure that changes applied to the release branch are
> included in the trunk.  A "merge" from the release to the trunk is
> a good way to be sure.

You can always do that merge privately and check that it has a zero diff
to the trunk.

-- 
David Kastrup

Re: State of the repository conversion

[Steinar Bang]

>>>>> David Kastrup <dak@gnu.org>:

> Stefan <monnier@iro.umontreal.ca> writes:
>>> See my other mail: why not install changes on the trunk, and then
>>> cherry-pick some of them to the release branch?

>> Because I want to be sure that changes applied to the release branch are
>> included in the trunk.  A "merge" from the release to the trunk is
>> a good way to be sure.

> You can always do that merge privately and check that it has a zero diff
> to the trunk.

If you merge with stop-before-commit and get some unexpected diffs, you
can fix them (usually by doing "git checkout" on the files with diff
from their master versions) and commit, and the diffs won't confuse
future merge results.

(I actually don't know if there is any point to making a commit of a zero
diff merge result...?)

Re: State of the repository conversion

[Eli Zaretskii]

> From: Stefan <monnier@iro.umontreal.ca>
> Cc: ofv@wanadoo.es,  emacs-devel@gnu.org
> Date: Thu, 20 Mar 2014 14:02:01 -0400
> 
> > See my other mail: why not install changes on the trunk, and then
> > cherry-pick some of them to the release branch?
> 
> Because I want to be sure that changes applied to the release branch are
> included in the trunk.

If you apply the changes to the trunk to begin with, this condition is
satisfied by definition, no?

Re: State of the repository conversion

[Stefan]

> If you apply the changes to the trunk to begin with, this condition is
> satisfied by definition, no?

Yup: "If"


        Stefan

Re: State of the repository conversion

[Eli Zaretskii]

> From: Stefan <monnier@iro.umontreal.ca>
> Cc: ofv@wanadoo.es,  emacs-devel@gnu.org
> Date: Thu, 20 Mar 2014 14:44:47 -0400
> 
> > If you apply the changes to the trunk to begin with, this condition is
> > satisfied by definition, no?
> 
> Yup: "If"

The method I was asking about had that "If" condition fulfilled.

Re: State of the repository conversion

[Steinar Bang]

>>>>> Eli Zaretskii <eliz@gnu.org>:

> See my other mail: why not install changes on the trunk, and then
> cherry-pick some of them to the release branch?

Here's why I does it the other way around (ie. fix on branch and then
merge to master): when a bug is reported on a released version, it feels
more natural to do the fix against what was released, and then get some
kind of patch or hotfix on that out the door, and then looking in to
moving that fix to master.

Taking an unreported fix from master and applying it to a relase branch
with a cherry pick also happens, but for that to happen, it has to be a
fix to an OMG!! condition (potential catastrophic data loss, or a high
enough crash rate to be annoying), and those bugs are thankfully few...

Re: State of the repository conversion

[Eli Zaretskii]

> From: Steinar Bang <sb@dod.no>
> Date: Thu, 20 Mar 2014 21:30:47 +0100
> 
> >>>>> Eli Zaretskii <eliz@gnu.org>:
> 
> > See my other mail: why not install changes on the trunk, and then
> > cherry-pick some of them to the release branch?
> 
> Here's why I does it the other way around (ie. fix on branch and then
> merge to master): when a bug is reported on a released version, it feels
> more natural to do the fix against what was released

Why is it more natural?  If the same bug exists on the trunk, you
could fix it in either of the two places first.

Re: State of the repository conversion

[Steinar Bang]

>>>>> Eli Zaretskii <eliz@gnu.org>:

> Why is it more natural?  If the same bug exists on the trunk, you
> could fix it in either of the two places first.

I live in the land of refactoring, and one thing that happens
immediately after a branch for release, is that people tend to go wild
with refactoring that has been delayed pending the release.

So it makes most sense to look for the bug where it has been reported.

As for merging the fix into master: the master may have changed so much
that the fix doesn't make sense anymore.

That said, a merge has a better change at finding the correct place to
apply, in the refactoried code, than a cherry-pick does.  But the merge
result needs to be inspected after the merge.

Re: State of the repository conversion

[Eli Zaretskii]

> From: Steinar Bang <sb@dod.no>
> Date: Thu, 20 Mar 2014 22:10:01 +0100
> 
> >>>>> Eli Zaretskii <eliz@gnu.org>:
> 
> > Why is it more natural?  If the same bug exists on the trunk, you
> > could fix it in either of the two places first.
> 
> I live in the land of refactoring, and one thing that happens
> immediately after a branch for release, is that people tend to go wild
> with refactoring that has been delayed pending the release.
> 
> So it makes most sense to look for the bug where it has been reported.
> 
> As for merging the fix into master: the master may have changed so much
> that the fix doesn't make sense anymore.

In both of these cases, the fix should not be applied to the trunk
anyway.  So this is not the situation I was describing, where a fix is
applicable to both the trunk and the branch.

When a fix is only applicable to the branch, just apply it there, and
don't worry to merge or cherry-pick anywhere else.

> That said, a merge has a better change at finding the correct place to
> apply, in the refactoried code, than a cherry-pick does.  But the merge
> result needs to be inspected after the merge.

What bzrmerge.el does, when we merge a fix from the branch, is
effectively a cherry-pick, but it also records the branch commit as
being present on the trunk.  If the fix is not applicable to the
trunk, we end up with a commit that didn't bring any changes in terms
of diff (because the changes will be edited out before committing
after the merge).  I'm not sure this will work with git, btw.

Re: State of the repository conversion

[Stefan]

> What bzrmerge.el does, when we merge a fix from the branch, is
> effectively a cherry-pick,

No, bzrmerge does not cherry pick in any sense of the word.  It does
a merge.  The only difference with "bzr merge" is that it does it with
more care, using hints from the commit messages to try and auto-resolve
some conflicts.

> I'm not sure this will work with git, btw.

There's nothing special in bzrmerge that should prevent it from working
in another VCS, so long as that VCS has a notion of "branch merge", basically.
E.g. I'm pretty sure we could have written a corresponding cvsmerge.el,
and AFAIK some other project has written something similar for svn.


        Stefan

Re: State of the repository conversion

[Steinar Bang]

>>>>> Stefan <monnier@iro.umontreal.ca>:

> There's nothing special in bzrmerge that should prevent it from
> working in another VCS, so long as that VCS has a notion of "branch
> merge", basically.

I would go so far as to say that the branch merge is THE main feature of
git.

Re: State of the repository conversion

[Eli Zaretskii]

> From: Stefan <monnier@iro.umontreal.ca>
> Cc: Steinar Bang <sb@dod.no>,  emacs-devel@gnu.org
> Date: Fri, 21 Mar 2014 08:40:45 -0400
> 
> > What bzrmerge.el does, when we merge a fix from the branch, is
> > effectively a cherry-pick,
> 
> No, bzrmerge does not cherry pick in any sense of the word.  It does
> a merge.

For crying out loud, Stefan, give me the benefit of the doubt that I
just _might_ know this!

I said "_effectively_ a cherry-pick".

> > I'm not sure this will work with git, btw.
> 
> There's nothing special in bzrmerge that should prevent it from working
> in another VCS, so long as that VCS has a notion of "branch merge", basically.

In some ancient discussion Linus said that a commit that doesn't bring
in any diffs is "madness" and shouldn't exist.  Or maybe I
misunderstood.

Re: State of the repository conversion

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Stefan <monnier@iro.umontreal.ca>
>> Cc: Steinar Bang <sb@dod.no>,  emacs-devel@gnu.org
>> Date: Fri, 21 Mar 2014 08:40:45 -0400
>> 
>> > What bzrmerge.el does, when we merge a fix from the branch, is
>> > effectively a cherry-pick,
>> 
>> No, bzrmerge does not cherry pick in any sense of the word.  It does
>> a merge.
>
> For crying out loud, Stefan, give me the benefit of the doubt that I
> just _might_ know this!
>
> I said "_effectively_ a cherry-pick".
>
>> > I'm not sure this will work with git, btw.
>> 
>> There's nothing special in bzrmerge that should prevent it from working
>> in another VCS, so long as that VCS has a notion of "branch merge", basically.
>
> In some ancient discussion Linus said that a commit that doesn't bring
> in any diffs is "madness" and shouldn't exist.  Or maybe I
> misunderstood.

You probably misunderstood.  A merge commit introduces a change in
topology.  There are even explicit merge strategies "theirs" and "ours"
that lead to a merge commit having a work tree matching one of its
parents.

-- 
David Kastrup

Re: State of the repository conversion

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Date: Fri, 21 Mar 2014 18:02:53 +0100
> 
> > In some ancient discussion Linus said that a commit that doesn't bring
> > in any diffs is "madness" and shouldn't exist.  Or maybe I
> > misunderstood.
> 
> You probably misunderstood.

Quite likely.  But maybe you will be kind enough to explain what Linus
meant in these two messages, when he said that an extra merge-commit
is wrong and has "zero semantic information in it"?

  https://lists.ubuntu.com/archives/bazaar/2006q4/018265.html
  https://lists.ubuntu.com/archives/bazaar/2006q4/018273.html

There's a definite difference between bzr and git that is alluded to
here, but what is it?  (You may wish reading the surrounding messages
for context.)

Thanks.

Re: State of the repository conversion

[David Caldwell]

[-- Attachment #1: Type: text/plain, Size: 1297 bytes --]

On 3/21/14, 11:35 AM, Eli Zaretskii wrote:
>> From: David Kastrup <dak@gnu.org>
>> Date: Fri, 21 Mar 2014 18:02:53 +0100
>>
>>> In some ancient discussion Linus said that a commit that doesn't bring
>>> in any diffs is "madness" and shouldn't exist.  Or maybe I
>>> misunderstood.
>>
>> You probably misunderstood.
> 
> Quite likely.  But maybe you will be kind enough to explain what Linus
> meant in these two messages, when he said that an extra merge-commit
> is wrong and has "zero semantic information in it"?
> 
>   https://lists.ubuntu.com/archives/bazaar/2006q4/018265.html
>   https://lists.ubuntu.com/archives/bazaar/2006q4/018273.html

He's saying that in the context of tracking a remote branch. He's saying
that if I'm pulling from your repo and I haven't made any changes, then
it's pretty rude of me to add random empty merges which only signify
when I happened to run the "pull" command. What I really want it to just
collect your changes and add them to my repo, using fast-forward merges
to move my branch to the latest commits.

In particular, I don't believe he is saying anything about merging
across release/master/feature branches. In fact, I've seen many people
advocate "git merge --no-ff" when merging important branches together.

-David


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4219 bytes --]

Re: State of the repository conversion

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Thu, 20 Mar 2014 05:25:22 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > E.g., the way we merge from the release branch is due
> > to the fact that bzr doesn't track cherry-picks.  Git does.
> 
> No, it doesn't.

Maybe with magit ;-)

From the command line, there's the -x switch to cherry-pick, which
does, sort of.  The question is, do we want that service?

> Which reminds me of a comment I made long time ago: applying fixes on
> the maintenance branch and then cherry-picking them into the development
> branch lowers the quality of the history of the development branch,
> which is the everlasting one, while the maintenance branch will
> eventually freeze and become almost irrelevant.

That's another issue that I think is worth discussing.  E.g., GDB
applies fixes to the trunk, and then cherry-picks some of them to the
release branch.  But then GDB doesn't keep the release branch for very
long, and never makes more than 2 versions out of it, so I'm not sure
that example is good for Emacs.

(I hope now you agree that using or not using magit is not the
important issue here.)

Re: State of the repository conversion

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

>> > E.g., the way we merge from the release branch is due
>> > to the fact that bzr doesn't track cherry-picks.  Git does.
>> 
>> No, it doesn't.
>
> Maybe with magit ;-)
>
> From the command line, there's the -x switch to cherry-pick, which
> does, sort of.

-x just adds a line to the commit message with the revision id of the
source commit. It's a convenience for the user, not something that git
will exploit later.

[snip]

Re: State of the repository conversion

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Thu, 20 Mar 2014 20:28:50 +0100
> 
> -x just adds a line to the commit message with the revision id of the
> source commit. It's a convenience for the user, not something that git
> will exploit later.

Yes, I know.  And we are that user.

Re: State of the repository conversion

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

>> -x just adds a line to the commit message with the revision id of the
>> source commit. It's a convenience for the user, not something that git
>> will exploit later.
>
> Yes, I know.  And we are that user.

Then, why are you not using it with bzr?

Re: State of the repository conversion

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Thu, 20 Mar 2014 21:31:41 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> -x just adds a line to the commit message with the revision id of the
> >> source commit. It's a convenience for the user, not something that git
> >> will exploit later.
> >
> > Yes, I know.  And we are that user.
> 
> Then, why are you not using it with bzr?

Because "bzr merge" doesn't have such an option.

Re: State of the repository conversion

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

>> >> -x just adds a line to the commit message with the revision id of the
>> >> source commit. It's a convenience for the user, not something that git
>> >> will exploit later.
>> >
>> > Yes, I know.  And we are that user.
>> 
>> Then, why are you not using it with bzr?
>
> Because "bzr merge" doesn't have such an option.

Was it requested to bzr developers? It doesn't seem hard to implement.
Actually, it is something you could achieve with a simple wrapper
script.

Git does not track cherry-picks and -x solves nothing unless you develop
some kind of poor-man's tracking yourself.

Re: State of the repository conversion

[Stephen J. Turnbull]

Eli Zaretskii writes:
 > > From: "Stephen J. Turnbull" <stephen@xemacs.org>
 > > Cc: esr@thyrsus.com,
 > >     emacs-devel@gnu.org
 > > Date: Thu, 20 Mar 2014 09:08:32 +0900
 > > 
 > > About the workflow, Eli Zaretskii writes:
 > > 
 > >  > It is better to draw upon the knowhow and experience of veteran git
 > >  > users among us to identify in advance the preferred workflows,
 > > 
 > > And the resulting recommendations will be ignored in favor of the
 > > convenience of the powers that be, as the recommendations for Bazaar
 > > were.
 > 
 > I think almost all of the Emacs developers use the workflow described
 > on the wiki.  I know I do.  So I'm not sure what your assessment is
 > based on.

The reaction of you (!!), Ken'ichi, and Richard to the version of
BzrForEmacsDevs that Karl wrote and I revised (with, IIRC, the
enthusiastic approval of Karl).

I just don't see any point in spending a lot of time trying to design
a great workflow in advance.  If you do, go right ahead and do it
yourself, and I wish you the good fortune that you're better at
channeling the "user to git: just stay out of my way" users than Karl
and I were.

But I see no point in asking Eric Raymond or git fans to get involved
in it.  The only point in that is for them to suggest things that work
well for them that make no sense to most users until they've had a fair
amount of experience.  And most users will go: "not now, please!"  I
suggest that you respect that reaction, and let the appropriate
workflow evolve over time.

Re: State of the repository conversion

[Eli Zaretskii]

> From: "Stephen J. Turnbull" <stephen@xemacs.org>
> Cc: esr@thyrsus.com,
>     emacs-devel@gnu.org
> Date: Fri, 21 Mar 2014 12:02:31 +0900
> 
>  > >  > It is better to draw upon the knowhow and experience of veteran git
>  > >  > users among us to identify in advance the preferred workflows,
>  > > 
>  > > And the resulting recommendations will be ignored in favor of the
>  > > convenience of the powers that be, as the recommendations for Bazaar
>  > > were.
>  > 
>  > I think almost all of the Emacs developers use the workflow described
>  > on the wiki.  I know I do.  So I'm not sure what your assessment is
>  > based on.
> 
> The reaction of you (!!), Ken'ichi, and Richard to the version of
> BzrForEmacsDevs that Karl wrote and I revised (with, IIRC, the
> enthusiastic approval of Karl).

What reaction?  Again, I'm using the workflow described there since
the very beginning.  I think so does Handa-san (judging by his
commits).

Re: State of the repository conversion

[Stefan-W. Hahn]

Mail von Eli Zaretskii, Wed, 19 Mar 2014 at 21:17:52 +0200:

Good morning,

> It is better to draw upon the knowhow and experience of veteran git
> users among us to identify in advance the preferred workflows, discuss
> them, and make decisions blessed by the Powers That Be.  We have ample

I haven't seen any hint to "note from the maintainer" [1] and the "what's
cooking" [2] from the git maintainer J. Hamano. Perhaps this can help in the
discussions. 

(He is maintaining his scripts in the todo branch in the git repository.)

[1] http://marc.info/?l=git&m=136320640722607&w=2
[2] http://marc.info/?l=git&m=139534976411660&w=2

With kind regards,
Stefan

-- 
Stefan-W. Hahn                          It is easy to make things.
                                        It is hard to make things simple.

Re: State of the repository conversion

[Andreas Schwab]

esr@thyrsus.com (Eric S. Raymond) writes:

>        So perhaps we should find a way of having two separate branches,
>        such that switching between them does not require a build if
>        nothing changed.

I'm using git-new-workdir to create separate work directories that share
the git object directory.  (There is work underway to integrate it into
git core, though that seemed to have stalled a bit.)

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: State of the repository conversion

[Eli Zaretskii]

> From: Andreas Schwab <schwab@linux-m68k.org>
> Date: Wed, 19 Mar 2014 22:25:56 +0100
> Cc: emacs-devel@gnu.org
> 
> I'm using git-new-workdir to create separate work directories that share
> the git object directory.  (There is work underway to integrate it into
> git core, though that seemed to have stalled a bit.)

At the time this came up, you said this feature is not ready yet for
prime time.  Is this still the case?  If so, perhaps we should ask the
git maintainers to try to gear up with it.

If this feature is already mature enough, perhaps we should recommend
it on the wiki as the solution for this issue.

Re: State of the repository conversion

[Steinar Bang]

>>>>> Eli Zaretskii <eliz@gnu.org>:

> If this feature is already mature enough, perhaps we should recommend
> it on the wiki as the solution for this issue.

I'd recommend against using new and little tested features in the
recommended workflow.  I saw some painful results from someone
_insisting_ to use git submodule, because it fit his perceived workflow.

Firstly, just switching branches inside the the working directory is
fast and easy, so coding on separate branches can easily be done inside
one working directory.

Secondly, if what you want is to have two working directories side by
side, because you eg. wish to do a long running build or test session in
one working directory, and you don't wish to pollute the upstream
repository with your short lived local branches, you can just create an
extra remote and push and pull from that.

You can create that extra remote locally on the disk of your computer,
or you can create a repository on github and use that.

Re: State of the repository conversion

[Glenn Morris]

Boy, I sure am glad that replacing bzrmerge.el with equivalent
functionality is a precondition for changing VCS.

Re: State of the repository conversion

[Steinar Bang]

>>>>> Glenn Morris <rgm@gnu.org>:

> Boy, I sure am glad that replacing bzrmerge.el with equivalent
> functionality is a precondition for changing VCS.

What's bzrmerge.el?

Re: State of the repository conversion

[Eli Zaretskii]

> From: Steinar Bang <sb@dod.no>
> Date: Thu, 20 Mar 2014 21:17:40 +0100
> 
> >>>>> Glenn Morris <rgm@gnu.org>:
> 
> > Boy, I sure am glad that replacing bzrmerge.el with equivalent
> > functionality is a precondition for changing VCS.
> 
> What's bzrmerge.el?

admin/bzrmerge.el is its full name.

Re: State of the repository conversion

[Eli Zaretskii]

> From: Steinar Bang <sb@dod.no>
> Date: Thu, 20 Mar 2014 08:46:25 +0100
> 
> Firstly, just switching branches inside the the working directory is
> fast and easy, so coding on separate branches can easily be done inside
> one working directory.

Switching is indeed fast, but building after the switch isn't.  See my
original explanation why this is different in Emacs than in some other
projects.  In a nutshell, you will many times need a full bootstrap,
sometimes with the configure step on top of that.

> Secondly, if what you want is to have two working directories side by
> side, because you eg. wish to do a long running build or test session in
> one working directory, and you don't wish to pollute the upstream
> repository with your short lived local branches, you can just create an
> extra remote and push and pull from that.
> 
> You can create that extra remote locally on the disk of your computer,
> or you can create a repository on github and use that.

These are the kinds of workflows I suggested to discuss.  I don't
doubt for a moment that there are no solutions, the question is which
one(s) are best suited to Emacs development and maintenance.

IOW, we need specific suggestions of workflows and procedures, rather
than general references to possible ways of going about this.

Re: State of the repository conversion

[Steinar Bang]

>>>>> Eli Zaretskii <eliz@gnu.org>:

> IOW, we need specific suggestions of workflows and procedures, rather
> than general references to possible ways of going about this.

Ok, here's how to make a local, er... remote

 1. Make an empty local repo
     cd ~/git
     git init --bare emacs-local
 2. Add it as a remote
     cd ~/git/emacs/
     git remote add local ../emacs-local
 3. Make a copy of the working directory:
     cd ~/git
     cp -a emacs emacs-copy

At this point the remote named "local" doesn't contain any branches, and
push and pull from either working directory will go against the same
remote (named "origin" if there is only one, and it has been given the
default name).

Now we'll make a branch in one of the working directories and push it to
the other, using the "local" remote (interesting terminology...).

 4. Make the branch and push it to local:
     cd ~/git/emacs
     git checkout -b sb-fix-annoying-bug
     git push -u local HEAD
 5. Fetch the branch into the other working directory
     cd ~/git/emacs-copy
     git fetch local
     git checkout sb-fix-annoying-bug

After 4 and 5, then sb-fix-annoying-bug will be a tracking branch in
both emacs and emacs-copy, but the upstream will be in "local" rather
than in "origin" (ie. push and pull when standing on that branch will go
to "local" rather than "origin").

Merge from sb-fix-annoying-bug to master will work fine, and can be done
from either working directory.

Downsides to a git repository backing multiple working directories, are:
 - 3 copies of the git repository on disk, rather than just one
 - An extra push needed after the commit in the workflow

Upside:
 - Only uses core git functionality, rather than a new and experimental
   feature 

(Personally I have learned to stay away from experimental git
features. Even if they work as advertised they may not be installed on
all git instances I use, and GUI tool support may be lacking)

Re: State of the repository conversion

[Eli Zaretskii]

> From: Steinar Bang <sb@dod.no>
> Date: Thu, 20 Mar 2014 22:00:03 +0100
> 
> >>>>> Eli Zaretskii <eliz@gnu.org>:
> 
> > IOW, we need specific suggestions of workflows and procedures, rather
> > than general references to possible ways of going about this.
> 
> Ok, here's how to make a local, er... remote

Thanks.  However, I cannot easily map this arrangement to what I had
in mind, namely, having a separate trunk and release branches, each
one having its working tree in a different directory.  What am I
missing?

As for feature branches, I think relatively short-lived ones will be
better off in the same directory as the trunk, because they aren't
expected to diverge too much, so having them co-located actually makes
a lot of sense and saves time.  (Since each build produces also a
numbered emacs-XX.YY.nn executable, it is at least in theory possible
to have several executables available at once.)  By contrast, feature
branches that make deep changes and this diverge a lot from the trunk
are better kept in separate directories, like the release branch.

> Downsides to a git repository backing multiple working directories, are:
>  - 3 copies of the git repository on disk, rather than just one

Doesn't --separate-git-dir solve that problem?

Re: State of the repository conversion

[Andreas Schwab]

Eli Zaretskii <eliz@gnu.org> writes:

>> Downsides to a git repository backing multiple working directories, are:
>>  - 3 copies of the git repository on disk, rather than just one
>
> Doesn't --separate-git-dir solve that problem?

No, the index cannot be shared between different working directories.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: State of the repository conversion

[Steinar Bang]

>>>>> Eli Zaretskii <eliz@gnu.org>:

> Thanks.  However, I cannot easily map this arrangement to what I had
> in mind, namely, having a separate trunk and release branches, each
> one having its working tree in a different directory.  What am I
> missing?

Ok, so you want separate working directories of the official master and
release branches?

I have satisfied this requirement for myself, by simply having separate
clones, something like this:
 cd ~/git
 git clone http://somewhere/emacs.git
 git clone --branch release-23.4 http://somewhere/emacs.git emacs-release-23.4
 git clone --branch release-24.0 http://somewhere/emacs.git emacs-release-24.0

Exchange of information between these branches can take place with push
and pull from the upstream repository.

> As for feature branches, I think relatively short-lived ones will be
> better off in the same directory as the trunk, because they aren't
> expected to diverge too much, so having them co-located actually makes
> a lot of sense and saves time.

By "co-located", do you mean one .git directory serving multiple working
directories?  If so, that was the problem I was trying to solve by using
a local "remote": exchanging branches across separate clones.

> (Since each build produces also a numbered emacs-XX.YY.nn executable,
> it is at least in theory possible to have several executables
> available at once.)  By contrast, feature branches that make deep
> changes and this diverge a lot from the trunk are better kept in
> separate directories, like the release branch.

Ok, separate clones would work here as well.

>> Downsides to a git repository backing multiple working directories, are:
>> - 3 copies of the git repository on disk, rather than just one

> Doesn't --separate-git-dir solve that problem?

I don't know.  I have never used this.

Is this the experimental multiple working directories for one git
directory feature people were talking about earlier in the thread?

Or is this something else? It is in the main git documentation, so
presumably this is a mainline feature...?

Will using this option confuse other git tools?

Re: State of the repository conversion

[Eli Zaretskii]

> From: Steinar Bang <sb@dod.no>
> Date: Fri, 21 Mar 2014 10:53:18 +0100
> 
> >>>>> Eli Zaretskii <eliz@gnu.org>:
> 
> > Thanks.  However, I cannot easily map this arrangement to what I had
> > in mind, namely, having a separate trunk and release branches, each
> > one having its working tree in a different directory.  What am I
> > missing?
> 
> Ok, so you want separate working directories of the official master and
> release branches?

Right.

> I have satisfied this requirement for myself, by simply having separate
> clones, something like this:
>  cd ~/git
>  git clone http://somewhere/emacs.git
>  git clone --branch release-23.4 http://somewhere/emacs.git emacs-release-23.4
>  git clone --branch release-24.0 http://somewhere/emacs.git emacs-release-24.0

Yes, that's the obvious possibility.  I thought there were others.

Thanks.

> > As for feature branches, I think relatively short-lived ones will be
> > better off in the same directory as the trunk, because they aren't
> > expected to diverge too much, so having them co-located actually makes
> > a lot of sense and saves time.
> 
> By "co-located", do you mean one .git directory serving multiple working
> directories?

No, I meant in the same working directory, using "git checkout" to
switch between branches.

> >> Downsides to a git repository backing multiple working directories, are:
> >> - 3 copies of the git repository on disk, rather than just one
> 
> > Doesn't --separate-git-dir solve that problem?
> 
> I don't know.  I have never used this.

Andreas said this is not a solution.

Re: State of the repository conversion

[Steinar Bang]

>>>>> Eli Zaretskii <eliz@gnu.org>:

>> I have satisfied this requirement for myself, by simply having separate
>> clones, something like this:
>> cd ~/git
>> git clone http://somewhere/emacs.git
>> git clone --branch release-23.4 http://somewhere/emacs.git emacs-release-23.4
>> git clone --branch release-24.0 http://somewhere/emacs.git emacs-release-24.0

> Yes, that's the obvious possibility.  I thought there were others.

All I've seen is the experimental feature mentioned earlier in the
thread (which I recommend against, based on my own experience with
earlier experimental features in git).

If space usage on your local disk is an issue, you can limit the clones
to just a single branch, but I don't think there's much to gain, since
the biggest part of the history will be shared anyway.

(And if you need to share branches between these clones that you don't
wish to push to the official emacs repo, that's what my local "remote"
setup earlier was for)

[snip!]
>> > Doesn't --separate-git-dir solve that problem?

>> I don't know.  I have never used this.

> Andreas said this is not a solution.

I was kinda relieved to see that.  I felt my entire understanding of git
slipping away from me...:-)

Re: State of the repository conversion

[Andreas Schwab]

Eli Zaretskii <eliz@gnu.org> writes:

> At the time this came up, you said this feature is not ready yet for
> prime time.

Did I?  I don't think so.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: State of the repository conversion

[Eli Zaretskii]

> From: Andreas Schwab <schwab@linux-m68k.org>
> Cc: esr@thyrsus.com,  emacs-devel@gnu.org
> Date: Thu, 20 Mar 2014 19:19:07 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > At the time this came up, you said this feature is not ready yet for
> > prime time.
> 
> Did I?  I don't think so.

I meant this part from
http://lists.gnu.org/archive/html/emacs-devel/2014-01/msg00864.html:

> There is work in progress to fully integrate git new-workdir into the
> git core, to remove its shortcomings.

Apologies if I misinterpreted what you meant.