Repairing the elpa branch

Repairing the elpa branch

[Glenn Morris]

As an experiment, I created a new branch `bzrtest'.
This is `elpa', but with the problematic r170 done as a flat commit
rather than a merge.

In fact, all merges post r170 (r279, 263, 247, 246, 239, 231, 180, 171)
are flat commits rather than merges. If capturing those merges is
considered important (it probably is), and if this experiment is
considered worth pursuing, I can try to do that.

Commits have the same authors, timestamps, logs, tags, and fixes
information.

Re: Repairing the elpa branch

[Stefan Monnier]

> As an experiment, I created a new branch `bzrtest'.
> This is `elpa', but with the problematic r170 done as a flat commit
> rather than a merge.
> In fact, all merges post r170 (r279, 263, 247, 246, 239, 231, 180, 171)
> are flat commits rather than merges.  If capturing those merges is
> considered important (it probably is), and if this experiment is
> considered worth pursuing, I can try to do that.

I think capturing those merges is important.
And the r170 merge is not special, IIUC.  It was just the first such
merges, so if you replace it with a flat commit, the next merge will
turn into the source of the problem.

> Commits have the same authors, timestamps, logs, tags, and fixes
> information.

Yes, but subsequent syncs with upstream are made more difficult if you
use flat commits.


        Stefan

Re: Repairing the elpa branch

[Chong Yidong]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> I think capturing those merges is important.
>
> subsequent syncs with upstream are made more difficult if you use flat
> commits.

I think it is a matter of picking the lesser of two evils.  Not being
able to check out the branch at all (without a non-obvious workaround)
seems to be much worse problem, compared with a bit of difficulty
syncing with upstream.  Since no one else has come up with a better
solution, maybe we should go with Glenn's.

Re: Repairing the elpa branch

[Stefan Monnier]

>> I think capturing those merges is important.
>> subsequent syncs with upstream are made more difficult if you use flat
>> commits.
> I think it is a matter of picking the lesser of two evils.  Not being
> able to check out the branch at all (without a non-obvious workaround)
> seems to be much worse problem, compared with a bit of difficulty
> syncing with upstream.  Since no one else has come up with a better
> solution, maybe we should go with Glenn's.

Another direction (which I think is preferable/needed in the longer
term) is to better support "one branch per ELPA package".


        Stefan

Re: Repairing the elpa branch

[Eli Zaretskii]

> From: Chong Yidong <cyd@gnu.org>
> Date: Mon, 14 Jan 2013 12:26:14 +0800
> Cc: emacs-devel@gnu.org
> 
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
> 
> > I think capturing those merges is important.
> >
> > subsequent syncs with upstream are made more difficult if you use flat
> > commits.
> 
> I think it is a matter of picking the lesser of two evils.  Not being
> able to check out the branch at all (without a non-obvious workaround)
> seems to be much worse problem, compared with a bit of difficulty
> syncing with upstream.  Since no one else has come up with a better
> solution, maybe we should go with Glenn's.

What about the alternative of using the fast-import plugin?  The
following works for me with bzr 2.5.1 and 2.6b1:

  bzr fast-export elpa elpa.fi
  bzr init elpa2
  bzr fast-import elpa.fi elpa2

After this, the new elpa2 branch is healthy: I can clone it without
triggering the crash.  It can be pushed to Savannah, I think.

(For some reason, the above doesn't work if the original branch 'elpa'
is inside of a shared repo, so if you have that, first make a
stand-alone branch using these commands:

  bzr co -r170 elpa DESTINATION
  cd DESTINATION
  bzr up

and make sure DESTINATION is outside of a shared repo.  Then run
fast-export on the stand-alone branch.)

The downside of this is, of course, that all the revision-ids will
change (the hash part of the revid's displayed by bzr will be
different), so users will have to abandon their current elpa checkouts
and clone a new branch, then copy their local commits to the new
branch.  But this will be the problem with "flat commits" as well, no?

Re: Repairing the elpa branch

[Glenn Morris]

Eli Zaretskii wrote:

> The downside of this is, of course, that all the revision-ids will
> change (the hash part of the revid's displayed by bzr will be
> different), so users will have to abandon their current elpa checkouts
> and clone a new branch, then copy their local commits to the new
> branch.

This will be an inevitable side-effect of whatever we do (well, all the
revids after r170's anyway), if we do anything.

Re: Repairing the elpa branch

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Cc: Chong Yidong <cyd@gnu.org>,  monnier@iro.umontreal.ca,  emacs-devel@gnu.org
> Date: Mon, 14 Jan 2013 12:10:27 -0500
> 
> Eli Zaretskii wrote:
> 
> > The downside of this is, of course, that all the revision-ids will
> > change (the hash part of the revid's displayed by bzr will be
> > different), so users will have to abandon their current elpa checkouts
> > and clone a new branch, then copy their local commits to the new
> > branch.
> 
> This will be an inevitable side-effect of whatever we do (well, all the
> revids after r170's anyway), if we do anything.

Then I suggest that you use the fast-import method, as it will take
much less time and effort (20 sec overall, on my machine).  I also
think it's much less error-prone, and it retains the merges that
Stefan wanted to preserve.

Re: Repairing the elpa branch

[Glenn Morris]

Eli Zaretskii wrote:

> Then I suggest that you use the fast-import method, as it will take
> much less time and effort (20 sec overall, on my machine).

Well, I scripted my method, so the time&effort is already spent. :)
(I also think it's not error-prone, since I diffed the final working
trees and the bzr logs and saw only expected differences.)

> I also think it's much less error-prone, and it retains the merges
> that Stefan wanted to preserve.

I know nothing about what fast-import does, but if you think it's fine,
fine by me. Does it preserve all the log info?

Re: Repairing the elpa branch

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Cc: cyd@gnu.org,  monnier@iro.umontreal.ca,  emacs-devel@gnu.org
> Date: Mon, 14 Jan 2013 12:28:51 -0500
> 
> Eli Zaretskii wrote:
> 
> > Then I suggest that you use the fast-import method, as it will take
> > much less time and effort (20 sec overall, on my machine).
> 
> Well, I scripted my method, so the time&effort is already spent. :)
> (I also think it's not error-prone, since I diffed the final working
> trees and the bzr logs and saw only expected differences.)

OK, then feel free to do what you think is better.

> > I also think it's much less error-prone, and it retains the merges
> > that Stefan wanted to preserve.
> 
> I know nothing about what fast-import does

bzr-export exports all the history in a portable format, so that you
could import foreign repositories (like from git to bzr or vice
versa).  Then bzr-import creates a new branch from that info.

> Does it preserve all the log info?

Yes.  In fact, I just diff'ed "bzr log --long" results between the
original and the new branch, and the difference I see are because the
"branch nick" changed.  There are only 4 non-trivial changes in the
log messages, where fast-import loses some information, see below:

 revno: 324
-fixes bug: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=13221
 committer: Stefan Monnier <monnier@iro.umontreal.ca>
-branch nick: elpa
+branch nick: trunk
 timestamp: Wed 2012-12-19 14:14:47 +0000
 message:
   * memory-usage: Bump version, belatedly.
@@ -432,9 +431,8 @@
   * Debbugs.wsdl (search_estRequest): Remove superfluous arguments.
 ------------------------------------------------------------
 revno: 286
-fixes bug: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=12749
 committer: Stefan Monnier <monnier@iro.umontreal.ca>
-branch nick: elpa
+branch nick: trunk
 timestamp: Sun 2012-10-28 19:13:01 +0000
 message:
   * num3-mode.el: Fix footer.  Reported by Jonas Bernoulli <jonas@bernoul.li>.
@@ -2259,9 +2411,9 @@
   Add bzg@gnu.org as the maintainer of register-alist.el and windresize.el
 ------------------------------------------------------------
 revno: 256
-author: Michal Nazarewicz <mina86@mina86.com>, Felix Lee <felix8a@gmail.com>
+author: Michal Nazarewicz <mina86@mina86.com>
 committer: Stefan Monnier <monnier@iro.umontreal.ca>
-branch nick: elpa
+branch nick: trunk
 timestamp: Wed 2012-09-26 19:54:55 +0000
 message:
   Add num3-mode
@@ -4454,9 +4868,9 @@
   Allow "nomail" to mean no mail is to be forwarded.
 ------------------------------------------------------------
 revno: 214
-author: Jonathan Kotta <jpkotta@gmail.com>  (tiny change)
+author: Jonathan Kotta <jpkotta@gmail.com>
 committer: Stefan Monnier <monnier@iro.umontreal.ca>
-branch nick: elpa
+branch nick: trunk
 timestamp: Mon 2012-05-21 04:46:36 +0000
 message:
   Add adaptive-wrap-extra-indent.

Re: Repairing the elpa branch

[Andreas Schwab]

Eli Zaretskii <eliz@gnu.org> writes:

> Yes.  In fact, I just diff'ed "bzr log --long" results between the
> original and the new branch, and the difference I see are because the
> "branch nick" changed.  There are only 4 non-trivial changes in the
> log messages, where fast-import loses some information, see below:

You should get those differences resolved if you use `bzr fast-export
--no-plain'.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Repairing the elpa branch

[Eli Zaretskii]

> From: Andreas Schwab <schwab@linux-m68k.org>
> Date: Mon, 14 Jan 2013 19:13:45 +0100
> Cc: cyd@gnu.org, monnier@iro.umontreal.ca, emacs-devel@gnu.org
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Yes.  In fact, I just diff'ed "bzr log --long" results between the
> > original and the new branch, and the difference I see are because the
> > "branch nick" changed.  There are only 4 non-trivial changes in the
> > log messages, where fast-import loses some information, see below:
> 
> You should get those differences resolved if you use `bzr fast-export
> --no-plain'.

Thanks.  Unfortunately, "bzr fast-import" on the resulting export hits
https://bugs.launchpad.net/bzr-fastimport/+bug/730876 and aborts:

  21:12:04 Starting import of 1092 commits ...
  ABORT: exception occurred processing commit :619
  bzr: ERROR: An inconsistent delta was supplied involving u'examples', 'examples-20130114191204-8aot101a793x4ou0-476'
  reason: repeated path

Re: Repairing the elpa branch

[Stefan Monnier]

>> > I think capturing those merges is important.
>> > subsequent syncs with upstream are made more difficult if you use flat
>> > commits.
>> I think it is a matter of picking the lesser of two evils.  Not being
>> able to check out the branch at all (without a non-obvious workaround)
>> seems to be much worse problem, compared with a bit of difficulty
>> syncing with upstream.  Since no one else has come up with a better
>> solution, maybe we should go with Glenn's.
> What about the alternative of using the fast-import plugin?  The
> following works for me with bzr 2.5.1 and 2.6b1:

This loses the merge info.


        Stefan

Re: Repairing the elpa branch

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Chong Yidong <cyd@gnu.org>,  Glenn Morris <rgm@gnu.org>,  emacs-devel@gnu.org
> Date: Mon, 14 Jan 2013 20:38:38 -0500
> 
> >> > I think capturing those merges is important.
> >> > subsequent syncs with upstream are made more difficult if you use flat
> >> > commits.
> >> I think it is a matter of picking the lesser of two evils.  Not being
> >> able to check out the branch at all (without a non-obvious workaround)
> >> seems to be much worse problem, compared with a bit of difficulty
> >> syncing with upstream.  Since no one else has come up with a better
> >> solution, maybe we should go with Glenn's.
> > What about the alternative of using the fast-import plugin?  The
> > following works for me with bzr 2.5.1 and 2.6b1:
> 
> This loses the merge info.

How do you mean "loses"?  After doing this, I still see all the merges
seen in the original branch, and as I wrote, "bzr log --include-merged"
looks the same, with 4 exceptions (that have nothing to do with
merges).

Re: Repairing the elpa branch

[Stefan Monnier]

> How do you mean "loses"?  After doing this, I still see all the merges
> seen in the original branch, and as I wrote, "bzr log --include-merged"
> looks the same, with 4 exceptions (that have nothing to do with
> merges).

Yes, you see merges, but they are from rewritten branches (with
different revids), so they're useless when you try to sync up with the
upstream branch because there's no common ancestor any more.


        Stefan

Re: Repairing the elpa branch

[Andreas Schwab]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Yes, you see merges, but they are from rewritten branches (with
> different revids), so they're useless when you try to sync up with the
> upstream branch because there's no common ancestor any more.

This will happen with _any_ rewriting.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Repairing the elpa branch

[Stefan Monnier]

>> Yes, you see merges, but they are from rewritten branches (with
>> different revids), so they're useless when you try to sync up with the
>> upstream branch because there's no common ancestor any more.
> This will happen with _any_ rewriting.

No: not if the rewrite only rewrites the mainline and not the branches.


        Stefan

Re: Repairing the elpa branch

[Stephen J. Turnbull]

Stefan Monnier writes:

 > >> Yes, you see merges, but they are from rewritten branches (with
 > >> different revids), so they're useless when you try to sync up with the
 > >> upstream branch because there's no common ancestor any more.
 > > This will happen with _any_ rewriting.
 > 
 > No: not if the rewrite only rewrites the mainline and not the branches.

AFAICS, for "recent" branches (including those that have been recently
sync'ed to mainline by merging from mainline), either they get
rewritten, or they get chopped off the mainline.

Reminds me of C. P. Snow's Three Laws of Thermodynamics.

Re: Repairing the elpa branch

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Eli Zaretskii <eliz@gnu.org>,  cyd@gnu.org,  emacs-devel@gnu.org
> Date: Tue, 15 Jan 2013 08:39:52 -0500
> 
> >> Yes, you see merges, but they are from rewritten branches (with
> >> different revids), so they're useless when you try to sync up with the
> >> upstream branch because there's no common ancestor any more.
> > This will happen with _any_ rewriting.
> 
> No: not if the rewrite only rewrites the mainline and not the branches.

But the common ancestors are (usually) on the mainline.  So you'd need
to keep the mainline rev-ids unaltered as well, no?

Re: Repairing the elpa branch

[Glenn Morris]

Eli Zaretskii wrote:

> But the common ancestors are (usually) on the mainline.  So you'd need
> to keep the mainline rev-ids unaltered as well, no?

Who the heck knows how merge works going between a standalone repo and a
repo into which it has been `bzr join'ed? Does it just compare
revision-ids and say: you seem to have x and y, but not z, so I'll try
merging z?

Re: Repairing the elpa branch

[Glenn Morris]

I think there's no way to preserve the info that Stefan wants without
using bzr join, and using bzr join causes precisely the problem we are
trying to avoid, so it's insoluble. (Unless someone invents a better bzr
join?) I had hoped that running bzr split on the current elpa branch
would give something that could be fed back to bzr join, and maybe would
not have the same issue, but bzr split does not DTRT.

Re: Repairing the elpa branch

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Cc: cyd@gnu.org,  schwab@linux-m68k.org,  Stefan Monnier <monnier@iro.umontreal.ca>,  emacs-devel@gnu.org
> Date: Tue, 15 Jan 2013 13:45:48 -0500
> 
> 
> I think there's no way to preserve the info that Stefan wants without
> using bzr join, and using bzr join causes precisely the problem we are
> trying to avoid, so it's insoluble. (Unless someone invents a better bzr
> join?) I had hoped that running bzr split on the current elpa branch
> would give something that could be fed back to bzr join, and maybe would
> not have the same issue, but bzr split does not DTRT.

Maybe we should ask on the Bazaar mailing list.  Perhaps someone there
will have an idea.  E.g., perhaps some Python could be written to
force "bzr fast-import" produce precisely the same rev-ids as in the
original branch.  Or something.

Re: Repairing the elpa branch

[Glenn Morris]

Eli Zaretskii wrote:

> Maybe we should ask on the Bazaar mailing list.  Perhaps someone there
> will have an idea.  E.g., perhaps some Python could be written to
> force "bzr fast-import" produce precisely the same rev-ids as in the
> original branch.  Or something.

I'm not optimistic. But maybe the --export-marks / --import-marks
options are related to this issue?

Re: Repairing the elpa branch

[Stefan Monnier]

> I think there's no way to preserve the info that Stefan wants without
> using bzr join, and using bzr join causes precisely the problem we are
> trying to avoid, so it's insoluble.

Right.  Note also that the `elpa' branch generated by those "bzr joins"
is actually perfectly fine, it just happens to trigger a bug later on
in bzr.


        Stefan

Re: Repairing the elpa branch

[Glenn Morris]

Stefan Monnier wrote:

> I think capturing those merges is important.

I'll try it then.

> And the r170 merge is not special, IIUC. 

Oh, I thought it was because it came from bzr-git, or bzr join, or
something?

> It was just the first such merges, so if you replace it with a flat
> commit, the next merge will turn into the source of the problem.

Then it seems pointless for me to try and copy the merges?

But r10, 28, 33, 76 were also merges...

Re: Repairing the elpa branch

[Stefan Monnier]

>> And the r170 merge is not special, IIUC. 
> Oh, I thought it was because it came from bzr-git, or bzr join, or
> something?

Yes (it's actually a bzr join of a bzr-git branch), but most of the
other merges do as well.

>> It was just the first such merges, so if you replace it with a flat
>> commit, the next merge will turn into the source of the problem.
> Then it seems pointless for me to try and copy the merges?
> But r10, 28, 33, 76 were also merges...

IIUC these are "normal merges" of Bzr branches of elpa (and AFAIK these
branches were very short lived and we don't need to keep track of that
merge metadata for future merges).


        Stefan

Re: Repairing the elpa branch

[Glenn Morris]

Glenn Morris wrote:

> Stefan Monnier wrote:
>
>> I think capturing those merges is important.
>
> I'll try it then.

AFAICS, since bzr merge does not track cherrypicks, the only way for me
to do what you ask is to use bzr join as you did. And since bzr split is
not the inverse of bzr join, there is no way for me to do so unless I
start from the same initial separate repos that you did. In which case I
would presumably run into exactly the same issue.