git mirror?

git mirror?

[Miles Bader]

What's the status/prognosis of the Emacs git mirror, now that the main
archive uses bzr?

Thanks,

-Miles

-- 
Cat, n. A soft, indestructible automaton provided by nature to be kicked when
things go wrong in the domestic circle.

Re: git mirror?

[Andreas Schwab]

Miles Bader <miles@gnu.org> writes:

> What's the status/prognosis of the Emacs git mirror, now that the main
> archive uses bzr?

I'm currently experimenting with fast-export/fast-import.  If that turns
out to be workable something based on it could be installed on savannah.
Chances are that the resulting git tree is identical to the one on
repo.or.cz.
 
Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: git mirror?

[Stefan Monnier]

>> What's the status/prognosis of the Emacs git mirror, now that the main
>> archive uses bzr?
> I'm currently experimenting with fast-export/fast-import.

Thank you.

> If that turns
> out to be workable something based on it could be installed on savannah.

Yes, that would good.

> Chances are that the resulting git tree is identical to the one on
> repo.or.cz.
 
That would be swell,


        Stefan

Re: git mirror?

[Andreas Schwab]

Anyone who wants to set up a git mirror can use something like this,
suitably adapted:

test last-run -nt last-run-ready || touch last-run
rsync -av --del bzr.sv.gnu.org:/srv/bzr/emacs/ emacs.bzr/
cd emacs.bzr
for b in *; do
  test $b/.bzr/branch/last-revision -ot ../last-run && continue
  echo $b
  bzr fast-export --marks=../bzr-marks $(test $b != trunk && echo -b $b) $b |
  (cd ../emacs.git; git fast-import --export-marks=../git-marks --import-marks=../git-marks)
done
cd ../emacs.git
git push --mirror ssh://repo.or.cz/srv/git/emacs.git
touch ../last-run-ready

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: git mirror?

[Andreas Schwab]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

>> Chances are that the resulting git tree is identical to the one on
>> repo.or.cz.
>  
> That would be swell,

That turned out to be not the case, because some commit messages
contained non-UTF-8 characters, which were mangled during bzr import.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: git mirror?

[Miles Bader]

Andreas Schwab <schwab@linux-m68k.org> writes:
>>> Chances are that the resulting git tree is identical to the one on
>>> repo.or.cz.
>>  
>> That would be swell,
>
> That turned out to be not the case, because some commit messages
> contained non-UTF-8 characters, which were mangled during bzr import.

I also notice an issue with RCS keywords being expanded differently (in
the CVS checkout I guess).

[I delete those from the repository as soon as I notice them, but people
keep adding new ones... and they're part of history forever I guess.   argh...]

-Miles

-- 
It wasn't the Exxon Valdez captain's driving that caused the Alaskan oil spill.
It was yours.  [Greenpeace advertisement, New York Times, 25 February 1990]

Re: git mirror?

[Andreas Schwab]

Miles Bader <miles@gnu.org> writes:

> I also notice an issue with RCS keywords being expanded differently (in
> the CVS checkout I guess).

I didn't expand them at all during the conversion to bzr.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: git mirror?

[Miles Bader]

Andreas Schwab <schwab@linux-m68k.org> writes:
>> I also notice an issue with RCS keywords being expanded differently (in
>> the CVS checkout I guess).
>
> I didn't expand them at all during the conversion to bzr.

Well anyway, they seem to be expanded in the two git mirrors, differently...

-Miles

-- 
The automobile has not merely taken over the street, it has dissolved the
living tissue of the city.  Its appetite for space is absolutely insatiable;
moving and parked, it devours urban land, leaving the buildings as mere
islands of habitable space in a sea of dangerous and ugly traffic.
[James Marston Fitch, New York Times, 1 May 1960]

Git mirror - bidrectional

[David Reitter]

[-- Attachment #1: Type: text/plain, Size: 935 bytes --]

Does anyone plan to set up facility to make the git mirror writeable and merge changes into the Bzr branch?
(Changes to ChangeLog files might lead to merge conflicts, but if that is solved, a bidirectional Git bridge would really be quite useful.)


On Dec 30, 2009, at 8:10 AM, Andreas Schwab wrote:

> Anyone who wants to set up a git mirror can use something like this,
> suitably adapted:
> 
> test last-run -nt last-run-ready || touch last-run
> rsync -av --del bzr.sv.gnu.org:/srv/bzr/emacs/ emacs.bzr/
> cd emacs.bzr
> for b in *; do
>  test $b/.bzr/branch/last-revision -ot ../last-run && continue
>  echo $b
>  bzr fast-export --marks=../bzr-marks $(test $b != trunk && echo -b $b) $b |
>  (cd ../emacs.git; git fast-import --export-marks=../git-marks --import-marks=../git-marks)
> done
> cd ../emacs.git
> git push --mirror ssh://repo.or.cz/srv/git/emacs.git
> touch ../last-run-ready
> 
> Andreas.


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 203 bytes --]

Re: Git mirror - bidrectional

[Ken Raeburn]

On Jan 12, 2010, at 14:27, David Reitter wrote:
> Does anyone plan to set up facility to make the git mirror writeable  
> and merge changes into the Bzr branch?
> (Changes to ChangeLog files might lead to merge conflicts, but if  
> that is solved, a bidirectional Git bridge would really be quite  
> useful.)

Does git have sufficient hooks to push back submitted changes to bzr  
in real time, and detect conflicts at that point, and report an error  
to the client without updating the git mirror (aside from maybe some  
new commits unreferenced by branch labels)?

Ken

Re: Git mirror - bidrectional

[David Reitter]

[-- Attachment #1: Type: text/plain, Size: 1513 bytes --]

On Jan 13, 2010, at 10:12 AM, Ken Raeburn wrote:

> On Jan 12, 2010, at 14:27, David Reitter wrote:
>> Does anyone plan to set up facility to make the git mirror writeable and merge changes into the Bzr branch?
>> (Changes to ChangeLog files might lead to merge conflicts, but if that is solved, a bidirectional Git bridge would really be quite useful.)
> 
> Does git have sufficient hooks to push back submitted changes to bzr in real time, and detect conflicts at that point, and report an error to the client without updating the git mirror (aside from maybe some new commits unreferenced by branch labels)?

Looking at the documentation, I see that the "update" hook is designed to do just that.  When a remote repository does a push, the new revisions are sent to the repository, but before the ref (i.e. branch pointer) is updated, this hook is invoked.  It can prevent the update to the ref.

If the hook fails this way, one could call the sync script to import the latest changes from the bzr side so that the client can then pull/merge those before attempting to push again.

What happens when the changes that originated on the git side are re-imported to the git repository?  Would we get double revisions? 

Also, would importing sidestep the new revisions on the git side because the "marks" file points to an earlier previous revision?  This would mean that users would have to constantly "pull -f" and rebase, right?


http://kernel.org/pub/software/scm/git-core/docs/githooks.html

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 203 bytes --]

Re: Git mirror - bidrectional

[Stephen J. Turnbull]

Ken Raeburn writes:

 > Does git have sufficient hooks to push back submitted changes to bzr  
 > in real time,

Probably.  There have been some more or less feeble attempts to create
bound branches in git with the post-commit hook.  That might be
usable.  There is also a post-update hook for push (see "git help
receive-pack").

 > and detect conflicts at that point,

Any conflicts in pushing to the bzr mainline will be detected by bzr,
before anything gets pushed to bzr.  But to get to that point, the
merge needs to have succeeded in the git mirror.

 > and report an error to the client without updating the git mirror
 > (aside from maybe some new commits unreferenced by branch labels)?

I think that would you would need is two refs per branch, one public
and one for the use of the mirror only.  Maybe even three (a public
read-only branch updated by the bzr->git process, a public write-only
branch whose only reader should be the git-bzr branch, and a private
ref used for locking as below).  (Note that something like this
three-branch method is how git remotes are currently implemented.  But
at this point I'm to tired to work out whether it would be an
improvement in any of the failure states.)

To update bzr, the git/bzr repo's post-update hook will

1.  Use "git branch -m public private" to lock the branch.  (Yes, we
    want readers locked out too, or you have a rebase fiasco when the
    bzr push fails.)  Before doing this, it should be verified that
    branch -m is atomic, but I can't imagine why the operation would
    exist if somebody didn't need it to be atomic.

2.  Tag the head.

3.  Push your new commits from local workspace to private.

4.  Push private to the bzr mainline with bzr.

5.  Delete the tag.

6.  Use "git branch -m private public" to unlock the branch.

If 4 fails, you recreate the public branch at the tag, delete the tag
and the private branch ref, and inform the user she lost.  Now you
have to resync the git mirror with bzr.

If 3 fails, you need to update local and try again.  First clean up by
recreating the public branch ref at the tag, delete the tag and
private branch ref.

If 1 fails, somebody else is updating, wait your turn.

2, 5, and 6 shouldn't fail unless git is buggy.

I think that should work, except that if 4 fails, resyncing to bzr
looks like it will need manual intervention.  It might not be that bad
if everybody who wants to use one of these implements their own, but
running one as a public service would probably be hairy.

Re: Git mirror - bidrectional

[Stephen J. Turnbull]

David Reitter writes:

 > Looking at the documentation, I see that the "update" hook is
 > designed to do just that.  When a remote repository does a push,
 > the new revisions are sent to the repository, but before the ref
 > (i.e. branch pointer) is updated, this hook is invoked.  It can
 > prevent the update to the ref.

Ah, this is better than what I just suggested.  The doc for
receive-pack isn't as clear as that for hooks.

The main thing that worries me in any such process is that bzr can
take a relatively long time to do a push.

 > What happens when the changes that originated on the git side are
 > re-imported to the git repository?  Would we get double revisions?

I think not.  This is the point of the fastimport format.  Although
it's possible that some metadata present in git objects is not
supplied by bzr.  In that case you would.

 > Also, would importing sidestep the new revisions on the git side
 > because the "marks" file points to an earlier previous revision?

I don't understand precisely what you mean by "importing", "sidestep",
or "new revisions on the git side".

Re: Git mirror - bidrectional

[David Reitter]

[-- Attachment #1: Type: text/plain, Size: 1641 bytes --]

On Jan 13, 2010, at 12:14 PM, Stephen J. Turnbull wrote:
> The main thing that worries me in any such process is that bzr can
> take a relatively long time to do a push.

Indeed, anything more than a few seconds would seem unusual if not unacceptable to the typical busy Git user.

The alternative would be to do it asynchronously and undo the push if it fails (a "git reset" is trivial).  The user
could then only be notified by e-mail.

Perhaps the first option can be tried to see what the delay is.


>> What happens when the changes that originated on the git side are
>> re-imported to the git repository?  Would we get double revisions?
> 
> I think not.  This is the point of the fastimport format.  Although
> it's possible that some metadata present in git objects is not
> supplied by bzr.  In that case you would.
> 
>> Also, would importing sidestep the new revisions on the git side
>> because the "marks" file points to an earlier previous revision?
> 
> I don't understand precisely what you mean by "importing", "sidestep",
> or "new revisions on the git side".


OK, if the fastimport mechanism aligns revisions in both directions then that's not going to be a problem:

from the fast-export man page:

" Any commits that have already been marked will not be exported again. If the backend uses a similar --import-marks
           file, this allows for incremental bidirectional exporting of the repository by keeping the marks the same across runs."

As a next step, could somebody with access to Bzr and the "marks" used for the `cz' mirror try out the export and the commit hook?




[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 203 bytes --]

Re: Git mirror - bidrectional

[Dan Nicolaescu]

David Reitter <david.reitter@gmail.com> writes:

  > Does anyone plan to set up facility to make the git mirror writeable
  > and merge changes into the Bzr branch?  (Changes to ChangeLog files
  > might lead to merge conflicts, but if that is solved, a
  > bidirectional Git bridge would really be quite useful.)

IMHO this would be a terrible idea.  Am I the only one that thinks that?
We already have a never ending thread about using Bzr, we don't need
another one about using git and another one about the git-bzr gateway.

We have an official VCS, that what everyone should be using.
Especially people that have write access.
If people want to use something else in private, fine, but please keep
it private.

We are already getting bug reports that mention git version numbers,
that generates more work than needed to figure out exactly what version
that is.

If bzr is not satisfactory for some reason, then please open a
discussion with arguments about replacing it, not just add yet more
noise about a different way of checking things in.

[And yes, I do like git better than bzr, but that's not the point here]

  > On Dec 30, 2009, at 8:10 AM, Andreas Schwab wrote:
  > 
  > > Anyone who wants to set up a git mirror can use something like this,
  > > suitably adapted:
  > > 
  > > test last-run -nt last-run-ready || touch last-run
  > > rsync -av --del bzr.sv.gnu.org:/srv/bzr/emacs/ emacs.bzr/
  > > cd emacs.bzr
  > > for b in *; do
  > >  test $b/.bzr/branch/last-revision -ot ../last-run && continue
  > >  echo $b
  > >  bzr fast-export --marks=../bzr-marks $(test $b != trunk && echo -b $b) $b |
  > >  (cd ../emacs.git; git fast-import --export-marks=../git-marks --import-marks=../git-marks)
  > > done
  > > cd ../emacs.git
  > > git push --mirror ssh://repo.or.cz/srv/git/emacs.git
  > > touch ../last-run-ready
  > > 
  > > Andreas.

Re: Git mirror - bidrectional

[David Reitter]

[-- Attachment #1: Type: text/plain, Size: 956 bytes --]

On Jan 13, 2010, at 2:28 PM, Dan Nicolaescu wrote:

> If bzr is not satisfactory for some reason, then please open a
> discussion with arguments about replacing it, not just add yet more
> noise about a different way of checking things in.


Chong, Stefan, do you want this discussion here?
I wasn't under the impression that this a desired discussion topic at this time. 

I don't think we need another mailing list to discuss "emacs development on the git mirror".

Your point is well taken about the git rev-ids.  Maybe we can set up the mirror  so that the Bzr revisions are mentioned in the commit messages.
I agree that a common VCS would be preferable, but one wouldn't want to risk alienating or losing contributions over it, right?  So, it would be mostly beneficial for Emacs development if more people can make commits or produce patches using their preferred tools.   An open solution, rather than a proprietary one, so to speak.  

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 203 bytes --]

Re: Git mirror - bidrectional

[Miles Bader]

Dan Nicolaescu <dann@ics.uci.edu> writes:
> If bzr is not satisfactory for some reason, then please open a
> discussion with arguments about replacing it, not just add yet more
> noise about a different way of checking things in.

Bzr was chosen for political reasons, and it's been very clear for a
long time that technical arguments against it would be ignored unless it
proved _completely_ unusable.  That has not happened, however clunky it
may be.

So absent the possibility of any change, people have to work around the
annoyance as best they can, and that is what is being discussed.  It's
sad that it comes down this, but that's life....

-Miles

-- 
Freebooter, n. A conqueror in a small way of business, whose annexations lack
of the sanctifying merit of magnitude.

Re: Git mirror - bidrectional

[Ken Raeburn]

On Jan 13, 2010, at 14:16, David Reitter wrote:
> Indeed, anything more than a few seconds would seem unusual if not  
> unacceptable to the typical busy Git user.

Perhaps some parallelism or commit-triggered bzr->git mirroring could  
help keep the delays down.  I don't think we need to get excessively  
clever unless it actually proves to be a problem, though.

Does the git protocol allow the server to send arbitrary messages for  
presentation to the user?  "Your patch is very important to us.   
Please stay on the line...." :-)

> The alternative would be to do it asynchronously and undo the push  
> if it fails (a "git reset" is trivial).  The user
> could then only be notified by e-mail.

I think I'd rather have the interactive delay.  If it proves to be too  
much, I can simply bury that window and look at it later.

Ken

Re: Git mirror - bidrectional

[David Reitter]

[-- Attachment #1: Type: text/plain, Size: 1024 bytes --]

On Jan 13, 2010, at 8:58 PM, Ken Raeburn wrote:

> Does the git protocol allow the server to send arbitrary messages for presentation to the user?  "Your patch is very important to us.  Please stay on the line...." :-)

Yes it does.  "Both standard output and standard error output are forwarded to git-send-pack on the other end, so you can simply echo messages for the user."

>> The alternative would be to do it asynchronously and undo the push if it fails (a "git reset" is trivial).  The user
>> could then only be notified by e-mail.
> 
> I think I'd rather have the interactive delay.  If it proves to be too much, I can simply bury that window and look at it later.

I tend to agree.  Failure after a long wait (context switch, cognitively speaking) is bad, but we can keep the likelihood of such failures down if we bzr>git sync often.

I would volunteer to set up an experimental server if I was willing to touch bzr and bzr importing/exporting again.
But the concept is in place it seems.   Anyone?

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 203 bytes --]

Re: Git mirror - bidrectional

[Stefan Monnier]

>> If bzr is not satisfactory for some reason, then please open a
>> discussion with arguments about replacing it, not just add yet more
>> noise about a different way of checking things in.
> Chong, Stefan, do you want this discussion here?

No.


        Stefan


PS: The only thing I'm interested in at this point is to improve the Bzr
repository, either with great new features and bugfixes to install, or
by improving its setup, e.g. allow bzr+ssh access, or improve Bzr's
performance and UI.