git-1.8.2 support pulling from and pushing to a bzr branch

git-1.8.2 support pulling from and pushing to a bzr branch

[Jonas Bernoulli]

Hello List

git-1.8.2, released in March, supports pulling from and pushing to bzr
branches.  This should make it possible to contribute to Emacs using git
without ever having to touch bzr directly.  Also read-only users won't
be affected by the git mirrors lagging behind anymore - they can just
pull themselves.

I was able to clone the Emacs trunk in about an hour.  Cloning the Cedet
trunk also succeeded.  The fast-{import,export} scripts used previously
were no longer able to do so.

There are two small problems:

(1) Only bzr branches can be cloned, not complete repositories.  So
    multiple remotes have to be added to track a complete repository.

(2) When pushing to a bzr branch 'git push' always claims that a new
    branch is being created.

And then there is a big problem:

The commit hashes are not identical to those from fast-{export,import}.
The problem is that bzr saves commit messages without a trailing newline
while git saves them with a trailing newline; and neither bzr-fast-export
nor git-fast-import take care of adding that missing newline.

While it is _possible_ in git to create a commit message without a
trailing newline using the plumbing command git-commit-tree, commits
created with git-commit do end with a newline.

A commit created in bzr using "the normal way" should result in a git
commit that also looks like it was created in git "the normal way".
Therefor when translating a commit from bzr to git the correct thing
to do is to append a newline.

I hope that despite this issue we can recreate the "official" git mirror
using the new transparent importer.  This will inconvenient users in the
short term but I think it is better to get this over with now.

  Best regards,
  Jonas



Ps: A little demo:

    $ bzr init /tmp/demo/bzr
    Created a standalone tree (format: 2a)                                              
    $ cd /tmp/demo/bzr
    $ bzr commit --unchanged -m "from bzr"
    Committing to: /tmp/demo/bzr/                                                       
    Committed revision 1.
    $ git clone bzr::file:///tmp/demo/bzr /tmp/demo/git
    Cloning into '/tmp/demo/git'...
    $ cd /tmp/demo/git
    $ git commit --allow-empty -m "from git"
    [master af354dd] from git
    $ git push origin master
    All changes applied successfully.
    To bzr::file:///tmp/demo/bzr
     * [new branch]      master -> master
    $ cd -
    /tmp/demo/bzr
    $ bzr commit --unchanged -m "from bzr again"
    Committing to: /tmp/demo/bzr/                                                       
    Committed revision 3.
    $ bzr log --line
    3: Jonas Bernoulli 2013-04-13 from bzr again
    2: Jonas Bernoulli 2013-04-13 from git
    1: Jonas Bernoulli 2013-04-13 from bzr
    $ cd -
    /tmp/demo/git
    $ git pull
    From bzr::file:///tmp/demo/bzr
       af354dd..c8a556c  master     -> origin/master
    Updating af354dd..c8a556c
    Fast-forward
    $ git log --oneline
    c8a556c from bzr again
    af354dd from git
    36d4ef4 from bzr

Pps: And now the difference between the new and old importer:

    $ git init /tmp/demo/git_fast-import
    Initialized empty Git repository in /tmp/demo/git_fast-import/.git/
    $ cd /tmp/demo/git_fast-import
    $ bzr fast-export ../bzr | git fast-import
    21:56:37 Calculating the revisions to include ...
    21:56:37 Starting export of 3 revisions ...
    21:56:37 Exported 3 revisions in 0:00:00
    git-fast-import statistics:
    <snip>
    $ git remote add git-remote-bzr ../git
    $ git fetch git-remote-bzr
    warning: no common commits
    remote: Counting objects: 4, done.
    remote: Compressing objects: 100% (3/3), done.
    remote: Total 4 (delta 2), reused 2 (delta 0)
    Unpacking objects: 100% (4/4), done.
    From ../git
     * [new branch]      master     -> git-remote-bzr/master
    $ git cat-file -p HEAD~2
    tree 4b825dc642cb6eb9a060e54bf8d69288fbee4904
    author Jonas Bernoulli <jonas@bernoul.li> 1365882997 +0200
    committer Jonas Bernoulli <jonas@bernoul.li> 1365882997 +0200
    
    from bzr%
    $ git cat-file -p HEAD~1
    tree 4b825dc642cb6eb9a060e54bf8d69288fbee4904
    parent 70f6af8a56ea7744de514d1e80b9a97ef75fc94b
    author Jonas Bernoulli <jonas@bernoul.li> 1365882997 +0200
    committer Jonas Bernoulli <jonas@bernoul.li> 1365882997 +0200
    
    from git
    $ git cat-file -p git-remote-bzr/master~2
    tree 4b825dc642cb6eb9a060e54bf8d69288fbee4904
    author Jonas Bernoulli <jonas@bernoul.li> 1365882997 +0200
    committer Jonas Bernoulli <jonas@bernoul.li> 1365882997 +0200
    
    from bzr
    $ git cat-file -p git-remote-bzr/master~1
    tree 4b825dc642cb6eb9a060e54bf8d69288fbee4904
    parent 36d4ef499813359d492162ed6a53d28cdfb9aa35
    author Jonas Bernoulli <jonas@bernoul.li> 1365882997 +0200
    committer Jonas Bernoulli <jonas@bernoul.li> 1365882997 +0200
    
    from git
    $ git cat-file -p HEAD~2 > a
    $ git cat-file -p git-remote-bzr/master~2 > b
    $ diff -u a b
    --- a   2013-04-13 21:56:37.866995633 +0200
    +++ b   2013-04-13 21:57:32.690996992 +0200
    @@ -2,4 +2,4 @@
     author Jonas Bernoulli <jonas@bernoul.li> 1365882997 +0200
     committer Jonas Bernoulli <jonas@bernoul.li> 1365882997 +0200
     
    -from bzr
    \ No newline at end of file
    +from bzr
    1 $ git show HEAD~2 > a
    $ git show git-remote-bzr/master~2 > b
    $ diff -u a b
    --- a   2013-04-13 21:57:32.698996994 +0200
    +++ b   2013-04-13 21:57:32.706996994 +0200
    @@ -1,4 +1,4 @@
    -commit 70f6af8a56ea7744de514d1e80b9a97ef75fc94b
    +commit 36d4ef499813359d492162ed6a53d28cdfb9aa35
     Author: Jonas Bernoulli <jonas@bernoul.li>
     Date:   Sat Apr 13 21:56:37 2013 +0200

Re: git-1.8.2 support pulling from and pushing to a bzr branch

[John Wiegley]

>>>>> Jonas Bernoulli <jonas@bernoul.li> writes:

> git-1.8.2, released in March, supports pulling from and pushing to bzr
> branches.  This should make it possible to contribute to Emacs using git
> without ever having to touch bzr directly.  Also read-only users won't
> be affected by the git mirrors lagging behind anymore - they can just
> pull themselves.

I found out about this support last week, so as you can imagine I immediately
jumped on it and cloned the Emacs BZR repository (which took a good long
while, but did succeed).

The resulting repository could not be updated (using git-fetch).  At first it
kept timing out trying to communicate with the remote bazaar server.  Then, it
started giving me an exception about "object 000000000 not existing", or
something like that.  I was never able to get the repository to function again
after that and deleted it.

So, I think this is a great idea, but I'm not sure it's any more ready for
prime-time than the old git-bzr machinery was.

John

Re: git-1.8.2 support pulling from and pushing to a bzr branch

[Γιώργος Κεραμίδας]

On Sat, 13 Apr 2013 17:22:32 -0500, "John Wiegley" <johnw@newartisans.com> wrote:
>>>>>> Jonas Bernoulli <jonas@bernoul.li> writes:
>> git-1.8.2, released in March, supports pulling from and pushing to bzr
>> branches.  This should make it possible to contribute to Emacs using git
>> without ever having to touch bzr directly.  Also read-only users won't
>> be affected by the git mirrors lagging behind anymore - they can just
>> pull themselves.
>
> I found out about this support last week, so as you can imagine I immediately
> jumped on it and cloned the Emacs BZR repository (which took a good long
> while, but did succeed).
>
> The resulting repository could not be updated (using git-fetch).  At first it
> kept timing out trying to communicate with the remote bazaar server.  Then, it
> started giving me an exception about "object 000000000 not existing", or
> something like that.  I was never able to get the repository to function again
> after that and deleted it.
>
> So, I think this is a great idea, but I'm not sure it's any more ready for
> prime-time than the old git-bzr machinery was.

Ditto.

I did the same experiment when I found out about the new support too.
Unfortunately, I also hit the "object 000000000 not existing" problem,
and failed to re-fetch anything from the git clone.  The existing git
mirror works fine for now I think.

Re: git-1.8.2 support pulling from and pushing to a bzr branch

[David Engster]

Γιώργος Κεραμίδας writes:
> Ditto.
>
> I did the same experiment when I found out about the new support too.
> Unfortunately, I also hit the "object 000000000 not existing" problem,
> and failed to re-fetch anything from the git clone.  The existing git
> mirror works fine for now I think.

Report these problems at

https://github.com/felipec/git/issues

though I'd suggest to first try the latest version from

https://github.com/felipec/git/blob/fc/remote/bzr/contrib/remote-helpers/git-remote-bzr

-David

Re: git-1.8.2 support pulling from and pushing to a bzr branch

[Giorgos Keramidas]

On 2013-04-15 21:06, David Engster <deng@randomsample.de> wrote:
>Γιώργος Κεραμίδας writes:
>> Ditto.
>>
>> I did the same experiment when I found out about the new support
>> too.  Unfortunately, I also hit the "object 000000000 not existing"
>> problem, and failed to re-fetch anything from the git clone.  The
>> existing git mirror works fine for now I think.
>
> Report these problems at
>
> https://github.com/felipec/git/issues
>
> though I'd suggest to first try the latest version from
>
> https://github.com/felipec/git/blob/fc/remote/bzr/contrib/remote-helpers/git-remote-bzr

I will repeat the test with this version.  Thanks!

Re: git-1.8.2 support pulling from and pushing to a bzr branch

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 746 bytes --]

On Sat, Apr 13 2013, John Wiegley wrote:

>
> The resulting repository could not be updated (using git-fetch).  At first it
> kept timing out trying to communicate with the remote bazaar server.  Then, it
> started giving me an exception about "object 000000000 not existing", or
> something like that.  I was never able to get the repository to function again
> after that and deleted it.

I've already reported the issue to the upstream author, and he has been
very responsive and helpful (hint hint).

So actually this problem is already fixed in git's git version of
git-remote-bzr, so go ahead and try it, it works like a charm!

-- 
Julien Danjou
# Free Software hacker # freelance consultant
# http://julien.danjou.info

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: git-1.8.2 support pulling from and pushing to a bzr branch

[joakim]

Julien Danjou <julien@danjou.info> writes:

> On Sat, Apr 13 2013, John Wiegley wrote:
>
>>
>> The resulting repository could not be updated (using git-fetch).  At first it
>> kept timing out trying to communicate with the remote bazaar server.  Then, it
>> started giving me an exception about "object 000000000 not existing", or
>> something like that.  I was never able to get the repository to function again
>> after that and deleted it.
>
> I've already reported the issue to the upstream author, and he has been
> very responsive and helpful (hint hint).
>
> So actually this problem is already fixed in git's git version of
> git-remote-bzr, so go ahead and try it, it works like a charm!

I also tried cloning trunk. It takes a long time. The resulting
directory is 12Gb. Subsequent pulls are fast. Adding a remote(the
xwidget branch) adds 1Gb.

I havent tested further.

-- 
Joakim Verona

Re: git-1.8.2 support pulling from and pushing to a bzr branch

[David Engster]

'joakim' writes:
> I also tried cloning trunk. It takes a long time. The resulting
> directory is 12Gb.

You have to pack it afterwards using 'git gc --agressive'.

-David

Re: git-1.8.2 support pulling from and pushing to a bzr branch

[joakim]

David Engster <deng@randomsample.de> writes:

> 'joakim' writes:
>> I also tried cloning trunk. It takes a long time. The resulting
>> directory is 12Gb.
>
> You have to pack it afterwards using 'git gc --agressive'.

I tried this. I took some time. (long enough that I started doing
something else) Afterwards the repo was 1gb in total.

I'm just reporting so people can have some clue what to expect. I'm
impressed it works at all, especially as compared to the various
fastimport based approaches.

>
> -David
>

-- 
Joakim Verona

Re: git-1.8.2 support pulling from and pushing to a bzr branch

[joakim]

joakim@verona.se writes:

> David Engster <deng@randomsample.de> writes:
>
>> 'joakim' writes:
>>> I also tried cloning trunk. It takes a long time. The resulting
>>> directory is 12Gb.
>>
>> You have to pack it afterwards using 'git gc --agressive'.
>
> I tried this. I took some time. (long enough that I started doing
> something else) Afterwards the repo was 1gb in total.
>
> I'm just reporting so people can have some clue what to expect. I'm
> impressed it works at all, especially as compared to the various
> fastimport based approaches.

I tried pushing a small patch to my xwidget branch using
git-remote-bzr. Pushing times out. I've contacted the author about the
issue.

>
>>
>> -David
>>

-- 
Joakim Verona

Re: git-1.8.2 support pulling from and pushing to a bzr branch

[joakim]

joakim@verona.se writes:

> joakim@verona.se writes:
>
>> David Engster <deng@randomsample.de> writes:
>>
>>> 'joakim' writes:
>>>> I also tried cloning trunk. It takes a long time. The resulting
>>>> directory is 12Gb.
>>>
>>> You have to pack it afterwards using 'git gc --agressive'.
>>
>> I tried this. I took some time. (long enough that I started doing
>> something else) Afterwards the repo was 1gb in total.
>>
>> I'm just reporting so people can have some clue what to expect. I'm
>> impressed it works at all, especially as compared to the various
>> fastimport based approaches.
>
> I tried pushing a small patch to my xwidget branch using
> git-remote-bzr. Pushing times out. I've contacted the author about the
> issue.

I have now pushed a test commit to my xwidget branch using the latest
version of Felipes git-remote-bzr.

https://raw.github.com/felipec/git/fc/master/contrib/remote-helpers/git-remote-bzr

Felipe was very helpful during this test. I think more of us should try
it out now.

>
>>
>>>
>>> -David
>>>

-- 
Joakim Verona

Re: git-1.8.2 support pulling from and pushing to a bzr branch

[Felipe Contreras]

Hi,

On Tue, Apr 30, 2013 at 5:24 AM,  <joakim@verona.se> wrote:
> I have now pushed a test commit to my xwidget branch using the latest
> version of Felipes git-remote-bzr.
>
> https://raw.github.com/felipec/git/fc/master/contrib/remote-helpers/git-remote-bzr
>
> Felipe was very helpful during this test. I think more of us should try
> it out now.

I just pushed a small fix so you can clone the whole repository.

These are my recommended instructions:

If you already have a bazaar branch, for example:

  % bzr branch bzr://bzr.savannah.gnu.org/emacs/trunk emacs-trunk

Then you do:

  % git init emacs-git
  % cd emacs-git
  % git remote add origin bzr::bzr://bzr.savannah.gnu.org/emacs
  % bzr init-repo --no-trees .git/bzr
  % bzr branch ../emacs-trunk .git/bzr/foobar # setup repo objects
  % rm -rf .git/bzr/foobar # remove cruft
  % git fetch origin

If something fails in the fetch phase, not everything is lost, you can
report the problem, and the information in the emacs-git repository
might be useful for debugging the issue. Then you can recover without
starting from scratch.

If you are feeling adventurous, you can of course try the whole thing in one go:

  % git clone bzr::bzr://bzr.savannah.gnu.org/emacs emacs-git

You will be able to keep track of all the branches, as if they were
native to git:

  % git branch --remote

  % git checkout -b xwidget origin/xwidget
  # hacking
  % git push

The only limitation is that you cannot use the idiom 'git push origin
cur-name:new-name', so the local branch should have the same name as
the remote branch.

Once you are happy with the setup, you should probably compress the git repo:

  % git gc --aggressive

Unfortunately, the emacs repo is too big, and too disperse (since it
came from a remote helper) for my poor machine to handle without
running out of memory, so, I have to do this:

  % git config gc.aggressiveWindow 50

Moreover, if you are only interested in a couple of branches, you can
specify them with:

  % git config remote-bzr.branches 'trunk, xwindow'

I also read a complaint about git always showing 'new branch', but
that's not a problem of this remote helper, it's in the git
infrastructure itself[1].

Good luck :)

[1] http://article.gmane.org/gmane.comp.version-control.git/220798

-- 
Felipe Contreras