Official Git mirror?

Official Git mirror?

[Oleksandr Gavenko (aka gavenkoa)]

As Bzr have network inefficient protocol I search for Git mirror.

There are 2 page say about mirrors:

   http://www.emacswiki.org/emacs/EmacsFromGit
   http://savannah.gnu.org/git/?group=emacs

Seems that

   http://repo.or.cz/w/emacs.git

and

   http://git.savannah.gnu.org/cgit/emacs.git

both up-date.

Which use and is they compatible?

Re: Official Git mirror?

[Eli Zaretskii]

> From: "Oleksandr Gavenko (aka gavenkoa)" <gavenkoa@gmail.com>
> Date: Sun, 20 Feb 2011 23:11:16 +0200
> 
> As Bzr have network inefficient protocol I search for Git mirror.

Bzr is no less efficient than Git.  If the initial checkout is slow,
try one of these two, for the first "bzr branch" command only:

  bzr branch nosmart+bzr://bzr.savannah.gnu.org/emacs/trunk

  bzr branch lp:emacs

Thereafter, just use bzr://bzr.savannah.gnu.org/emacs/trunk, it is
fast for resyncing.

> There are 2 page say about mirrors:
> 
>    http://www.emacswiki.org/emacs/EmacsFromGit
>    http://savannah.gnu.org/git/?group=emacs
> 
> Seems that
> 
>    http://repo.or.cz/w/emacs.git
> 
> and
> 
>    http://git.savannah.gnu.org/cgit/emacs.git
> 
> both up-date.
> 
> Which use and is they compatible?

They are all outdated by several hours (up to 1 day).  You are well
advised to use bzr instead.

Re: Official Git mirror?

[Oleksandr Gavenko (aka gavenkoa)]

On 20.02.2011 23:35, Eli Zaretskii wrote:
>> From: "Oleksandr Gavenko (aka gavenkoa)"<gavenkoa@gmail.com>
>> Date: Sun, 20 Feb 2011 23:11:16 +0200
>>
>> As Bzr have network inefficient protocol I search for Git mirror.
>
> Bzr is no less efficient than Git.  If the initial checkout is slow,
> try one of these two, for the first "bzr branch" command only:
>
>    bzr branch nosmart+bzr://bzr.savannah.gnu.org/emacs/trunk
>
>    bzr branch lp:emacs
>
> Thereafter, just use bzr://bzr.savannah.gnu.org/emacs/trunk, it is
> fast for resyncing.
>
>> There are 2 page say about mirrors:
>>
>>     http://www.emacswiki.org/emacs/EmacsFromGit
>>     http://savannah.gnu.org/git/?group=emacs
>>
>> Seems that
>>
>>     http://repo.or.cz/w/emacs.git
>>
>> and
>>
>>     http://git.savannah.gnu.org/cgit/emacs.git
>>
>> both up-date.
>>
>> Which use and is they compatible?
>
> They are all outdated by several hours (up to 1 day).  You are well
> advised to use bzr instead.
>
Can I update http://www.emacswiki.org/emacs/EmacsFromGit by your text?

Re: Official Git mirror?

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: "Oleksandr Gavenko (aka gavenkoa)" <gavenkoa@gmail.com>
>> Date: Sun, 20 Feb 2011 23:11:16 +0200
>> 
>> As Bzr have network inefficient protocol I search for Git mirror.
>
> Bzr is no less efficient than Git.  If the initial checkout is slow,
> try one of these two, for the first "bzr branch" command only:
>
>   bzr branch nosmart+bzr://bzr.savannah.gnu.org/emacs/trunk
>
>   bzr branch lp:emacs

oscar@qcore:~/dev/ff$ time bzr branch lp:emacs
Branched 103368 revision(s).                                                                                                                        
real    17m41.424s
user    7m56.250s
sys     0m8.240s

oscar@qcore:~/dev/ff/git-emacs$ time git clone git://repo.or.cz/emacs.git git-emacs
Initialized empty Git repository in /home/oscar/dev/ff/git-emacs/git-emacs/.git/
remote: Counting objects: 597983, done.
remote: Compressing objects: 100% (122520/122520), done.
remote: Total 597983 (delta 474994), reused 597778 (delta 474826)
Receiving objects: 100% (597983/597983), 521.43 MiB | 1.00 MiB/s, done.
Resolving deltas: 100% (474994/474994), done.
real    11m42.616s
user    2m36.630s
sys     0m14.820s


Please note that git downloads all 51 branches that exists or existed on
Savannah while bzr gets just `trunk'.

Git keeps the pipe downloading data at full speed all the time, while
bzr fluctuates a lot, including several long pauses, possibly because
the server is doing some CPU-intensive work for preparing the data.

Maybe the differences are not big enough to notice by most people that
update their Emacs mirrors from time to time, but it is not accurate to
say that bzr's network protocol is no less efficient than git.

Re: Official Git mirror?

[Leo]

On 2011-02-21 13:36 +0800, Óscar Fuentes wrote:
> Maybe the differences are not big enough to notice by most people that
> update their Emacs mirrors from time to time, but it is not accurate to
> say that bzr's network protocol is no less efficient than git.

Thanks for the excellent data points.

Leo

Re: Official Git mirror?

[Eli Zaretskii]

> From: "Oleksandr Gavenko (aka gavenkoa)" <gavenkoa@gmail.com>
> Date: Sun, 20 Feb 2011 23:59:16 +0200
> 
> Can I update http://www.emacswiki.org/emacs/EmacsFromGit by your text?

I already did.

Re: Official Git mirror?

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Mon, 21 Feb 2011 06:36:48 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: "Oleksandr Gavenko (aka gavenkoa)" <gavenkoa@gmail.com>
> >> Date: Sun, 20 Feb 2011 23:11:16 +0200
> >> 
> >> As Bzr have network inefficient protocol I search for Git mirror.
> >
> > Bzr is no less efficient than Git.  If the initial checkout is slow,
> > try one of these two, for the first "bzr branch" command only:
> >
> >   bzr branch nosmart+bzr://bzr.savannah.gnu.org/emacs/trunk
> >
> >   bzr branch lp:emacs
> 
> oscar@qcore:~/dev/ff$ time bzr branch lp:emacs
> Branched 103368 revision(s).                                                                                                                        
> real    17m41.424s
> user    7m56.250s
> sys     0m8.240s
> 
> oscar@qcore:~/dev/ff/git-emacs$ time git clone git://repo.or.cz/emacs.git git-emacs
> Initialized empty Git repository in /home/oscar/dev/ff/git-emacs/git-emacs/.git/
> remote: Counting objects: 597983, done.
> remote: Compressing objects: 100% (122520/122520), done.
> remote: Total 597983 (delta 474994), reused 597778 (delta 474826)
> Receiving objects: 100% (597983/597983), 521.43 MiB | 1.00 MiB/s, done.
> Resolving deltas: 100% (474994/474994), done.
> real    11m42.616s
> user    2m36.630s
> sys     0m14.820s

I don't think 11 min are significantly better than 17.  Seems like a
good price to pay for being always in sync with the repository.  And
this is just the initial checkout, the differences get even smaller
for routine day-to-day operation.

Yours is just one example, I get almost the same times, and sometimes
slightly faster times for bzr than for git.  Here's one example where
I got the same times:

  bzr:
  real    0m14.437s
  user    0m2.516s
  sys     0m0.308s

  git:
  real    13m59.655s
  user    7m55.702s
  sys     0m18.321s

Btw, what is your bzr version?  You could get faster downloads with
the latest versions (2.2+).

> Please note that git downloads all 51 branches that exists or existed on
> Savannah while bzr gets just `trunk'.

Most people don't need the other branches, so it's just ballast.

> Git keeps the pipe downloading data at full speed all the time, while
> bzr fluctuates a lot, including several long pauses, possibly because
> the server is doing some CPU-intensive work for preparing the data.

The nosmart+ option prevents the server from wasting CPU cycles when
everything is needed to be downloaded anyway.

> Maybe the differences are not big enough to notice by most people that
> update their Emacs mirrors from time to time, but it is not accurate to
> say that bzr's network protocol is no less efficient than git.

I did testing on several machines, and the average is almost the same,
in terms of elapsed time.  On some machines, git is slightly faster,
on others it's the other way around.

And I doubt that many people care about the trade-off between CPU,
file I/O, and network I/O.  The OP was talking about the network
protocol, but I'm quite sure he actually cares about the elapsed time
of the initial download.

Bottom line: I'd advise using bzr, because the advantages of using the
same tool as the Emacs developers (same revision IDs etc.) outweigh
the disadvantages of a slightly slower operation, even on GNU/Linux.

Re: Official Git mirror?

[Tim X]

Óscar Fuentes <ofv@wanadoo.es> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> From: "Oleksandr Gavenko (aka gavenkoa)" <gavenkoa@gmail.com>
>>> Date: Sun, 20 Feb 2011 23:11:16 +0200
>>> 
>>> As Bzr have network inefficient protocol I search for Git mirror.
>>
>> Bzr is no less efficient than Git.  If the initial checkout is slow,
>> try one of these two, for the first "bzr branch" command only:
>>
>>   bzr branch nosmart+bzr://bzr.savannah.gnu.org/emacs/trunk
>>
>>   bzr branch lp:emacs
>
> oscar@qcore:~/dev/ff$ time bzr branch lp:emacs
> Branched 103368 revision(s).
> real    17m41.424s
> user    7m56.250s
> sys     0m8.240s
>
> oscar@qcore:~/dev/ff/git-emacs$ time git clone git://repo.or.cz/emacs.git git-emacs
> Initialized empty Git repository in /home/oscar/dev/ff/git-emacs/git-emacs/.git/
> remote: Counting objects: 597983, done.
> remote: Compressing objects: 100% (122520/122520), done.
> remote: Total 597983 (delta 474994), reused 597778 (delta 474826)
> Receiving objects: 100% (597983/597983), 521.43 MiB | 1.00 MiB/s, done.
> Resolving deltas: 100% (474994/474994), done.
> real    11m42.616s
> user    2m36.630s
> sys     0m14.820s
>
>
> Please note that git downloads all 51 branches that exists or existed on
> Savannah while bzr gets just `trunk'.
>
> Git keeps the pipe downloading data at full speed all the time, while
> bzr fluctuates a lot, including several long pauses, possibly because
> the server is doing some CPU-intensive work for preparing the data.
>
> Maybe the differences are not big enough to notice by most people that
> update their Emacs mirrors from time to time, but it is not accurate to
> say that bzr's network protocol is no less efficient than git.
>
>

Such comparisons are meaingless. There are two many variables not
accounted for, such as network, server load, client/server versions and
even differences in repositories. 

However, what really matters is the updates rather than a fresh full
branching/cloning as you only do the long initial copy once. 

Personally, I prefer git, but use bzr just as much as a number of
projects I work on use bzr. I've found the initial checkouts to be
fairly close and later updates to also be about the same. If you setup
things according to the instructions on the emacs wiki for bzr, creating
a new branch copy is extremely fast. For example

tcross@puma:~/bzr/emacs$ time bzr branch http://bzr.savannah.gnu.org/r/emacs/trunk/ new
Branched 103370 revision(s).                                                        

real	0m44.743s
user	0m3.800s
sys	0m0.400s

which created a fresh new branch to work on. My preference for git is
that it just fits how I think and my preferred workflow over bzr.
However, functionally, I find them pretty much equivalent.

The real point to note, as mentioned by Eli, is that the git repositories
are frequently hours behind bzr. As we are talking about development
sources, this can be important. There are a number of times I've updated
only to find that the current snapshot either fails to build or has a
bug. However, this is usually fixed very quickly. In fact, this happened
to me last week. I reported the problem on the dev list and it was fixed
within about an hour. 

With the git copy, you may have to wait a day to get that update. In
fact, there was a post just after mine from someone using git who ran
into the same problem. The response he got was that he had to wait 24
hours until his git master was updated. If you ever run into issues and
want to raise/discuss them on the dev list, you will also get better
results if your based on the bzr branch as people don't know if your
being affected by a bug which may have already been fixed in bzr but has
not yet been copied over to git. Consequently, the common response is to
tell someone to wait 24 hours, update and see if the bug still exists,
if it does, come back and raise it again.

For example, the current git version is 

commit 0cc4633247929de05e1ae58cea225137074764c0
Author: Paul Eggert <eggert@cs.ucla.edu>
Date:   Sun Feb 20 11:46:08 2011 -0800

while the current bzr revision is

revno: 103370
committer: Chong Yidong <cyd@stupidchicken.com>
branch nick: trunk
timestamp: Mon 2011-02-21 01:03:36 -0500

and there is a difference of 5 commits between the two. 

Given the typical difference in refresh/update times between git and bzr
are minimal, I think its worth sticking with bzr. If you ever plan to
contribute bug fixes or other code, bzr will also be a better way to go
as your patches will likely be better accepted or you may even be able
to push changes up to get them merged in etc. While this is still
possible with git, it is a bit harder - anything that makes things
easier is more likely to increase the likelihood your contributions are
accepted. Reporting issues is easier as well as you have the bzr revno
readily available. 

Of course it all comes down to what you want and what you expect to do.
If all you want is a fairly up-to-date version of the dev code and you
don't plan to work on bugs, report problems or participate in
development, then the git clone is probably fine. However, if you plan
to work on bugs/submissions, participate in development etc, bzr is
really the only sane choice. 

Tim

-- 
tcross (at) rapttech dot com dot au

Re: Official Git mirror? -- Request

[Perry Smith]

On Feb 21, 2011, at 2:52 AM, Tim X wrote:

> Óscar Fuentes <ofv@wanadoo.es> writes:
> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>>>> From: "Oleksandr Gavenko (aka gavenkoa)" <gavenkoa@gmail.com>
>>>> Date: Sun, 20 Feb 2011 23:11:16 +0200
>>>> 

I wish folks would pick one SCM and stick with it.  Porting the SCM to
AIX just so I can check out some project is a pain.  There seems to be
at least a half dozen SCMs right now.  They all do exactly the same
thing.  Why are so many people recreating the exact same wheel?  The
current nightmare of SCMs makes me dream of the old days when everyone
used SVN.

Any chance GNU could pick git or the git folks pick
bzr?

pedz

Re: Official Git mirror? -- Request

[andrea crotti]

[-- Attachment #1: Type: text/plain, Size: 957 bytes --]

2011/2/21 Perry Smith <pedzsan@gmail.com>

>
>
> I wish folks would pick one SCM and stick with it.  Porting the SCM to
> AIX just so I can check out some project is a pain.  There seems to be
> at least a half dozen SCMs right now.  They all do exactly the same
> thing.  Why are so many people recreating the exact same wheel?  The
> current nightmare of SCMs makes me dream of the old days when everyone
> used SVN.
>
> Any chance GNU could pick git or the git folks pick
> bzr?
>
> pedz
>
>
> I agree that it's really a mess, and not mentioning mercurial/darcs and
others, but I don't agree on missing SVN ;)

The thing is that everyone is more productive with what uses most of the
time (I use git for example), and maybe there are also some "political"
reasons behind the adoption of one or the other.

SVN was not too bad for that, it could be used as a backend and then anyone
could work with git/svn whatever.
But it's surely not the optimal path.

[-- Attachment #2: Type: text/html, Size: 1304 bytes --]

Re: Official Git mirror?

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

> I don't think 11 min are significantly better than 17.  Seems like a
> good price to pay for being always in sync with the repository.  And
> this is just the initial checkout, the differences get even smaller
> for routine day-to-day operation.
>
> Yours is just one example, I get almost the same times, and sometimes
> slightly faster times for bzr than for git.  Here's one example where
> I got the same times:
>
>   bzr:
>   real    0m14.437s
>   user    0m2.516s
>   sys     0m0.308s
>
>   git:
>   real    13m59.655s
>   user    7m55.702s
>   sys     0m18.321s

The times quoted above seems wrong (0m14s for bzr?)

> Btw, what is your bzr version?  You could get faster downloads with
> the latest versions (2.2+).

2.2.1

>> Please note that git downloads all 51 branches that exists or existed on
>> Savannah while bzr gets just `trunk'.
>
> Most people don't need the other branches, so it's just ballast.

I have no issues with your recommendation of using bzr. I'm discussing
your assertion about bzr protocol's efficiency compared to git.

>> Git keeps the pipe downloading data at full speed all the time, while
>> bzr fluctuates a lot, including several long pauses, possibly because
>> the server is doing some CPU-intensive work for preparing the data.
>
> The nosmart+ option prevents the server from wasting CPU cycles when
> everything is needed to be downloaded anyway.

Then the question is: why is it not enabled by default when bzr clones a
branch from scratch?

>> Maybe the differences are not big enough to notice by most people that
>> update their Emacs mirrors from time to time, but it is not accurate to
>> say that bzr's network protocol is no less efficient than git.
>
> I did testing on several machines, and the average is almost the same,
> in terms of elapsed time.  On some machines, git is slightly faster,
> on others it's the other way around.

Bzr is quite CPU- and memory-intensive, to the point of being almost
unbearable when cloning a large branch (i.e. Emacs) on a netbook. Maybe
the machines that work faster for you are the more powerful ones?

> And I doubt that many people care about the trade-off between CPU,
> file I/O, and network I/O.  The OP was talking about the network
> protocol, but I'm quite sure he actually cares about the elapsed time
> of the initial download.

Agreed.

> Bottom line: I'd advise using bzr, because the advantages of using the
> same tool as the Emacs developers (same revision IDs etc.) outweigh
> the disadvantages of a slightly slower operation, even on GNU/Linux.

Agreed too, except for the case where the user is already familiar with
git and wants to keep local changes or do experimental hacking. Apart
from differences on workflow and performance, whose value may be
subjetive, developing with emacs+git is a very pleasant experience
thanks to packages like magit.el and others. Bzr still has a long path
to walk on terms of Emacs integration.

Re: Official Git mirror?

[Óscar Fuentes]

Tim X <timx@nospam.dev.null> writes:

>> Maybe the differences are not big enough to notice by most people that
>> update their Emacs mirrors from time to time, but it is not accurate to
>> say that bzr's network protocol is no less efficient than git.
>
> Such comparisons are meaingless. There are two many variables not
> accounted for, such as network, server load, client/server versions and
> even differences in repositories. 

Okay, let's repeat the experiment on a controlled environment.

LAN 100 Mb/s, idle.

A quad-core 2.4GHz Q6600 machine with 8 GB RAM, Kubuntu 10.10 64 bits (let's
call this machine `qcore')

A single-core single-thread Pentium M 2.0GHz with 2 GB RAM, Kubuntu
10.10 32 bits (let's name this one `single')

git version 1.7.1
bzr version 2.2.1

                                        real  /  user  /  sys
git ssh://qcore (from single)           5m34s    4m28s    0m27s
bzr nosmart+bzr://qcore (from single)   9m54s    9m21s    0m9s
bzr bzr://qcore (from single)           7m37s    6m15s    0m8s
git ssh://single (from qcore)           2m50s    2m23s    0m7s
bzr nosmart+bzr://single (from qcore)   6m47s    6m27s    0m3s
bzr bzr://single (from qcore)           6m24s    4m17s    0m3s

On all cases the operation seems CPU-bound on the *client*. Git shows a
CPU spike on the server at the beginning, on the "counting objects"
phase. The Bzr server shows no significant CPU usage when connected with
the "nosmart" option. Else it pegs one CPU core for a large part of the
phase where revisions are sent, and uses up to 1.5 GB of RAM for a short
moment, although most of the time it is about 400 MB (on the 64bit
machine). Neither bzr nor git clients seems to exploit the multiple
cores while cloning. The bzr client uses a lot of memory (almost 600 MB
on the 64 bit machine, almost 400 on the 32 bits machine.)

It seems that "nosmart" is used for compensating for servers with busy
CPUs at the cost of more data sent to the client and more work offloaded
to it.

> However, what really matters is the updates rather than a fresh full
> branching/cloning as you only do the long initial copy once. 

We are discussing protocol efficiency. It is reasonable to expect that
any shortcoming on the cloning process will show in the update
operation. Moreover, on the timings I used an special case for bzr
(nosmart) that may tilt the results to its favor if we extrapolate the
results to update operations.

My experience updating emacs bzr from Launchpad (with the smart
protocol) is that it is noticeably slower than updating emacs git from
repo.or.cz. (To address just one factor, network latency is important on
short operations, but pinging to repo.or.cz is 50% slower than pinging
to launchpad.net from here.)

> Personally, I prefer git, but use bzr just as much as a number of
> projects I work on use bzr. I've found the initial checkouts to be
> fairly close and later updates to also be about the same.

I guess that your anecdotal experience is as good as mine.

[snip]

> The real point to note, as mentioned by Eli, is that the git repositories
> are frequently hours behind bzr.

That's a different issue. No discussion about that.

[snip]

Re: Official Git mirror?

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Mon, 21 Feb 2011 17:09:03 +0100
> 
> >   bzr:
> >   real    0m14.437s
> >   user    0m2.516s
> >   sys     0m0.308s
> >
> >   git:
> >   real    13m59.655s
> >   user    7m55.702s
> >   sys     0m18.321s
> 
> The times quoted above seems wrong (0m14s for bzr?)

Yes, sorry.  A copy/paste error.  I remembered one of my trials took
14 min.

> I have no issues with your recommendation of using bzr. I'm discussing
> your assertion about bzr protocol's efficiency compared to git.

I didn't assert that, at least didn't mean to.  I interpreted the OP's
complaint as referring to the elapsed time it takes, and answered
that.  Sorry if my wording was misleading.

> >> Git keeps the pipe downloading data at full speed all the time, while
> >> bzr fluctuates a lot, including several long pauses, possibly because
> >> the server is doing some CPU-intensive work for preparing the data.
> >
> > The nosmart+ option prevents the server from wasting CPU cycles when
> > everything is needed to be downloaded anyway.
> 
> Then the question is: why is it not enabled by default when bzr clones a
> branch from scratch?

I don't think the Bazaar developers saw this kind of data until now.
You will see a discussion about this on the Bazaar mailing list.  I
hope they will find a solution soon.  I also hope the server on
savannah will be upgraded to something similar to Launchpad.

> Bzr is quite CPU- and memory-intensive, to the point of being almost
> unbearable when cloning a large branch (i.e. Emacs) on a netbook.

That's not true, at least not wrt CPU.  Your own data refutes this:

    real    17m41.424s
    user    7m56.250s
    sys     0m8.240s

Here are a few of my data points, with different machines and
different network bandwidths:

    real    49m17.067s
    user    14m36.890s
    sys     0m14.250s

    real    01h04m01.629s
    user    00h20m54.484s
    sys     00h00m57.046s

    real    01h10m09.873s
    user    00h40m22.046s
    sys     00h05m36.921s

    real    16m30.189s
    user    15m22.090s
    sys     0m14.560s

    real    02h28m35.032s
    user    00h20m36.921s
    sys     00h00m50.750s

    real    30m23.956s
    user    11m33.760s
    sys     0m17.730s

In all but one case, the CPU time is 1/3 to 1/7 of the elapsed time.
That's not how a CPU-bound app looks like.

> Maybe the machines that work faster for you are the more powerful
> ones?

No, they are on faster networks.

Re: Official Git mirror?

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Mon, 21 Feb 2011 19:17:28 +0100
> 
> It seems that "nosmart" is used for compensating for servers with busy
> CPUs

No, it's used to compensate for overly "smart" server when there's no
win in being smart, because you need to send everything anyway.  The
"smart" part is for sending less data, which is not going to win for
the initial checkout.

> at the cost of more data sent to the client and more work offloaded
> to it.

The data ratio is 5.5:7.2, so it is indeed larger, but not awfully
so.  As for CPU, I wrote elsewhere that the client doesn't need too
much for the initial checkout.

> We are discussing protocol efficiency. It is reasonable to expect that
> any shortcoming on the cloning process will show in the update
> operation.

I don't see that in my work.  Updates are of comparable speed,
although bzr indeed typically pulls more data through the wire.

Re: Official Git mirror?

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

>> It seems that "nosmart" is used for compensating for servers with busy
>> CPUs
>
> No, it's used to compensate for overly "smart" server when there's no
> win in being smart, because you need to send everything anyway.

But then the plan is failing, because the timings I posted show that
"smart" wins over "nosmart" even when a weak server is servicing a
mighty client over a fast network.

> The "smart" part is for sending less data, which is not going to win
> for the initial checkout.

You said on the other post that cloning time is network-bound. So being
smart and sending less data would be better.

[snip]

Re: Official Git mirror?

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

[snip]

>> Bzr is quite CPU- and memory-intensive, to the point of being almost
>> unbearable when cloning a large branch (i.e. Emacs) on a netbook.
>
> That's not true, at least not wrt CPU.  Your own data refutes this:
>
>     real    17m41.424s
>     user    7m56.250s
>     sys     0m8.240s

First: that's on a fast desktop machine. My experience says that the netbook
(currently unavailable) on the same network will take 6 times more CPU
time. I recall having to copy the bzr Emacs repository from the desktop
machine to the netbook (instead of cloning it with bzr) after waiting
for more than an hour and losing patience. That was on the local
network.

Second: bzr is downloading approx. 450 MB over a 1 MB/s ADSL line. On
the best case it would take 7.5 minutes (it takes 9.8). For this specific
case of cloning the bzr emacs repo on Launchpad, saying that it is mostly
network-bound is accurate, but not by much. As soon as you start using a
slightly faster network or a slower machine the CPU time dominates.

> Here are a few of my data points, with different machines and
> different network bandwidths:

[snip]

> In all but one case, the CPU time is 1/3 to 1/7 of the elapsed time.
> That's not how a CPU-bound app looks like.

Without precise specs of the machines and networks for each timing, it
is hard to interpret those results.

>> Maybe the machines that work faster for you are the more powerful
>> ones?
>
> No, they are on faster networks.

Re: Official Git mirror?

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Mon, 21 Feb 2011 20:40:37 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> It seems that "nosmart" is used for compensating for servers with busy
> >> CPUs
> >
> > No, it's used to compensate for overly "smart" server when there's no
> > win in being smart, because you need to send everything anyway.
> 
> But then the plan is failing, because the timings I posted show that
> "smart" wins over "nosmart" even when a weak server is servicing a
> mighty client over a fast network.

Sometimes it indeed makes no significant difference, but sometimes it
wins big time.  Observe:

 bzr://bzr.savannah.gnu.org/emacs/trunk

    real    45m4.820s
    user    15m58.380s
    sys     0m12.910s

    Transferred: 540480KiB (199.9K/s r:540403K w:77K)

  bzr branch nosmart+bzr://bzr.savannah.gnu.org/emacs/trunk


    real    16m30.189s
    user    15m22.090s
    sys     0m14.560s

    Transferred: 780914KiB (789.2K/s r:780640K w:275K)

In the thread I mentioned on the Bazaar list, someone else also
reported a huge speedup:

> over a 3 Mbit/s connection:
> 
>    bzr://bzr.savannah.gnu.org/emacs/trunk
>    6949.356  Transferred: 469739kB (67.6kB/s r:469659kB w:80kB)
> 
>    nosmart+bzr://bzr.savannah.gnu.org/emacs/trunk
>    2919.117  Transferred: 524353kB (179.7kB/s r:524162kB w:191kB)

That's almost 2 hours slashed to 48 minutes, an almost 3-fold speedup.

> > The "smart" part is for sending less data, which is not going to win
> > for the initial checkout.
> 
> You said on the other post that cloning time is network-bound. So being
> smart and sending less data would be better.

Not if "being smart" wastes CPU cycles on the server side and causes
it to use the available bandwidth less efficiently.  See the network
throughput figures above, reported by bzr on .bzr.log.

Re: Official Git mirror?

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

>> But then the plan is failing, because the timings I posted show that
>> "smart" wins over "nosmart" even when a weak server is servicing a
>> mighty client over a fast network.
>
> Sometimes it indeed makes no significant difference, but sometimes it
> wins big time.  Observe:
>
>  bzr://bzr.savannah.gnu.org/emacs/trunk
>
>     real    45m4.820s
>     user    15m58.380s
>     sys     0m12.910s
>
>     Transferred: 540480KiB (199.9K/s r:540403K w:77K)
>
>   bzr branch nosmart+bzr://bzr.savannah.gnu.org/emacs/trunk
>
>
>     real    16m30.189s
>     user    15m22.090s
>     sys     0m14.560s
>
>     Transferred: 780914KiB (789.2K/s r:780640K w:275K)
>
> In the thread I mentioned on the Bazaar list, someone else also
> reported a huge speedup:
>
>> over a 3 Mbit/s connection:
>> 
>>    bzr://bzr.savannah.gnu.org/emacs/trunk
>>    6949.356  Transferred: 469739kB (67.6kB/s r:469659kB w:80kB)
>> 
>>    nosmart+bzr://bzr.savannah.gnu.org/emacs/trunk
>>    2919.117  Transferred: 524353kB (179.7kB/s r:524162kB w:191kB)
>
> That's almost 2 hours slashed to 48 minutes, an almost 3-fold speedup.
>
>> > The "smart" part is for sending less data, which is not going to win
>> > for the initial checkout.
>> 
>> You said on the other post that cloning time is network-bound. So being
>> smart and sending less data would be better.
>
> Not if "being smart" wastes CPU cycles on the server side and causes
> it to use the available bandwidth less efficiently.  See the network
> throughput figures above, reported by bzr on .bzr.log.

Eli, I re-quote what you said:

Oscar:
>> >> It seems that "nosmart" is used for compensating for servers with busy
>> >> CPUs
Eli:
>> > No, it's used to compensate for overly "smart" server when there's no
>> > win in being smart, because you need to send everything anyway.

You are contradicting yourself, because your data above is more evidence
supporting my hypotheses of nosmart being a tricky effective with
CPU-starved servers.

Besides, cloning from Savannah is almost twice as slow than from
Launchpad for me. If it is because network distance considerations
(pinging to it is 3 times slower than to Launchpad) or because the
server is CPU and/or bandwidth starved, I don't know. Your timings
comparing smart/nosmart seem to point to the CPU, though.

Re: Official Git mirror?

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Mon, 21 Feb 2011 21:57:30 +0100
> 
> >> > The "smart" part is for sending less data, which is not going to win
> >> > for the initial checkout.
> >> 
> >> You said on the other post that cloning time is network-bound. So being
> >> smart and sending less data would be better.
> >
> > Not if "being smart" wastes CPU cycles on the server side and causes
> > it to use the available bandwidth less efficiently.  See the network
> > throughput figures above, reported by bzr on .bzr.log.
> 
> Eli, I re-quote what you said:
> 
> Oscar:
> >> >> It seems that "nosmart" is used for compensating for servers with busy
> >> >> CPUs
> Eli:
> >> > No, it's used to compensate for overly "smart" server when there's no
> >> > win in being smart, because you need to send everything anyway.
> 
> You are contradicting yourself, because your data above is more evidence
> supporting my hypotheses of nosmart being a tricky effective with
> CPU-starved servers.

I see no contradiction.  The smart server uses more CPU on the server
side, and by that causes large gaps in sending data, while it "thinks"
what to send next.  When it needs to send hundreds of MBs, this adds
up to many minutes.  And in the initial checkout case, the invested
CPU time goes to waste anyway, because there's no way of being "smart"
when you need to send all the data downstream.

As an example, look at the "Finding revisions" stage, the first stage
of "bzr branch".  It takes less than a minute with "nosmart", and
around 20 min (!) with a "smart" server.  It tries to be smart about
which revisions to send, only to discover that it eventually needs to
send everything.

IOW, the "smart server" should be made aware that this is an initial
checkout into a fresh repository.  I guess this was not done yet
because for small projects the slowdown is barely visible.

> Besides, cloning from Savannah is almost twice as slow than from
> Launchpad for me. If it is because network distance considerations
> (pinging to it is 3 times slower than to Launchpad) or because the
> server is CPU and/or bandwidth starved, I don't know. Your timings
> comparing smart/nosmart seem to point to the CPU, though.

Launchpad is faster because its server is a newer bzr than that on
Savannah.  I'm not sure if the network bandwidth is also a factor
here, but it could be.

Re: Official Git mirror?

[Oleksandr Gavenko]

On 2011-02-21 6:58, Eli Zaretskii wrote:
>> From: "Oleksandr Gavenko (aka gavenkoa)"<gavenkoa@gmail.com>
>> Date: Sun, 20 Feb 2011 23:59:16 +0200
>>
>> Can I update http://www.emacswiki.org/emacs/EmacsFromGit by your text?
>
> I already did.
>
I also worry about http://www.emacswiki.org/emacs/EmacsFromGit
(you update 'BzrForEmacsDevs' page).

What is a status of git://git.savannah.gnu.org/emacs.git
Note that link to it present on http://savannah.gnu.org/projects/emacs

How frequency mirrors update:

   lp:emacs
   git://git.savannah.gnu.org/emacs.git
   git://repo.or.cz/emacs.git

We hear in this thread that git repo outdated for about 1 day.
How about Launchpad mirror?

-- 
Best regards!

Re: Official Git mirror?

[Eli Zaretskii]

> From: Oleksandr Gavenko <gavenkoa@gmail.com>
> Date: Mon, 21 Feb 2011 23:29:53 +0000
> 
> What is a status of git://git.savannah.gnu.org/emacs.git
> Note that link to it present on http://savannah.gnu.org/projects/emacs

It's a git mirror of the Emacs repository, intended for those who
prefer to use git as their VCS.

> How frequency mirrors update:
> 
>    lp:emacs
>    git://git.savannah.gnu.org/emacs.git
>    git://repo.or.cz/emacs.git
> 
> We hear in this thread that git repo outdated for about 1 day.

That is my experience, but I don't have any definitive answers.

> How about Launchpad mirror?

You will have to ask the maintainers of these sites.  I have no idea.
One thing is clear: they all fall behind the bzr repo by hours.

Re: Official Git mirror?

[Giorgos Keramidas]

On Mon, 21 Feb 2011 22:33:23 +0200, Eli Zaretskii <eliz@gnu.org> wrote:
> Sometimes it indeed makes no significant difference, but sometimes it
> wins big time.  Observe:
>
>  bzr://bzr.savannah.gnu.org/emacs/trunk
>
>     real    45m4.820s
>     user    15m58.380s
>     sys     0m12.910s
>
>     Transferred: 540480KiB (199.9K/s r:540403K w:77K)
>
>   bzr branch nosmart+bzr://bzr.savannah.gnu.org/emacs/trunk
>
>
>     real    16m30.189s
>     user    15m22.090s
>     sys     0m14.560s
>
>     Transferred: 780914KiB (789.2K/s r:780640K w:275K)

I started a new 'bzr branch' this morning, when I got to the office,
using the nosmart option.  It's been running for more than an hour now,
it has transferred 500 MB of data, and all this on a laptop running
Ubuntu 10.10 LTS with ample free disk space and connected to the very
fast internal network of Google.

This is a *lot* of data for something that is supposed to fetch only the
trunk of Emacs.  It took 3989.501 real / 581.270 user / 15.990 sys
seconds to fetch everything and the resulting .bzr/ directory contains
233 MB of data:

  gkeramidas@gkeramidas-glaptop:~/bzr/trunk$ du -sh .bzr/
  233M    .bzr/

This is not very fast, but I'll keep using the bzr branch to see how
future updates go.

Re: Official Git mirror?

[Eli Zaretskii]

> From: Giorgos Keramidas <keramida@ceid.upatras.gr>
> Newsgroups: gnu.emacs.help
> Date: Wed, 23 Feb 2011 10:16:10 +0100
> 
> I started a new 'bzr branch' this morning, when I got to the office,
> using the nosmart option.  It's been running for more than an hour now,
> it has transferred 500 MB of data, and all this on a laptop running
> Ubuntu 10.10 LTS with ample free disk space and connected to the very
> fast internal network of Google.

Like I said, the nosmart operation moves about 780MB of data on the
initial "bzr branch" command.  The resulting branch directory is
around 690MB.  With git, I get 632MB and 780MB respectively (but that
includes all the branches, so it should be larger).

> This is a *lot* of data for something that is supposed to fetch only the
> trunk of Emacs.

The trunk (or, actually, the first branch you download into a shared
repository) brings in the bulk of the revision history.  Additional
branches are mush smaller and their checkouts are much faster, because
they share a lot of revisions with the trunk, and "bzr branch" then
brings only the revisions that are not on the trunk.  Try another
branch command for emacs-23, for example.

> It took 3989.501 real / 581.270 user / 15.990 sys
> seconds to fetch everything and the resulting .bzr/ directory contains
> 233 MB of data:
> 
>   gkeramidas@gkeramidas-glaptop:~/bzr/trunk$ du -sh .bzr/
>   233M    .bzr/

About an hour is what we can get now, yes (I got as little as 16 min,
but that was on a system sitting on the same LAN as
bzr.savannah.gnu.org; otherwise, I get between 0:45 and 1:10).

> This is not very fast, but I'll keep using the bzr branch to see how
> future updates go.

Please don't forget NOT to use nosmart+ after the initial branch
command.

Re: Official Git mirror?

[jun yu]

I think git that is more quickly.

2011/2/23 Eli Zaretskii <eliz@gnu.org>:
>> From: Giorgos Keramidas <keramida@ceid.upatras.gr>
>> Newsgroups: gnu.emacs.help
>> Date: Wed, 23 Feb 2011 10:16:10 +0100
>>
>> I started a new 'bzr branch' this morning, when I got to the office,
>> using the nosmart option.  It's been running for more than an hour now,
>> it has transferred 500 MB of data, and all this on a laptop running
>> Ubuntu 10.10 LTS with ample free disk space and connected to the very
>> fast internal network of Google.
>
> Like I said, the nosmart operation moves about 780MB of data on the
> initial "bzr branch" command.  The resulting branch directory is
> around 690MB.  With git, I get 632MB and 780MB respectively (but that
> includes all the branches, so it should be larger).
>
>> This is a *lot* of data for something that is supposed to fetch only the
>> trunk of Emacs.
>
> The trunk (or, actually, the first branch you download into a shared
> repository) brings in the bulk of the revision history.  Additional
> branches are mush smaller and their checkouts are much faster, because
> they share a lot of revisions with the trunk, and "bzr branch" then
> brings only the revisions that are not on the trunk.  Try another
> branch command for emacs-23, for example.
>
>> It took 3989.501 real / 581.270 user / 15.990 sys
>> seconds to fetch everything and the resulting .bzr/ directory contains
>> 233 MB of data:
>>
>>   gkeramidas@gkeramidas-glaptop:~/bzr/trunk$ du -sh .bzr/
>>   233M    .bzr/
>
> About an hour is what we can get now, yes (I got as little as 16 min,
> but that was on a system sitting on the same LAN as
> bzr.savannah.gnu.org; otherwise, I get between 0:45 and 1:10).
>
>> This is not very fast, but I'll keep using the bzr branch to see how
>> future updates go.
>
> Please don't forget NOT to use nosmart+ after the initial branch
> command.
>
>

Re: Official Git mirror?

[Vagn Johansen]

Eli Zaretskii <eliz@gnu.org> writes:

>> What is a status of git://git.savannah.gnu.org/emacs.git
>> Note that link to it present on http://savannah.gnu.org/projects/emacs
>
> It's a git mirror of the Emacs repository, intended for those who
> prefer to use git as their VCS.

Is there a recommended way of submitting patches when using the git
mirror?

etc/CONTRIBUTE has no describe this.

-- 
Vagn Johansen

Re: Official Git mirror?

[Eli Zaretskii]

> From: Vagn Johansen <gonz808@hotmail.com>
> Date: Thu, 24 Feb 2011 20:31:48 +0100
> 
> Is there a recommended way of submitting patches when using the git
> mirror?

A simple diff will do.