"bzr commit" takes 17 minutes?

"bzr commit" takes 17 minutes?

[Paul Eggert]

When I do even trivial bzr commits to the Emacs trunk
it takes an amazingly long time:

> $ time bzr commit -m 'Elaborate my most-recent ChangeLog entry.'
> Committing to: bzr+ssh://eggert@bzr.savannah.gnu.org/emacs/trunk/
> modified ChangeLog
  [here is where the long delay occurs]
> Committed revision 103419.
> 
> real    17m33.115s
> user    0m0.390s
> sys     0m0.170s

Are other committers observing such bad performance?
What could explain the performance problem?

I'm pretty well connected to the Internet,
and have a reasonably fast machine, so I don't
think it's my processor or network connection.
I don't see a similar problem with git commits
to Savannah, for example.

Re: "bzr commit" takes 17 minutes?

[Andreas Schwab]

Paul Eggert <eggert@cs.ucla.edu> writes:

> When I do even trivial bzr commits to the Emacs trunk
> it takes an amazingly long time:

Probably a repack on the server.  Unlike git, bzr does not know how to
reuse deltas.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: "bzr commit" takes 17 minutes?

[Eli Zaretskii]

> Date: Fri, 25 Feb 2011 02:24:47 -0800
> From: Paul Eggert <eggert@cs.ucla.edu>
> 
> When I do even trivial bzr commits to the Emacs trunk
> it takes an amazingly long time:
> 
> > $ time bzr commit -m 'Elaborate my most-recent ChangeLog entry.'
> > Committing to: bzr+ssh://eggert@bzr.savannah.gnu.org/emacs/trunk/
> > modified ChangeLog
>   [here is where the long delay occurs]
> > Committed revision 103419.
> > 
> > real    17m33.115s
> > user    0m0.390s
> > sys     0m0.170s

Can you show the corresponding trace from your ~/.bzr.log?

> Are other committers observing such bad performance?

Not that long, never.  A commit is typically 1-2 min for me.  Which is
also slow, but not as bad as 17 min.  Is the above time exceptional,
or do you get that as a matter of routine?  You can gather statistics
from your ~/.bzr.log file.

Re: "bzr commit" takes 17 minutes?

[Eli Zaretskii]

> From: Andreas Schwab <schwab@linux-m68k.org>
> Date: Fri, 25 Feb 2011 11:47:00 +0100
> Cc: Emacs Development <emacs-devel@gnu.org>
> 
> Paul Eggert <eggert@cs.ucla.edu> writes:
> 
> > When I do even trivial bzr commits to the Emacs trunk
> > it takes an amazingly long time:
> 
> Probably a repack on the server.

Only if it happens relatively rarely, but not if this time is typical.

Re: "bzr commit" takes 17 minutes?

[Eli Zaretskii]

> Date: Fri, 25 Feb 2011 13:05:53 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: emacs-devel@gnu.org
> 
> Not that long, never.  A commit is typically 1-2 min for me.  Which is
> also slow, but not as bad as 17 min.  Is the above time exceptional,
> or do you get that as a matter of routine?  You can gather statistics
> from your ~/.bzr.log file.

Something _is_ strange with bzr.savanna.gnu.org.  It just took me a
full minute to get a single revision with "bzr up", something that
should take 20 sec at most.  And "bzr up" in the emacs-23 branch took
2.5 minutes(!).

It worked fine just a couple of hours ago.

I'll ask on Savannah-hackers-public about this.

Re: "bzr commit" takes 17 minutes?

[Eli Zaretskii]

> Date: Fri, 25 Feb 2011 13:23:01 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: 
> 
> > Date: Fri, 25 Feb 2011 13:05:53 +0200
> > From: Eli Zaretskii <eliz@gnu.org>
> > Cc: emacs-devel@gnu.org
> > 
> > Not that long, never.  A commit is typically 1-2 min for me.  Which is
> > also slow, but not as bad as 17 min.  Is the above time exceptional,
> > or do you get that as a matter of routine?  You can gather statistics
> > from your ~/.bzr.log file.
> 
> Something _is_ strange with bzr.savanna.gnu.org.  It just took me a
> full minute to get a single revision with "bzr up", something that
> should take 20 sec at most.  And "bzr up" in the emacs-23 branch took
> 2.5 minutes(!).

I just committed a small change to the emacs-23 branch, and
experienced the same 17-minute time.  From my .bzr.log:

  [ 4564] 2011-02-25 13:47:21.246 INFO: Committed revision 100508.
  1019.921  Committed revid eliz@gnu.org-20110225113047-ah523retq71cj6ca as revno 100508.
  1020.296  Transferred: 310kB (0.3kB/s r:137kB w:173kB)

17 minutes to write 170KB!

I would guess some bad CPU or other resource problem on the server,
because when it starts sending data downstream, I see my normal
network throughput figures (hundreds of KB/sec).  So it doesn't look
like network congestion.

Re: "bzr commit" takes 17 minutes?

[Eli Zaretskii]

> From: Eli Zaretskii <eliz@gnu.org>
> Cc: 
> 
> I just committed a small change to the emacs-23 branch, and
> experienced the same 17-minute time.  From my .bzr.log:
> 
>   [ 4564] 2011-02-25 13:47:21.246 INFO: Committed revision 100508.
>   1019.921  Committed revid eliz@gnu.org-20110225113047-ah523retq71cj6ca as revno 100508.
>   1020.296  Transferred: 310kB (0.3kB/s r:137kB w:173kB)
> 
> 17 minutes to write 170KB!
> 
> I would guess some bad CPU or other resource problem on the server,
> because when it starts sending data downstream, I see my normal
> network throughput figures (hundreds of KB/sec).  So it doesn't look
> like network congestion.

And this is what Jim Meyering found out:

> Whoa.
> The load average was above 90:
> 
>   top - 11:40:56 up 65 days,  8:59,  3 users,  load average: 94.12, 90.53, 81.92
>   Tasks: 1545 total,  43 running, 1500 sleeping,   1 stopped,   1 zombie
>   Cpu(s): 69.9%us, 29.8%sy,  0.2%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.2%si,  0.0%st
>   Mem:   6291652k total,  4883792k used,  1407860k free,   128692k buffers
>   Swap: 12582904k total,  7998212k used,  4584692k free,  1229580k cached
> 
> I suspect a cracking attempt, since there were lots of cgit.cgi
> scripts running and many cgit.cgi segfaults logged in dmesg.
> 
> I've made the cgit.cgi script non-executable for now and
> killed all the running cgit.cgi processes, and things seem
> to be returning to normal.

If this slowness returns, please report to
savannah-hackers-public@gnu.org, in the thread that I started (and Jim
replied).

Re: "bzr commit" takes 17 minutes?

[Thierry Volpiatto]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Fri, 25 Feb 2011 13:23:01 +0200
>> From: Eli Zaretskii <eliz@gnu.org>
>> Cc: 
>> 
>> > Date: Fri, 25 Feb 2011 13:05:53 +0200
>> > From: Eli Zaretskii <eliz@gnu.org>
>> > Cc: emacs-devel@gnu.org
>> > 
>> > Not that long, never.  A commit is typically 1-2 min for me.  Which is
>> > also slow, but not as bad as 17 min.  Is the above time exceptional,
>> > or do you get that as a matter of routine?  You can gather statistics
>> > from your ~/.bzr.log file.
>> 
>> Something _is_ strange with bzr.savanna.gnu.org.  It just took me a
>> full minute to get a single revision with "bzr up", something that
>> should take 20 sec at most.  And "bzr up" in the emacs-23 branch took
>> 2.5 minutes(!).
>
> I just committed a small change to the emacs-23 branch, and
> experienced the same 17-minute time.  From my .bzr.log:
>
>   [ 4564] 2011-02-25 13:47:21.246 INFO: Committed revision 100508.
>   1019.921  Committed revid eliz@gnu.org-20110225113047-ah523retq71cj6ca as revno 100508.
>   1020.296  Transferred: 310kB (0.3kB/s r:137kB w:173kB)
>
> 17 minutes to write 170KB!
>
> I would guess some bad CPU or other resource problem on the server,
> because when it starts sending data downstream, I see my normal
> network throughput figures (hundreds of KB/sec).  So it doesn't look
> like network congestion.
>
I had a (another) try with bzr this morning to get emacs24 from my laptop:

--8<---------------cut here---------------start------------->8---
thierry@thierry-MM061:~/download$ time bzr branch nosmart+bzr://bzr.savannah.gnu.org/emacs/trunk emacs-24-bzr
^Cbzr: interrupted - Fetching revisions:Inserting stream:Estimate 692419/1165640

real    43m55.394s
user    10m31.447s
sys     0m21.089s
--8<---------------cut here---------------end--------------->8---

I didn't have time to finish and abort after 43mn...

-- 
A+ Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997 

Re: "bzr commit" takes 17 minutes?

[Glenn Morris]

Eli Zaretskii wrote:

>> I suspect a cracking attempt, since there were lots of cgit.cgi
>> scripts running and many cgit.cgi segfaults logged in dmesg.

Damn git, slowing down my bzr connections...

;)

Re: "bzr commit" takes 17 minutes?

[Eli Zaretskii]

> From: Thierry Volpiatto <thierry.volpiatto@gmail.com>
> Date: Fri, 25 Feb 2011 18:41:17 +0100
> 
> I had a (another) try with bzr this morning to get emacs24 from my laptop:

You chose a bad day to try...

> --8<---------------cut here---------------start------------->8---
> thierry@thierry-MM061:~/download$ time bzr branch nosmart+bzr://bzr.savannah.gnu.org/emacs/trunk emacs-24-bzr
> ^Cbzr: interrupted - Fetching revisions:Inserting stream:Estimate 692419/1165640
> 
> real    43m55.394s
> user    10m31.447s
> sys     0m21.089s

Try again, it's back to normal now.

And for the initial "bzr branch", 44 minutes is not too much, at least
as long as the current version of bzr is not upgraded on Savannah.  It
takes around 1 hour.

Re: "bzr commit" takes 17 minutes?

[Thierry Volpiatto]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Thierry Volpiatto <thierry.volpiatto@gmail.com>
>> Date: Fri, 25 Feb 2011 18:41:17 +0100
>> 
>> I had a (another) try with bzr this morning to get emacs24 from my laptop:
>
> You chose a bad day to try...
>
>> --8<---------------cut here---------------start------------->8---
>> thierry@thierry-MM061:~/download$ time bzr branch nosmart+bzr://bzr.savannah.gnu.org/emacs/trunk emacs-24-bzr
>> ^Cbzr: interrupted - Fetching revisions:Inserting stream:Estimate 692419/1165640
>> 
>> real    43m55.394s
>> user    10m31.447s
>> sys     0m21.089s
>
> Try again, it's back to normal now.
Thanks.

> And for the initial "bzr branch", 44 minutes is not too much, at least
> as long as the current version of bzr is not upgraded on Savannah.  It
> takes around 1 hour.
Yes, but when i stop at 44mn, i was at: Estimate 692419/1165640.
But well, if bzr is not updgraded on savannah servers, this explain all.

-- 
A+ Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997 

Savannah slowdown due to cgit.cgi [was Re: "bzr commit" takes 17 minutes?]

[Glenn Morris]

>> I've made the cgit.cgi script non-executable for now and
>> killed all the running cgit.cgi processes, and things seem
>> to be returning to normal.

Maybe it should be replaced with a little note "disabled due to
instability; please contact us if you are interested in maintaining it."

Cough, cough.

Re: "bzr commit" takes 17 minutes?

[Eli Zaretskii]

> From: Thierry Volpiatto <thierry.volpiatto@gmail.com>
> Cc: emacs-devel@gnu.org
> Date: Fri, 25 Feb 2011 19:13:09 +0100
> 
> > And for the initial "bzr branch", 44 minutes is not too much, at least
> > as long as the current version of bzr is not upgraded on Savannah.  It
> > takes around 1 hour.
> Yes, but when i stop at 44mn, i was at: Estimate 692419/1165640.

The 1165640 figure is an overestimation, it actually finishes at 700MB
plus change.

Re: Savannah slowdown due to cgit.cgi [was Re: "bzr commit" takes 17 minutes?]

[Andrew W. Nosenko]

On Fri, Feb 25, 2011 at 20:30, Glenn Morris <rgm@gnu.org> wrote:
>
>>> I've made the cgit.cgi script non-executable for now and
>>> killed all the running cgit.cgi processes, and things seem
>>> to be returning to normal.
>
> Maybe it should be replaced with a little note "disabled due to
> instability; please contact us if you are interested in maintaining it."

As far, as I understand, You need to disable not cgit itself, but
cloning through cgit.
    http://permalink.gmane.org/gmane.comp.gnome.devel.announce/155

-- 
Andrew W. Nosenko <andrew.w.nosenko@gmail.com>

Re: "bzr commit" takes 17 minutes?

[Miles Bader]

Glenn Morris <rgm@gnu.org> writes:
>>> I suspect a cracking attempt, since there were lots of cgit.cgi
>>> scripts running and many cgit.cgi segfaults logged in dmesg.
>
> Damn git, slowing down my bzr connections...

Incidentally, I'd hope that cgit.cgi runs without write access to
anything (except temp storage); anyone know...?

-Miles

-- 
「すっごい」と呟いてる。「へんてこなもんばっかり」
「そんなにへんてこ？」
「へんてこへんてこ」
そう言われると見たくなってしまう。

Re: Savannah slowdown due to cgit.cgi [was Re: "bzr commit" takes 17 minutes?]

[Eli Zaretskii]

> Date: Sat, 26 Feb 2011 02:45:49 +0200
> From: "Andrew W. Nosenko" <andrew.w.nosenko@gmail.com>
> Cc: Eli Zaretskii <eliz@gnu.org>, eggert@cs.ucla.edu, emacs-devel@gnu.org
> 
> On Fri, Feb 25, 2011 at 20:30, Glenn Morris <rgm@gnu.org> wrote:
> >
> >>> I've made the cgit.cgi script non-executable for now and
> >>> killed all the running cgit.cgi processes, and things seem
> >>> to be returning to normal.
> >
> > Maybe it should be replaced with a little note "disabled due to
> > instability; please contact us if you are interested in maintaining it."
> 
> As far, as I understand, You need to disable not cgit itself, but
> cloning through cgit.
>     http://permalink.gmane.org/gmane.comp.gnome.devel.announce/155

You are missing the subtle humor in Glenn's message.

It's useless to discuss this seriously here, since no one of the
people who administrate Savannah are listening.