bzr + savannah.gnu.org slow again

bzr + savannah.gnu.org slow again

[Paul Eggert]

Savannah.gnu.org is very slow for bzr commits on the Emacs trunk,
with symptoms much like they were about 24 hours ago.  Perhaps the
same thing is happening again?

My most recent attempt to commit is taking more than 20 minutes.
It's not done yet, so I don't know how long it will take exactly.

Here are the symptoms at the shell level:

$ bzr commit -m 'Merge: lib-src changes mostly to avoid GCC warnings'
Committing to: bzr+ssh://eggert@bzr.savannah.gnu.org/emacs/trunk/
modified lib-src/ChangeLog
modified lib-src/ebrowse.c
modified lib-src/emacsclient.c
modified lib-src/etags.c
modified lib-src/fakemail.c
modified lib-src/make-docfile.c
[ It's hanging here now. ]


Here is my ~/.bzr.log:

Fri 2011-02-25 21:53:48 -0800
0.073  bazaar version: 2.2.2
0.073  bzr arguments: [u'commit', u'-m', u'Merge: lib-src changes mostly to avoid GCC warnings']
0.096  looking for plugins in /u/cs/fac/eggert/.bazaar/plugins
0.096  looking for plugins in /usr/local/cs/Python-2.6.6/lib/python2.6/site-packages/bzrlib/plugins
0.117  encoding stdout as sys.stdout encoding 'UTF-8'
0.245  opening working tree '/w/fac.2/cs/eggert/src/gnu/emacs/trunk'
0.566  ssh implementation is OpenSSH
38.887  preparing to commit
[10929] 2011-02-25 21:54:36.575 INFO: Committing to: bzr+ssh://eggert@bzr.savannah.gnu.org/emacs/trunk/
47.711  Selecting files for commit with filter None
[10929] 2011-02-25 21:54:37.032 INFO: modified lib-src/ChangeLog
[10929] 2011-02-25 21:54:37.033 INFO: modified lib-src/ebrowse.c
[10929] 2011-02-25 21:54:37.033 INFO: modified lib-src/emacsclient.c
[10929] 2011-02-25 21:54:37.033 INFO: modified lib-src/etags.c
[10929] 2011-02-25 21:54:37.034 INFO: modified lib-src/fakemail.c
[10929] 2011-02-25 21:54:37.034 INFO: modified lib-src/make-docfile.c
48.761  Using fetch logic to copy between CHKInventoryRepository('file:///w/fac.2/cs/eggert/src/gnu/emacs/.bzr/repository/')(RepositoryFormat2a()) and RemoteRepository(bzr+ssh://eggert@bzr.savannah.gnu.org/emacs/.bzr/)(RemoteRepositoryFormat(_network_name='Bazaar repository format 2a (needs bzr 1.16 or later)\n'))
48.761  fetch up to rev {eggert@cs.ucla.edu-20110226055436-mfvjfrh702e68yqk}
142.288  stripping trailing bytes from groupcompress block 627289 => 258611


It is now Fri Feb 25 22:20:20 PST 2011, so it's about
27 minutes and counting.

Re: bzr + savannah.gnu.org slow again

[Paul Eggert]

Following up on my previous message, my commit to the Emacs trunk
took over 50 minutes (!).

Here's the finished .bzr.log entry:

Fri 2011-02-25 21:53:48 -0800
0.073  bazaar version: 2.2.2
0.073  bzr arguments: [u'commit', u'-m', u'Merge: lib-src changes mostly to avoid GCC warnings']
0.096  looking for plugins in /u/cs/fac/eggert/.bazaar/plugins
0.096  looking for plugins in /usr/local/cs/Python-2.6.6/lib/python2.6/site-packages/bzrlib/plugins
0.117  encoding stdout as sys.stdout encoding 'UTF-8'
0.245  opening working tree '/w/fac.2/cs/eggert/src/gnu/emacs/trunk'
0.566  ssh implementation is OpenSSH
38.887  preparing to commit
[10929] 2011-02-25 21:54:36.575 INFO: Committing to: bzr+ssh://eggert@bzr.savannah.gnu.org/emacs/trunk/
47.711  Selecting files for commit with filter None
[10929] 2011-02-25 21:54:37.032 INFO: modified lib-src/ChangeLog
[10929] 2011-02-25 21:54:37.033 INFO: modified lib-src/ebrowse.c
[10929] 2011-02-25 21:54:37.033 INFO: modified lib-src/emacsclient.c
[10929] 2011-02-25 21:54:37.033 INFO: modified lib-src/etags.c
[10929] 2011-02-25 21:54:37.034 INFO: modified lib-src/fakemail.c
[10929] 2011-02-25 21:54:37.034 INFO: modified lib-src/make-docfile.c
48.761  Using fetch logic to copy between CHKInventoryRepository('file:///w/fac.2/cs/eggert/src/gnu/emacs/.bzr/repository/')(RepositoryFormat2a()) and RemoteRepository(bzr+ssh://eggert@bzr.savannah.gnu.org/emacs/.bzr/)(RemoteRepositoryFormat(_network_name='Bazaar repository format 2a (needs bzr 1.16 or later)\n'))
48.761  fetch up to rev {eggert@cs.ucla.edu-20110226055436-mfvjfrh702e68yqk}
142.288  stripping trailing bytes from groupcompress block 627289 => 258611
[10929] 2011-02-25 22:43:50.604 INFO: Committed revision 103424.
3001.740  Committed revid eggert@cs.ucla.edu-20110226055436-mfvjfrh702e68yqk as revno 103424.
3002.897  return code 0

Re: bzr + savannah.gnu.org slow again

[Eli Zaretskii]

> Date: Fri, 25 Feb 2011 22:50:33 -0800
> From: Paul Eggert <eggert@cs.ucla.edu>
> Cc: Savannah-hackers-public@gnu.org
> 
> Following up on my previous message, my commit to the Emacs trunk
> took over 50 minutes (!).

I get "only" 5.5 minutes here, about 2 hours after Paul's commit:

  331.016  Committed revid eliz@gnu.org-20110226074438-z9zfono0mhgzr4ri as revno 103425.
  331.547  Transferred: 244kB (0.7kB/s r:137kB w:107kB)

But something's definitely wrong again.  "bzr up" again took 3.5
minutes, which is about 4-5 times longer than it usually takes for me.

Re: [Savannah-hackers-public] Re: bzr + savannah.gnu.org slow again

[Jim Meyering]

Eli Zaretskii wrote:
>> Date: Fri, 25 Feb 2011 22:50:33 -0800
>> From: Paul Eggert <eggert@cs.ucla.edu>
>> Cc: Savannah-hackers-public@gnu.org
>>
>> Following up on my previous message, my commit to the Emacs trunk
>> took over 50 minutes (!).
>
> I get "only" 5.5 minutes here, about 2 hours after Paul's commit:
>
>   331.016 Committed revid eliz@gnu.org-20110226074438-z9zfono0mhgzr4ri
> as revno 103425.
>   331.547  Transferred: 244kB (0.7kB/s r:137kB w:107kB)
>
> But something's definitely wrong again.  "bzr up" again took 3.5
> minutes, which is about 4-5 times longer than it usually takes for me.

Load average was 150 -- mostly due to cgit.cgi processes.
I killed them, again.  It may be robots that need to be told to go away.

FYI, now we're using the latest cgit: 0.8.3.4.

Re: [Savannah-hackers-public] Re: bzr + savannah.gnu.org slow again

[Eli Zaretskii]

> From: Jim Meyering <jim@meyering.net>
> Cc: Paul Eggert <eggert@cs.ucla.edu>,  Savannah-hackers-public@gnu.org,  emacs-devel@gnu.org
> Date: Sat, 26 Feb 2011 09:13:50 +0100
> 
> Eli Zaretskii wrote:
> >> Date: Fri, 25 Feb 2011 22:50:33 -0800
> >> From: Paul Eggert <eggert@cs.ucla.edu>
> >> Cc: Savannah-hackers-public@gnu.org
> >>
> >> Following up on my previous message, my commit to the Emacs trunk
> >> took over 50 minutes (!).
> >
> > I get "only" 5.5 minutes here, about 2 hours after Paul's commit:
> >
> >   331.016 Committed revid eliz@gnu.org-20110226074438-z9zfono0mhgzr4ri
> > as revno 103425.
> >   331.547  Transferred: 244kB (0.7kB/s r:137kB w:107kB)
> >
> > But something's definitely wrong again.  "bzr up" again took 3.5
> > minutes, which is about 4-5 times longer than it usually takes for me.
> 
> Load average was 150 -- mostly due to cgit.cgi processes.
> I killed them, again.  It may be robots that need to be told to go away.

Thanks.  I still get very slow commits, half an hour after you did it:

  41.953  fetch up to rev {eliz@gnu.org-20110226083656-wtcszqlzmytktl8w}
  [ 1288] 2011-02-26 10:43:20.466 INFO: Committed revision 103426.
  393.406  Committed revid eliz@gnu.org-20110226083656-wtcszqlzmytktl8w as revno 103426.
  393.812  Transferred: 307kB (0.8kB/s r:137kB w:170kB)

That's 6.5 minutes, with most of it sitting and waiting for the commit
to complete on the server.

So either the problem is back, or it takes bzr.savannah.gnu.org some
time to recover.

Re: [Savannah-hackers-public] Re: bzr + savannah.gnu.org slow again

[Eli Zaretskii]

> Date: Sat, 26 Feb 2011 10:49:33 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: eggert@cs.ucla.edu, Savannah-hackers-public@gnu.org, emacs-devel@gnu.org
> 
> Thanks.  I still get very slow commits, half an hour after you did it:
> 
>   41.953  fetch up to rev {eliz@gnu.org-20110226083656-wtcszqlzmytktl8w}
>   [ 1288] 2011-02-26 10:43:20.466 INFO: Committed revision 103426.
>   393.406  Committed revid eliz@gnu.org-20110226083656-wtcszqlzmytktl8w as revno 103426.
>   393.812  Transferred: 307kB (0.8kB/s r:137kB w:170kB)
> 
> That's 6.5 minutes, with most of it sitting and waiting for the commit
> to complete on the server.

Btw, git.savannah.gnu.org is also affected: it took me a whopping 5
minutes to do a "git pull" (from gnulib) that said it's pulling 1837
objects.

Re: [Savannah-hackers-public] Re: bzr + savannah.gnu.org slow again

[Glenn Morris]

At 23.30 PST on Sunday 27 Feb, it is very slow again. ~ 35 mins for a ~ 5
line commit. It was alright mid-afternoon PST.

Can you not just disable the cgit service (it seems to still be
available) until there is a solution for this problem? Being able to
commit is more important.

Re: [Savannah-hackers-public] Re: bzr + savannah.gnu.org slow again

[Jim Meyering]

Glenn Morris wrote:
> At 23.30 PST on Sunday 27 Feb, it is very slow again. ~ 35 mins for a ~ 5
> line commit. It was alright mid-afternoon PST.
>
> Can you not just disable the cgit service (it seems to still be
> available) until there is a solution for this problem? Being able to
> commit is more important.

Hi Glenn,

I've diagnosed and fixed the problem.  git.sv is now running a new binary.
The trigger was a buggy spider that was provoking an infloop in cgit.
Once there were 15-20 cgit.cgi process stuck in this infloop, the
system would become essentially unusable.

I've just fixed the cgit bug and reported it privately to upstream
and as http://bugzilla.redhat.com/680905 (private, for now).

Re: [Savannah-hackers-public] Re: bzr + savannah.gnu.org slow again

[Glenn Morris]

Jim Meyering wrote:

> I've diagnosed and fixed the problem.

Great, thank you.