[git] puplic repo url incorrect or server overloaded?

[git] puplic repo url incorrect or server overloaded?

[H. Dieter Wilhelm]

Hello,

I'm using the address from the EmacsWiki
(http://www.emacswiki.org/emacs/GitForEmacsDevs#toc4)

    git://git.sv.gnu.org/emacs

It seems to be correct since `git remote show ALIAS' shows seemingly the
right content. But with `git fetch ALIAS' I'm getting

      fatal: read error: Connection reset by peer

or sometimes no response but no download seems to happen...

Is this just a sign that the server is overloaded or am I'm doing
something wrong?

Thanks
        Dieter
-- 
Best wishes
H. Dieter Wilhelm
Darmstadt, Germany

Re: [git] puplic repo url incorrect or server overloaded?

[Stephen Berman]

On Thu, 13 Nov 2014 13:43:50 +0100 dieter@duenenhof-wilhelm.de (H. Dieter Wilhelm) wrote:

> Hello,
>
> I'm using the address from the EmacsWiki
> (http://www.emacswiki.org/emacs/GitForEmacsDevs#toc4)
>
>     git://git.sv.gnu.org/emacs
>
> It seems to be correct since `git remote show ALIAS' shows seemingly the
> right content. But with `git fetch ALIAS' I'm getting
>
>       fatal: read error: Connection reset by peer
>
> or sometimes no response but no download seems to happen...
>
> Is this just a sign that the server is overloaded or am I'm doing
> something wrong?

According to http://savannah.gnu.org/git/?group=emacs it should be this:

git clone git://git.savannah.gnu.org/emacs.git

(I assume "sv" and "savannah" are the same machine, but the ".git"
extension may be necessary.  I used the corresponding invocation for
members fours hours ago and it worked find.)

Steve Berman

Re: [git] puplic repo url incorrect or server overloaded?

[H. Dieter Wilhelm]

Stephen Berman <stephen.berman@gmx.net> writes:

> On Thu, 13 Nov 2014 13:43:50 +0100 dieter@duenenhof-wilhelm.de (H. Dieter Wilhelm) wrote:
>
>> Hello,
>>
>> I'm using the address from the EmacsWiki
>> (http://www.emacswiki.org/emacs/GitForEmacsDevs#toc4)
>>
>>     git://git.sv.gnu.org/emacs
>>
>> It seems to be correct since `git remote show ALIAS' shows seemingly the
>> right content. But with `git fetch ALIAS' I'm getting
>>
>>       fatal: read error: Connection reset by peer
>>
>> or sometimes no response but no download seems to happen...
>>
>> Is this just a sign that the server is overloaded or am I'm doing
>> something wrong?
>
> According to http://savannah.gnu.org/git/?group=emacs it should be this:
>
> git clone git://git.savannah.gnu.org/emacs.git

> (I assume "sv" and "savannah" are the same machine, but the ".git"
> extension may be necessary.  I used the corresponding invocation for
> members fours hours ago and it worked find.)

I see, thanks, I'm trying it now with the .git extension...

-- 
Best wishes
H. Dieter Wilhelm
Darmstadt, Germany

Re: [git] puplic repo url incorrect or server overloaded?

[Bob Proulx]

H. Dieter Wilhelm wrote:
> Stephen Berman writes:
> > H. Dieter Wilhelm wrote:
> > >     git://git.sv.gnu.org/emacs
> > > Is this just a sign that the server is overloaded or am I'm doing
> > > something wrong?

The server was overloaded.  I wrote that in my other message but
wanted to address these points separately.

> > According to http://savannah.gnu.org/git/?group=emacs it should be this:
> >
> > git clone git://git.savannah.gnu.org/emacs.git
> 
> > (I assume "sv" and "savannah" are the same machine, 

Yes.  The sv is an alias to savannah.  A typing aid.  The canonical
name is vcs.savannah.gnu.org but any of the aliases to it are fine
too.  You will also see aliases for cvs and bzr and others too.

> > but the ".git" extension may be necessary.
> 
> I see, thanks, I'm trying it now with the .git extension...

Whether a clone has a .git on the end or not is not important.  The
git clone will add a .git to the end automatically.

  git clone git://git.sv.gnu.org/emacs

Is the same as:

  git clone git://git.sv.gnu.org/emacs.git

Therefore I tend to leave it off to shorten things.  Otherwise there
is "git" in the command line about four different times!  At some
point it sounds like South Park's "Marklar marklar marklar".

Your problem was unrelated to the .git on the end or not.  Either way
there is fine.  Your problem was the overloaded server as I wrote
about in my other message.

Bob

Re: [git] tracking instead of cloining; was: puplic repo url incorrect or server overloaded?

[H. Dieter Wilhelm]

Stephen Berman <stephen.berman@gmx.net> writes:

> On Thu, 13 Nov 2014 13:43:50 +0100 dieter@duenenhof-wilhelm.de (H. Dieter Wilhelm) wrote:
>
>> Hello,
>>
>> I'm using the address from the EmacsWiki
>> (http://www.emacswiki.org/emacs/GitForEmacsDevs#toc4)
>>
>>     git://git.sv.gnu.org/emacs
>>
>> It seems to be correct since `git remote show ALIAS' shows seemingly the
>> right content. But with `git fetch ALIAS' I'm getting
>>
>>       fatal: read error: Connection reset by peer
>>
>> or sometimes no response but no download seems to happen...
>>
>> Is this just a sign that the server is overloaded or am I'm doing
>> something wrong?
>
> According to http://savannah.gnu.org/git/?group=emacs it should be this:
>
> git clone git://git.savannah.gnu.org/emacs.git

Right! Cloning is working for me as well.

But I dabbled the last weeks already with the git mirror and now I would
like to track additionally the new Emacs repo with besides my existing
stuff, then rebase my branches and delete the old bzr mirror branches.
To that end I did

  git remote add ALIAS git://git.savannah.gnu.org/emacs.git

and when trying to load it with

  git fetch ALIAS

nothing happens, where am I wrong?

        Dieter
-- 
Best wishes
H. Dieter Wilhelm
Darmstadt, Germany

Re: [git] puplic repo url incorrect or server overloaded?

[Bob Proulx]

H. Dieter Wilhelm wrote:
>     git://git.sv.gnu.org/emacs
> ...
>       fatal: read error: Connection reset by peer
> ...
> Is this just a sign that the server is overloaded or am I'm doing
> something wrong?

There has been some miscommunication all around.  Let me bend
everyone's ear for a moment and fill in some behind the scenes
background information.

Emacs has made another attempt at converting from bzr to git.  I am
one of the volunteers that helps administer the Savannah systems and
have been helping them with this process on the admin side.  

Recently esr made another conversion and uploaded 13G of source to the
git repository.  

  vcs:~# du -sh /srv/git/emacs.git
  13G     /srv/git/emacs.git
    
I had set up the initial empty repository and didn't realize that the
recent upload was 13G.  Then it was announced that was the new
repository.  I didn't know it had been announced that this was the new
source location.  But I did notice that vcs browned out and needed
help.  I rebooted it twice to rescue it.  I had to stop all of the git
daemons at another two times.

Think what happens.  Everyone and their dog tries to download a fresh
copy of the repository.  At 13G each!  The concurrency limit was 40
concurrent processes.  Each of those were trying to download 13G.
That would take hours each.  It starved out all other projects from
being able to access their git repositories.

Worse is that it overloaded the system to the point that it was
inaccessible.  Turns out that it can't handle 40 simultaneous git
downloads of 13G each.  It pushed the system over a tipping point to
where other processes couldn't finish faster than new processes were
started.  The last system load logged was 350 before it stopped
responding at all.

Alerts notified me of the problem.  It was unresponsive.  I and the
FSF admins were in conference about the problem.  We rebooted the
server to rescue it.  Things seemed okay for a bit.  Then the load
average would creep up again.  We lost it again.  I needed to reboot
it again.  Figured out that it was git that was doing it.  I was
forced to disable git in order to keep the system alive.  Shut down
git and kill all of the git downloads.

Of course initially it wasn't known that it was the new 13G emacs
repository that was the problem.  That became apparent only after
digging into the problem.  The git daemons don't log what they are
doing and all of the projects share the pool.  It was just that git
was overloading the system.  I reduced the limits on git resources.  I
added more virtual memory.  I reconfigured overcommit off to avoid the
oom from killing critical processes.  Turned git on again and watched
the process list closely.  Figure out that everyone is downloading
emacs.

During this time your emacs git clone and others were probably getting
reset more than once.  What you were seeing was problems from this.

When we figured out that the emacs repository was new and 13G in size
and that was the problem!  I asked esr to repack it and upload it
again.  I moved the 13G repository out of the way to prevent the
continuing problem.  Kill all of the troubled git downloads and
restart git so that other projects could function again.  esr repacked
the emacs archive and uploaded it again.  The new repacked repository
is now only 200M in size.  As you can imagine that makes a world of
difference!

  vcs:/srv/git/emacs.git# du -sh
  200M    .

Basically now at this time everything is back to operating normally.
I still have the reduced process limits for git in place though.
Because at any time some other project might do the same thing.

This is all discussed at length that is the firehose that is
emacs-devel.  Unfortunately I don't have the time to read it right now
so as a note to others I am only reacting and helping when people CC
me on tasks from there.

And that is the behind the scenes of yesterday.

Bob