bzr not working?

bzr not working?

[Tim Cross]

This morning I am unable to access the bzr repository. When attmepting
with native bzr method, I get



tcross@blind-mole:~/bzr/emacs/latest$ bzr pull
Using saved parent location: bzr://bzr.savannah.gnu.org/emacs/trunk/
bzr: ERROR: Connection closed: Unexpected end of message. Please check
connectivity and permissions, and report a bug if problems persist.

and if I try using the http method, I get

tcross@blind-mole:~/bzr/emacs/trunk$ bzr pull
Using saved parent location: http://bzr.savannah.gnu.org/r/emacs/trunk/
bzr: ERROR: Cannot lock
LockDir(filtered-161262092:///emacs/trunk/.bzr/branch/lock):
Permission denied:
"/srv/bzr/emacs/trunk/.bzr/branch/lock/w447esrl9l.tmp": [Errno 13]
Permission denied:
'/srv/bzr/emacs/trunk/.bzr/branch/lock/w447esrl9l.tmp'

Are others having problems and if so, does it need to be reported to
savannah sys admins or has that been done?

Tim


-- 
Tim Cross
Phone: 0428 212 217

Re: bzr not working?

[Carsten Mattner]

On Mon, Oct 10, 2011 at 12:24 AM, Tim Cross <theophilusx@gmail.com> wrote:
> This morning I am unable to access the bzr repository. When attmepting
> with native bzr method, I get
>
>
>
> tcross@blind-mole:~/bzr/emacs/latest$ bzr pull
> Using saved parent location: bzr://bzr.savannah.gnu.org/emacs/trunk/
> bzr: ERROR: Connection closed: Unexpected end of message. Please check
> connectivity and permissions, and report a bug if problems persist.

 bzr://bzr.savannah.gnu.org/emacs/trunk/ works for me.

Re: bzr not working?

[Tim Cross]

On Mon, Oct 10, 2011 at 9:41 AM, Carsten Mattner
<carstenmattner@googlemail.com> wrote:
> On Mon, Oct 10, 2011 at 12:24 AM, Tim Cross <theophilusx@gmail.com> wrote:
>> This morning I am unable to access the bzr repository. When attmepting
>> with native bzr method, I get
>>
>>
>>
>> tcross@blind-mole:~/bzr/emacs/latest$ bzr pull
>> Using saved parent location: bzr://bzr.savannah.gnu.org/emacs/trunk/
>> bzr: ERROR: Connection closed: Unexpected end of message. Please check
>> connectivity and permissions, and report a bug if problems persist.
>
>  bzr://bzr.savannah.gnu.org/emacs/trunk/ works for me.
>

Hmmm, well I cannot get it to work. I even tried a completely fresh
checkout and still get the same error messag

The command I'm using for a completely new checkout is

bzr branch bzr://bzr.savannah.gnu.org/emacs/trunk

is something wrong/missing?

Tim




-- 
Tim Cross
Phone: 0428 212 217

Re: bzr not working?

[Eli Zaretskii]

> Date: Mon, 10 Oct 2011 09:24:47 +1100
> From: Tim Cross <theophilusx@gmail.com>
> 
> This morning I am unable to access the bzr repository.

If it still doesn't work when you read this, please post the
information to savannah-hackers-public@gnu.org.

Re: bzr not working?

[Tim Cross]

On Mon, Oct 10, 2011 at 2:56 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>> Date: Mon, 10 Oct 2011 09:24:47 +1100
>> From: Tim Cross <theophilusx@gmail.com>
>>
>> This morning I am unable to access the bzr repository.
>
> If it still doesn't work when you read this, please post the
> information to savannah-hackers-public@gnu.org.
>

OK, thanks Eli. have sent them a message with all the info I have.

Tim


-- 
Tim Cross
Phone: 0428 212 217

Re: bzr not working?

[Eli Zaretskii]

> Date: Mon, 10 Oct 2011 16:01:14 +1100
> From: Tim Cross <theophilusx@gmail.com>
> Cc: emacs-devel@gnu.org
> 
> On Mon, Oct 10, 2011 at 2:56 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> >> Date: Mon, 10 Oct 2011 09:24:47 +1100
> >> From: Tim Cross <theophilusx@gmail.com>
> >>
> >> This morning I am unable to access the bzr repository.
> >
> > If it still doesn't work when you read this, please post the
> > information to savannah-hackers-public@gnu.org.
> >
> 
> OK, thanks Eli. have sent them a message with all the info I have.

FWIW, I just tried the same command as you showed, and it worked just
fine for me.

Re: bzr not working?

[Tim Cross]

On Mon, Oct 10, 2011 at 4:59 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>> Date: Mon, 10 Oct 2011 16:01:14 +1100
>> From: Tim Cross <theophilusx@gmail.com>
>> Cc: emacs-devel@gnu.org
>>
>> On Mon, Oct 10, 2011 at 2:56 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>> >> Date: Mon, 10 Oct 2011 09:24:47 +1100
>> >> From: Tim Cross <theophilusx@gmail.com>
>> >>
>> >> This morning I am unable to access the bzr repository.
>> >
>> > If it still doesn't work when you read this, please post the
>> > information to savannah-hackers-public@gnu.org.
>> >
>>
>> OK, thanks Eli. have sent them a message with all the info I have.
>
> FWIW, I just tried the same command as you showed, and it worked just
> fine for me.
>

OK, so that makes two who can issue the same command as me and it
would seem only me who cannot.

Now the question is why does it work for you and CD but not me? Could
there be some sort of cached authentication that is making it work for
you guys and not me? I will try when I get home from my home network
and see if it works there.

Tim



-- 
Tim Cross

Re: bzr not working?

[Eli Zaretskii]

> Date: Mon, 10 Oct 2011 17:08:54 +1100
> From: Tim Cross <theophilusx@gmail.com>
> Cc: emacs-devel@gnu.org
> 
> Now the question is why does it work for you and CD but not me? Could
> there be some sort of cached authentication that is making it work for
> you guys and not me?

AFAIK, there's no authentication when you access the server via the
bzr:// protocol.

Try running the same command with the -Dhpss switch, i.e.

  bzr -Dhpss pull bzr://bzr.savannah.gnu.org/emacs/trunk/

Then look in your ~/.bzr.log file for clues.  savannah-hackers-public
guys will probably request this info anyway.

Re: bzr not working?

[Eli Zaretskii]

> Date: Mon, 10 Oct 2011 17:08:54 +1100
> From: Tim Cross <theophilusx@gmail.com>
> Cc: emacs-devel@gnu.org
> 
> OK, so that makes two who can issue the same command as me and it
> would seem only me who cannot.

I "succeeded" to reproduce this on one machine to which I have
access.  So you are not alone.  Posted the info to
savannah-hackers-public.

Re: bzr not working?

[Tim Cross]

On Mon, Oct 10, 2011 at 6:39 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>> Date: Mon, 10 Oct 2011 17:08:54 +1100
>> From: Tim Cross <theophilusx@gmail.com>
>> Cc: emacs-devel@gnu.org
>>
>> OK, so that makes two who can issue the same command as me and it
>> would seem only me who cannot.
>
> I "succeeded" to reproduce this on one machine to which I have
> access.  So you are not alone.  Posted the info to
> savannah-hackers-public.
>

Got home and tried it on my home machine. Exactly same setup, using 64
bit xubuntu instead of 32 bit and all works fine with no problems! On
the other machine, I was able to access other bzr repositories i.e.
the VM repo on launchpad, with no problems, so whatever the issue is,
it seems specific to the emacs repo on savannah (and maybe other bzr
repos on savannah???).

Will try your diagnostic suggestions tomorrow when I return to work.
Thanks for the help and thanks for indicating that perhaps its not
just me!

Tim


-- 
Tim Cross
Phone: 0428 212 217

Re: bzr not working?

[Eli Zaretskii]

> Date: Mon, 10 Oct 2011 19:00:34 +1100
> From: Tim Cross <theophilusx@gmail.com>
> Cc: emacs-devel@gnu.org
> 
> On Mon, Oct 10, 2011 at 6:39 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> >> Date: Mon, 10 Oct 2011 17:08:54 +1100
> >> From: Tim Cross <theophilusx@gmail.com>
> >> Cc: emacs-devel@gnu.org
> >>
> >> OK, so that makes two who can issue the same command as me and it
> >> would seem only me who cannot.
> >
> > I "succeeded" to reproduce this on one machine to which I have
> > access.  So you are not alone.  Posted the info to
> > savannah-hackers-public.
> >
> 
> Got home and tried it on my home machine. Exactly same setup, using 64
> bit xubuntu instead of 32 bit and all works fine with no problems! On
> the other machine, I was able to access other bzr repositories i.e.
> the VM repo on launchpad, with no problems, so whatever the issue is,
> it seems specific to the emacs repo on savannah (and maybe other bzr
> repos on savannah???).

Is it possible that you got the error after approximately 1 hour?  Or
was it almost immediate?  For "bzr pull", I'd expect the latter, but a
full initial "bzr branch" can take an hour, depending on your
connection speed.

Re: bzr not working?

[Tim Cross]

On Mon, Oct 10, 2011 at 7:27 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>> Date: Mon, 10 Oct 2011 19:00:34 +1100
>> From: Tim Cross <theophilusx@gmail.com>
>> Cc: emacs-devel@gnu.org
>>
>> On Mon, Oct 10, 2011 at 6:39 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>> >> Date: Mon, 10 Oct 2011 17:08:54 +1100
>> >> From: Tim Cross <theophilusx@gmail.com>
>> >> Cc: emacs-devel@gnu.org
>> >>
>> >> OK, so that makes two who can issue the same command as me and it
>> >> would seem only me who cannot.
>> >
>> > I "succeeded" to reproduce this on one machine to which I have
>> > access.  So you are not alone.  Posted the info to
>> > savannah-hackers-public.
>> >
>>
>> Got home and tried it on my home machine. Exactly same setup, using 64
>> bit xubuntu instead of 32 bit and all works fine with no problems! On
>> the other machine, I was able to access other bzr repositories i.e.
>> the VM repo on launchpad, with no problems, so whatever the issue is,
>> it seems specific to the emacs repo on savannah (and maybe other bzr
>> repos on savannah???).
>
> Is it possible that you got the error after approximately 1 hour?  Or
> was it almost immediate?  For "bzr pull", I'd expect the latter, but a
> full initial "bzr branch" can take an hour, depending on your
> connection speed.
>

I'm on a fairly fast link, so I would expect a bzr branch to only take
about 25-30 min max (going from memory).

The current error occurs quite quickly in both cases - no real
difference in time between issuing the commands and getting the error
for either the pull or branch.

Tim


-- 
Tim Cross

Re: bzr not working?

[Tim Cross]

Has anyone heard anything more regarding a resolution for this issue?
I've heard nothing from the savannah admins and stil cannot access the
bzr repository.

Given the recent discussions regarding git and if git is now being
updated regularly, I'm thinking maybe the simplest solution is to just
switch to git as it has better emacs integration anyway.

Tim


On Mon, Oct 10, 2011 at 10:53 PM, Tim Cross <theophilusx@gmail.com> wrote:
> On Mon, Oct 10, 2011 at 7:27 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>>> Date: Mon, 10 Oct 2011 19:00:34 +1100
>>> From: Tim Cross <theophilusx@gmail.com>
>>> Cc: emacs-devel@gnu.org
>>>
>>> On Mon, Oct 10, 2011 at 6:39 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>>> >> Date: Mon, 10 Oct 2011 17:08:54 +1100
>>> >> From: Tim Cross <theophilusx@gmail.com>
>>> >> Cc: emacs-devel@gnu.org
>>> >>
>>> >> OK, so that makes two who can issue the same command as me and it
>>> >> would seem only me who cannot.
>>> >
>>> > I "succeeded" to reproduce this on one machine to which I have
>>> > access.  So you are not alone.  Posted the info to
>>> > savannah-hackers-public.
>>> >
>>>
>>> Got home and tried it on my home machine. Exactly same setup, using 64
>>> bit xubuntu instead of 32 bit and all works fine with no problems! On
>>> the other machine, I was able to access other bzr repositories i.e.
>>> the VM repo on launchpad, with no problems, so whatever the issue is,
>>> it seems specific to the emacs repo on savannah (and maybe other bzr
>>> repos on savannah???).
>>
>> Is it possible that you got the error after approximately 1 hour?  Or
>> was it almost immediate?  For "bzr pull", I'd expect the latter, but a
>> full initial "bzr branch" can take an hour, depending on your
>> connection speed.
>>
>
> I'm on a fairly fast link, so I would expect a bzr branch to only take
> about 25-30 min max (going from memory).
>
> The current error occurs quite quickly in both cases - no real
> difference in time between issuing the commands and getting the error
> for either the pull or branch.
>
> Tim
>
>
> --
> Tim Cross
>



-- 
Tim Cross
Phone: 0428 212 217

Re: bzr not working?

[Eli Zaretskii]

> Date: Wed, 12 Oct 2011 10:35:13 +1100
> From: Tim Cross <theophilusx@gmail.com>
> Cc: emacs-devel@gnu.org
> 
> Has anyone heard anything more regarding a resolution for this issue?
> I've heard nothing from the savannah admins and stil cannot access the
> bzr repository.

You didn't subscribe to savannah-hackers-public, and evidently don't
read the archives, so you are unaware of the continuing discussion on
this matter.  Please look in the archives, if you are interested.

In a nutshell, after a day and a half of testing on 3 different
machine, my conclusion is that the problem I succeeded to reproduce is
a different problem.  It evidently causes any bzr connection that
takes longer than 1 hour to be forcibly dropped after precisely 1
hour.  (Due to the long time, I can only reproduce this with an
initial "bzr branch" command, and only on machines with slow
connection.)  There's probably some timeout somewhere on savannah, and
savannah admins will be looking for it.

But that is not your problem, because you reported a much faster
disconnect.  So I suggest that you run "bzr -Dhpss pull" on the
machine where this happens, and post the last portion of the .bzr.log
file as a followup to your original message.  That should give the
savannah admins information about what happens in the transport that
causes the problem.

TIA

Re: bzr not working?

[Tim Cross]

On Wed, Oct 12, 2011 at 7:43 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>> Date: Wed, 12 Oct 2011 10:35:13 +1100
>> From: Tim Cross <theophilusx@gmail.com>
>> Cc: emacs-devel@gnu.org
>>
>> Has anyone heard anything more regarding a resolution for this issue?
>> I've heard nothing from the savannah admins and stil cannot access the
>> bzr repository.
>
> You didn't subscribe to savannah-hackers-public, and evidently don't
> read the archives, so you are unaware of the continuing discussion on
> this matter.  Please look in the archives, if you are interested.
>
> In a nutshell, after a day and a half of testing on 3 different
> machine, my conclusion is that the problem I succeeded to reproduce is
> a different problem.  It evidently causes any bzr connection that
> takes longer than 1 hour to be forcibly dropped after precisely 1
> hour.  (Due to the long time, I can only reproduce this with an
> initial "bzr branch" command, and only on machines with slow
> connection.)  There's probably some timeout somewhere on savannah, and
> savannah admins will be looking for it.
>
> But that is not your problem, because you reported a much faster
> disconnect.  So I suggest that you run "bzr -Dhpss pull" on the
> machine where this happens, and post the last portion of the .bzr.log
> file as a followup to your original message.  That should give the
> savannah admins information about what happens in the transport that
> causes the problem.
>
> TIA
>
>

I have sent another message to the savannah list with the output from
-Dhpss. I also tried accessing another bzr repository on savannah
(color-theme) and got the same error. However, I can access bzr
repositories on other hosts, such as launchpad.net with no problems,
so it does seem to be savannah specific. I can also use the git
mirrors on savannah with no problems, which is what I'm doing now to
update my emacs code.

While I would prefer to use bzr as it is the 'official' repository,
using git is actually more convenient, seems faster and has better
emacs integration. Now that the git mirror is being updated regularly,
I'm happy to use it for the time being, so this issue is less urgent
for me than it was.

Tim

-- 
Tim Cross
Phone: 0428 212 217

Re: bzr not working?

[Tim Cross]

On Thu, Oct 13, 2011 at 10:13 AM, Tim Cross <theophilusx@gmail.com> wrote:
> On Wed, Oct 12, 2011 at 7:43 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>>> Date: Wed, 12 Oct 2011 10:35:13 +1100
>>> From: Tim Cross <theophilusx@gmail.com>
>>> Cc: emacs-devel@gnu.org
>>>
>>> Has anyone heard anything more regarding a resolution for this issue?
>>> I've heard nothing from the savannah admins and stil cannot access the
>>> bzr repository.
>>
>> You didn't subscribe to savannah-hackers-public, and evidently don't
>> read the archives, so you are unaware of the continuing discussion on
>> this matter.  Please look in the archives, if you are interested.
>>
>> In a nutshell, after a day and a half of testing on 3 different
>> machine, my conclusion is that the problem I succeeded to reproduce is
>> a different problem.  It evidently causes any bzr connection that
>> takes longer than 1 hour to be forcibly dropped after precisely 1
>> hour.  (Due to the long time, I can only reproduce this with an
>> initial "bzr branch" command, and only on machines with slow
>> connection.)  There's probably some timeout somewhere on savannah, and
>> savannah admins will be looking for it.
>>
>> But that is not your problem, because you reported a much faster
>> disconnect.  So I suggest that you run "bzr -Dhpss pull" on the
>> machine where this happens, and post the last portion of the .bzr.log
>> file as a followup to your original message.  That should give the
>> savannah admins information about what happens in the transport that
>> causes the problem.
>>
>> TIA
>>
>>
>
> I have sent another message to the savannah list with the output from
> -Dhpss. I also tried accessing another bzr repository on savannah
> (color-theme) and got the same error. However, I can access bzr
> repositories on other hosts, such as launchpad.net with no problems,
> so it does seem to be savannah specific. I can also use the git
> mirrors on savannah with no problems, which is what I'm doing now to
> update my emacs code.
>
> While I would prefer to use bzr as it is the 'official' repository,
> using git is actually more convenient, seems faster and has better
> emacs integration. Now that the git mirror is being updated regularly,
> I'm happy to use it for the time being, so this issue is less urgent
> for me than it was.
>
> Tim
>

This issue has now been resolved. A quick update for the record and in
case anyone else encounters a similar issue.

The root cause of the problem was the deep packet inspection used by
the firewall used by my employer. The firewall was configured to drop
connections which were using 'unknown' protocols. At some point,
either the bzr protocols changed or the vendor changed the signatures
used to identify protocols (it looks like bzr may have changed, but
not certain about this). When this change occurred, the bzr protocol
was no longer recognized and the firewall dropped the connection.

The confusing/misleading aspect of this is that the nature of some
firewalls which use deep packet inspection is that they initially
allow some traffic through in order to perform the analysis and then
drop the connection if unrecognized. This gives the impression you
have gotten 'out' of your network and are connected tot he remote
server and that it is the remote server which drops the connection.

I have assisted our network admins in gathering sample bzr protocl
data, which they will submit to the firewall vendor so that they can
update their signatures for identifying bzr protocls. This should then
allow explicit configuration of what to do with bzr connections i.e.
allow/block depending on policy etc.

There is also an unrelated issue of savannah's bzr dropping
connections after 1 hour. This was done by savannah sys admins to deal
with runaway bzr processes - they configured a 'hard' 60min timeout
for bzr connections. I believe this has been increased to 120min and
is being reviewed. Apparently, newer versions of bzr have both
built-in connection timeouts and improved detection for idle
connections. Therefore, it is likely this hard coded limit will be
removed once savannah has updated bzr to a sufficiently recent
version.  I have no details of time lines or even firm confirmation
if/when these updates/changes will be performed and only mention it so
that anyone on a slow connection is aware that
bzr://bzr.savannah.gnu.org will timeout after 120min. I believe this
only affects users of the native bzr protocol and not those accessing
bzr repositories over http. Contact savannah if you have
concerns/issues.

Tim
-- 
Tim Cross