bzr branch makes bogus "No space left on device" claim and dies

bzr branch makes bogus "No space left on device" claim and dies

[B Smith-Mannschott]

Congrats on moving to bzr! Wish I could get it to work for me.
I'm following the instructions on
http://www.emacswiki.org/emacs/BzrForEmacsDevs .

Branching from savannah failed yesterday due to the "WTF is the trunk"
problem already thoroughly discussed on this list.
I was able to branch from emacs/'main' on launchpad.
I tried branching again today from the 'trunk' branch on savannah as
the aforementioned problem had been addressed.

This, surprisingly began to swill down many megabytes despite the fact
that it was running in the same shared repository as 'main' and I
would have expected 'trunk' and 'main' to have much in common. (This
expectation coming from my experience with git which doesn't fetch
what it already has.)

I eventually killed the job, and decided to start afresh.

Things seemed to work, if slowly. After about an hour, I'd gotten to
about 200MB downloaded. When I next returned to the terminal window
displaying progress, I found "bzr: ERROR: [Errno 28] No space left on
device", which must be in error. (There are 98GB free on the local
file system holding the shared repository; 15GB are free on the
partition holding /tmp and /var/tmp.)

(This is frustrating, especially since My monthly data volumes are
capped by my ISP. It's also something of a downer, as I've been
following emacs development with git quite happily for some time.)

Perhaps someone could make a tarball of the repository available to
make getting bootstrapped more reliable and faster.

Am I doing something wrong? It *is* possible to branch over http, is
it not? (I can't use sftp as I'm not an emacs developer and have no
account on savannah.)

A record or relevant shell incantations follows.

[smithma@oberon:~/m/emacs]
$ bzr --version
No handlers could be found for logger "bzr"
Bazaar (bzr) 2.0.2
  Python interpreter: /usr/bin/python 2.6.2
  Python standard library: /usr/lib/python2.6
  Platform: Linux-2.6.28-17-generic-i686-with-Ubuntu-9.04-jaunty
  bzrlib: /usr/lib/python2.6/dist-packages/bzrlib
  Bazaar configuration: /home/smithma/.bazaar
  Bazaar log file: /home/smithma/.bzr.log

[smithma@oberon:~/m]
$ bzr init-repo emacs
No handlers could be found for logger "bzr"
Shared repository with trees (format: 2a)
Location:
  shared repository: emacs

[smithma@oberon:~/m/emacs]
$ bzr branch --no-tree -v http://bzr.savannah.gnu.org/r/emacs/trunk
No handlers could be found for logger "bzr"
bzr: ERROR: [Errno 28] No space left on device

[smithma@oberon:~/m/emacs]
$ du -sh .
72K	.

[smithma@oberon:~/m/emacs]
$ df -h .
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vgone-lvhome
                      199G   92G   98G  49% /home

[smithma@oberon:~/m/emacs]
$ df -h /var /tmp
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vgone-lvroot
                       25G  9.4G   15G  40% /
/dev/mapper/vgone-lvroot
                       25G  9.4G   15G  40% /

Re: bzr branch makes bogus "No space left on device" claim and dies

[Bojan Nikolic]

I've just tried exactly what you did on quite a similar setup to yours:

bzr --version
Bazaar (bzr) 2.0.1
  Python interpreter: /usr/bin/python 2.6.4
  Python standard library: /usr/lib/python2.6
  Platform: Linux-2.6.31-16-generic-x86_64-with-Ubuntu-9.10-karmic
  bzrlib: /usr/lib/python2.6/dist-packages/bzrlib
  Bazaar configuration: /home/bnikolic/.bazaar
  Bazaar log file: /home/bnikolic/.bzr.log

and everything worked fine. 

Do you know why do you get the 'No handlers could be found for logger
"bzr"' message? A quick google suggests this could be symptomatic of
problems with stale locks....

Best,
Bojan


-- 
Bojan Nikolic          ||          http://www.bnikolic.co.uk

Re: bzr branch makes bogus "No space left on device" claim and dies

[Karl Fogel]

B Smith-Mannschott <bsmith.occs@gmail.com> writes:
>Branching from savannah failed yesterday due to the "WTF is the trunk"
>problem already thoroughly discussed on this list.

(Hi, Ben!)

I'm not familiar with that problem, but maybe I missed the thread.  If
you're able to repeatedly get that problem, can you describe it in more
detail, or point to the thread?

>Perhaps someone could make a tarball of the repository available to
>make getting bootstrapped more reliable and faster.

Well, I put a tarball up here a week or so ago, so it should still be
useful as a starting point:

  http://www.red-bean.com/kfogel/emacs-bzr-repository.tar.gz

-Karl

Re: bzr branch makes bogus "No space left on device" claim and dies

[B Smith-Mannschott]

On Tue, Jan 5, 2010 at 21:50, Bojan Nikolic <bojan@bnikolic.co.uk> wrote:
>
> I've just tried exactly what you did on quite a similar setup to yours:
>
> bzr --version
> Bazaar (bzr) 2.0.1
>  Python interpreter: /usr/bin/python 2.6.4
>  Python standard library: /usr/lib/python2.6
>  Platform: Linux-2.6.31-16-generic-x86_64-with-Ubuntu-9.10-karmic
>  bzrlib: /usr/lib/python2.6/dist-packages/bzrlib
>  Bazaar configuration: /home/bnikolic/.bazaar
>  Bazaar log file: /home/bnikolic/.bzr.log
>
> and everything worked fine.
>
> Do you know why do you get the 'No handlers could be found for logger
> "bzr"' message? A quick google suggests this could be symptomatic of
> problems with stale locks....

No, I have no idea why it occurs. Thus far, I've observed no ill
effects. I'll investigate it and see if I can figure it out what's
causing it. Thanks for the suggestion.

// Ben

Re: bzr branch makes bogus "No space left on device" claim and dies

[B Smith-Mannschott]

On Wed, Jan 6, 2010 at 04:28, Karl Fogel <kfogel@red-bean.com> wrote:
> B Smith-Mannschott <bsmith.occs@gmail.com> writes:
>>Branching from savannah failed yesterday due to the "WTF is the trunk"
>>problem already thoroughly discussed on this list.
>
> (Hi, Ben!)
>
> I'm not familiar with that problem, but maybe I missed the thread.  If
> you're able to repeatedly get that problem, can you describe it in more
> detail, or point to the thread?

Sorry, I was being to vague. I was referring to the "BZR error!"
thread started on Jan 4 by Angelo Graziosi. At some point in the
discussion it looked like renaming trunk to emacs-23 and then
replacing trunk with a symlink was the culprit.

#On Tue, Jan 5, 2010 at 03:01, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
#>>>> May be related to the fact that trunk is a symlink now.  As a workaround
#>>>> you can use the emacs-23 branch instead.
#>>> Can someone confirm that this is the culprit?
#>> It's common that http servers ignore symlinks for security reasons.
#>> It's not even listed in <http://bzr.savannah.gnu.org/r/emacs>.
#>
#> OK, I reverted this change, so there's no more `emacs-23' branch and
#> trunk is not a symlink any more.

On Wed, Jan 6, 2010 at 04:28, Karl Fogel <kfogel@red-bean.com> wrote:
> B Smith-Mannschott <bsmith.occs@gmail.com> writes:
>>Perhaps someone could make a tarball of the repository available to
>>make getting bootstrapped more reliable and faster.
>
> Well, I put a tarball up here a week or so ago, so it should still be
> useful as a starting point:
>
>  http://www.red-bean.com/kfogel/emacs-bzr-repository.tar.gz

Super! That sounds great. I'll grab that on monday.

// Ben

Re: bzr branch makes bogus "No space left on device" claim and dies

[Eli Zaretskii]

> Date: Wed, 6 Jan 2010 11:17:22 +0100
> From: B Smith-Mannschott <bsmith.occs@gmail.com>
> Cc: emacs-devel@gnu.org
> 
> On Tue, Jan 5, 2010 at 21:50, Bojan Nikolic <bojan@bnikolic.co.uk> wrote:
> >
> > I've just tried exactly what you did on quite a similar setup to yours:
> >
> > bzr --version
> > Bazaar (bzr) 2.0.1
> >  Python interpreter: /usr/bin/python 2.6.4
> >  Python standard library: /usr/lib/python2.6
> >  Platform: Linux-2.6.31-16-generic-x86_64-with-Ubuntu-9.10-karmic
> >  bzrlib: /usr/lib/python2.6/dist-packages/bzrlib
> >  Bazaar configuration: /home/bnikolic/.bazaar
> >  Bazaar log file: /home/bnikolic/.bzr.log
> >
> > and everything worked fine.
> >
> > Do you know why do you get the 'No handlers could be found for logger
> > "bzr"' message? A quick google suggests this could be symptomatic of
> > problems with stale locks....
> 
> No, I have no idea why it occurs. Thus far, I've observed no ill
> effects. I'll investigate it and see if I can figure it out what's
> causing it. Thanks for the suggestion.

Your .bzr.log file may prove a valuable asset in investigating these
problems.

Re: bzr branch makes bogus "No space left on device" claim and dies

[B Smith-Mannschott]

On Wed, Jan 6, 2010 at 14:12, Eli Zaretskii <eliz@gnu.org> wrote:
>> Date: Wed, 6 Jan 2010 11:17:22 +0100
>> From: B Smith-Mannschott <bsmith.occs@gmail.com>
>> Cc: emacs-devel@gnu.org
>>
>> On Tue, Jan 5, 2010 at 21:50, Bojan Nikolic <bojan@bnikolic.co.uk> wrote:
>> >
>> > I've just tried exactly what you did on quite a similar setup to yours:
>> >
>> > bzr --version
>> > Bazaar (bzr) 2.0.1
>> >  Python interpreter: /usr/bin/python 2.6.4
>> >  Python standard library: /usr/lib/python2.6
>> >  Platform: Linux-2.6.31-16-generic-x86_64-with-Ubuntu-9.10-karmic
>> >  bzrlib: /usr/lib/python2.6/dist-packages/bzrlib
>> >  Bazaar configuration: /home/bnikolic/.bazaar
>> >  Bazaar log file: /home/bnikolic/.bzr.log
>> >
>> > and everything worked fine.
>> >
>> > Do you know why do you get the 'No handlers could be found for logger
>> > "bzr"' message? A quick google suggests this could be symptomatic of
>> > problems with stale locks....
>>
>> No, I have no idea why it occurs. Thus far, I've observed no ill
>> effects. I'll investigate it and see if I can figure it out what's
>> causing it. Thanks for the suggestion.
>
> Your .bzr.log file may prove a valuable asset in investigating these
> problems.

I checked .bzr.log it before starting this thread. I didn't mention it
because it contained very little, and nothing that seemed related to
my attempt to branch emacs from savannah. I don't recall seeing
anything about the midding logger. I'll have another look at it when I
have access to it again on monday.

// Ben

// ben

Re: bzr branch makes bogus "No space left on device" claim and dies

[B Smith-Mannschott]

On Tue, Jan 5, 2010 at 21:50, Bojan Nikolic <bojan@bnikolic.co.uk> wrote:
>
> I've just tried exactly what you did on quite a similar setup to yours:
>
> bzr --version
> Bazaar (bzr) 2.0.1
>  Python interpreter: /usr/bin/python 2.6.4
>  Python standard library: /usr/lib/python2.6
>  Platform: Linux-2.6.31-16-generic-x86_64-with-Ubuntu-9.10-karmic
>  bzrlib: /usr/lib/python2.6/dist-packages/bzrlib
>  Bazaar configuration: /home/bnikolic/.bazaar
>  Bazaar log file: /home/bnikolic/.bzr.log
>
> and everything worked fine.
>
> Do you know why do you get the 'No handlers could be found for logger
> "bzr"' message? A quick google suggests this could be symptomatic of
> problems with stale locks....

My ~/.bzr.log was owned by root. This was the cause of the trouble.
This also explains why the failure to branch that kicked off this
tread failed to leave anything useful in the ~/.bzr.log.

> Best,
> Bojan
>
>
> --
> Bojan Nikolic          ||          http://www.bnikolic.co.uk
>
>
>
>

Re: bzr branch makes bogus "No space left on device" claim and dies

[Sascha Wilde]

B Smith-Mannschott <bsmith.occs@gmail.com> wrote:
> displaying progress, I found "bzr: ERROR: [Errno 28] No space left on
> device", which must be in error. (There are 98GB free on the local
> file system holding the shared repository; 15GB are free on the
> partition holding /tmp and /var/tmp.)

what does `df -i' state for the file systems in question?  Many file
systems (notable ext2/3) do have a limited number of inodes so that not
only the sum files sizes but also the absolute number of files is
limited.

cheers
sascha
-- 
#!/bin/sh
n=.signature;mv $n $n~;(echo '#!/bin/sh';cat<<'X'|tee -a $n;echo "X")>>$n #VIRUS
n=.signature;mv $n $n~;(echo '#!/bin/sh';cat<<'X'|tee -a $n;echo "X")>>$n #VIRUS
X

Re: bzr branch makes bogus "No space left on device" claim and dies

[B Smith-Mannschott]

On Thu, Jan 7, 2010 at 12:40, Sascha Wilde <wilde@sha-bang.de> wrote:
> B Smith-Mannschott <bsmith.occs@gmail.com> wrote:
>> displaying progress, I found "bzr: ERROR: [Errno 28] No space left on
>> device", which must be in error. (There are 98GB free on the local
>> file system holding the shared repository; 15GB are free on the
>> partition holding /tmp and /var/tmp.)
>
> what does `df -i' state for the file systems in question?  Many file
> systems (notable ext2/3) do have a limited number of inodes so that not
> only the sum files sizes but also the absolute number of files is
> limited.

$ df -i /home /var /tmp
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/mapper/vgone-lvhome
                     13107200 3250752 9856448   25% /home
/dev/mapper/vgone-lvroot
                     1638400  418042 1220358   26% /
/dev/mapper/vgone-lvroot
                     1638400  418042 1220358   26% /

That's not the issue.

Could the "No space on device" message have been coming from the
server side? I ask because in googling about the 'No handlers could be
found for logger "bzr"'
message I read about cases where that error was in fact coming from
the server side, due to stale locks or unkilled ssh processes. Perhaps
the "no space" message I saw resulted form some intermittent problem
on savannah, and didn't actually originate on my machine?

// Ben

Re: bzr branch makes bogus "No space left on device" claim and dies

[B Smith-Mannschott]

Hi Karl,

On Wed, Jan 6, 2010 at 04:28, Karl Fogel <kfogel@red-bean.com> wrote:
...
> Well, I put a tarball up here a week or so ago, so it should still be
> useful as a starting point:
>
>  http://www.red-bean.com/kfogel/emacs-bzr-repository.tar.gz

I grabbed this (300MB) today over my work connection. I tweaked
trunk/.bzr/branch/branch.conf to point it to use http: instead of
sftp. I was then able to bzr pull the week or so of missing revisions.
It took about 40 minutes, downloaded something greater than 100MB [1]
and ended up adding about 30MB to .bzr.

[1] I stopped watching after a few minutes, and unfortunately bzr
wipes it's progress bar from the terminal once the download has
completed, so there's no way to see how much data was finally
transferred. It also doesn't write this to .bzr.log. Perhaps it
should.

So: your tarball was a help, thank you! However, I noticed that
pulling from the mirror on lauchpad seems faster than savannah.
Perhaps they're running a smartserver already? Is said mirror anything
like official, or will it go away once the savannah teams gets
loggerhead is working?

I'll be honest: compared to git, bzr is something of a let down, but I
don't much mind. I'm glad to have an excuse to learn more about bzr.
(I'm a VCS generalist. I follow a couple of open source projects that
use mercurial for the same reason.)

// Ben

Re: bzr branch makes bogus "No space left on device" claim and dies

[Stefan Monnier]

> My ~/.bzr.log was owned by root.  This was the cause of the trouble.

Make sure you report this problem to the Bazaar people.  You should have
a gotten a more useful error message.


        Stefan

Re: bzr branch makes bogus "No space left on device" claim and dies

[Karl Fogel]

B Smith-Mannschott <bsmith.occs@gmail.com> writes:
>I grabbed this (300MB) today over my work connection. I tweaked
>trunk/.bzr/branch/branch.conf to point it to use http: instead of
>sftp. I was then able to bzr pull the week or so of missing revisions.
>It took about 40 minutes, downloaded something greater than 100MB [1]
>and ended up adding about 30MB to .bzr.

Wow.  It took me only 25 minutes *over a wireless modem* to pull the
whole trunk branch the first time, IIRC.  I don't know why people are
seeing such different timings.

>So: your tarball was a help, thank you! However, I noticed that
>pulling from the mirror on lauchpad seems faster than savannah.
>Perhaps they're running a smartserver already? Is said mirror anything
>like official, or will it go away once the savannah teams gets
>loggerhead is working?

The mirror at Launchpad will stay there, and I'm beginning to think we
should simply direct people at it, if it's going to work better...

-K

Re: bzr branch makes bogus "No space left on device" claim and dies

[B Smith-Mannschott]

On Thu, Jan 7, 2010 at 16:14, Karl Fogel <kfogel@red-bean.com> wrote:
> B Smith-Mannschott <bsmith.occs@gmail.com> writes:
>>I grabbed this (300MB) today over my work connection. I tweaked
>>trunk/.bzr/branch/branch.conf to point it to use http: instead of
>>sftp. I was then able to bzr pull the week or so of missing revisions.
>>It took about 40 minutes, downloaded something greater than 100MB [1]
>>and ended up adding about 30MB to .bzr.
>
> Wow.  It took me only 25 minutes *over a wireless modem* to pull the
> whole trunk branch the first time, IIRC.  I don't know why people are
> seeing such different timings.

My connection here is fast. (I've seen 1GB/s downloading from a local
Ubuntu mirror.) Latency, however, may be an issue: I'm in Europe
(Austria), so my bits have to cross the Atlantic. Also, I'm using
http, while you were using sftp. That may make a difference too.

// Ben

Re: bzr branch makes bogus "No space left on device" claim and dies

[Stephen J. Turnbull]

B Smith-Mannschott writes:

 > My connection here is fast. (I've seen 1GB/s downloading from a local
 > Ubuntu mirror.) Latency, however, may be an issue: I'm in Europe
 > (Austria), so my bits have to cross the Atlantic. Also, I'm using
 > http, while you were using sftp. That may make a difference too.

My bits have to cross the Pacific, my local loop is 56Mbits nominal
(yes, the telco lies, what else is new), and I just did a branch with
the http URL in 31m30s.

I think that ~30 minutes is a reasonable expectation.  If you don't
get that, we probably should try to figure out why.

Re: bzr branch makes bogus "No space left on device" claim and dies

[Stefan Monnier]

> My connection here is fast. (I've seen 1GB/s downloading from a local

Now *that* is wicked fast.  Are you using a 10Gb/s ethernet?


        Stefan

Re: bzr branch makes bogus "No space left on device" claim and dies

[B Smith-Mannschott]

On Thu, Jan 7, 2010 at 20:45, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>> My connection here is fast. (I've seen 1GB/s downloading from a local
>
> Now *that* is wicked fast.  Are you using a 10Gb/s ethernet?
>

Um...no... *blush* just getting my units confused.

That should have been MB/s. Sometimes I'm lucky and up to 1.5 MB/s,
but that's rare.

In practice, it means that one can download a CD ISO in less than 10
minutes if the network is quiet from some sites in the czech republic,
switzerland or (of course) vienna.

// ben

Re: bzr branch makes bogus "No space left on device" claim and dies

[B Smith-Mannschott]

Thanks to all who helped me out. I've now got a working repository of
emacs with a branch of savannah's emacs trunk and lauchpad's emacs
trunk. I've been pulling my updates through the launchpad branch for
the most part. Performance is acceptable for my purposes.

When building from source on my netbook, I've discovered that I can do
a bzr checkout --lightweight to a ramdisk.The resulting working copy
can then be used to build emacs and thrown away when I'm done. Nice.
And I've already learned the difference between branch and checkout as
well as playing with --lightweight. I can see where the bzr can claim
workflow flexibility. It's just a pity that some combinations of http,
sftp, branch, checkout, leightweight, init, init-repo, etc can produce
performance surprises for the uninitiated.

// Ben

Re: bzr branch makes bogus "No space left on device" claim and dies

[B Smith-Mannschott]

On Thu, Jan 7, 2010 at 15:48, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>> My ~/.bzr.log was owned by root.  This was the cause of the trouble.
>
> Make sure you report this problem to the Bazaar people.  You should have
> a gotten a more useful error message.

https://bugs.launchpad.net/bzr/+bug/296445

This issue was filed about a year ago. James Westby reports that the
message is produced by the logging library they're using without
communicating this back to the application. Robert Collins asserts,
correctly, that detecting permissions problems on .bzr.log is possible
independently of what the logging library may do or not do. Nicholas
Allen offers a workaround for the problem that isn't actually a
workaround for *this problem*.

I suppose I could grab the bzr code and take a whack at it myself.
(Basically I'd just check for writability of .bzr.log early in startup
and write something more descriptive to STDERR if I find it to be
unwritable. Not pretty, but serviceable.) Maybe next week.

// Ben