Surely 'bzr branch' shouldn't be this slow?

Surely 'bzr branch' shouldn't be this slow?

[Alan Mackenzie]

Hi, all,

Having created a local copy of the bzr repository in my directory
~/emacs/emacs.bzr/trunk, I then went to create the now familiar
quickfixes branch from it, with this command:

    $ time bzr branch emacs.bzr/trunk/ quickfixes/

.  This took over 39 minutes at 100% CPU usage.  For comparison, a full
build on my machine takes ~33 minutes.

What on Earth is going on here?  Surely creating a branch (which,
essentially, is just copying or linking files) should take at most a few
seconds.

Is there some configuration setting (or something) I've missed?

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Surely 'bzr branch' shouldn't be this slow?

[Óscar Fuentes]

Alan Mackenzie <acm@muc.de> writes:

> Hi, all,
>
> Having created a local copy of the bzr repository in my directory
> ~/emacs/emacs.bzr/trunk, I then went to create the now familiar
> quickfixes branch from it, with this command:
>
>     $ time bzr branch emacs.bzr/trunk/ quickfixes/
>
> .  This took over 39 minutes at 100% CPU usage.  For comparison, a full
> build on my machine takes ~33 minutes.
>
> What on Earth is going on here?  Surely creating a branch (which,
> essentially, is just copying or linking files) should take at most a few
> seconds.
>
> Is there some configuration setting (or something) I've missed?

If you followed the recommended setup, your quickfixes/ branch should be
created within emacs.bzr/, which is the shared repo. Branching within a
shared repo is much faster because the new branch shares its history
with the previous one and the required time is reduced to creating the
working tree (the files you edit). Try it.

-- 
Óscar

Re: Surely 'bzr branch' shouldn't be this slow?

[Alan Mackenzie]

Hi, Óscar,

On Sun, Jan 03, 2010 at 06:59:10PM +0100, Óscar Fuentes wrote:
> Alan Mackenzie <acm@muc.de> writes:

> > Hi, all,

> > Having created a local copy of the bzr repository in my directory
> > ~/emacs/emacs.bzr/trunk, I then went to create the now familiar
> > quickfixes branch from it, with this command:

> >     $ time bzr branch emacs.bzr/trunk/ quickfixes/

> > .  This took over 39 minutes at 100% CPU usage.  For comparison, a full
> > build on my machine takes ~33 minutes.

> > What on Earth is going on here?  Surely creating a branch (which,
> > essentially, is just copying or linking files) should take at most a few
> > seconds.

> > Is there some configuration setting (or something) I've missed?

> If you followed the recommended setup, your quickfixes/ branch should be
> created within emacs.bzr/, which is the shared repo. Branching within a
> shared repo is much faster because the new branch shares its history
> with the previous one and the required time is reduced to creating the
> working tree (the files you edit). Try it.

Ah, right.  The .../quickfixes/ needs to be physically under .../trunk,
not just "related" to it.  Thanks!

But it still feels a bit strange that creating a branch in the "wrong"
place takes up 39 minutes of processing.

> -- 
> Óscar

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Surely 'bzr branch' shouldn't be this slow?

[Jason Rumney]

Alan Mackenzie wrote:

>>> Having created a local copy of the bzr repository in my directory
>>> ~/emacs/emacs.bzr/trunk, I then went to create the now familiar
>>> quickfixes branch from it, with this command:
>>>       
>
>   
>>>     $ time bzr branch emacs.bzr/trunk/ quickfixes/
>>>       
>
> Ah, right.  The .../quickfixes/ needs to be physically under .../trunk,
> not just "related" to it.  Thanks!
>   

I fear you may be heading for another 39 minute wait now.  quickfixes 
needs to be under emacs/emacs.bzr,
alongside trunk/, not under it.

Re: Surely 'bzr branch' shouldn't be this slow?

[Óscar Fuentes]

Jason Rumney <jasonr@gnu.org> writes:

> Alan Mackenzie wrote:
>
>>>> Having created a local copy of the bzr repository in my directory
>>>> ~/emacs/emacs.bzr/trunk, I then went to create the now familiar
>>>> quickfixes branch from it, with this command:
>>>>       
>>
>>   
>>>>     $ time bzr branch emacs.bzr/trunk/ quickfixes/
>>>>       
>>
>> Ah, right.  The .../quickfixes/ needs to be physically under .../trunk,
>> not just "related" to it.  Thanks!
>>   
>
> I fear you may be heading for another 39 minute wait now.  quickfixes
> needs to be under emacs/emacs.bzr,
> alongside trunk/, not under it.

As far as the directories containing the branches are under the
directory that acts as the shared repository, you will benefit from the
fast operations the shared branch brings in. You can move and rename
branches within the shared repository (as long as they are not
referenced by other branches: if you work with the recommended
decentralized workflow, renaming `trunk' will require some tweaking on
the other branches.)

39 minutes for branching outside the shared repository corresponds to a
slow machine. It is 10 minutes on a GNU/Linux 2.4 GHz 64 bits
worksation-class machine. Bazaar is no speed daemon, but in this case it
has to process more than 200 MB of data (not merely copying it around.)

-- 
Óscar

Re: Surely 'bzr branch' shouldn't be this slow?

[Eli Zaretskii]

> From: Óscar_Fuentes <ofv@wanadoo.es>
> Date: Wed, 06 Jan 2010 15:06:37 +0100
> 
> 39 minutes for branching outside the shared repository corresponds to a
> slow machine. It is 10 minutes on a GNU/Linux 2.4 GHz 64 bits
> worksation-class machine.

I don't think the time it takes to make the original branch (or a
branch outside the shared repo) is CPU-bound.  It took me 50 minutes
on a 3GHz Windows machine with a single hyperthreaded core, and the
CPU time used by bzr never went up to more than 25%.

It's possible that the speed of the network connection is much more
important.  Mine is 3.5Mbps, in case it matters.

Re: Surely 'bzr branch' shouldn't be this slow?

[Stefan Monnier]

>> 39 minutes for branching outside the shared repository corresponds to a
>> slow machine. It is 10 minutes on a GNU/Linux 2.4 GHz 64 bits
>> worksation-class machine.

> I don't think the time it takes to make the original branch (or a
> branch outside the shared repo) is CPU-bound.  It took me 50 minutes
> on a 3GHz Windows machine with a single hyperthreaded core, and the
> CPU time used by bzr never went up to more than 25%.

It's both CPU and IO bound, last I tried.
It's just not very efficient.


        Stefan

Re: Surely 'bzr branch' shouldn't be this slow?

[Yavor Doganov]

Eli Zaretskii wrote:
> I don't think the time it takes to make the original branch (or a
> branch outside the shared repo) is CPU-bound.

It certainly is!  I've been fighting for 3 days now to branch on one
of my most powerful machines at home (AMD Duron 1.3 GHz, 500 MB RAM)
without success, and I gave up today.  Both sftp/http methods get
stuck around 275/300 MB, and the machine is experiencing sky high
loads; I can't even login.  I'm tired of killing bzr and restarting
the process; it always fails.

At work, on a Pentium IV 2 GHz/1G RAM it took me about an hour and a
half with a slower connection.

I guess if I compress the shared repo and scp it to my home machine,
it would work, right?

Re: Surely 'bzr branch' shouldn't be this slow?

[Lennart Borgman]

On Thu, Jan 7, 2010 at 8:44 AM, Yavor Doganov <yavor@gnu.org> wrote:
> Eli Zaretskii wrote:
>> I don't think the time it takes to make the original branch (or a
>> branch outside the shared repo) is CPU-bound.
>
> It certainly is!  I've been fighting for 3 days now to branch on one
> of my most powerful machines at home (AMD Duron 1.3 GHz, 500 MB RAM)
> without success, and I gave up today.  Both sftp/http methods get
> stuck around 275/300 MB, and the machine is experiencing sky high
> loads; I can't even login.  I'm tired of killing bzr and restarting
> the process; it always fails.
>
> At work, on a Pentium IV 2 GHz/1G RAM it took me about an hour and a
> half with a slower connection.
>
> I guess if I compress the shared repo and scp it to my home machine,
> it would work, right?


Just a comment: I have not even tried yet to use bzr. According to
what I am reading it looks impossible at the moment.

Am I right or?

Re: Surely 'bzr branch' shouldn't be this slow?

[Juanma Barranquero]

On Thu, Jan 7, 2010 at 09:37, Lennart Borgman <lennart.borgman@gmail.com> wrote:

> According to
> what I am reading it looks impossible at the moment.
>
> Am I right or?

Yes, of course.

    Juanma

Re: Surely 'bzr branch' shouldn't be this slow?

[Stephen Berman]

On Thu, 7 Jan 2010 09:37:15 +0100 Lennart Borgman <lennart.borgman@gmail.com> wrote:

> On Thu, Jan 7, 2010 at 8:44 AM, Yavor Doganov <yavor@gnu.org> wrote:
>> Eli Zaretskii wrote:
>>> I don't think the time it takes to make the original branch (or a
>>> branch outside the shared repo) is CPU-bound.
>>
>> It certainly is!  I've been fighting for 3 days now to branch on one
>> of my most powerful machines at home (AMD Duron 1.3 GHz, 500 MB RAM)
>> without success, and I gave up today.  Both sftp/http methods get
>> stuck around 275/300 MB, and the machine is experiencing sky high
>> loads; I can't even login.  I'm tired of killing bzr and restarting
>> the process; it always fails.
>>
>> At work, on a Pentium IV 2 GHz/1G RAM it took me about an hour and a
>> half with a slower connection.
>>
>> I guess if I compress the shared repo and scp it to my home machine,
>> it would work, right?
>
>
> Just a comment: I have not even tried yet to use bzr. According to
> what I am reading it looks impossible at the moment.
>
> Am I right or?

Maybe I was lucky, but I successfully branched the trunk via http last
night, on my first try.  It took about 50 minutes for bzr to download
281 MB, then, as reported above, it appeared to stall for more than 10
minutes, but the bzr process continued to consume 80-90% CPU.  (This is
with an Athlon XP 2100+ (1.7 GHz) and 2 GB RAM over a rather slow
wireless connection.)  Just as I was about to kill it, it finished.  I
then made a quickfixes branch from the (now local) trunk, which took
only about 10 seconds, and bootstrapped from the quickfixes branch
without a problem.

Steve Berman

Re: Surely 'bzr branch' shouldn't be this slow?

[Eli Zaretskii]

> From: Lennart Borgman <lennart.borgman@gmail.com>
> Date: Thu, 7 Jan 2010 09:37:15 +0100
> Cc: emacs-devel@gnu.org
> 
> Just a comment: I have not even tried yet to use bzr. According to
> what I am reading it looks impossible at the moment.
> 
> Am I right or?

It's both possible and practical.  I use it on 2 different Windows
machines and on 1 GNU/Linux one, and it works quite well.  The initial
"bzr branch" takes a long time, but that's all.

Re: Surely 'bzr branch' shouldn't be this slow?

[Lennart Borgman]

On Thu, Jan 7, 2010 at 8:54 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Lennart Borgman <lennart.borgman@gmail.com>
>> Date: Thu, 7 Jan 2010 09:37:15 +0100
>> Cc: emacs-devel@gnu.org
>>
>> Just a comment: I have not even tried yet to use bzr. According to
>> what I am reading it looks impossible at the moment.
>>
>> Am I right or?
>
> It's both possible and practical.  I use it on 2 different Windows
> machines and on 1 GNU/Linux one, and it works quite well.  The initial
> "bzr branch" takes a long time, but that's all.


And with a 2Mbit/s connection and an Alan style pc, 1GB RAM?

Re: Surely 'bzr branch' shouldn't be this slow?

[Jason Earl]

Lennart Borgman <lennart.borgman@gmail.com> writes:

> On Thu, Jan 7, 2010 at 8:54 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>>> From: Lennart Borgman <lennart.borgman@gmail.com>
>>> Date: Thu, 7 Jan 2010 09:37:15 +0100
>>> Cc: emacs-devel@gnu.org
>>>
>>> Just a comment: I have not even tried yet to use bzr. According to
>>> what I am reading it looks impossible at the moment.
>>>
>>> Am I right or?
>>
>> It's both possible and practical.  I use it on 2 different Windows
>> machines and on 1 GNU/Linux one, and it works quite well.  The initial
>> "bzr branch" takes a long time, but that's all.
>
>
> And with a 2Mbit/s connection and an Alan style pc, 1GB RAM?

I use it with a netbook with 1GB of Ram and no swap.  It works fine.  I
will admit that I have switched to pulling from launchpad, but I did my
initial pull from Savannah.

Jason

Re: Surely 'bzr branch' shouldn't be this slow?

[Lennart Borgman]

On Thu, Jan 7, 2010 at 10:48 PM, Jason Earl <jearl@notengoamigos.org> wrote:
> Lennart Borgman <lennart.borgman@gmail.com> writes:
>
>> On Thu, Jan 7, 2010 at 8:54 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>>>> From: Lennart Borgman <lennart.borgman@gmail.com>
>>>> Date: Thu, 7 Jan 2010 09:37:15 +0100
>>>> Cc: emacs-devel@gnu.org
>>>>
>>>> Just a comment: I have not even tried yet to use bzr. According to
>>>> what I am reading it looks impossible at the moment.
>>>>
>>>> Am I right or?
>>>
>>> It's both possible and practical.  I use it on 2 different Windows
>>> machines and on 1 GNU/Linux one, and it works quite well.  The initial
>>> "bzr branch" takes a long time, but that's all.
>>
>>
>> And with a 2Mbit/s connection and an Alan style pc, 1GB RAM?
>
> I use it with a netbook with 1GB of Ram and no swap.  It works fine.  I
> will admit that I have switched to pulling from launchpad, but I did my
> initial pull from Savannah.

How long did the initial pull take on this machine? (Sorry if you have
told already. Are the figures perhaps on the wiki?) How fast is your
internet?

And doing the initial pull from Launchpad instead how much faster is it?

Re: Surely 'bzr branch' shouldn't be this slow?

[Jason Earl]

Lennart Borgman <lennart.borgman@gmail.com> writes:

> On Thu, Jan 7, 2010 at 10:48 PM, Jason Earl <jearl@notengoamigos.org> wrote:
>> Lennart Borgman <lennart.borgman@gmail.com> writes:
>>
>>> On Thu, Jan 7, 2010 at 8:54 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>>>>> From: Lennart Borgman <lennart.borgman@gmail.com>
>>>>> Date: Thu, 7 Jan 2010 09:37:15 +0100
>>>>> Cc: emacs-devel@gnu.org
>>>>>
>>>>> Just a comment: I have not even tried yet to use bzr. According to
>>>>> what I am reading it looks impossible at the moment.
>>>>>
>>>>> Am I right or?
>>>>
>>>> It's both possible and practical.  I use it on 2 different Windows
>>>> machines and on 1 GNU/Linux one, and it works quite well.  The initial
>>>> "bzr branch" takes a long time, but that's all.
>>>
>>>
>>> And with a 2Mbit/s connection and an Alan style pc, 1GB RAM?
>>
>> I use it with a netbook with 1GB of Ram and no swap.  It works fine.  I
>> will admit that I have switched to pulling from launchpad, but I did my
>> initial pull from Savannah.
>
> How long did the initial pull take on this machine? (Sorry if you have
> told already. Are the figures perhaps on the wiki?) How fast is your
> internet?

Just for fun I did it again into a new repository.  While there were
times when my netbook struggled, for the most part it appeared that my
crappy wireless network connection was the bottleneck.

time bzr branch http://bzr.savannah.gnu.org/r/emacs/trunk/ trunk
Branched 99273 revision(s).                                                                

real    56m30.346s
user    34m10.160s
sys     0m32.638s

Then I tested to see how long it took to see if there were more
revisions.  There weren't any.

time bzr pull
Using saved parent location: http://bzr.savannah.gnu.org/r/emacs/trunk/
No revisions to pull.                                                                      

real    0m8.109s
user    0m1.672s
sys     0m0.272s

This is on a wireless network.

> And doing the initial pull from Launchpad instead how much faster is it?

Launchpad actually took at little longer:

time bzr pull --remember lp:emacs
No revisions to pull.                                                                      

real    0m10.246s
user    0m1.108s
sys     0m0.180s

The advantage of using launchpad is that you will never get the
pathologically bad updates where bzr pulls down 100M of data to make 2M
worth of changes.  The disadvantage is that it is a mirror that updates
only every 8 hours.

Jason

Re: Surely 'bzr branch' shouldn't be this slow?

[Lennart Borgman]

On Fri, Jan 8, 2010 at 3:07 AM, Jason Earl <jearl@notengoamigos.org> wrote:
> Lennart Borgman <lennart.borgman@gmail.com> writes:
>
>> On Thu, Jan 7, 2010 at 10:48 PM, Jason Earl <jearl@notengoamigos.org> wrote:
>>> Lennart Borgman <lennart.borgman@gmail.com> writes:
>>>
>>>> On Thu, Jan 7, 2010 at 8:54 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>>>>>> From: Lennart Borgman <lennart.borgman@gmail.com>
>>>>>> Date: Thu, 7 Jan 2010 09:37:15 +0100
>>>>>> Cc: emacs-devel@gnu.org
>>>>>>
>>>>>> Just a comment: I have not even tried yet to use bzr. According to
>>>>>> what I am reading it looks impossible at the moment.
>>>>>>
>>>>>> Am I right or?
>>>>>
>>>>> It's both possible and practical.  I use it on 2 different Windows
>>>>> machines and on 1 GNU/Linux one, and it works quite well.  The initial
>>>>> "bzr branch" takes a long time, but that's all.
>>>>
>>>>
>>>> And with a 2Mbit/s connection and an Alan style pc, 1GB RAM?
>>>
>>> I use it with a netbook with 1GB of Ram and no swap.  It works fine.  I
>>> will admit that I have switched to pulling from launchpad, but I did my
>>> initial pull from Savannah.
>>
>> How long did the initial pull take on this machine? (Sorry if you have
>> told already. Are the figures perhaps on the wiki?) How fast is your
>> internet?
>
> Just for fun I did it again into a new repository.  While there were
> times when my netbook struggled, for the most part it appeared that my
> crappy wireless network connection was the bottleneck.
>
> time bzr branch http://bzr.savannah.gnu.org/r/emacs/trunk/ trunk
> Branched 99273 revision(s).
>
> real    56m30.346s
> user    34m10.160s
> sys     0m32.638s
>
> Then I tested to see how long it took to see if there were more
> revisions.  There weren't any.
>
> time bzr pull
> Using saved parent location: http://bzr.savannah.gnu.org/r/emacs/trunk/
> No revisions to pull.
>
> real    0m8.109s
> user    0m1.672s
> sys     0m0.272s
>
> This is on a wireless network.
>
>> And doing the initial pull from Launchpad instead how much faster is it?
>
> Launchpad actually took at little longer:
>
> time bzr pull --remember lp:emacs
> No revisions to pull.
>
> real    0m10.246s
> user    0m1.108s
> sys     0m0.180s
>
> The advantage of using launchpad is that you will never get the
> pathologically bad updates where bzr pulls down 100M of data to make 2M
> worth of changes.  The disadvantage is that it is a mirror that updates
> only every 8 hours.


Thanks Jason, very clear and instructive. The road for me at the
moment seems to be Launchpad. The 8 hours does not matter right now,
but the the 100 MB does.

Re: Surely 'bzr branch' shouldn't be this slow?

[Eli Zaretskii]

> From: Lennart Borgman <lennart.borgman@gmail.com>
> Date: Thu, 7 Jan 2010 21:09:02 +0100
> Cc: emacs-devel@gnu.org
> 
> On Thu, Jan 7, 2010 at 8:54 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> >> From: Lennart Borgman <lennart.borgman@gmail.com>
> >> Date: Thu, 7 Jan 2010 09:37:15 +0100
> >> Cc: emacs-devel@gnu.org
> >>
> >> Just a comment: I have not even tried yet to use bzr. According to
> >> what I am reading it looks impossible at the moment.
> >>
> >> Am I right or?
> >
> > It's both possible and practical.  I use it on 2 different Windows
> > machines and on 1 GNU/Linux one, and it works quite well.  The initial
> > "bzr branch" takes a long time, but that's all.
> 
> 
> And with a 2Mbit/s connection and an Alan style pc, 1GB RAM?

That's about what I have here, maybe slightly less.  I won't expect
any special trouble.  Why won't you try it?

Re: Surely 'bzr branch' shouldn't be this slow?

[Stefan Monnier]

> It certainly is!  I've been fighting for 3 days now to branch on one
> of my most powerful machines at home (AMD Duron 1.3 GHz, 500 MB RAM)
> without success, and I gave up today.  Both sftp/http methods get
> stuck around 275/300 MB, and the machine is experiencing sky high
> loads; I can't even login.

Hard to believe the load goes high.  But it probably thrashes wildly
because Bzr ends up using more than 500MB.  That would also explain the
fact you can't login, which shouldn't be a problem with a high load.

> I guess if I compress the shared repo and scp it to my home machine,
> it would work, right?

Yes it would work just fine and be much faster.  But since it's
compressed already, you can skip the first step.


        Stefan

Re: Surely 'bzr branch' shouldn't be this slow?

[Yavor Doganov]

Stefan Monnier wrote:
> > Both sftp/http methods get stuck around 275/300 MB, and the
> > machine is experiencing sky high loads; I can't even login.
> 
> Hard to believe the load goes high.

It's pretty decent (0.5-1.5) in the first 1 hr or so, when the bzr
progress indicator is moving.  After it becomes (or at least appears
to be) stuck, the load goes nuts, increasing from 1.5 to 15 for a few
minutes, staying high for a while, falling back again, then up, etc.

> But it probably thrashes wildly because Bzr ends up using more than
> 500MB.

Indeed, it eats the memory + 500 MB swap, but there's still 500 MB
swap availaible, so I can't explain myself this behavior.

> > I guess if I compress the shared repo and scp it to my home machine,
> > it would work, right?
> 
> Yes it would work just fine and be much faster.

Yes, it worked nicely, although the download rate at the receiving end
was very low.  At least I finally have the sources, and can do `bzr
pull', etc.  Thanks.

Re: Surely 'bzr branch' shouldn't be this slow?

[Stefan Monnier]

>> But it probably thrashes wildly because Bzr ends up using more than
>> 500MB.
> Indeed, it eats the memory + 500 MB swap, but there's still 500 MB
> swap availaible, so I can't explain myself this behavior.

The amount of swap left is mostly irrelevant.  What matters is that
there more memory in active use than the amount of RAM you have, so the
system spends its time moving data back&forth between RAM and swap,
which brings the whole system to a crawl.

http://en.wikipedia.org/wiki/Thrash_%28computer_science%29


        Stefan

Re: Surely 'bzr branch' shouldn't be this slow?

[Lennart Borgman]

On Thu, Jan 7, 2010 at 8:43 PM, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>>> But it probably thrashes wildly because Bzr ends up using more than
>>> 500MB.
>> Indeed, it eats the memory + 500 MB swap, but there's still 500 MB
>> swap availaible, so I can't explain myself this behavior.
>
> The amount of swap left is mostly irrelevant.  What matters is that
> there more memory in active use than the amount of RAM you have, so the
> system spends its time moving data back&forth between RAM and swap,
> which brings the whole system to a crawl.
>
> http://en.wikipedia.org/wiki/Thrash_%28computer_science%29


Is it the design or the algorithm that are the problem?

Thrashing [was: Surely 'bzr branch' shouldn't be this slow?]

[Stephen J. Turnbull]

Lennart Borgman writes:

 > Is it the design or the algorithm that are the problem?

Neither.  It's Intel advertising that makes people think that GHz are
as useful as GB.  If you have a choice to make between buying more GHz
and buying more GB, as a rule of thumb you won't go wrong by taking
the latter.

Re: Thrashing [was: Surely 'bzr branch' shouldn't be this slow?]

[Lennart Borgman]

On Thu, Jan 7, 2010 at 9:26 PM, Stephen J. Turnbull <stephen@xemacs.org> wrote:
> Lennart Borgman writes:
>
>  > Is it the design or the algorithm that are the problem?
>
> Neither.  It's Intel advertising that makes people think that GHz are
> as useful as GB.  If you have a choice to make between buying more GHz
> and buying more GB, as a rule of thumb you won't go wrong by taking
> the latter.


Are you saying the algorithm is future proof? ;-)

Re: Surely 'bzr update' shouldn't be this slow? [was: branch]

[Alan Mackenzie]

Hi, Óscar,

On Wed, Jan 06, 2010 at 03:06:37PM +0100, Óscar Fuentes wrote:
> Jason Rumney <jasonr@gnu.org> writes:

> > Alan Mackenzie wrote:

> >>>> Having created a local copy of the bzr repository in my directory
> >>>> ~/emacs/emacs.bzr/trunk, I then went to create the now familiar
> >>>> quickfixes branch from it, with this command:



> >>>>     $ time bzr branch emacs.bzr/trunk/ quickfixes/


> >> Ah, right.  The .../quickfixes/ needs to be physically under
> >> .../trunk, not just "related" to it.  Thanks!


> > I fear you may be heading for another 39 minute wait now.  quickfixes
> > needs to be under emacs/emacs.bzr, alongside trunk/, not under it.

Thanks!  Did that, and it took 33 seconds.  That's OK.

> As far as the directories containing the branches are under the
> directory that acts as the shared repository, you will benefit from the
> fast operations the shared branch brings in. You can move and rename
> branches within the shared repository (as long as they are not
> referenced by other branches: if you work with the recommended
> decentralized workflow, renaming `trunk' will require some tweaking on
> the other branches.)

OK.  Hopefully the other branches are referred to via a relative path
rather than an absolute one.

> 39 minutes for branching outside the shared repository corresponds to a
> slow machine. It is 10 minutes on a GNU/Linux 2.4 GHz 64 bits
> worksation-class machine. Bazaar is no speed daemon, but in this case it
> has to process more than 200 MB of data (not merely copying it around.)

I beg your pardon?  My machine is a 1.2 GHz Athlon.  That is NOT a slow
machine by any measure, except that even faster machines are now common.
Why does Bazaar need to "process" this data?  It's essentially doing
copying, with some accompanying administrivia.  Is it doing heavy number
crunching in Python, when it really needs a C module, or something like
that?

I just did a 'bzr update' on my .../trunk.  It took 23 minutes,
transferring nearly 200Mb to/from savannah in the process.  This compares
with all our source files (.c, .h, .el) being ~64Mb.  Could it be that
'bzr update' just downloads the whole repository again?  Or has somebody
else raised this issue on another thread that I've missed?

There seems to be a substantial mismatch between the assumptions of the
bzr project and the realities of the Emacs project.  My impression is
that bzr is so slow as to be barely usable at the moment.

> -- 
> Óscar

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Surely 'bzr update' shouldn't be this slow?

[Óscar Fuentes]

Alan Mackenzie <acm@muc.de> writes:

[snip]

>> 39 minutes for branching outside the shared repository corresponds to a
>> slow machine. It is 10 minutes on a GNU/Linux 2.4 GHz 64 bits
>> worksation-class machine. Bazaar is no speed daemon, but in this case it
>> has to process more than 200 MB of data (not merely copying it around.)
>
> I beg your pardon?  My machine is a 1.2 GHz Athlon.  That is NOT a slow
> machine by any measure, except that even faster machines are now
> common.

I think that that is the definition of "slow machine" :-)

> Why does Bazaar need to "process" this data?  It's essentially doing
> copying, with some accompanying administrivia.

It is not copying, and in that case it can't simply copy the data. A
shared repository contains data from all the branches it encloses. When
you branch from it, only the data that belongs to the branch you want
must be transferred.

I don't know why Bazaar does not copy the files under .bzr when you
clone a branch when it is on the same format that the target,
though. A good question for it ml.

> Is it doing heavy number crunching in Python, when it really needs a C
> module, or something like that?

The inner loops are written in C.

> I just did a 'bzr update' on my .../trunk.  It took 23 minutes,
> transferring nearly 200Mb to/from savannah in the process.  This compares
> with all our source files (.c, .h, .el) being ~64Mb.  Could it be that
> 'bzr update' just downloads the whole repository again?  Or has somebody
> else raised this issue on another thread that I've missed?

Yup, that was recently discussed here. It was an exceptional case (I
hope).

> There seems to be a substantial mismatch between the assumptions of the
> bzr project and the realities of the Emacs project.  My impression is
> that bzr is so slow as to be barely usable at the moment.

Apart from the 200MB download, do you think that bzr is too slow on your
daily Emacs work? Which operations are too slow for you?

Sometimes I work on a netbook, which maybe is comparable to your 1.2GHz
Athlon on CPU power, and some operations are slow indeed (>20 seconds
for the log of a file.) I raised this on the Bazaar ml, but most people
there think that Bazaar is fast enough and no extensive work is planned
for performance improvements.

-- 
Óscar

Re: Surely 'bzr update' shouldn't be this slow?

[Lennart Borgman]

On Thu, Jan 7, 2010 at 2:56 PM, Óscar Fuentes <ofv@wanadoo.es> wrote:

> Sometimes I work on a netbook, which maybe is comparable to your 1.2GHz
> Athlon on CPU power, and some operations are slow indeed (>20 seconds
> for the log of a file.) I raised this on the Bazaar ml, but most people
> there think that Bazaar is fast enough and no extensive work is planned
> for performance improvements.


If that is the case I really wonder what is wrong. We use to complain
that software from MS needs faster and faster machines.

It should of course not be like that for free software. It should not
pollute. It should not stop people who can't afford faster machines.

Re: Surely 'bzr update' shouldn't be this slow?

[Jan Djärv]

Lennart Borgman skrev:
> On Thu, Jan 7, 2010 at 2:56 PM, Óscar Fuentes <ofv@wanadoo.es> wrote:
> 
>> Sometimes I work on a netbook, which maybe is comparable to your 1.2GHz
>> Athlon on CPU power, and some operations are slow indeed (>20 seconds
>> for the log of a file.) I raised this on the Bazaar ml, but most people
>> there think that Bazaar is fast enough and no extensive work is planned
>> for performance improvements.
> 
> 
> If that is the case I really wonder what is wrong. 

With bzr or the bzr developers? :-)
A bzr pull for the trunk takes 25-30 seconds when there is nothing to pull.
CVS took 4-5 seconds in the same case.  This is critical, as pulling from the 
trunk is done quite often.  Other operations are also slow (commit to the 
trunk for instance).  It is on the edge of usable.

Maybe it is bad to compare to git:

- Bzr is slow!
- Compared to what?
- Git.
- Well, git is superfast compared to anything, that means bzr is not really
   slow.

	Jan D.

Re: Surely 'bzr update' shouldn't be this slow?

[Alan Mackenzie]

Hi, Óscar,

On Thu, Jan 07, 2010 at 02:56:02PM +0100, Óscar Fuentes wrote:
> Alan Mackenzie <acm@muc.de> writes:

> [snip]

> > I beg your pardon?  My machine is a 1.2 GHz Athlon.  That is NOT a
> > slow machine by any measure, except that even faster machines are now
> > common.

> I think that that is the definition of "slow machine" :-)

It's a machine which is more than fast enough for every aspect of Emacs
development other than running bzr.

> > Why does Bazaar need to "process" this data?  It's essentially doing
> > copying, with some accompanying administrivia.

> It is not copying, and in that case it can't simply copy the data. A
> shared repository contains data from all the branches it encloses. When
> you branch from it, only the data that belongs to the branch you want
> must be transferred.

It IS copying, conceptually - the content of the branched repository
contains a portion of the original branch, and nothing else.  So it has
to do a bit of filtering on the files' logs.  There are ~2000 files in
Emacs, and that copying/filtering took 39 minutes, about 2300 seconds.
It's taking over a second per file (at ~100% CPU usage).  A second is
about how long my PC takes to compile each C file in .../emacs/src.

> I don't know why Bazaar does not copy the files under .bzr when you
> clone a branch when it is on the same format that the target,
> though. A good question for it ml.

> > Is it doing heavy number crunching in Python, when it really needs a C
> > module, or something like that?

> The inner loops are written in C.

> > I just did a 'bzr update' on my .../trunk.  It took 23 minutes,
> > transferring nearly 200Mb to/from savannah in the process.  This
> > compares with all our source files (.c, .h, .el) being ~64Mb.  Could
> > it be that 'bzr update' just downloads the whole repository again?
> > Or has somebody else raised this issue on another thread that I've
> > missed?

> Yup, that was recently discussed here. It was an exceptional case (I
> hope).

There have been, perhaps, ~100 files updated since I first downloaded the
repository last Friday.  For each changed file, bzr transferred ~2Mb
between my PC and savannah.  Why?  This is ludicrous.

Hopefully it was an exceptional case, but I'd not changed my .../trunk at
all since downloading it, so anything exceptional was at the savannah
end.

I'm about to fix a bug which will involve ~100 bytes change to a C file
and ~200 bytes log message and ChangeLog addition.  How much will the bzr
commit operation transfer?  Hopefully, several kilobytes, no more.

Any chance you could point out that other thread to me?

> > There seems to be a substantial mismatch between the assumptions of
> > the bzr project and the realities of the Emacs project.  My
> > impression is that bzr is so slow as to be barely usable at the
> > moment.

> Apart from the 200MB download, do you think that bzr is too slow on your
> daily Emacs work? Which operations are too slow for you?

Yes, bzr is too slow for me.  My first checkout took, perhaps, an hour
and a half, but I can cope with that.  'bzr branch' (to a random place)
took 40 minutes.  'bzr branch' to the Right Place took a few seconds, and
this is the only bzr thing I've done so far which has been reasonable.
'bzr update' took 23 minutes, and this is the killer, the operation which
will make Emacs development such a frustrating, miserable experience; on
CVS, it would have been faster on my 33MHz 486 with 33kbaud modem.

> Sometimes I work on a netbook, which maybe is comparable to your 1.2GHz
> Athlon on CPU power, and some operations are slow indeed (>20 seconds
> for the log of a file.) I raised this on the Bazaar ml, but most people
> there think that Bazaar is fast enough and no extensive work is planned
> for performance improvements.

Their basic assumptions don't match the Emacs project, for whatever
reason.

> -- 
> Óscar

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Surely 'bzr update' shouldn't be this slow?

[Juanma Barranquero]

On Thu, Jan 7, 2010 at 15:52, Alan Mackenzie <acm@muc.de> wrote:

> Hopefully it was an exceptional case, but I'd not changed my .../trunk at
> all since downloading it, so anything exceptional was at the savannah
> end.

Yes.

> I'm about to fix a bug which will involve ~100 bytes change to a C file
> and ~200 bytes log message and ChangeLog addition.  How much will the bzr
> commit operation transfer?  Hopefully, several kilobytes, no more.

src/ChangeLog is already ~734 KiB.

> Yes, bzr is too slow for me.  My first checkout took, perhaps, an hour
> and a half, but I can cope with that.

OK.

> 'bzr branch' (to a random place)
> took 40 minutes.

Slow, but it's not a common operation (branching and not using a
shared repo, I mean).

> 'bzr update' took 23 minutes, and this is the killer, the operation which
> will make Emacs development such a frustrating, miserable experience; on
> CVS, it would have been faster on my 33MHz 486 with 33kbaud modem.

I just did "bzr up" and it took < 9s. You are generalizing from what
is already been described as an exceptional situation.

> Their basic assumptions don't match the Emacs project, for whatever
> reason.

Their basic assumption, which seems quite reasonable, is that people
will use the smart server, not sftp, for such a big project.

    Juanma

Re: Surely 'bzr update' shouldn't be this slow?

[Karl Fogel]

Juanma Barranquero <lekktu@gmail.com> writes:
>> Their basic assumptions don't match the Emacs project, for whatever
>> reason.
>
>Their basic assumption, which seems quite reasonable, is that people
>will use the smart server, not sftp, for such a big project.

Yes.  I *think* we are having a savannah problem here, not a bzr
problem.  People may find it faster to use the Launchpad.net mirror of
Emacs until Savannah gets bzr+ssh:// access (smart server).

-Karl

Re: Surely 'bzr update' shouldn't be this slow?

[Glenn Morris]

Karl Fogel wrote:

> Yes.  I *think* we are having a savannah problem here, not a bzr
> problem.  People may find it faster to use the Launchpad.net mirror of
> Emacs until Savannah gets bzr+ssh:// access (smart server).

Have you tried asking rms to ask the Savannah admins to please make
improving support for GNU bzr a high priority?

Re: Surely 'bzr update' shouldn't be this slow?

[Karl Fogel]

Glenn Morris <rgm@gnu.org> writes:
>Karl Fogel wrote:
>
>> Yes.  I *think* we are having a savannah problem here, not a bzr
>> problem.  People may find it faster to use the Launchpad.net mirror of
>> Emacs until Savannah gets bzr+ssh:// access (smart server).
>
>Have you tried asking rms to ask the Savannah admins to please make
>improving support for GNU bzr a high priority?

I've made similar requests of RMS before, and his response was:

  "Mail the Savannah admins list and CC me."

which seems like a reasonable course of action.

Someone here want to do it?

Re: Surely 'bzr update' shouldn't be this slow?

[Glenn Morris]

Karl Fogel wrote:

> Someone here want to do it?

I see Stefan (and you) already said pretty much all there is to say
back in November:

https://savannah.gnu.org/support/?107077#comment13

It tailed off into a discussion of commit emails, but that's resolved
now isn't it?

Re: Surely 'bzr update' shouldn't be this slow?

[Eli Zaretskii]

> From: Karl Fogel <kfogel@red-bean.com>
> Date: Thu, 07 Jan 2010 15:48:00 -0500
> Cc: =?utf-8?Q?=C3=93scar?= Fuentes <ofv@wanadoo.es>,
> 	Juanma Barranquero <lekktu@gmail.com>, emacs-devel@gnu.org,
> 	Alan Mackenzie <acm@muc.de>
> 
> Glenn Morris <rgm@gnu.org> writes:
> >Karl Fogel wrote:
> >
> >> Yes.  I *think* we are having a savannah problem here, not a bzr
> >> problem.  People may find it faster to use the Launchpad.net mirror of
> >> Emacs until Savannah gets bzr+ssh:// access (smart server).
> >
> >Have you tried asking rms to ask the Savannah admins to please make
> >improving support for GNU bzr a high priority?
> 
> I've made similar requests of RMS before, and his response was:
> 
>   "Mail the Savannah admins list and CC me."
> 
> which seems like a reasonable course of action.
> 
> Someone here want to do it?

Done.

Re: Surely 'bzr update' shouldn't be this slow?

[Óscar Fuentes]

Alan Mackenzie <acm@muc.de> writes:

[snip]

>> > Why does Bazaar need to "process" this data?  It's essentially doing
>> > copying, with some accompanying administrivia.
>
>> It is not copying, and in that case it can't simply copy the data. A
>> shared repository contains data from all the branches it encloses. When
>> you branch from it, only the data that belongs to the branch you want
>> must be transferred.
>
> It IS copying, conceptually - the content of the branched repository
> contains a portion of the original branch, and nothing else.  So it has
> to do a bit of filtering on the files' logs.  There are ~2000 files in
> Emacs, and that copying/filtering took 39 minutes, about 2300 seconds.
> It's taking over a second per file (at ~100% CPU usage).  A second is
> about how long my PC takes to compile each C file in .../emacs/src.

Things are not so simple. Please note that bzr stores almost 100,000
revisions of Emacs development in a bit more that 200 MB. And each
revision is related to other revisions and that relations needs
traversing. It is obvious that the storage system is far from trivial,
so any complexity assumption based on concepts like files or chunks of
bytes are, very likely, wrong.

>> Yup, that was recently discussed here. It was an exceptional case (I
>> hope).
>
> There have been, perhaps, ~100 files updated since I first downloaded the
> repository last Friday.  For each changed file, bzr transferred ~2Mb
> between my PC and savannah.  Why?  This is ludicrous.
>
> Hopefully it was an exceptional case, but I'd not changed my .../trunk at
> all since downloading it, so anything exceptional was at the savannah
> end.
>
> I'm about to fix a bug which will involve ~100 bytes change to a C file
> and ~200 bytes log message and ChangeLog addition.  How much will the bzr
> commit operation transfer?  Hopefully, several kilobytes, no more.

When bzr communicates with upstream by the http or sftp protocols, it
needs to read or write lots of data because the needed information is
scattered all over the repository files.

In theory, installing the Bazaar smart server in Savannah should fix
those problems.

> Any chance you could point out that other thread to me?

The subthread starts here:

http://article.gmane.org/gmane.emacs.devel/119410

>> Apart from the 200MB download, do you think that bzr is too slow on your
>> daily Emacs work? Which operations are too slow for you?
>
> Yes, bzr is too slow for me.  My first checkout took, perhaps, an hour
> and a half, but I can cope with that.  'bzr branch' (to a random place)
> took 40 minutes.  'bzr branch' to the Right Place took a few seconds, and
> this is the only bzr thing I've done so far which has been reasonable.
> 'bzr update' took 23 minutes, and this is the killer, the operation which
> will make Emacs development such a frustrating, miserable experience; on
> CVS, it would have been faster on my 33MHz 486 with 33kbaud modem.

If the exceptionally long time that it took for certain update is the
biggest problem you found so far, there is no reason to worry. Because
it is not usual (except for that case, bzr needs less than 3 minutes for
updating on my netbook) and because if it becomes too frequent people
here will force the Savannah maintainers to install the smart server.

[snip]

-- 
Óscar

Re: Surely 'bzr update' shouldn't be this slow?

[Stephen J. Turnbull]

Alan Mackenzie writes:

 > OK.  Hopefully the other branches are referred to via a relative path
 > rather than an absolute one.

Paths are resolved in the usual way on GNU/Linux and other POSIX
systems (ie, the shell globs, evaluates environment variable
references, and interprets special path elements like ~), and bzr
checks for an absolute path and otherwise resolves relative to CWD.

Windows is more complex but that's not relevant to you, right?

 > On Thu, Jan 07, 2010 at 02:56:02PM +0100, Óscar Fuentes wrote:
 > > Alan Mackenzie <acm@muc.de> writes:

 > > > Why does Bazaar need to "process" this data?  It's essentially doing
 > > > copying, with some accompanying administrivia.
 > 
 > > It is not copying, [...].
 > 
 > It IS copying, conceptually

Alan, don't be contrary.  Óscar is just telling you the facts, and
AIUI *he* would not have chosen bzr if it were up to him.  He's just
trying to help you and others make the best use of the official VCS.

 > - the content of the branched repository contains a portion of the
 > original branch, and nothing else.  So it has to do a bit of
 > filtering on the files' logs.

No, it has to do a lot more than that.  What it is doing is
conceptually most like garbage collection.  The actual content of a
repository is stored as a set of compressed archives of file objects,
revision objects, and patches, plus some indicies.  (These archives
are called "packs.")  The "processing" goes through those objects,
throws out the ones not needed for the branch you request, and
reconstructs minimal packs and indicies for them.

 > There have been, perhaps, ~100 files updated since I first downloaded the
 > repository last Friday.  For each changed file, bzr transferred ~2Mb
 > between my PC and savannah.  Why?  This is ludicrous.
 > 
 > Hopefully it was an exceptional case, but I'd not changed my .../trunk at
 > all since downloading it, so anything exceptional was at the savannah
 > end.

Exactly!  At this date, the main cause of that particular slowness is
not bzr, it's savannah-hackers, which chose the absolute worst
available way to implement bzr support.  (They had their reasons,
security-related, which given Savannah's history as a target seem
reasonably important to me.)

 > I'm about to fix a bug which will involve ~100 bytes change to a C file
 > and ~200 bytes log message and ChangeLog addition.  How much will the bzr
 > commit operation transfer?  Hopefully, several kilobytes, no more.

As much as it needs to, but until the issue with Savannah gets
addressed, the minimum you should hope for is one pack and the index
file, typically a few MB.

You'll know when it gets fixed because there will be an announcement
that you should use "bzr+ssh" URLs, not "http" or "sftp".

 > Any chance you could point out that other thread to me?

The issue that bzr can be damned slow if you don't tune your workflow
has been pretty well constantly active since late 2007.  I guess it
just faded into the rest of the background noise for you. :-)

 > Yes, bzr is too slow for me.  My first checkout took, perhaps, an hour
 > and a half, but I can cope with that.

By "checkout" you mean "bzr branch http://savannah.../path/to/emacs/trunk",
right?  ("bzr checkout" does something completely different and should
not take more than a handful of seconds.  Please don't remark about the
randomness of bzr command names, this particular choice makes sense in
the appropriate context.)

The 90-minute initial branch is something we'll just have to deal
with (probably you personally won't actually need to do it again!)

BTW, "bzr branch http://bzr.savannah.gnu.org/r/emacs/trunk emacs" just
now took 31:30 on my MacBook Pro (2.4GHz Intel Core2 Duo, 2GB RAM, Mac
OS X 10.5.8, Python 2.6.4) with bzr 2.0.1.  As others report, this is
CPU-bound, using 97% of one core.

 > 'bzr branch' (to a random place) took 40 minutes.

Are you sure the "random place" wasn't your CD-RW?  Just joking, but
that seems awfully long to me ("bzr branch emacs emacs.1" took 584
seconds, under 10 minutes for me, same box as above).  The only recent
reports I've seen on the Bazaar list in that league involve Windows
file systems or NFS mounts, so I wonder if there's something unusual
about your setup.  Are you on NFS?  If not, may I ask you to confirm
that your disk performance doesn't normally noticably suck, etc?

 > 'bzr branch' to the Right Place took a few seconds, and
 > this is the only bzr thing I've done so far which has been
 > reasonable.

It's also one of the things you're likely to do fairly frequently, and
minimizing its cost removes an important barrier to adopting best
practice.  So that's a good sign, even if so far it's rare in your
experience.

 > 'bzr update' took 23 minutes,

This is probably a one-off.  See the "BZR error!" thread, where Stefan
explains.

And the underlying problem is probably the Savannah misconfiguration,
which will eventually get fixed.

In any case, once you get the branch(es) you need locally, most
operations can be done locally, and with a little bit of care are all
reasonably fast.  Even while the Savannah issue remains open, once
Stefan et al finish tuning the arrangement of stuff in the main repo
(probably already done, actually), if you update your local branch
mirrors at least a couple of times a week you can probably hope to
limit each download to 10 or 20MB.

Or you can use the Launchpad mirror, as others have suggested.

 > Their basic assumptions don't match the Emacs project, for whatever
 > reason.

bzr was chosen for political reasons (it was the only GNU project
among the contenders), not because it was the best fit to existing
Emacs project practice.  The Emacs project therefore must adapt, and
that's part of the deal for Emacs developers.

It really shouldn't be that hard, but managing transitions in the GNU
Project is hard because the admins are spread so thinly.  Expect a few
more bumps, but really, there's no reason to expect that problems will
continue to be this frequent for more than a couple more weeks.

Re: Surely 'bzr update' shouldn't be this slow?

[Stefan Monnier]

> I beg your pardon?  My machine is a 1.2 GHz Athlon.  That is NOT a slow
> machine by any measure, except that even faster machines are now common.

Agreed.  I use a similar machine every once in a while (Thinkpad X30)
and it's perfectly useable.  I haven't used it extensively for Emacs
hacking of late, but I have done some "bzr updates" of Emacs there and
didn't find it unbearable.
Note that Bzr is pretty memory hungry as well, so that might make
a larger difference (I have 768MB in my smallest/oldest/slowest
machine).

> Why does Bazaar need to "process" this data?  It's essentially doing
> copying, with some accompanying administrivia.  Is it doing heavy number
> crunching in Python, when it really needs a C module, or something like
> that?

The technical reason is that the repository (whether shared or not)
contains all the info for the branch you're copying, plus some more
(for other branches that are, or have been stored there, or other branches
which you have referred to from there (e.g. in a diff, merge, ...)).
So Bzr basically does a GC along the way, only extracting the bits that
are necessary, which means it unpacks and then repacks pretty much the
whole thing.

> I just did a 'bzr update' on my .../trunk.  It took 23 minutes,
> transferring nearly 200Mb to/from savannah in the process.  This compares
> with all our source files (.c, .h, .el) being ~64Mb.  Could it be that
> 'bzr update' just downloads the whole repository again?

Yes.  As mentioned, if the remote repository could be accessed via
bzr+ssh rather than sftp, it would be a bit better.

> Or has somebody else raised this issue on another thread that
> I've missed?

Of course.


        Stefan

Re: Surely 'bzr branch' shouldn't be this slow?

[Stefan Monnier]

> But it still feels a bit strange that creating a branch in the "wrong"
> place takes up 39 minutes of processing.

There's no question it's long.  It's the same reason why the original
checkout over the network is slow: a large part of the time is spent
waiting for Bazaar to decide what to do next and where to put what,
rather than waiting for the data to arrive from the remote end.


        Stefan

Re: Surely 'bzr branch' shouldn't be this slow?

[Karl Fogel]

Alan Mackenzie <acm@muc.de> writes:
>Ah, right.  The .../quickfixes/ needs to be physically under .../trunk,
>not just "related" to it.  Thanks!

Wait, hunh?  No, they should be siblings... something sounds wrong here.

-Karl