Re: bug#8025: 24.0.50; vc-bzr does not perform initial commit

Re: bug#8025: 24.0.50; vc-bzr does not perform initial commit

[Christoph Scholtes]

I looked into this issue yesterday and I narrowed it down to the 
following function in `vc-bzr.el': `vc-bzr-state-heuristic'.

Apparently, when there is a bare repo and when trying to register the 
first file, the heuristic function, which parses the dir-state file, 
fails and reports the file unregistered all the time, even after it has 
been registered. It starts working correctly when you register the file 
and commit the file externally. I have not been able to figure out why 
the parsing fails.

It also works when bypassing the heuristic function and calling 
`vc-bzr-state' all the time.

I am wondering whether the heuristic function should be removed/simplified.

There is a comment at the beginning of the function:
   ;; `bzr status' was excruciatingly slow with large histories and
   ;; pending merges, so try to avoid using it until they fix their
   ;; performance problems.
   ;; This function tries first to parse Bzr internal file
   ;; `checkout/dirstate', but it may fail if Bzr internal file format
   ;; has changed.  As a safeguard, the `checkout/dirstate' file is
   ;; only parsed if it contains the string `#bazaar dirstate flat
   ;; format 3' in the first line.

`excruciatingly slow' does not mean anything to me. Is this still an 
issue with the latest version of bzr? If we can be reasonably sure it is 
not, we should just have it call `vc-bzr-state' all the time and 
eliminate this fragile construct of parsing the file manually. I think 
if bzr has a performance issue bzr should be fixed not emacs.

So, my question is: do I spent the time trying to fix this issue or can 
we agree to make bzr work like any of the other backends and call its 
native stat function all the time?

Also, does anybody have actual performance numbers that would support 
keeping this function around?

Christoph

bug#8025: 24.0.50; vc-bzr does not perform initial commit

[Eli Zaretskii]

> Date: Tue, 01 Mar 2011 09:19:47 -0700
> From: Christoph Scholtes <cschol2112@googlemail.com>
> CC: emacs-devel@gnu.org, Eli Zaretskii <eliz@gnu.org>
> 
> So, my question is: do I spent the time trying to fix this issue or can 
> we agree to make bzr work like any of the other backends and call its 
> native stat function all the time?

It would be good to at least understand why it fails.  That file is
full of binary nulls, so this could be something specific to Windows.

> Also, does anybody have actual performance numbers that would support 
> keeping this function around?

Can you give performance numbers with the current code in the Emacs
trunk branch?

Re: bug#8025: 24.0.50; vc-bzr does not perform initial commit

[Glenn Morris]

Also remember there is the annoying bzr locking issue. Eg doing `bzr up'
renders `bzr status' unusable so long as it is running. So it is not
just the speed of the status command that is a factor.

Personally I'd like to see the heuristic stay (based on zero real data).

Re: bug#8025: 24.0.50; vc-bzr does not perform initial commit

[Glenn Morris]

Very lightly tested patch. Seems like not all of the fields are present
in a freshly minted repo.

*** lisp/vc/vc-bzr.el	2011-02-19 21:23:51 +0000
--- lisp/vc/vc-bzr.el	2011-03-02 03:58:40 +0000
***************
*** 210,222 ****
                                 ;; was executable the last time bzr checked?
                                 "[^\0]*\0"
                                 "[^\0]*\0"       ;?
!                                "\\([^\0]*\\)\0" ;"a/f/d" a=added?
                                 "\\([^\0]*\\)\0" ;sha1 again?
                                 "\\([^\0]*\\)\0" ;size again?
                                 ;; y/n.  Whether or not the repo thinks
                                 ;; the file should be executable?
                                 "\\([^\0]*\\)\0"
!                                "[^\0]*\0" ;last revid?
                                 ;; There are more fields when merges are pending.
                                 )
                         nil t)
--- 210,222 ----
                                 ;; was executable the last time bzr checked?
                                 "[^\0]*\0"
                                 "[^\0]*\0"       ;?
!                                "\\(?:\\([^\0]*\\)\0" ;"a/f/d" a=added?
                                 "\\([^\0]*\\)\0" ;sha1 again?
                                 "\\([^\0]*\\)\0" ;size again?
                                 ;; y/n.  Whether or not the repo thinks
                                 ;; the file should be executable?
                                 "\\([^\0]*\\)\0"
!                                "[^\0]*\0\\)?" ;last revid?
                                 ;; There are more fields when merges are pending.
                                 )
                         nil t)
***************
*** 225,230 ****
--- 225,231 ----
                        ;; first size seems to correspond to the file with
                        ;; conflict markers).
                        (cond
+                        ((not (match-beginning 4)) 'added)
                         ((eq (char-after (match-beginning 1)) ?a) 'removed)
                         ((eq (char-after (match-beginning 4)) ?a) 'added)
                         ((or (and (eq (string-to-number (match-string 3))

Re: bug#8025: 24.0.50; vc-bzr does not perform initial commit

[Christoph Scholtes]

Eli Zaretskii <eliz@gnu.org> writes:

> It would be good to at least understand why it fails.  

I agree. I just saw that Glenn started looking into it.

> That file is full of binary nulls, so this could be something specific
> to Windows.

No, unfortunately it is not. I tested it on ArchLinux with the latest
emacs trunk and bzr 2.3.0 and it behaves exactly the same.

> Can you give performance numbers with the current code in the Emacs
> trunk branch?

I did some testing with elp on Windows.

Test case: Instrument functions with `vc-' prefix with
elp-instrument-package.  Modified BUGS in the trunk root. C-x v v to
commit with buffer asking to enter commit message.  C-x k to kill buffer
and abort commit.  Get results with elp-results.

1. Using stock vc-bzr.el from the trunk.

Result:
vc-bzr-state-heuristic     3     0.174            0.0579999999
                                                  ^^^^^^^^^^^^

2. Using modified vc-bzr.el forcing vc-bzr-state-heuristic function to use
vc-bzr-state function instead of its own logic.

Result:
vc-bzr-state-heuristic     3     1.5230000000     0.5076666666
                                                  ^^^^^^^^^^^^

Roughly an order of 10 difference, but 0.5s is not really `excruciatingly
slow' in my book.

Christoph

Re: bug#8025: 24.0.50; vc-bzr does not perform initial commit

[Christoph Scholtes]

Glenn Morris <rgm@gnu.org> writes:

> Also remember there is the annoying bzr locking issue. Eg doing `bzr up'
> renders `bzr status' unusable so long as it is running. So it is not
> just the speed of the status command that is a factor.

OK, but one could argue that this is a bzr issue and should be fixed
there and not "dealt-with" in emacs. Do you know if it has been reported
to the bzr team?

> Personally I'd like to see the heuristic stay (based on zero real data).

In general, I have no problem with that. As long as it works. ;)

Re: bug#8025: 24.0.50; vc-bzr does not perform initial commit

[Christoph Scholtes]

Glenn Morris <rgm@gnu.org> writes:

> Very lightly tested patch. Seems like not all of the fields are present
> in a freshly minted repo.

Thanks Glenn. This patch fixed the problem in my initial bug
report. Everything now works as expected.

Re: bug#8025: 24.0.50; vc-bzr does not perform initial commit

[Stefan Monnier]

> `excruciatingly slow' does not mean anything to me.

Like several seconds.
Current bzr is much better in this respect.  If we can fix the function,
of course it's still better since it's not only faster but even works on
remote hosts and in the absence of a bzr binary.


        Stefan

bug#8025: 24.0.50; vc-bzr does not perform initial commit

[Eli Zaretskii]

> From: Christoph Scholtes <cschol2112@googlemail.com>
> Cc: 8025@debbugs.gnu.org,  emacs-devel@gnu.org
> Date: Tue, 01 Mar 2011 22:55:46 -0700
> 
> 1. Using stock vc-bzr.el from the trunk.
> 
> Result:
> vc-bzr-state-heuristic     3     0.174            0.0579999999
>                                                   ^^^^^^^^^^^^
> 
> 2. Using modified vc-bzr.el forcing vc-bzr-state-heuristic function to use
> vc-bzr-state function instead of its own logic.
> 
> Result:
> vc-bzr-state-heuristic     3     1.5230000000     0.5076666666
>                                                   ^^^^^^^^^^^^
> 
> Roughly an order of 10 difference, but 0.5s is not really `excruciatingly
> slow' in my book.

No, it isn't.  Although it would be interesting to see the same test
with a cold cache.

Re: bug#8025: 24.0.50; vc-bzr does not perform initial commit

[Eli Zaretskii]

> From: Christoph Scholtes <cschol2112@googlemail.com>
> Cc: Eli Zaretskii <eliz@gnu.org>,  8025@debbugs.gnu.org,  emacs-devel@gnu.org
> Date: Tue, 01 Mar 2011 23:00:22 -0700
> 
> Glenn Morris <rgm@gnu.org> writes:
> 
> > Also remember there is the annoying bzr locking issue. Eg doing `bzr up'
> > renders `bzr status' unusable so long as it is running. So it is not
> > just the speed of the status command that is a factor.
> 
> OK, but one could argue that this is a bzr issue and should be fixed
> there

It cannot be fixed in bzr, not unless they redesign and rewrite the
branch locking code from scratch.  Atomicity of transactions is
important, so locks on the directory or some of its files must be used
at least at some strategic points of time.