bug#8025: 24.0.50; vc-bzr does not perform initial commit

bug#8025: 24.0.50; vc-bzr does not perform initial commit

[Christoph]

Running emacs 24.0.50 r103325 on Windows 7.

In a cmd window in `C:/' do the following:
mkdir "my temp"
cd "my temp"
echo test > test.txt
Open test.txt with Emacs.
M-x vc-next-action

Minibuffer output: Registering (c:/my temp/test.txt)... done

M-x vc-next-action

Minibuffer output: Previous master file has vanished.  Make a new one? (y
or n)

Answering yes, the file is registered again. Answering no aborts the
process.

However, the next action should be committing the file to the repository. 

If I commit the file externally in a cmd window with `hg commit', then
edit the file in Emacs and do M-x vc-next-action, the file is committed
correctly.



In GNU Emacs 24.0.50.1 (i386-mingw-nt6.1.7600)
 of 2011-02-12 on MARVIN
Windowing system distributor `Microsoft Corp.', version 6.1.7600
configured using `configure --with-gcc (4.5) --cflags -IC:/Progra~2/GnuWin32/include -ID:/devel/emacs/libXpm-3.5.8/include -ID:/devel/emacs/libXpm-3.5.8/src'

Important settings:
  value of $LC_ALL: nil
  value of $LC_COLLATE: nil
  value of $LC_CTYPE: nil
  value of $LC_MESSAGES: nil
  value of $LC_MONETARY: nil
  value of $LC_NUMERIC: nil
  value of $LC_TIME: nil
  value of $LANG: ENU
  value of $XMODIFIERS: nil
  locale-coding-system: cp1252
  default enable-multibyte-characters: t

Major mode: Fundamental

Minor modes in effect:
  ido-everywhere: t
  yas/global-mode: t
  yas/minor-mode: t
  global-auto-revert-mode: t
  tooltip-mode: t
  mouse-wheel-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  size-indication-mode: t
  column-number-mode: t
  line-number-mode: t

Recent input:
C-x RET r e p o <tab> r t - e m <tab> <return> v c 
- b a r <backspace> <backspace> z r SPC d o e s SPC 
n i t <backspace> <backspace> o t SPC C-g C-x P C-g 
C-x M-p M-p C-g C-x C-f M-p c : / m y <return> C-s 
<return> C-x RET M-[ M-p C-n C-x RET v c - n e x t 
C-g C-x RET v c - n e x t <tab> <return> C-x RET v 
<backspace> v c - n e x t <tab> <return> n C-d C-x 
C-s C-x b m e s s <return> <up> <up> <up> <up> <up> 
<up> <up> <down> <down> C-x RET <up> C-g C-x RET r 
e p o r t - e <tab> <return>

Recent messages:
Quit [2 times]
Loading vc-bzr...done
byte-code: End of buffer
read-extended-command: Command attempted to use minibuffer while in minibuffer
Quit
Registering (c:/my temp/test.txt)... done
Previous master file has vanished.  Make a new one? (y or n)  n
vc-register: Aborted
Saving file c:/my temp/test.txt...
Wrote c:/my temp/test.txt
Quit

Load-path shadows:
None found.

Features:
(shadow sort mail-extr message rfc822 mml mml-sec mm-decode mm-bodies
mm-encode mail-parse rfc2231 rfc2047 rfc2045 ietf-drums mailabbrev
gmm-utils mailheader vc-sccs vc-svn vc-cvs vc-rcs vc vc-dispatcher
vc-bzr sha1 hex-util emacsbug url-util url-parse auth-source netrc
gnus-util time-date url-vars mm-util mail-prsvr help-mode view server
js2-mode-autoloads rainbow-mode-autoloads package re-builder dired+
dired-x ediff-merg ediff-diff ediff-wind ediff-mult ediff-help
ediff-init ediff-util dired-aux ibuffer nav nav-tags python-21 python
nav-bufs anything-config warnings browse-url semantic/util-modes
semantic/util semantic semantic/tag semantic/lex semantic/fw loaddefs
eieio byte-opt bytecomp byte-compile mode-local cedet imenu bookmark pp
dired rx ffap thingatpt anything google-c-style cc-mode cc-fonts
cc-menus cc-cmds cc-styles cc-align cc-engine cc-vars cc-defs
grep-o-matic grep compile comint browse-kill-ring+ browse-kill-ring
second-sel ido yasnippet dropdown-list derived easy-mmode assoc
etags-table etags ring remember zenburn color-theme edmacro kmacro
wid-edit cl sendmail regexp-opt mail-utils reporter easymenu uniquify
advice help-fns advice-preload autorevert tooltip ediff-hook vc-hooks
lisp-float-type mwheel dos-w32 disp-table ls-lisp w32-win w32-vars
tool-bar dnd fontset image fringe lisp-mode register page menu-bar
rfn-eshadow timer select scroll-bar mouse jit-lock font-lock syntax
facemenu font-core frame cham georgian utf-8-lang misc-lang vietnamese
tibetan thai tai-viet lao korean japanese hebrew greek romanian slovak
czech european ethiopic indian cyrillic chinese case-table epa-hook
jka-cmpr-hook help simple abbrev button minibuffer faces cus-face files
text-properties overlay md5 base64 format env code-pages mule custom
widget hashtable-print-readable backquote make-network-process multi-tty
emacs)

bug#8025: 24.0.50; vc-bzr does not perform initial commit

[Christoph]

Christoph <cschol2112@googlemail.com> writes:

> Running emacs 24.0.50 r103325 on Windows 7.

This happens with the following bzr versions:

bzr 2.2.3
bzr 2.3.0

Christoph

bug#8025: 24.0.50; vc-bzr does not perform initial commit

[Eli Zaretskii]

> From: Christoph <cschol2112@googlemail.com>
> Date: Sat, 12 Feb 2011 12:18:11 -0700
> Cc: 
> 
> If I commit the file externally in a cmd window with `hg commit', then
                                                        ^^^^^^^^^
I guess you meant "bzr commit".

bug#8025: 24.0.50; vc-bzr does not perform initial commit

[Christoph]

Eli Zaretskii <eliz@gnu.org> writes:

> I guess you meant "bzr commit".

Yes. Sorry. I was troubleshooting two different issues and got confused.

bug#8025: 24.0.50; vc-bzr does not perform initial commit

[Christoph Scholtes]

I looked into this issue yesterday and I narrowed it down to the 
following function in `vc-bzr.el': `vc-bzr-state-heuristic'.

Apparently, when there is a bare repo and when trying to register the 
first file, the heuristic function, which parses the dir-state file, 
fails and reports the file unregistered all the time, even after it has 
been registered. It starts working correctly when you register the file 
and commit the file externally. I have not been able to figure out why 
the parsing fails.

It also works when bypassing the heuristic function and calling 
`vc-bzr-state' all the time.

I am wondering whether the heuristic function should be removed/simplified.

There is a comment at the beginning of the function:
   ;; `bzr status' was excruciatingly slow with large histories and
   ;; pending merges, so try to avoid using it until they fix their
   ;; performance problems.
   ;; This function tries first to parse Bzr internal file
   ;; `checkout/dirstate', but it may fail if Bzr internal file format
   ;; has changed.  As a safeguard, the `checkout/dirstate' file is
   ;; only parsed if it contains the string `#bazaar dirstate flat
   ;; format 3' in the first line.

`excruciatingly slow' does not mean anything to me. Is this still an 
issue with the latest version of bzr? If we can be reasonably sure it is 
not, we should just have it call `vc-bzr-state' all the time and 
eliminate this fragile construct of parsing the file manually. I think 
if bzr has a performance issue bzr should be fixed not emacs.

So, my question is: do I spent the time trying to fix this issue or can 
we agree to make bzr work like any of the other backends and call its 
native stat function all the time?

Also, does anybody have actual performance numbers that would support 
keeping this function around?

Christoph

Re: bug#8025: 24.0.50; vc-bzr does not perform initial commit

[Christoph Scholtes]

I looked into this issue yesterday and I narrowed it down to the 
following function in `vc-bzr.el': `vc-bzr-state-heuristic'.

Apparently, when there is a bare repo and when trying to register the 
first file, the heuristic function, which parses the dir-state file, 
fails and reports the file unregistered all the time, even after it has 
been registered. It starts working correctly when you register the file 
and commit the file externally. I have not been able to figure out why 
the parsing fails.

It also works when bypassing the heuristic function and calling 
`vc-bzr-state' all the time.

I am wondering whether the heuristic function should be removed/simplified.

There is a comment at the beginning of the function:
   ;; `bzr status' was excruciatingly slow with large histories and
   ;; pending merges, so try to avoid using it until they fix their
   ;; performance problems.
   ;; This function tries first to parse Bzr internal file
   ;; `checkout/dirstate', but it may fail if Bzr internal file format
   ;; has changed.  As a safeguard, the `checkout/dirstate' file is
   ;; only parsed if it contains the string `#bazaar dirstate flat
   ;; format 3' in the first line.

`excruciatingly slow' does not mean anything to me. Is this still an 
issue with the latest version of bzr? If we can be reasonably sure it is 
not, we should just have it call `vc-bzr-state' all the time and 
eliminate this fragile construct of parsing the file manually. I think 
if bzr has a performance issue bzr should be fixed not emacs.

So, my question is: do I spent the time trying to fix this issue or can 
we agree to make bzr work like any of the other backends and call its 
native stat function all the time?

Also, does anybody have actual performance numbers that would support 
keeping this function around?

Christoph

bug#8025: 24.0.50; vc-bzr does not perform initial commit

[Eli Zaretskii]

> Date: Tue, 01 Mar 2011 09:19:47 -0700
> From: Christoph Scholtes <cschol2112@googlemail.com>
> CC: emacs-devel@gnu.org, Eli Zaretskii <eliz@gnu.org>
> 
> So, my question is: do I spent the time trying to fix this issue or can 
> we agree to make bzr work like any of the other backends and call its 
> native stat function all the time?

It would be good to at least understand why it fails.  That file is
full of binary nulls, so this could be something specific to Windows.

> Also, does anybody have actual performance numbers that would support 
> keeping this function around?

Can you give performance numbers with the current code in the Emacs
trunk branch?

bug#8025: 24.0.50; vc-bzr does not perform initial commit

[Glenn Morris]

Also remember there is the annoying bzr locking issue. Eg doing `bzr up'
renders `bzr status' unusable so long as it is running. So it is not
just the speed of the status command that is a factor.

Personally I'd like to see the heuristic stay (based on zero real data).

Re: bug#8025: 24.0.50; vc-bzr does not perform initial commit

[Glenn Morris]

Also remember there is the annoying bzr locking issue. Eg doing `bzr up'
renders `bzr status' unusable so long as it is running. So it is not
just the speed of the status command that is a factor.

Personally I'd like to see the heuristic stay (based on zero real data).

bug#8025: 24.0.50; vc-bzr does not perform initial commit

[Glenn Morris]

Very lightly tested patch. Seems like not all of the fields are present
in a freshly minted repo.

*** lisp/vc/vc-bzr.el	2011-02-19 21:23:51 +0000
--- lisp/vc/vc-bzr.el	2011-03-02 03:58:40 +0000
***************
*** 210,222 ****
                                 ;; was executable the last time bzr checked?
                                 "[^\0]*\0"
                                 "[^\0]*\0"       ;?
!                                "\\([^\0]*\\)\0" ;"a/f/d" a=added?
                                 "\\([^\0]*\\)\0" ;sha1 again?
                                 "\\([^\0]*\\)\0" ;size again?
                                 ;; y/n.  Whether or not the repo thinks
                                 ;; the file should be executable?
                                 "\\([^\0]*\\)\0"
!                                "[^\0]*\0" ;last revid?
                                 ;; There are more fields when merges are pending.
                                 )
                         nil t)
--- 210,222 ----
                                 ;; was executable the last time bzr checked?
                                 "[^\0]*\0"
                                 "[^\0]*\0"       ;?
!                                "\\(?:\\([^\0]*\\)\0" ;"a/f/d" a=added?
                                 "\\([^\0]*\\)\0" ;sha1 again?
                                 "\\([^\0]*\\)\0" ;size again?
                                 ;; y/n.  Whether or not the repo thinks
                                 ;; the file should be executable?
                                 "\\([^\0]*\\)\0"
!                                "[^\0]*\0\\)?" ;last revid?
                                 ;; There are more fields when merges are pending.
                                 )
                         nil t)
***************
*** 225,230 ****
--- 225,231 ----
                        ;; first size seems to correspond to the file with
                        ;; conflict markers).
                        (cond
+                        ((not (match-beginning 4)) 'added)
                         ((eq (char-after (match-beginning 1)) ?a) 'removed)
                         ((eq (char-after (match-beginning 4)) ?a) 'added)
                         ((or (and (eq (string-to-number (match-string 3))

Re: bug#8025: 24.0.50; vc-bzr does not perform initial commit

[Glenn Morris]

Very lightly tested patch. Seems like not all of the fields are present
in a freshly minted repo.

*** lisp/vc/vc-bzr.el	2011-02-19 21:23:51 +0000
--- lisp/vc/vc-bzr.el	2011-03-02 03:58:40 +0000
***************
*** 210,222 ****
                                 ;; was executable the last time bzr checked?
                                 "[^\0]*\0"
                                 "[^\0]*\0"       ;?
!                                "\\([^\0]*\\)\0" ;"a/f/d" a=added?
                                 "\\([^\0]*\\)\0" ;sha1 again?
                                 "\\([^\0]*\\)\0" ;size again?
                                 ;; y/n.  Whether or not the repo thinks
                                 ;; the file should be executable?
                                 "\\([^\0]*\\)\0"
!                                "[^\0]*\0" ;last revid?
                                 ;; There are more fields when merges are pending.
                                 )
                         nil t)
--- 210,222 ----
                                 ;; was executable the last time bzr checked?
                                 "[^\0]*\0"
                                 "[^\0]*\0"       ;?
!                                "\\(?:\\([^\0]*\\)\0" ;"a/f/d" a=added?
                                 "\\([^\0]*\\)\0" ;sha1 again?
                                 "\\([^\0]*\\)\0" ;size again?
                                 ;; y/n.  Whether or not the repo thinks
                                 ;; the file should be executable?
                                 "\\([^\0]*\\)\0"
!                                "[^\0]*\0\\)?" ;last revid?
                                 ;; There are more fields when merges are pending.
                                 )
                         nil t)
***************
*** 225,230 ****
--- 225,231 ----
                        ;; first size seems to correspond to the file with
                        ;; conflict markers).
                        (cond
+                        ((not (match-beginning 4)) 'added)
                         ((eq (char-after (match-beginning 1)) ?a) 'removed)
                         ((eq (char-after (match-beginning 4)) ?a) 'added)
                         ((or (and (eq (string-to-number (match-string 3))

bug#8025: 24.0.50; vc-bzr does not perform initial commit

[Christoph Scholtes]

Eli Zaretskii <eliz@gnu.org> writes:

> It would be good to at least understand why it fails.  

I agree. I just saw that Glenn started looking into it.

> That file is full of binary nulls, so this could be something specific
> to Windows.

No, unfortunately it is not. I tested it on ArchLinux with the latest
emacs trunk and bzr 2.3.0 and it behaves exactly the same.

> Can you give performance numbers with the current code in the Emacs
> trunk branch?

I did some testing with elp on Windows.

Test case: Instrument functions with `vc-' prefix with
elp-instrument-package.  Modified BUGS in the trunk root. C-x v v to
commit with buffer asking to enter commit message.  C-x k to kill buffer
and abort commit.  Get results with elp-results.

1. Using stock vc-bzr.el from the trunk.

Result:
vc-bzr-state-heuristic     3     0.174            0.0579999999
                                                  ^^^^^^^^^^^^

2. Using modified vc-bzr.el forcing vc-bzr-state-heuristic function to use
vc-bzr-state function instead of its own logic.

Result:
vc-bzr-state-heuristic     3     1.5230000000     0.5076666666
                                                  ^^^^^^^^^^^^

Roughly an order of 10 difference, but 0.5s is not really `excruciatingly
slow' in my book.

Christoph

Re: bug#8025: 24.0.50; vc-bzr does not perform initial commit

[Christoph Scholtes]

Eli Zaretskii <eliz@gnu.org> writes:

> It would be good to at least understand why it fails.  

I agree. I just saw that Glenn started looking into it.

> That file is full of binary nulls, so this could be something specific
> to Windows.

No, unfortunately it is not. I tested it on ArchLinux with the latest
emacs trunk and bzr 2.3.0 and it behaves exactly the same.

> Can you give performance numbers with the current code in the Emacs
> trunk branch?

I did some testing with elp on Windows.

Test case: Instrument functions with `vc-' prefix with
elp-instrument-package.  Modified BUGS in the trunk root. C-x v v to
commit with buffer asking to enter commit message.  C-x k to kill buffer
and abort commit.  Get results with elp-results.

1. Using stock vc-bzr.el from the trunk.

Result:
vc-bzr-state-heuristic     3     0.174            0.0579999999
                                                  ^^^^^^^^^^^^

2. Using modified vc-bzr.el forcing vc-bzr-state-heuristic function to use
vc-bzr-state function instead of its own logic.

Result:
vc-bzr-state-heuristic     3     1.5230000000     0.5076666666
                                                  ^^^^^^^^^^^^

Roughly an order of 10 difference, but 0.5s is not really `excruciatingly
slow' in my book.

Christoph

bug#8025: 24.0.50; vc-bzr does not perform initial commit

[Christoph Scholtes]

Glenn Morris <rgm@gnu.org> writes:

> Also remember there is the annoying bzr locking issue. Eg doing `bzr up'
> renders `bzr status' unusable so long as it is running. So it is not
> just the speed of the status command that is a factor.

OK, but one could argue that this is a bzr issue and should be fixed
there and not "dealt-with" in emacs. Do you know if it has been reported
to the bzr team?

> Personally I'd like to see the heuristic stay (based on zero real data).

In general, I have no problem with that. As long as it works. ;)

Re: bug#8025: 24.0.50; vc-bzr does not perform initial commit

[Christoph Scholtes]

Glenn Morris <rgm@gnu.org> writes:

> Also remember there is the annoying bzr locking issue. Eg doing `bzr up'
> renders `bzr status' unusable so long as it is running. So it is not
> just the speed of the status command that is a factor.

OK, but one could argue that this is a bzr issue and should be fixed
there and not "dealt-with" in emacs. Do you know if it has been reported
to the bzr team?

> Personally I'd like to see the heuristic stay (based on zero real data).

In general, I have no problem with that. As long as it works. ;)

bug#8025: 24.0.50; vc-bzr does not perform initial commit

[Christoph Scholtes]

Glenn Morris <rgm@gnu.org> writes:

> Very lightly tested patch. Seems like not all of the fields are present
> in a freshly minted repo.

Thanks Glenn. This patch fixed the problem in my initial bug
report. Everything now works as expected.

Re: bug#8025: 24.0.50; vc-bzr does not perform initial commit

[Christoph Scholtes]

Glenn Morris <rgm@gnu.org> writes:

> Very lightly tested patch. Seems like not all of the fields are present
> in a freshly minted repo.

Thanks Glenn. This patch fixed the problem in my initial bug
report. Everything now works as expected.

bug#8025: 24.0.50; vc-bzr does not perform initial commit

[Stefan Monnier]

> `excruciatingly slow' does not mean anything to me.

Like several seconds.
Current bzr is much better in this respect.  If we can fix the function,
of course it's still better since it's not only faster but even works on
remote hosts and in the absence of a bzr binary.


        Stefan

Re: bug#8025: 24.0.50; vc-bzr does not perform initial commit

[Stefan Monnier]

> `excruciatingly slow' does not mean anything to me.

Like several seconds.
Current bzr is much better in this respect.  If we can fix the function,
of course it's still better since it's not only faster but even works on
remote hosts and in the absence of a bzr binary.


        Stefan

bug#8025: 24.0.50; vc-bzr does not perform initial commit

[Eli Zaretskii]

> From: Christoph Scholtes <cschol2112@googlemail.com>
> Cc: 8025@debbugs.gnu.org,  emacs-devel@gnu.org
> Date: Tue, 01 Mar 2011 22:55:46 -0700
> 
> 1. Using stock vc-bzr.el from the trunk.
> 
> Result:
> vc-bzr-state-heuristic     3     0.174            0.0579999999
>                                                   ^^^^^^^^^^^^
> 
> 2. Using modified vc-bzr.el forcing vc-bzr-state-heuristic function to use
> vc-bzr-state function instead of its own logic.
> 
> Result:
> vc-bzr-state-heuristic     3     1.5230000000     0.5076666666
>                                                   ^^^^^^^^^^^^
> 
> Roughly an order of 10 difference, but 0.5s is not really `excruciatingly
> slow' in my book.

No, it isn't.  Although it would be interesting to see the same test
with a cold cache.

bug#8025: 24.0.50; vc-bzr does not perform initial commit

[Eli Zaretskii]

> From: Christoph Scholtes <cschol2112@googlemail.com>
> Cc: Eli Zaretskii <eliz@gnu.org>,  8025@debbugs.gnu.org,  emacs-devel@gnu.org
> Date: Tue, 01 Mar 2011 23:00:22 -0700
> 
> Glenn Morris <rgm@gnu.org> writes:
> 
> > Also remember there is the annoying bzr locking issue. Eg doing `bzr up'
> > renders `bzr status' unusable so long as it is running. So it is not
> > just the speed of the status command that is a factor.
> 
> OK, but one could argue that this is a bzr issue and should be fixed
> there

It cannot be fixed in bzr, not unless they redesign and rewrite the
branch locking code from scratch.  Atomicity of transactions is
important, so locks on the directory or some of its files must be used
at least at some strategic points of time.

Re: bug#8025: 24.0.50; vc-bzr does not perform initial commit

[Eli Zaretskii]

> From: Christoph Scholtes <cschol2112@googlemail.com>
> Cc: Eli Zaretskii <eliz@gnu.org>,  8025@debbugs.gnu.org,  emacs-devel@gnu.org
> Date: Tue, 01 Mar 2011 23:00:22 -0700
> 
> Glenn Morris <rgm@gnu.org> writes:
> 
> > Also remember there is the annoying bzr locking issue. Eg doing `bzr up'
> > renders `bzr status' unusable so long as it is running. So it is not
> > just the speed of the status command that is a factor.
> 
> OK, but one could argue that this is a bzr issue and should be fixed
> there

It cannot be fixed in bzr, not unless they redesign and rewrite the
branch locking code from scratch.  Atomicity of transactions is
important, so locks on the directory or some of its files must be used
at least at some strategic points of time.