Incomplete output from "cvs annotate"

Incomplete output from "cvs annotate"

[Kim F. Storm]

In latest CVS emacs on GNU/Linux (redhat9.0), I noticed that the following
sequence: 

        C-h C-n         (open NEWS)
        C-x v g         (annotate it)

produces a very incomplete annotate buffer (approx 2/3 of the output is missing).

I then tried the same with 21.2, and got similar, not not quite as bad results.


I have tried to dig into what's going on, but I don't understand...

Here's what I have found out so far:

1) The bug is related to "call-process".

2) I can only make the problem appear with cvs annotate.

3) If I have "cvs -z6" in my ~/.cvsrc the problem is much more severe,
   but it is still present without it.


To narrow down the cause of this error, the following exhibits the error:

In *scratch*, do

        M-x cd (switch to emacs/etc) RET
        (call-process "/usr/bin/cvs" nil t nil "annotate" "NEWS")

This inserts the annotate output directly in *scratch*; the number of
lines inserted should equal the number of lines in NEWS, but it is
more like 4000 than 12400.

If I externally do "cvs annotate NEWS > XYZ", and then call
        (call-process "/bin/cat" nil t nil "XYZ")
in emacs, the entire file is inserted correctly.


I then wrote a small wrapper for cvs annotate:

#include <stdio.h>
#include <errno.h>

main()
{
  FILE *f = popen("/usr/bin/cvs -z6 annotate vc.el", "r");
  int fd = fileno(f);
  char buf[4096*16];
  int xx, tot=0;

  while ((xx = read(fd, buf, sizeof(buf))) > 0)
    {
      fprintf(stderr, "wrote %d[%d]\n", write(1, buf, xx), errno);
    }
  exit(0);
}


When I run this in emacs:

(call-process "mytest" nil t nil)

the output contains some >>wrote 4096[0]<< strings as expected, but
after a while this changes to >> wrote -1[11]<< meaning that the
write would have blocked... 11=EAGAIN ...!?

So what's going on?  As an experiment, I modified the above wrapper to
repeat the write:

#include <stdio.h>
#include <errno.h>

main()
{
  FILE *f = popen("/usr/bin/cvs -z6 annotate vc.el", "r");
  int fd = fileno(f);
  char buf[4096*16];
  int xx, tot=0;

  while ((xx = read(fd, buf, sizeof(buf))) > 0)
    {
      while (xx > 0) {
	int jj = write(1, buf, xx);
	if (jj < 0 && errno == EAGAIN)
	  sleep(1);
	else
	  xx -= jj;
      }
    }
  
  exit(0);
}


The result is good => the annotate output is now received correctly
and in full by emacs (but of course, very slowly -- that can be tuned
of course).


In a further quest, I wrote a small program which simply wrote
a buffer of 4096 bytes 100 times, and reading the output from
that program (instead of cvs) works with no problems at all
(as do all other examples of using call-process that I have tried).


So the big questions are -- and this is where I'm stuck:

        What is so special with cvs annotate and call-process?

        Why is the pipe opened by call-process -- and which cvs (as
        well as my wrapper around cvs) writes to -- in non-blocking
        state?

        I cannot find the place in emacs_open which sets file modes
        to O_NONBLOCK or O_NDELAY.  Can you?
        

-- 
Kim F. Storm <storm@cua.dk> http://www.cua.dk

Re: Incomplete output from "cvs annotate"

[Andreas Schwab]

storm@cua.dk (Kim F. Storm) writes:

> In latest CVS emacs on GNU/Linux (redhat9.0), I noticed that the following
> sequence: 
>
>         C-h C-n         (open NEWS)
>         C-x v g         (annotate it)
>
> produces a very incomplete annotate buffer (approx 2/3 of the output is missing).
>
[...]
> So the big questions are -- and this is where I'm stuck:
>
>         What is so special with cvs annotate and call-process?

Nothing, it can happen with any cvs command over ssh (you can get partial
checkins, for example).

>         Why is the pipe opened by call-process -- and which cvs (as
>         well as my wrapper around cvs) writes to -- in non-blocking
>         state?

See <http://mail.gnu.org/archive/html/bug-cvs/2002-07/msg00423.html>.  The
problem is that ssh makes stderr non-blocking and Emacs connects stdout
and stderr together, so that stdout becomes non-blocking as well.
Workaround is to separate stdout and stderr again, eg. by using this
script as CVS_RSH:

#!/bin/bash
exec 2> >(exec cat >&2 2>/dev/null)
exec ssh "$@"

Andreas.

-- 
Andreas Schwab, SuSE Labs, schwab@suse.de
SuSE Linux AG, Maxfeldstraße 5, 90409 Nürnberg, Germany
Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Incomplete output from "cvs annotate"

[Simon Josefsson]

Andreas Schwab <schwab@suse.de> writes:

>>         Why is the pipe opened by call-process -- and which cvs (as
>>         well as my wrapper around cvs) writes to -- in non-blocking
>>         state?
>
> See <http://mail.gnu.org/archive/html/bug-cvs/2002-07/msg00423.html>.  The
> problem is that ssh makes stderr non-blocking and Emacs connects stdout
> and stderr together, so that stdout becomes non-blocking as well.
> Workaround is to separate stdout and stderr again, eg. by using this
> script as CVS_RSH:
>
> #!/bin/bash
> exec 2> >(exec cat >&2 2>/dev/null)
> exec ssh "$@"

This workaround isn't a good solution.  Why do Emacs connect stdout
and stderr together?  IMHO, it shouldn't.  If the data need to be
collapsed into the same buffer, it should happen inside Emacs, not by
cloning the fd.  So using a BUFFER cons cell in VC for call-process
isn't a solution, using just one BUFFER should work.

My experience with this bug is that 'C-x v u' leads to corrupt files,
so I've more or less stopped using that command, and do 'rm FILE'
followed by 'cvs upd FILE' in a M-x shell instead.  It is not easily
reproducible, so I haven't reported it, but this discussion reminded
me.

Re: Incomplete output from "cvs annotate"

[Andreas Schwab]

Simon Josefsson <jas@extundo.com> writes:

> Andreas Schwab <schwab@suse.de> writes:
>
>>>         Why is the pipe opened by call-process -- and which cvs (as
>>>         well as my wrapper around cvs) writes to -- in non-blocking
>>>         state?
>>
>> See <http://mail.gnu.org/archive/html/bug-cvs/2002-07/msg00423.html>.  The
>> problem is that ssh makes stderr non-blocking and Emacs connects stdout
>> and stderr together, so that stdout becomes non-blocking as well.
>> Workaround is to separate stdout and stderr again, eg. by using this
>> script as CVS_RSH:
>>
>> #!/bin/bash
>> exec 2> >(exec cat >&2 2>/dev/null)
>> exec ssh "$@"
>
> This workaround isn't a good solution.  Why do Emacs connect stdout
> and stderr together?

You can either redirect stderr to a file, discard it completely, or
capture it in the same buffer as stdout.  In the latter case Emacs just
uses the same fd for both, which corresponds to 2>&1 in the shell.

> IMHO, it shouldn't.  If the data need to be collapsed into the same
> buffer, it should happen inside Emacs, not by cloning the fd.  So using
> a BUFFER cons cell in VC for call-process isn't a solution, using just
> one BUFFER should work.

This is what VC is doing.

Andreas.

-- 
Andreas Schwab, SuSE Labs, schwab@suse.de
SuSE Linux AG, Maxfeldstraße 5, 90409 Nürnberg, Germany
Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Incomplete output from "cvs annotate"

[Simon Josefsson]

Andreas Schwab <schwab@suse.de> writes:

> Simon Josefsson <jas@extundo.com> writes:
>
>> Andreas Schwab <schwab@suse.de> writes:
>>
>>>>         Why is the pipe opened by call-process -- and which cvs (as
>>>>         well as my wrapper around cvs) writes to -- in non-blocking
>>>>         state?
>>>
>>> See <http://mail.gnu.org/archive/html/bug-cvs/2002-07/msg00423.html>.  The
>>> problem is that ssh makes stderr non-blocking and Emacs connects stdout
>>> and stderr together, so that stdout becomes non-blocking as well.
>>> Workaround is to separate stdout and stderr again, eg. by using this
>>> script as CVS_RSH:
>>>
>>> #!/bin/bash
>>> exec 2> >(exec cat >&2 2>/dev/null)
>>> exec ssh "$@"
>>
>> This workaround isn't a good solution.  Why do Emacs connect stdout
>> and stderr together?
>
> You can either redirect stderr to a file, discard it completely, or
> capture it in the same buffer as stdout.  In the latter case Emacs just
> uses the same fd for both, which corresponds to 2>&1 in the shell.
>
>> IMHO, it shouldn't.  If the data need to be collapsed into the same
>> buffer, it should happen inside Emacs, not by cloning the fd.  So using
>> a BUFFER cons cell in VC for call-process isn't a solution, using just
>> one BUFFER should work.
>
> This is what VC is doing.

So what's the problem?  If VC is using a cons cell, I'd assume the
same fd isn't used for both stdout and stderr, but people's experience
seem to indicate the bug is still around.  Was VC changed recently,
perhaps?  I haven't seen my C-x v u problem for a while, although I
have not used it all that often lately.

If VC is using a cons cell, and that leads to using different fd's for
stdout and stderr, I believe the problem above isn't the same problem
as what we are seeing now.

Re: Incomplete output from "cvs annotate"

[Andreas Schwab]

Simon Josefsson <jas@extundo.com> writes:

> Andreas Schwab <schwab@suse.de> writes:
>
>> Simon Josefsson <jas@extundo.com> writes:
>>
>>> IMHO, it shouldn't.  If the data need to be collapsed into the same
>>> buffer, it should happen inside Emacs, not by cloning the fd.  So using
>>> a BUFFER cons cell in VC for call-process isn't a solution, using just
>>> one BUFFER should work.
>>
>> This is what VC is doing.
>
> So what's the problem?  If VC is using a cons cell,

I think you misunderstood me, VC is using a single buffer for both
stdout and stderr, which is why the bug occurs.

Andreas.

-- 
Andreas Schwab, SuSE Labs, schwab@suse.de
SuSE Linux AG, Maxfeldstraße 5, 90409 Nürnberg, Germany
Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Incomplete output from "cvs annotate"

[Simon Josefsson]

Andreas Schwab <schwab@suse.de> writes:

> Simon Josefsson <jas@extundo.com> writes:
>
>> Andreas Schwab <schwab@suse.de> writes:
>>
>>> Simon Josefsson <jas@extundo.com> writes:
>>>
>>>> IMHO, it shouldn't.  If the data need to be collapsed into the same
>>>> buffer, it should happen inside Emacs, not by cloning the fd.  So using
>>>> a BUFFER cons cell in VC for call-process isn't a solution, using just
>>>> one BUFFER should work.
>>>
>>> This is what VC is doing.
>>
>> So what's the problem?  If VC is using a cons cell,
>
> I think you misunderstood me, VC is using a single buffer for both
> stdout and stderr, which is why the bug occurs.

Ah, right.  I thought you were saying the opposite.  Sorry.

Re: Incomplete output from "cvs annotate"

[Eli Zaretskii]

> From: Simon Josefsson <jas@extundo.com>
> Date: Mon, 19 Jan 2004 15:59:45 +0100
> 
> If the data need to be collapsed into the same buffer, it should
> happen inside Emacs, not by cloning the fd.

You cannot DTRT to put data from different descriptors together
because you don't know their relative timing.  Causing them to be
written to the same descriptor emulates what happens in the
interactive case, when both stdout and stderr are connected to the
same device.

Re: Incomplete output from "cvs annotate"

[Simon Josefsson]

"Eli Zaretskii" <eliz@elta.co.il> writes:

>> From: Simon Josefsson <jas@extundo.com>
>> Date: Mon, 19 Jan 2004 15:59:45 +0100
>> 
>> If the data need to be collapsed into the same buffer, it should
>> happen inside Emacs, not by cloning the fd.
>
> You cannot DTRT to put data from different descriptors together
> because you don't know their relative timing.

Wouldn't select on both fd's and reading from the first one available,
and then sending that to the buffer, work?  Perhaps if there is data
in both stdout and stderr, write the stderr data.  Surely there is
some similar logic somewhere in bash, the kernel or somewhere.

Requiring another script to be installed is not good, IMHO.  Can't the
script be simulated in elisp somehow?

Re: Incomplete output from "cvs annotate"

[Kim F. Storm]

Simon Josefsson <jas@extundo.com> writes:

> Wouldn't select on both fd's and reading from the first one available,
> and then sending that to the buffer, work?  Perhaps if there is data
> in both stdout and stderr, write the stderr data.  Surely there is
> some similar logic somewhere in bash, the kernel or somewhere.

That might help in some cases, but there's no guarantee that it will
avoid losing data.  

In any case, your idea will not fix the case for cvs, as pcs-cvs
relies on the sequence of the data on those streams (cvs is definitely
a mess here), so if you split the flows, there's really no way to
ensure the proper sequence anymore -- no matter how you read from
those two flows, there's no way to tell how to merge them.

> 
> Requiring another script to be installed is not good, IMHO.  Can't the
> script be simulated in elisp somehow?

The problem is that the script must be inserted between SSH and CVS,
not between CVS and emacs.  

Last time I checked, neither SSH nor CVS understood Elisp :-)


-- 
Kim F. Storm <storm@cua.dk> http://www.cua.dk

Re: Incomplete output from "cvs annotate"

[Simon Josefsson]

storm@cua.dk (Kim F. Storm) writes:

>> Requiring another script to be installed is not good, IMHO.  Can't the
>> script be simulated in elisp somehow?
>
> The problem is that the script must be inserted between SSH and CVS,
> not between CVS and emacs.  

Ah.  I see.  Then perhaps this shouldn't be solved for emacs only, but
in ssh/cvs instead.

> Last time I checked, neither SSH nor CVS understood Elisp :-)

Bummer.  (Implement the CVS protocol in elisp...?)

Re: Incomplete output from "cvs annotate"

[Kim F. Storm]

Simon Josefsson <jas@extundo.com> writes:

> > Last time I checked, neither SSH nor CVS understood Elisp :-)
> 
> Bummer.  (Implement the CVS protocol in elisp...?)

You can start here :-) 

;;; cvscli.el --- cvs client commands

;; Copyright (C) 1999,2004 Kim F. Storm <storm@cua.dk>
;;                    All rights reserved.

;; Run CVS commands towards CVS server directly in emacs.

(defvar cvscli-server-connection nil
  "Current connection to cvs server")

(defvar cvscli-current-config nil
  "Current cvs server/directory configuration.
This is a list with 6 elements: (DIR REPOSITORY PASSWD USER SERVER ROOT)")

(defvar cvscli-passwd-alist nil
  "Passwords alist to use for cvs connections (from .cvspass).")

(defvar cvscli-keep-connection t
  "*When non-nil, keep connection to cvs server.")

(defun cvscli-open-connection (dir)
  (let* 
      ((config (cvscli-get-config dir))
       (action
	(catch 'oc
	  (if (or (null cvscli-server-connection)
		  (eq (process-status cvscli-server-connection) 'closed))
	      (throw 'oc 'open))	; no current connection
	  (if (or (not (string-equal (nth 4 cvscli-current-config) (nth 4 config)))
		  (not (string-equal (nth 3 cvscli-current-config) (nth 3 config)))
		  (not (string-equal (nth 5 cvscli-current-config) (nth 5 config))))
	      (throw 'oc 'reopen))	; wrong server, user, or root
	  (if (not (string-equal (nth 1 cvscli-current-config) (nth 1 config)))
	      (throw 'oc 'setrep))	; wrong repository
	  'done)))

    (if (eq action 'reopen)
	(progn
	  (cvscli-close-connection)
	  (setq action 'open)))

    (if (eq action 'open)
	(if (catch 'cvserr
	      (if (not (setq cvscli-server-connection (open-network-stream "cvscli" "*vc-info*" (nth 4 config) 2401)))
		  (throw 'cvserr t))

	      (save-excursion
		(set-buffer (process-buffer cvscli-server-connection))
		(erase-buffer)
		(set-process-coding-system cvscli-server-connection 'raw-text-unix 'raw-text-unix)
		;; (set-process-filter cvscli-server-connection 'cvscli-check-file-filter)
		(cvscli-send-string (concat "BEGIN AUTH REQUEST\n"
					    (nth 5 config) "\n"
					    (nth 3 config) "\n"
					    (nth 2 config) "\n"
					    "END AUTH REQUEST\n"))

		;; response: I LOVE YOU\n
		(accept-process-output cvscli-server-connection 3)
		(goto-char (point-min))
		(if (not (looking-at "I LOVE YOU"))
		    (progn
		      (cvscli-close-connection)
		      (throw 'cvserr t)))

		(erase-buffer)
		(cvscli-send-string (concat "Root " (nth 5 config)  "\n"))
		(cvscli-send-string "\
Valid-responses ok error\
 Valid-requests Checked-in New-entry Checksum Copy-file Updated Created\
 Update-existing Merged Patched Mode Removed Remove-entry Set-static-directory\
 Clear-static-directory Set-sticky Clear-sticky Template Set-checkin-prog\
 Set-update-prog Notified Module-expansion M E F
UseUnchanged
Global_option -r
Case
"))
	      (setq action 'setrep)
	      nil)
	    (cvscli-close-connection)))

    (if (eq action 'setrep)
	(progn
	  (cvscli-send-string (concat "Directory .\n" (nth 1 config) "\n"))
	  (setq action 'done)))

    (setq cvscli-current-config config))

  (if cvscli-server-connection
      (save-excursion
	(set-buffer (process-buffer cvscli-server-connection))
	(erase-buffer)))

  cvscli-server-connection)


(defun cvscli-close-connection ()
  (if cvscli-server-connection
      (progn
	(if (eq (process-status cvscli-server-connection) 'open)
	    (delete-process cvscli-server-connection))
	; (kill-buffer (process-buffer cvscli-server-connection))
	(setq cvscli-server-connection nil))))
  
(defun cvscli-send-string (str &optional resp)
  ;; (message "Send: %s" str)
  (process-send-string cvscli-server-connection str)
  (if resp
      (let (ok done)
	(save-excursion
	  (set-buffer (process-buffer cvscli-server-connection))
	  (while (not done)
	    (if (not (accept-process-output cvscli-server-connection 3))
	    (setq done t)
	  (let ((pm (process-mark cvscli-server-connection)) s)
	    (cond
	     ((= pm 3)
	      (setq ok (string-equal (buffer-substring (- pm 3) pm) "ok\n")))
	     ((> pm 3)
	      (setq ok (string-equal (buffer-substring (- pm 4) pm) "\nok\n"))))
	    (setq done ok))))
	  (if ok
	      (progn
		(set-marker (process-mark cvscli-server-connection) (- (point-max) 3))
		(delete-region (- (point-max) 3) (point-max))))
	  ok))
    t))

(defun cvscli-check-file (file &optional dir)
  (if (null dir)
      (setq dir default-directory))
  (let (entry ok fbuf) 
    (setq ok
	  (and (cvscli-open-connection dir)
	       (setq entry (cvscli-get-entry dir file))
	       (cvscli-send-string (concat "Argument " file 
					   "\nEntry /" file "/" (nth 1 entry) "//" (nth 3 entry) "/" (or (nth 4 entry) "")))
	       (save-excursion 
		 (if (and (setq fbuf (find-buffer-visiting (concat dir file)))
			  (set-buffer fbuf)
			  (not buffer-read-only))
		     (save-restriction
		       (widen)
		       (cvscli-send-string (concat "\nModified " file "\nu=rw,g=rw,o=rw\n" (- (point-max) (point-min)) "\n"))
		       (process-send-region cvscli-server-connection (point-min) (point-max)))
		   
		   (cvscli-send-string (concat "\nUnchanged " file "\n")))
		 t)
	       (cvscli-send-string "status\n" t)
	       (cvscli-send-string (concat "Argument " file "\neditors\n") t)
	       (cvscli-send-string (concat "Argument " file "\nlog\n") t)))
    (if ok
	(save-excursion
	  (set-buffer (process-buffer cvscli-server-connection))
	  (goto-char (point-min))
	  (while (and
		  (not (looking-at "head:"))
		       (search-forward-regexp "^M " nil t))
	    (replace-match "" nil t))))

    (if (not cvscli-keep-connection)
	cvscli-close-connection)
    ok))
      
(defun cvscli-get-config (dir)
  (if (and cvscli-current-config
	   (string-equal dir (car cvscli-current-config)))
      cvscli-current-config
    (if (null cvscli-passwd-alist)
	(let ((pw (expand-file-name "~/.cvspass")))
	  (if (file-exists-p pw)
	      (let ((buf (find-file-noselect pw)) p s)
		(save-excursion
		  (set-buffer buf)
		  (goto-char (point-min))
		  (while (not (eobp))
		    (setq s (point))
		    (if (not (search-forward " " nil t))
			(forward-line 1)
		      (setq s (buffer-substring s (1- (point))))
		      (setq p (point))
		      (end-of-line)
		      (setq p (buffer-substring p (point)))
		      (setq cvscli-passwd-alist (cons (cons s p) cvscli-passwd-alist))
		      (forward-char 1))))
		(kill-buffer buf)))))
    (let ((rep (concat dir "/CVS/Repository")) 
	  (root (concat dir "/CVS/Root")) 
	  config)
      (if (and (file-exists-p root)
	       (file-exists-p rep))
	  (progn
	    (let ((buf (find-file-noselect root)) pw rep)
	      (save-excursion
		(set-buffer buf)
		(setq rep (buffer-substring (point-min) (1- (point-max))))
		(setq config (split-string rep "[:@]")))
	      (kill-buffer buf)
	      (if config
		  (setcar config 
			  (and (setq pw (assoc rep cvscli-passwd-alist)) (cdr pw)))))
	    (let ((buf (find-file-noselect rep)) s)
	      (save-excursion
		(set-buffer buf)
		(setq config (cons (buffer-substring (point-min) (1- (point-max))) config)))
	      (kill-buffer buf))))
      (and config
	   (cons dir config)))))

(defun cvscli-get-entry (dir file)
  (let ((entry (concat dir "/CVS/Entries")) s)
    (if (file-exists-p entry)
	(let ((buf (find-file-noselect entry)))
	  (save-excursion
	    (set-buffer buf)
	    (goto-char (point-min))
	    (if (search-forward-regexp (concat "^/" file "/") nil t)
		(let (b)
		  (beginning-of-line)
		  (setq b (point))
		  (end-of-line)
		  (setq s (split-string (buffer-substring b (point)) "/")))))
	  (kill-buffer buf)))
    s))


(defun cvscli-check-file-filter (process output-string)
  (let ((old-buffer (current-buffer)))
    (unwind-protect
	(let ((moving))
	  (set-buffer (process-buffer process))
	  (setq moving (= (point) (process-mark process)))
	  (save-excursion
	    ;; Insert the text, moving the process-marker.
	    (goto-char (process-mark process))
	    (insert output-string)
	    (set-marker (process-mark process) (point)))

	  ;;(while (string-match "\r" filtered-string)
	  ;;     (setq filtered-string
	  ;;   (replace-match "" nil nil filtered-string)))
	  (if moving (goto-char (process-mark process))))
      (set-buffer old-buffer))))

  

(defun vc-ocvs-fetch-master-properties (file fail-ok)
  ;; Fetch those properties of FILE that are stored in the CVS repository file.
  (save-excursion
    ;; Call "cvs emacs" in the right directory, passing only the
    ;; nondirectory part of the file name -- otherwise CVS might 
    ;; silently give a wrong result.
    (let ((default-directory (file-name-directory file)))
      (or (cvscli-check-file (file-name-nondirectory file) default-directory)
	  (vc-simple-command 0 "cvs" (file-name-nondirectory file) "emacs")))
    (set-buffer (get-buffer "*vc-info*"))
    (vc-parse-buffer
     ;; CVS 1.3 says "RCS Version:", other releases "RCS Revision:",
     ;; and CVS 1.4a1 says "Repository revision:".
     '(("\\(RCS Version\\|RCS Revision\\|Repository revision\\):[\t ]+\\([0-9.]+\\)" 2)
       ("^File: [^ \t]+[ \t]+Status: \\(.*\\)" 1))
     file
     '(vc-latest-version vc-cvs-status))
    (vc-parse-buffer
     '(("Sticky Tag:[ \t]*\\([^\n ]+\\)" 1)
       ("Sticky Date:[ \t]*\\([^\n ]+\\)" 1)
       ("Sticky Options:[ \t]*\\([^\n ]+\\)" 1))
     file
     '(vc-sticky-tag vc-sticky-date vc-sticky-options))
    (let ((stag (vc-file-getprop file 'vc-sticky-tag))
	  (sdate (vc-file-getprop file 'vc-sticky-date))
	  (soptions (vc-file-getprop file 'vc-sticky-options)))
      (if (and stag (string-match stag "(none)"))
	  (vc-file-setprop file 'vc-sticky-tag nil))
      (if (and sdate (string-match sdate "(none)"))
	  (vc-file-setprop file 'vc-sticky-date nil))
      (if (and soptions (string-match soptions "(none)"))
	  (vc-file-setprop file 'vc-sticky-options nil)))

    ;; Translate those status values that we understand into symbols.
    ;; Any other value is converted to nil.
    (let ((status (vc-file-getprop file 'vc-cvs-status)))
      (cond 
       ((string-match "Up-to-date" status)
	(vc-file-setprop file 'vc-cvs-status 'up-to-date)
	(vc-file-setprop file 'vc-checkout-time
			 (nth 5 (file-attributes file))))
       ((vc-file-setprop file 'vc-cvs-status
			 (cond 
			  ((string-match "Locally Modified"    status) 'locally-modified)
			  ((string-match "Needs Merge"         status) 'needs-merge)
			  ((string-match "Needs \\(Checkout\\|Patch\\)" status) 
			   'needs-checkout)
			  ((string-match "Unresolved Conflict" status) 'unresolved-conflict)
			  ((string-match "Locally Added"       status) 'locally-added)
			  (t 'unknown)
			  )))))
    (vc-parse-locks file (buffer-substring-no-properties (point-min) (point-max)))
    (vc-parse-buffer
     '(("^Head: \\(.*\\)" 1))
     file
     '(vc-latest-version))
    ))


-- 
Kim F. Storm <storm@cua.dk> http://www.cua.dk

Re: Incomplete output from "cvs annotate"

[Eli Zaretskii]

> From: Simon Josefsson <jas@extundo.com>
> Date: Tue, 20 Jan 2004 13:01:54 +0100
> >
> > You cannot DTRT to put data from different descriptors together
> > because you don't know their relative timing.
> 
> Wouldn't select on both fd's and reading from the first one available,
> and then sending that to the buffer, work?

Not really, since Emacs only checks if input from some process is
available when it is not otherwise busy doing something.  On a fast
machine, getting to that a second too late can lose you your world.

> Surely there is some similar logic somewhere in bash, the kernel or
> somewhere.

Probably, but the important case is when stdout and stderr are written
one after another, but before Emacs gets to see the first one.

Re: Incomplete output from "cvs annotate"

[Andreas Schwab]

Simon Josefsson <jas@extundo.com> writes:

> Wouldn't select on both fd's and reading from the first one available,
> and then sending that to the buffer, work?  Perhaps if there is data
> in both stdout and stderr, write the stderr data.  Surely there is
> some similar logic somewhere in bash, the kernel or somewhere.

The only place where multiplexing is required is in ssh, as it has to
transmit two data streams over a single channel.  The kernel just copies
the data from the source to the sink as soon as it is available, and the
shell is not involved in any data copying, it only sets up the stream
connections.

Andreas.

-- 
Andreas Schwab, SuSE Labs, schwab@suse.de
SuSE Linux AG, Maxfeldstraße 5, 90409 Nürnberg, Germany
Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Incomplete output from "cvs annotate"

[Kim F. Storm]

Andreas Schwab <schwab@suse.de> writes:

> storm@cua.dk (Kim F. Storm) writes:
> 
> > In latest CVS emacs on GNU/Linux (redhat9.0), I noticed that the following
> > sequence: 
> >
> >         C-h C-n         (open NEWS)
> >         C-x v g         (annotate it)
> >
> > produces a very incomplete annotate buffer (approx 2/3 of the output is missing).
> >
> [...]
> > So the big questions are -- and this is where I'm stuck:
> >
> >         What is so special with cvs annotate and call-process?
> 
> Nothing, it can happen with any cvs command over ssh (you can get partial
> checkins, for example).
> 
> >         Why is the pipe opened by call-process -- and which cvs (as
> >         well as my wrapper around cvs) writes to -- in non-blocking
> >         state?
> 
> See <http://mail.gnu.org/archive/html/bug-cvs/2002-07/msg00423.html>.  The
> problem is that ssh makes stderr non-blocking and Emacs connects stdout
> and stderr together, so that stdout becomes non-blocking as well.
> Workaround is to separate stdout and stderr again, eg. by using this
> script as CVS_RSH:
> 
> #!/bin/bash
> exec 2> >(exec cat >&2 2>/dev/null)
> exec ssh "$@"
> 
> Andreas.


Thanks for the pointer.

Knowing the cause, there is a simple work-arond in emacs, as the
following call-process usage avoids having emacs tying stderr into
stdout; instead, it redirects stderr to /dev/null:

  (call-process "/usr/bin/cvs" nil '(t nil) nil "annotate" "vc.el")

I suppose the cvs backend could use this to avoid the problems.

Actually, I think merging stdout and stderr when calling cvs (or some
other vc tool) is generally a bad thing, as you risk ending up with
error messages in checked out files, so perhaps cvs (and vc in
general) should do this sort of thing always.

I haven't checked pcl-cvs, but I suppose it could use a similar
remedy for the problems it has with cvs+ssh?!

-- 
Kim F. Storm <storm@cua.dk> http://www.cua.dk

Re: Incomplete output from "cvs annotate"

[Stefan Monnier]

> I haven't checked pcl-cvs, but I suppose it could use a similar
> remedy for the problems it has with cvs+ssh?!

PCL-CVS uses both stdout and stderr, so redirecting it to /dev/null is not
an option.  Redirecting it to a separate file is not an option either
because the interleaving is also relevant (i.e. used by PCL-CVS).

It is really a bug between CVS, ssh, and libc (which might not appear with
all libc, but appear with glibc and with Mac OS X's libc as well
appearently).  Andreas Schwab's wrapper script is the best workaround
I know, and I don't think people have figured out whether the bug is in CVS,
SSH, or libc.


        Stefan

Re: Incomplete output from "cvs annotate"

[Kim F. Storm]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> > I haven't checked pcl-cvs, but I suppose it could use a similar
> > remedy for the problems it has with cvs+ssh?!
> 
> PCL-CVS uses both stdout and stderr, so redirecting it to /dev/null is not
> an option.  Redirecting it to a separate file is not an option either
> because the interleaving is also relevant (i.e. used by PCL-CVS).

Yes, cvs stdout/stderr usage is a mess...

> 
>                Andreas Schwab's wrapper script is the best workaround
> I know, and I don't think people have figured out whether the bug is in CVS,
> SSH, or libc.

In any case, this problem should be prominently documented in PROBLEMS
_and_ INSTALL.CVS.

Maybe we could install Andreas' script in etc and point people to use that.

We could even let vc-cvs and pcl-cvs override CVS_RSH to point to that
script if the original value is "ssh"

-- 
Kim F. Storm <storm@cua.dk> http://www.cua.dk

Re: Incomplete output from "cvs annotate"

[Stefan Monnier]

> Yes, cvs stdout/stderr usage is a mess...

To the point where CVS IIRC has (at least had, but I doubt they managed to
get rid of it) special code for when stdout = stderr to make sure the
messages are properly interleaved in that case.


        Stefan

Re: Incomplete output from "cvs annotate"

[Richard Stallman]

    It is really a bug between CVS, ssh, and libc (which might not appear with
    all libc, but appear with glibc and with Mac OS X's libc as well
    appearently).  Andreas Schwab's wrapper script is the best workaround
    I know, and I don't think people have figured out whether the bug is in CVS,
    SSH, or libc.

I think  we should document this recipe.  But where?

Re: Incomplete output from "cvs annotate"

[Kevin Rodgers]

Kim F. Storm wrote:

> Knowing the cause, there is a simple work-arond in emacs, as the
> following call-process usage avoids having emacs tying stderr into
> stdout; instead, it redirects stderr to /dev/null:
> 
>   (call-process "/usr/bin/cvs" nil '(t nil) nil "annotate" "vc.el")

Discarding standard error is not a good idea.  Too bad STDERR-FILE can't

simply be a buffer (or a buffer name):

(call-process "/usr/bin/cvs" nil '(t "*Messages*") nil "annotate" "vc.el")

-- 
Kevin Rodgers