bug#470: pgg-gpg.el - pgg-gpg-process-region timing problem

bug#470: pgg-gpg.el - pgg-gpg-process-region timing problem

[Thien-Thi Nguyen]

() "ken manheimer" <ken.manheimer@gmail.com>
() Sun, 22 Jun 2008 13:37:11 -0400

   pggprob-patch.txt, with a single-line modification to
   pgg-gpg-process-region to work around the problem.  both
   are attached.  since the workaround is merely a timing
   delay, it's clearly not a fix!

Please find below an UNTESTED patch that addresses the
problem in two ways:
 - use a downward funarg sentinel for complete monitoring
 - reduce unnecessary data motion

Does it give good results?

(Authors added to cc.)

thi

bug#486: pgg-gpg.el - pgg-gpg-process-region timing problem

[ken manheimer]

[-- Attachment #1: Type: text/plain, Size: 5144 bytes --]

i am not sure how to proceed with this problem.  as demonstrated by
the pggprob.el script, i consistently get encryption and decryption
problems without the contrived delay in my pgg-gpg-process-region
patch.  (the problem does not happen if i just do a regular
pgg-decrypt - i think the crucial thing is that a symmetric passphrase
is being passed into pgg-gpg-process-region.)

i have tried everything i could think of to exclude local quirks from
the problem, including starting emacs with '--no-site-file -q' and
eliminating everything from my ~/.gnupg/gpg.conf file.  i have
reproduced the problem (and workaround) consistently in kubuntu 8.04
emacs-snapshot package and a recent CVS HEAD checkout.  in fact, i
have incidentally reinstalled my kubuntu 8.04 OS recently, on both my
main 2.0 Ghz T43 thinkpad and on a 750 Mhz a20p thinkpad.  the problem
continues consistently on the t43 but does *not* occur on the a20p,
which is not contrary to the sensitivities of a timing problem.

i have also tried thien-thi nguyen's alternative approaches, but
neither worked for me, hanging in different places.

i am pretty sure this is a real problem, and something that warrants
attention.  i realize that the difficulty in repeating the problem
elsewhere increases the challenge in addressing it, and am open to
cooperating with people to try things, if that would help.  the
consistent effectiveness of my workaround should help provide clues
about the nature of the problem, at least.

any suggestions?
--
ken
http://myriadicity.net

On Sun, Jun 22, 2008 at 1:37 PM, ken manheimer <ken.manheimer@gmail.com> wrote:
> allout encryption, which depends on pgg.el &c, has been failing due to
> a weird timing problem i've narrowed down to the
> pgg-gpg-process-region function.  this is in emacs 23.0.60 built from
> a CVS HEAD checkout of an hour or so ago, and some earlier 23.0.60
> versions.  i've reproduced the problem using emacs -q and regularly
> launched emacs, etc.
>
> the problem is that pgg-gpg-process-region often returns a nil status
> despite successful decryption, unless a delay (eg, (sleep-for .3) is
> inserted between process start and later interaction with the process.
>  with the delay, the status is something like 207 or 208, while
> without it the status is nil - yet, on examination the content is
> sucessfully decrypted.
>
> the size of the delay needs to be increased when the machine is
> heavily loaded, and can be reduced (on my thinkpad t43 2 GHz laptop,
> 3Gb ram, running kubuntu 8.0.4 +) to no lower than (sleep-for .2).
> rarely, the proper status is returned without the delay, depending on
> the machine load and other things i can't identify.  considering the
> real-time sensitivity of the problem, your mileage will almost
> certainly vary!
>
> i originally thought the problem might be in gpg, but now figure it
> resides either in the emacs process mechanism, pgg's use of that, or
> somewhere in the combination of emacs and pgg.
>
> i've created a simple .el script, pggprob.el, with minimal code and
> data to produce the problem, and a patch, pggprob-patch.txt, with a
> single-line modification to pgg-gpg-process-region to work around the
> problem.  both are attached.  since the workaround is merely a timing
> delay, it's clearly not a fix!  i'm hoping this is enough to enable
> others to reproduce the problem, and investigate...
> --
> ken
> http://myriadicity.net
>
> If Emacs crashed, and you have the Emacs process in the gdb debugger,
> please include the output from the following gdb commands:
>    `bt full' and `xbacktrace'.
> If you would like to further debug the crash, please read the file
> /home/klm/src/emacs-HEAD/etc/DEBUG for instructions.
>
> In GNU Emacs 23.0.60.1 (i686-pc-linux-gnu, GTK+ Version 2.12.9)
>  of 2008-06-22 on twist
> Windowing system distributor `The X.Org Foundation', version 11.0.10400090
> Important settings:
>  value of $LC_ALL: nil
>  value of $LC_COLLATE: nil
>  value of $LC_CTYPE: nil
>  value of $LC_MESSAGES: nil
>  value of $LC_MONETARY: nil
>  value of $LC_NUMERIC: nil
>  value of $LC_TIME: nil
>  value of $LANG: en_US.UTF-8
>  value of $XMODIFIERS: nil
>  locale-coding-system: utf-8-unix
>  default-enable-multibyte-characters: t
>
> Major mode: Emacs-Lisp
>
> Minor modes in effect:
>  tooltip-mode: t
>  tool-bar-mode: t
>  mouse-wheel-mode: t
>  menu-bar-mode: t
>  file-name-shadow-mode: t
>  global-font-lock-mode: t
>  font-lock-mode: t
>  blink-cursor-mode: t
>  global-auto-composition-mode: t
>  auto-composition-mode: t
>  auto-encryption-mode: t
>  auto-compression-mode: t
>  line-number-mode: t
>  transient-mark-mode: t
>
> Recent input:
> C-x C-f <M-backspace> <M-backspace> <M-backspace> l
> i b / e m a <tab> a l l <tab> s c r <tab> p g g p <tab>
> <return> M-x e v a l - c u r <tab> <return> M-. C-g
> M-> C-x C-e M-x e m a c s <M-backspace> r e p o r t
> - e m <tab> <return>
>
> Recent messages:
> For information about GNU Emacs and the GNU system, type C-h C-a.
> Source file `/home/klm/src/emacs-HEAD/lisp/pgg-gpg.el' newer than
> byte-compiled file
> Quit
> Mark set
> nil
>



-- 
ken
http://myriadicity.net

[-- Attachment #2: pggprob.el --]
[-- Type: application/octet-stream, Size: 671 bytes --]

;; demonstrate timing problem in pgg-gpg-process-region

(setq passphrase "a"
      verifier "-----BEGIN PGP MESSAGE-----
Version: GnuPG v1.4.6 (GNU/Linux)

jA0EAwMCo72fNGB1rNxgySv+rxdkPv46nVxQ5fg4yT78ldgVcDseD7Mre4Fhjifn
VTA1/PdmetVfFb9+
=MXF6
-----END PGP MESSAGE-----"
      count 0)

(defun decrypt (passphrase text)
  (with-current-buffer (get-buffer-create "*pgg-test*")
    (erase-buffer)
    (insert text)
    (pgg-decrypt nil nil passphrase)))

;; the following yields nil w/out the patched delay, 208 with it.
;; examination of the actual results buffer shows that the verifier is
;; successfully decrypted in both cases, however.
(decrypt passphrase verifier)

[-- Attachment #3: pgg-patch.txt --]
[-- Type: text/plain, Size: 409 bytes --]

--- pgg-gpg.el.~1.27.~	2008-05-06 03:57:46.000000000 -0400
+++ pgg-gpg.el	2008-06-22 13:24:14.000000000 -0400
@@ -86,6 +86,7 @@
 	    (setq process
 		  (apply #'start-process "*GnuPG*" errors-buffer
 			 program args)))
+          (sleep-for .3)                ; klm: timing delay
 	  (set-process-sentinel process #'ignore)
 	  (when passphrase
 	    (setq passphrase-with-newline (concat passphrase "\n"))

bug#465: marked as done (pgg-gpg.el - pgg-gpg-process-region timing problem)

[Emacs bug Tracking System]

[-- Attachment #1: Type: text/plain, Size: 865 bytes --]


Your message dated Sat, 02 Aug 2008 15:08:30 -0400
with message-id <873aln2q1d.fsf@stupidchicken.com>
and subject line Re: pgg-gpg.el - pgg-gpg-process-region timing problem
has caused the Emacs bug report #465,
regarding pgg-gpg.el - pgg-gpg-process-region timing problem
to be marked as done.

This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
bug report if necessary, and/or fix the problem forthwith.

(NB: If you are a system administrator and have no idea what this
message is talking about, this may indicate a serious mail system
misconfiguration somewhere. Please contact don@donarmstrong.com
immediately.)


-- 
465: http://emacsbugs.donarmstrong.com/cgi-bin/bugreport.cgi?bug=465
Emacs Bug Tracking System
Contact don@donarmstrong.com with problems

[-- Attachment #2: Type: message/rfc822, Size: 8269 bytes --]

[-- Attachment #2.1.1: Type: text/plain, Size: 3349 bytes --]

allout encryption, which depends on pgg.el &c, has been failing due to
a weird timing problem i've narrowed down to the
pgg-gpg-process-region function.  this is in emacs 23.0.60 built from
a CVS HEAD checkout of an hour or so ago, and some earlier 23.0.60
versions.  i've reproduced the problem using emacs -q and regularly
launched emacs, etc.

the problem is that pgg-gpg-process-region often returns a nil status
despite successful decryption, unless a delay (eg, (sleep-for .3) is
inserted between process start and later interaction with the process.
 with the delay, the status is something like 207 or 208, while
without it the status is nil - yet, on examination the content is
sucessfully decrypted.

the size of the delay needs to be increased when the machine is
heavily loaded, and can be reduced (on my thinkpad t43 2 GHz laptop,
3Gb ram, running kubuntu 8.0.4 +) to no lower than (sleep-for .2).
rarely, the proper status is returned without the delay, depending on
the machine load and other things i can't identify.  considering the
real-time sensitivity of the problem, your mileage will almost
certainly vary!

i originally thought the problem might be in gpg, but now figure it
resides either in the emacs process mechanism, pgg's use of that, or
somewhere in the combination of emacs and pgg.

i've created a simple .el script, pggprob.el, with minimal code and
data to produce the problem, and a patch, pggprob-patch.txt, with a
single-line modification to pgg-gpg-process-region to work around the
problem.  both are attached.  since the workaround is merely a timing
delay, it's clearly not a fix!  i'm hoping this is enough to enable
others to reproduce the problem, and investigate...
--
ken
http://myriadicity.net

If Emacs crashed, and you have the Emacs process in the gdb debugger,
please include the output from the following gdb commands:
    `bt full' and `xbacktrace'.
If you would like to further debug the crash, please read the file
/home/klm/src/emacs-HEAD/etc/DEBUG for instructions.

In GNU Emacs 23.0.60.1 (i686-pc-linux-gnu, GTK+ Version 2.12.9)
 of 2008-06-22 on twist
Windowing system distributor `The X.Org Foundation', version 11.0.10400090
Important settings:
  value of $LC_ALL: nil
  value of $LC_COLLATE: nil
  value of $LC_CTYPE: nil
  value of $LC_MESSAGES: nil
  value of $LC_MONETARY: nil
  value of $LC_NUMERIC: nil
  value of $LC_TIME: nil
  value of $LANG: en_US.UTF-8
  value of $XMODIFIERS: nil
  locale-coding-system: utf-8-unix
  default-enable-multibyte-characters: t

Major mode: Emacs-Lisp

Minor modes in effect:
  tooltip-mode: t
  tool-bar-mode: t
  mouse-wheel-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  global-auto-composition-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent input:
C-x C-f <M-backspace> <M-backspace> <M-backspace> l
i b / e m a <tab> a l l <tab> s c r <tab> p g g p <tab>
<return> M-x e v a l - c u r <tab> <return> M-. C-g
M-> C-x C-e M-x e m a c s <M-backspace> r e p o r t
- e m <tab> <return>

Recent messages:
For information about GNU Emacs and the GNU system, type C-h C-a.
Source file `/home/klm/src/emacs-HEAD/lisp/pgg-gpg.el' newer than
byte-compiled file
Quit
Mark set
nil

[-- Attachment #2.1.2: pggprob.el --]
[-- Type: application/octet-stream, Size: 671 bytes --]

;; demonstrate timing problem in pgg-gpg-process-region

(setq passphrase "a"
      verifier "-----BEGIN PGP MESSAGE-----
Version: GnuPG v1.4.6 (GNU/Linux)

jA0EAwMCo72fNGB1rNxgySv+rxdkPv46nVxQ5fg4yT78ldgVcDseD7Mre4Fhjifn
VTA1/PdmetVfFb9+
=MXF6
-----END PGP MESSAGE-----"
      count 0)

(defun decrypt (passphrase text)
  (with-current-buffer (get-buffer-create "*pgg-test*")
    (erase-buffer)
    (insert text)
    (pgg-decrypt nil nil passphrase)))

;; the following yields nil w/out the patched delay, 208 with it.
;; examination of the actual results buffer shows that the verifier is
;; successfully decrypted in both cases, however.
(decrypt passphrase verifier)

[-- Attachment #2.1.3: pgg-patch.txt --]
[-- Type: text/plain, Size: 409 bytes --]

--- pgg-gpg.el.~1.27.~	2008-05-06 03:57:46.000000000 -0400
+++ pgg-gpg.el	2008-06-22 13:24:14.000000000 -0400
@@ -86,6 +86,7 @@
 	    (setq process
 		  (apply #'start-process "*GnuPG*" errors-buffer
 			 program args)))
+          (sleep-for .3)                ; klm: timing delay
 	  (set-process-sentinel process #'ignore)
 	  (when passphrase
 	    (setq passphrase-with-newline (concat passphrase "\n"))

[-- Attachment #3: Type: message/rfc822, Size: 1087 bytes --]

From: Chong Yidong <cyd@stupidchicken.com>
To: 465-done@emacsbugs.donarmstrong.com
Subject: Re: pgg-gpg.el - pgg-gpg-process-region timing problem
Date: Sat, 02 Aug 2008 15:08:30 -0400
Message-ID: <873aln2q1d.fsf@stupidchicken.com>

I've checked in a fix for this (verified by Ken Manheimer via private
email).  This fix is in pgg-gpg.el, which I think was calling
accept-process-output in a slightly incorrect way.

bug#469: pgg-gpg.el - pgg-gpg-process-region timing problem

[Thien-Thi Nguyen]

ChangeLog entry:

	Use process sentinel to monitor gpg subprocess status.

	* pgg-gpg.el (pgg-gpg-process-region): Fix omission bug:
	Use process sentinel to update process status and exit-status.
	Reported by ken manheimer <ken.manheimer@gmail.com>.

patch:
*** pgg-gpg.el	6 May 2008 07:57:46 -0000	1.27
--- pgg-gpg.el	23 Jun 2008 08:40:50 -0000
***************
*** 70,111 ****
  	    ,@pgg-gpg-extra-args ,@args))
  	 (output-buffer pgg-output-buffer)
  	 (errors-buffer pgg-errors-buffer)
! 	 (orig-mode (default-file-modes))
! 	 (process-connection-type nil)
! 	 (inhibit-redisplay t)
! 	 process status exit-status
! 	 passphrase-with-newline
! 	 encoded-passphrase-with-new-line)
      (with-current-buffer (get-buffer-create errors-buffer)
        (buffer-disable-undo)
        (erase-buffer))
      (unwind-protect
! 	(progn
  	  (set-default-file-modes 448)
! 	  (let ((coding-system-for-write 'binary))
! 	    (setq process
! 		  (apply #'start-process "*GnuPG*" errors-buffer
! 			 program args)))
! 	  (set-process-sentinel process #'ignore)
  	  (when passphrase
! 	    (setq passphrase-with-newline (concat passphrase "\n"))
! 	    (if pgg-passphrase-coding-system
! 		(progn
! 		  (setq encoded-passphrase-with-new-line
  			(encode-coding-string
! 			 passphrase-with-newline
! 			 (coding-system-change-eol-conversion
! 			  pgg-passphrase-coding-system 'unix)))
! 		  (pgg-clear-string passphrase-with-newline))
! 	      (setq encoded-passphrase-with-new-line passphrase-with-newline
! 		    passphrase-with-newline nil))
! 	    (process-send-string process encoded-passphrase-with-new-line))
  	  (process-send-region process start end)
  	  (process-send-eof process)
! 	  (while (eq 'run (process-status process))
  	    (accept-process-output process 5))
- 	  (setq status (process-status process)
- 		exit-status (process-exit-status process))
  	  (delete-process process)
  	  (with-current-buffer (get-buffer-create output-buffer)
  	    (buffer-disable-undo)
--- 70,104 ----
  	    ,@pgg-gpg-extra-args ,@args))
  	 (output-buffer pgg-output-buffer)
  	 (errors-buffer pgg-errors-buffer)
! 	 (inhibit-redisplay t))
      (with-current-buffer (get-buffer-create errors-buffer)
        (buffer-disable-undo)
        (erase-buffer))
      (unwind-protect
! 	(let* ((coding-system-for-write 'binary)
! 	       (process (let ((orig (default-file-modes)))
  			  (set-default-file-modes 448)
! 			  (prog1 (apply #'start-process "*GnuPG*"
! 					errors-buffer program args)
! 			    (set-default-file-modes orig))))
! 	       (status (process-status process))
! 	       exit-status)
! 	  (set-process-sentinel
! 	   process (lambda (process reason)
! 		     (setq status (process-status process)
! 			   exit-status (process-exit-status process))))
  	  (when passphrase
! 	    (process-send-string
! 	     process (if pgg-passphrase-coding-system
  			 (encode-coding-string
! 			  passphrase
! 			  pgg-passphrase-coding-system)
! 		       passphrase))
! 	    (process-send-string process "\n"))
  	  (process-send-region process start end)
  	  (process-send-eof process)
! 	  (while (eq 'run status)
  	    (accept-process-output process 5))
  	  (delete-process process)
  	  (with-current-buffer (get-buffer-create output-buffer)
  	    (buffer-disable-undo)
***************
*** 120,134 ****
  		(error "%s exited abnormally: '%s'" program exit-status))
  	    (if (= 127 exit-status)
  		(error "%s could not be found" program))))
-       (if passphrase-with-newline
- 	  (pgg-clear-string passphrase-with-newline))
-       (if encoded-passphrase-with-new-line
- 	  (pgg-clear-string encoded-passphrase-with-new-line))
-       (if (and process (eq 'run (process-status process)))
- 	  (interrupt-process process))
        (if (file-exists-p output-file-name)
! 	  (delete-file output-file-name))
!       (set-default-file-modes orig-mode))))
  
  (defun pgg-gpg-possibly-cache-passphrase (passphrase &optional key notruncate)
    (if (and passphrase
--- 113,120 ----
  		(error "%s exited abnormally: '%s'" program exit-status))
  	    (if (= 127 exit-status)
  		(error "%s could not be found" program))))
        (if (file-exists-p output-file-name)
! 	  (delete-file output-file-name)))))
  
  (defun pgg-gpg-possibly-cache-passphrase (passphrase &optional key notruncate)
    (if (and passphrase

bug#471: pgg-gpg.el - pgg-gpg-process-region timing problem

[Thien-Thi Nguyen]

Delving into this, i got a little carried away... full func below.
This adds GOOD_PASSPHRASE checking (necessary for GNUPG 1.4.6, at
least) and some other coding-system related finessing, but removes
the sentinel proposed in the first attempt.  In its place, we zonk
the baleful `process-adaptive-read-buffering' and remove timeout
parameters from `accept-process-output' calls.

With this func things seem to work fine for `pgg-sign' on a small
buffer, and mostly for large buffers (270KB), but sometimes (~10%)
with large buffers, i see a "file error: bad address *GnuPG*".  Hmmm...

thi

______________________________________________________________________
(defun pgg-gpg-process-region (start end passphrase program args)
  (let* ((use-agent (and (null passphrase) (pgg-gpg-use-agent-p)))
	 (output-file-name (pgg-make-temp-file "pgg-output"))
	 (args
	  `("--status-fd" "2"
	    ,@(if use-agent '("--use-agent")
		(if passphrase '("--passphrase-fd" "0")))
	    "--yes" ; overwrite
	    "--output" ,output-file-name
	    ,@pgg-gpg-extra-args ,@args))
	 (output-buffer pgg-output-buffer)
	 (errors-buffer pgg-errors-buffer)
	 (orig-mode (default-file-modes))
	 (inhibit-redisplay t))
    (with-current-buffer (get-buffer-create errors-buffer)
      (buffer-disable-undo)
      (erase-buffer))
    (unwind-protect
	(let* ((coding-system-for-write 'binary)
	       ;; GNUPG 1.4.6 does not terminate on bad passphrase, eg:
	       ;;   [GNUPG:] BAD_PASSPHRASE (long hex # here)
	       ;;   gpg: skipped "ttn": bad passphrase
	       ;;   gpg: [stdin]: clearsign failed: bad passphrase
	       ;; so we need to check that condition ourselves and bail out.
	       ;;
	       ;; To check if the passphrase is accepted, we need to parse the
	       ;; errors-buffer, but `process-adaptive-read-buffering' non-nil
	       ;; sometimes prevents it from filling.  So turn it off.
	       (process-adaptive-read-buffering (not passphrase))
	       (process (progn
			  (set-default-file-modes 448)
			  (apply #'start-process "*GnuPG*"
				 errors-buffer program args)))
	       (status (process-status process))

	       exit-status)
	  (set-process-sentinel process nil)
	  (when passphrase
	    (let ((coding-system-for-write (or pgg-passphrase-coding-system
					       'binary)))
	      (process-send-string process passphrase))
	    (process-send-string process "\n")
	    ;; Bail out if passphrase is not accepted.
	    ;; MAINTAIN ME: Tested against GNUPG 1.4.6.
	    (let (result)
	      (while (not result)
		(accept-process-output process)
		(with-current-buffer errors-buffer
		  (save-excursion
		    (goto-char (point-min))
		    (when (re-search-forward
			   ;; BGM: BAD, GOOD, MISSING.
			   "^.GNUPG:. \\([BGM][A-Z]+\\)_PASSPHRASE"
			   nil t)
		      (setq result (match-string 1))))))
	      (unless (string= "GOOD" result)
		(error "Passphrase no good"))))
	  (process-send-region process start end)
	  (process-send-eof process)
	  ;; TODO: Re-enable `process-adaptive-read-buffering' here.
	  (while (eq 'run (setq status (process-status process)))
	    (accept-process-output process))
	  (delete-process process)
	  (setq exit-status (process-exit-status process))
	  (with-current-buffer (get-buffer-create output-buffer)
	    (buffer-disable-undo)
	    (erase-buffer)
	    (if (file-exists-p output-file-name)
		(let ((coding-system-for-read (if pgg-text-mode
						  'raw-text
						'binary)))
		  (insert-file-contents output-file-name)))
	    (set-buffer errors-buffer)
	    (if (memq status '(stop signal))
		(error "%s exited abnormally: '%s'" program exit-status))
	    (if (= 127 exit-status)
		(error "%s could not be found" program))))
      (if (file-exists-p output-file-name)
	  (delete-file output-file-name))
      (when (get-process "*GnuPG*")
	(kill-process "*GnuPG*"))
      (set-default-file-modes orig-mode))))

bug#469: marked as done (pgg-gpg.el - pgg-gpg-process-region timing problem)

[Emacs bug Tracking System]

[-- Attachment #1: Type: text/plain, Size: 865 bytes --]


Your message dated Sat, 02 Aug 2008 15:08:30 -0400
with message-id <873aln2q1d.fsf@stupidchicken.com>
and subject line Re: pgg-gpg.el - pgg-gpg-process-region timing problem
has caused the Emacs bug report #465,
regarding pgg-gpg.el - pgg-gpg-process-region timing problem
to be marked as done.

This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
bug report if necessary, and/or fix the problem forthwith.

(NB: If you are a system administrator and have no idea what this
message is talking about, this may indicate a serious mail system
misconfiguration somewhere. Please contact don@donarmstrong.com
immediately.)


-- 
465: http://emacsbugs.donarmstrong.com/cgi-bin/bugreport.cgi?bug=465
Emacs Bug Tracking System
Contact don@donarmstrong.com with problems

[-- Attachment #2: Type: message/rfc822, Size: 6606 bytes --]

From: Thien-Thi Nguyen <ttn@gnuvola.org>
To: "ken manheimer" <ken.manheimer@gmail.com>
Cc: wilde@sha-bang.de, emacs-pretest-bug@gnu.org, ueno@unixuser.org, emacs-devel <emacs-devel@gnu.org>
Subject: Re: pgg-gpg.el - pgg-gpg-process-region timing problem
Date: Mon, 23 Jun 2008 10:43:05 +0200
Message-ID: <87r6ao8rty.fsf@ambire.localdomain>

ChangeLog entry:

	Use process sentinel to monitor gpg subprocess status.

	* pgg-gpg.el (pgg-gpg-process-region): Fix omission bug:
	Use process sentinel to update process status and exit-status.
	Reported by ken manheimer <ken.manheimer@gmail.com>.

patch:
*** pgg-gpg.el	6 May 2008 07:57:46 -0000	1.27
--- pgg-gpg.el	23 Jun 2008 08:40:50 -0000
***************
*** 70,111 ****
  	    ,@pgg-gpg-extra-args ,@args))
  	 (output-buffer pgg-output-buffer)
  	 (errors-buffer pgg-errors-buffer)
! 	 (orig-mode (default-file-modes))
! 	 (process-connection-type nil)
! 	 (inhibit-redisplay t)
! 	 process status exit-status
! 	 passphrase-with-newline
! 	 encoded-passphrase-with-new-line)
      (with-current-buffer (get-buffer-create errors-buffer)
        (buffer-disable-undo)
        (erase-buffer))
      (unwind-protect
! 	(progn
  	  (set-default-file-modes 448)
! 	  (let ((coding-system-for-write 'binary))
! 	    (setq process
! 		  (apply #'start-process "*GnuPG*" errors-buffer
! 			 program args)))
! 	  (set-process-sentinel process #'ignore)
  	  (when passphrase
! 	    (setq passphrase-with-newline (concat passphrase "\n"))
! 	    (if pgg-passphrase-coding-system
! 		(progn
! 		  (setq encoded-passphrase-with-new-line
  			(encode-coding-string
! 			 passphrase-with-newline
! 			 (coding-system-change-eol-conversion
! 			  pgg-passphrase-coding-system 'unix)))
! 		  (pgg-clear-string passphrase-with-newline))
! 	      (setq encoded-passphrase-with-new-line passphrase-with-newline
! 		    passphrase-with-newline nil))
! 	    (process-send-string process encoded-passphrase-with-new-line))
  	  (process-send-region process start end)
  	  (process-send-eof process)
! 	  (while (eq 'run (process-status process))
  	    (accept-process-output process 5))
- 	  (setq status (process-status process)
- 		exit-status (process-exit-status process))
  	  (delete-process process)
  	  (with-current-buffer (get-buffer-create output-buffer)
  	    (buffer-disable-undo)
--- 70,104 ----
  	    ,@pgg-gpg-extra-args ,@args))
  	 (output-buffer pgg-output-buffer)
  	 (errors-buffer pgg-errors-buffer)
! 	 (inhibit-redisplay t))
      (with-current-buffer (get-buffer-create errors-buffer)
        (buffer-disable-undo)
        (erase-buffer))
      (unwind-protect
! 	(let* ((coding-system-for-write 'binary)
! 	       (process (let ((orig (default-file-modes)))
  			  (set-default-file-modes 448)
! 			  (prog1 (apply #'start-process "*GnuPG*"
! 					errors-buffer program args)
! 			    (set-default-file-modes orig))))
! 	       (status (process-status process))
! 	       exit-status)
! 	  (set-process-sentinel
! 	   process (lambda (process reason)
! 		     (setq status (process-status process)
! 			   exit-status (process-exit-status process))))
  	  (when passphrase
! 	    (process-send-string
! 	     process (if pgg-passphrase-coding-system
  			 (encode-coding-string
! 			  passphrase
! 			  pgg-passphrase-coding-system)
! 		       passphrase))
! 	    (process-send-string process "\n"))
  	  (process-send-region process start end)
  	  (process-send-eof process)
! 	  (while (eq 'run status)
  	    (accept-process-output process 5))
  	  (delete-process process)
  	  (with-current-buffer (get-buffer-create output-buffer)
  	    (buffer-disable-undo)
***************
*** 120,134 ****
  		(error "%s exited abnormally: '%s'" program exit-status))
  	    (if (= 127 exit-status)
  		(error "%s could not be found" program))))
-       (if passphrase-with-newline
- 	  (pgg-clear-string passphrase-with-newline))
-       (if encoded-passphrase-with-new-line
- 	  (pgg-clear-string encoded-passphrase-with-new-line))
-       (if (and process (eq 'run (process-status process)))
- 	  (interrupt-process process))
        (if (file-exists-p output-file-name)
! 	  (delete-file output-file-name))
!       (set-default-file-modes orig-mode))))
  
  (defun pgg-gpg-possibly-cache-passphrase (passphrase &optional key notruncate)
    (if (and passphrase
--- 113,120 ----
  		(error "%s exited abnormally: '%s'" program exit-status))
  	    (if (= 127 exit-status)
  		(error "%s could not be found" program))))
        (if (file-exists-p output-file-name)
! 	  (delete-file output-file-name)))))
  
  (defun pgg-gpg-possibly-cache-passphrase (passphrase &optional key notruncate)
    (if (and passphrase



[-- Attachment #3: Type: message/rfc822, Size: 1087 bytes --]

From: Chong Yidong <cyd@stupidchicken.com>
To: 465-done@emacsbugs.donarmstrong.com
Subject: Re: pgg-gpg.el - pgg-gpg-process-region timing problem
Date: Sat, 02 Aug 2008 15:08:30 -0400
Message-ID: <873aln2q1d.fsf@stupidchicken.com>

I've checked in a fix for this (verified by Ken Manheimer via private
email).  This fix is in pgg-gpg.el, which I think was calling
accept-process-output in a slightly incorrect way.

bug#472: pgg-gpg.el - pgg-gpg-process-region timing problem

[ken manheimer]

hi, thi.  thanks for the responses - i'm glad to see investigation into this.

i've tried both the patches, and both hang for me on the pggprob.el
test.  the hang is interruptable with Ctl-G keyboard quit.  i tried
stepping through the second patched version with edebug (including
disabling the '(inhibit-redisplay t)' let-binding, which thwarts
edebug stepping), and found the hang happens at the first
'(accept-process-output process)'.  (it hangs at the second one if i
don't provide a passphrase.)

i'm not sure what's happening.  in case it helps, gpg 1.4.6 is what's
running on my system.
-- 
ken
http://myriadicity.net



On Mon, Jun 23, 2008 at 9:17 AM, Thien-Thi Nguyen <ttn@gnuvola.org> wrote:
> Delving into this, i got a little carried away... full func below.
> This adds GOOD_PASSPHRASE checking (necessary for GNUPG 1.4.6, at
> least) and some other coding-system related finessing, but removes
> the sentinel proposed in the first attempt.  In its place, we zonk
> the baleful `process-adaptive-read-buffering' and remove timeout
> parameters from `accept-process-output' calls.
>
> With this func things seem to work fine for `pgg-sign' on a small
> buffer, and mostly for large buffers (270KB), but sometimes (~10%)
> with large buffers, i see a "file error: bad address *GnuPG*".  Hmmm...
>
> thi
>
> ______________________________________________________________________
> (defun pgg-gpg-process-region (start end passphrase program args)
>  (let* ((use-agent (and (null passphrase) (pgg-gpg-use-agent-p)))
>         (output-file-name (pgg-make-temp-file "pgg-output"))
>         (args
>          `("--status-fd" "2"
>            ,@(if use-agent '("--use-agent")
>                (if passphrase '("--passphrase-fd" "0")))
>            "--yes" ; overwrite
>            "--output" ,output-file-name
>            ,@pgg-gpg-extra-args ,@args))
>         (output-buffer pgg-output-buffer)
>         (errors-buffer pgg-errors-buffer)
>         (orig-mode (default-file-modes))
>         (inhibit-redisplay t))
>    (with-current-buffer (get-buffer-create errors-buffer)
>      (buffer-disable-undo)
>      (erase-buffer))
>    (unwind-protect
>        (let* ((coding-system-for-write 'binary)
>               ;; GNUPG 1.4.6 does not terminate on bad passphrase, eg:
>               ;;   [GNUPG:] BAD_PASSPHRASE (long hex # here)
>               ;;   gpg: skipped "ttn": bad passphrase
>               ;;   gpg: [stdin]: clearsign failed: bad passphrase
>               ;; so we need to check that condition ourselves and bail out.
>               ;;
>               ;; To check if the passphrase is accepted, we need to parse the
>               ;; errors-buffer, but `process-adaptive-read-buffering' non-nil
>               ;; sometimes prevents it from filling.  So turn it off.
>               (process-adaptive-read-buffering (not passphrase))
>               (process (progn
>                          (set-default-file-modes 448)
>                          (apply #'start-process "*GnuPG*"
>                                 errors-buffer program args)))
>               (status (process-status process))
>
>               exit-status)
>          (set-process-sentinel process nil)
>          (when passphrase
>            (let ((coding-system-for-write (or pgg-passphrase-coding-system
>                                               'binary)))
>              (process-send-string process passphrase))
>            (process-send-string process "\n")
>            ;; Bail out if passphrase is not accepted.
>            ;; MAINTAIN ME: Tested against GNUPG 1.4.6.
>            (let (result)
>              (while (not result)
>                (accept-process-output process)
>                (with-current-buffer errors-buffer
>                  (save-excursion
>                    (goto-char (point-min))
>                    (when (re-search-forward
>                           ;; BGM: BAD, GOOD, MISSING.
>                           "^.GNUPG:. \\([BGM][A-Z]+\\)_PASSPHRASE"
>                           nil t)
>                      (setq result (match-string 1))))))
>              (unless (string= "GOOD" result)
>                (error "Passphrase no good"))))
>          (process-send-region process start end)
>          (process-send-eof process)
>          ;; TODO: Re-enable `process-adaptive-read-buffering' here.
>          (while (eq 'run (setq status (process-status process)))
>            (accept-process-output process))
>          (delete-process process)
>          (setq exit-status (process-exit-status process))
>          (with-current-buffer (get-buffer-create output-buffer)
>            (buffer-disable-undo)
>            (erase-buffer)
>            (if (file-exists-p output-file-name)
>                (let ((coding-system-for-read (if pgg-text-mode
>                                                  'raw-text
>                                                'binary)))
>                  (insert-file-contents output-file-name)))
>            (set-buffer errors-buffer)
>            (if (memq status '(stop signal))
>                (error "%s exited abnormally: '%s'" program exit-status))
>            (if (= 127 exit-status)
>                (error "%s could not be found" program))))
>      (if (file-exists-p output-file-name)
>          (delete-file output-file-name))
>      (when (get-process "*GnuPG*")
>        (kill-process "*GnuPG*"))
>      (set-default-file-modes orig-mode))))

bug#471: marked as done (pgg-gpg.el - pgg-gpg-process-region timing problem)

[Emacs bug Tracking System]

[-- Attachment #1: Type: text/plain, Size: 865 bytes --]


Your message dated Sat, 02 Aug 2008 15:08:30 -0400
with message-id <873aln2q1d.fsf@stupidchicken.com>
and subject line Re: pgg-gpg.el - pgg-gpg-process-region timing problem
has caused the Emacs bug report #465,
regarding pgg-gpg.el - pgg-gpg-process-region timing problem
to be marked as done.

This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
bug report if necessary, and/or fix the problem forthwith.

(NB: If you are a system administrator and have no idea what this
message is talking about, this may indicate a serious mail system
misconfiguration somewhere. Please contact don@donarmstrong.com
immediately.)


-- 
465: http://emacsbugs.donarmstrong.com/cgi-bin/bugreport.cgi?bug=465
Emacs Bug Tracking System
Contact don@donarmstrong.com with problems

[-- Attachment #2: Type: message/rfc822, Size: 6041 bytes --]

From: Thien-Thi Nguyen <ttn@gnuvola.org>
To: "ken manheimer" <ken.manheimer@gmail.com>
Cc: wilde@sha-bang.de, emacs-pretest-bug@gnu.org, ueno@unixuser.org, emacs-devel <emacs-devel@gnu.org>
Subject: Re: pgg-gpg.el - pgg-gpg-process-region timing problem
Date: Mon, 23 Jun 2008 15:17:39 +0200
Message-ID: <873an4b898.fsf@ambire.localdomain>

Delving into this, i got a little carried away... full func below.
This adds GOOD_PASSPHRASE checking (necessary for GNUPG 1.4.6, at
least) and some other coding-system related finessing, but removes
the sentinel proposed in the first attempt.  In its place, we zonk
the baleful `process-adaptive-read-buffering' and remove timeout
parameters from `accept-process-output' calls.

With this func things seem to work fine for `pgg-sign' on a small
buffer, and mostly for large buffers (270KB), but sometimes (~10%)
with large buffers, i see a "file error: bad address *GnuPG*".  Hmmm...

thi

______________________________________________________________________
(defun pgg-gpg-process-region (start end passphrase program args)
  (let* ((use-agent (and (null passphrase) (pgg-gpg-use-agent-p)))
	 (output-file-name (pgg-make-temp-file "pgg-output"))
	 (args
	  `("--status-fd" "2"
	    ,@(if use-agent '("--use-agent")
		(if passphrase '("--passphrase-fd" "0")))
	    "--yes" ; overwrite
	    "--output" ,output-file-name
	    ,@pgg-gpg-extra-args ,@args))
	 (output-buffer pgg-output-buffer)
	 (errors-buffer pgg-errors-buffer)
	 (orig-mode (default-file-modes))
	 (inhibit-redisplay t))
    (with-current-buffer (get-buffer-create errors-buffer)
      (buffer-disable-undo)
      (erase-buffer))
    (unwind-protect
	(let* ((coding-system-for-write 'binary)
	       ;; GNUPG 1.4.6 does not terminate on bad passphrase, eg:
	       ;;   [GNUPG:] BAD_PASSPHRASE (long hex # here)
	       ;;   gpg: skipped "ttn": bad passphrase
	       ;;   gpg: [stdin]: clearsign failed: bad passphrase
	       ;; so we need to check that condition ourselves and bail out.
	       ;;
	       ;; To check if the passphrase is accepted, we need to parse the
	       ;; errors-buffer, but `process-adaptive-read-buffering' non-nil
	       ;; sometimes prevents it from filling.  So turn it off.
	       (process-adaptive-read-buffering (not passphrase))
	       (process (progn
			  (set-default-file-modes 448)
			  (apply #'start-process "*GnuPG*"
				 errors-buffer program args)))
	       (status (process-status process))

	       exit-status)
	  (set-process-sentinel process nil)
	  (when passphrase
	    (let ((coding-system-for-write (or pgg-passphrase-coding-system
					       'binary)))
	      (process-send-string process passphrase))
	    (process-send-string process "\n")
	    ;; Bail out if passphrase is not accepted.
	    ;; MAINTAIN ME: Tested against GNUPG 1.4.6.
	    (let (result)
	      (while (not result)
		(accept-process-output process)
		(with-current-buffer errors-buffer
		  (save-excursion
		    (goto-char (point-min))
		    (when (re-search-forward
			   ;; BGM: BAD, GOOD, MISSING.
			   "^.GNUPG:. \\([BGM][A-Z]+\\)_PASSPHRASE"
			   nil t)
		      (setq result (match-string 1))))))
	      (unless (string= "GOOD" result)
		(error "Passphrase no good"))))
	  (process-send-region process start end)
	  (process-send-eof process)
	  ;; TODO: Re-enable `process-adaptive-read-buffering' here.
	  (while (eq 'run (setq status (process-status process)))
	    (accept-process-output process))
	  (delete-process process)
	  (setq exit-status (process-exit-status process))
	  (with-current-buffer (get-buffer-create output-buffer)
	    (buffer-disable-undo)
	    (erase-buffer)
	    (if (file-exists-p output-file-name)
		(let ((coding-system-for-read (if pgg-text-mode
						  'raw-text
						'binary)))
		  (insert-file-contents output-file-name)))
	    (set-buffer errors-buffer)
	    (if (memq status '(stop signal))
		(error "%s exited abnormally: '%s'" program exit-status))
	    (if (= 127 exit-status)
		(error "%s could not be found" program))))
      (if (file-exists-p output-file-name)
	  (delete-file output-file-name))
      (when (get-process "*GnuPG*")
	(kill-process "*GnuPG*"))
      (set-default-file-modes orig-mode))))




[-- Attachment #3: Type: message/rfc822, Size: 1087 bytes --]

From: Chong Yidong <cyd@stupidchicken.com>
To: 465-done@emacsbugs.donarmstrong.com
Subject: Re: pgg-gpg.el - pgg-gpg-process-region timing problem
Date: Sat, 02 Aug 2008 15:08:30 -0400
Message-ID: <873aln2q1d.fsf@stupidchicken.com>

I've checked in a fix for this (verified by Ken Manheimer via private
email).  This fix is in pgg-gpg.el, which I think was calling
accept-process-output in a slightly incorrect way.

bug#474: pgg-gpg.el - pgg-gpg-process-region timing problem

[Thien-Thi Nguyen]

() "ken manheimer" <ken.manheimer@gmail.com>
() Mon, 23 Jun 2008 13:44:52 -0400

   the hang happens at the first
   '(accept-process-output process)'.

What happens if you convert
  (accept-process-output process)
to
  (accept-process-output)
?

thi

bug#472: marked as done (pgg-gpg.el - pgg-gpg-process-region timing problem)

[Emacs bug Tracking System]

[-- Attachment #1: Type: text/plain, Size: 865 bytes --]


Your message dated Sat, 02 Aug 2008 15:08:30 -0400
with message-id <873aln2q1d.fsf@stupidchicken.com>
and subject line Re: pgg-gpg.el - pgg-gpg-process-region timing problem
has caused the Emacs bug report #465,
regarding pgg-gpg.el - pgg-gpg-process-region timing problem
to be marked as done.

This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
bug report if necessary, and/or fix the problem forthwith.

(NB: If you are a system administrator and have no idea what this
message is talking about, this may indicate a serious mail system
misconfiguration somewhere. Please contact don@donarmstrong.com
immediately.)


-- 
465: http://emacsbugs.donarmstrong.com/cgi-bin/bugreport.cgi?bug=465
Emacs Bug Tracking System
Contact don@donarmstrong.com with problems

[-- Attachment #2: Type: message/rfc822, Size: 8828 bytes --]

From: "ken manheimer" <ken.manheimer@gmail.com>
To: "Thien-Thi Nguyen" <ttn@gnuvola.org>
Cc: wilde@sha-bang.de, emacs-pretest-bug@gnu.org, ueno@unixuser.org, emacs-devel <emacs-devel@gnu.org>
Subject: Re: pgg-gpg.el - pgg-gpg-process-region timing problem
Date: Mon, 23 Jun 2008 13:44:52 -0400
Message-ID: <2cd46e7f0806231044w6410b825s9611ba713e9ac353@mail.gmail.com>

hi, thi.  thanks for the responses - i'm glad to see investigation into this.

i've tried both the patches, and both hang for me on the pggprob.el
test.  the hang is interruptable with Ctl-G keyboard quit.  i tried
stepping through the second patched version with edebug (including
disabling the '(inhibit-redisplay t)' let-binding, which thwarts
edebug stepping), and found the hang happens at the first
'(accept-process-output process)'.  (it hangs at the second one if i
don't provide a passphrase.)

i'm not sure what's happening.  in case it helps, gpg 1.4.6 is what's
running on my system.
-- 
ken
http://myriadicity.net



On Mon, Jun 23, 2008 at 9:17 AM, Thien-Thi Nguyen <ttn@gnuvola.org> wrote:
> Delving into this, i got a little carried away... full func below.
> This adds GOOD_PASSPHRASE checking (necessary for GNUPG 1.4.6, at
> least) and some other coding-system related finessing, but removes
> the sentinel proposed in the first attempt.  In its place, we zonk
> the baleful `process-adaptive-read-buffering' and remove timeout
> parameters from `accept-process-output' calls.
>
> With this func things seem to work fine for `pgg-sign' on a small
> buffer, and mostly for large buffers (270KB), but sometimes (~10%)
> with large buffers, i see a "file error: bad address *GnuPG*".  Hmmm...
>
> thi
>
> ______________________________________________________________________
> (defun pgg-gpg-process-region (start end passphrase program args)
>  (let* ((use-agent (and (null passphrase) (pgg-gpg-use-agent-p)))
>         (output-file-name (pgg-make-temp-file "pgg-output"))
>         (args
>          `("--status-fd" "2"
>            ,@(if use-agent '("--use-agent")
>                (if passphrase '("--passphrase-fd" "0")))
>            "--yes" ; overwrite
>            "--output" ,output-file-name
>            ,@pgg-gpg-extra-args ,@args))
>         (output-buffer pgg-output-buffer)
>         (errors-buffer pgg-errors-buffer)
>         (orig-mode (default-file-modes))
>         (inhibit-redisplay t))
>    (with-current-buffer (get-buffer-create errors-buffer)
>      (buffer-disable-undo)
>      (erase-buffer))
>    (unwind-protect
>        (let* ((coding-system-for-write 'binary)
>               ;; GNUPG 1.4.6 does not terminate on bad passphrase, eg:
>               ;;   [GNUPG:] BAD_PASSPHRASE (long hex # here)
>               ;;   gpg: skipped "ttn": bad passphrase
>               ;;   gpg: [stdin]: clearsign failed: bad passphrase
>               ;; so we need to check that condition ourselves and bail out.
>               ;;
>               ;; To check if the passphrase is accepted, we need to parse the
>               ;; errors-buffer, but `process-adaptive-read-buffering' non-nil
>               ;; sometimes prevents it from filling.  So turn it off.
>               (process-adaptive-read-buffering (not passphrase))
>               (process (progn
>                          (set-default-file-modes 448)
>                          (apply #'start-process "*GnuPG*"
>                                 errors-buffer program args)))
>               (status (process-status process))
>
>               exit-status)
>          (set-process-sentinel process nil)
>          (when passphrase
>            (let ((coding-system-for-write (or pgg-passphrase-coding-system
>                                               'binary)))
>              (process-send-string process passphrase))
>            (process-send-string process "\n")
>            ;; Bail out if passphrase is not accepted.
>            ;; MAINTAIN ME: Tested against GNUPG 1.4.6.
>            (let (result)
>              (while (not result)
>                (accept-process-output process)
>                (with-current-buffer errors-buffer
>                  (save-excursion
>                    (goto-char (point-min))
>                    (when (re-search-forward
>                           ;; BGM: BAD, GOOD, MISSING.
>                           "^.GNUPG:. \\([BGM][A-Z]+\\)_PASSPHRASE"
>                           nil t)
>                      (setq result (match-string 1))))))
>              (unless (string= "GOOD" result)
>                (error "Passphrase no good"))))
>          (process-send-region process start end)
>          (process-send-eof process)
>          ;; TODO: Re-enable `process-adaptive-read-buffering' here.
>          (while (eq 'run (setq status (process-status process)))
>            (accept-process-output process))
>          (delete-process process)
>          (setq exit-status (process-exit-status process))
>          (with-current-buffer (get-buffer-create output-buffer)
>            (buffer-disable-undo)
>            (erase-buffer)
>            (if (file-exists-p output-file-name)
>                (let ((coding-system-for-read (if pgg-text-mode
>                                                  'raw-text
>                                                'binary)))
>                  (insert-file-contents output-file-name)))
>            (set-buffer errors-buffer)
>            (if (memq status '(stop signal))
>                (error "%s exited abnormally: '%s'" program exit-status))
>            (if (= 127 exit-status)
>                (error "%s could not be found" program))))
>      (if (file-exists-p output-file-name)
>          (delete-file output-file-name))
>      (when (get-process "*GnuPG*")
>        (kill-process "*GnuPG*"))
>      (set-default-file-modes orig-mode))))



[-- Attachment #3: Type: message/rfc822, Size: 1087 bytes --]

From: Chong Yidong <cyd@stupidchicken.com>
To: 465-done@emacsbugs.donarmstrong.com
Subject: Re: pgg-gpg.el - pgg-gpg-process-region timing problem
Date: Sat, 02 Aug 2008 15:08:30 -0400
Message-ID: <873aln2q1d.fsf@stupidchicken.com>

I've checked in a fix for this (verified by Ken Manheimer via private
email).  This fix is in pgg-gpg.el, which I think was calling
accept-process-output in a slightly incorrect way.

bug#476: pgg-gpg.el - pgg-gpg-process-region timing problem

[Stefan Monnier]

>    the hang happens at the first
>    '(accept-process-output process)'.

> What happens if you convert
>   (accept-process-output process)
> to
>   (accept-process-output)
> ?

If that fixes it, it means either that `process' is not the right
process, or that we may have a bug in the C code.  So if that fixes the
bug, please don't just install such a workaround without trying to find
the root cause first.


        Stefan

bug#474: marked as done (pgg-gpg.el - pgg-gpg-process-region timing problem)

[Emacs bug Tracking System]

[-- Attachment #1: Type: text/plain, Size: 865 bytes --]


Your message dated Sat, 02 Aug 2008 15:08:30 -0400
with message-id <873aln2q1d.fsf@stupidchicken.com>
and subject line Re: pgg-gpg.el - pgg-gpg-process-region timing problem
has caused the Emacs bug report #465,
regarding pgg-gpg.el - pgg-gpg-process-region timing problem
to be marked as done.

This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
bug report if necessary, and/or fix the problem forthwith.

(NB: If you are a system administrator and have no idea what this
message is talking about, this may indicate a serious mail system
misconfiguration somewhere. Please contact don@donarmstrong.com
immediately.)


-- 
465: http://emacsbugs.donarmstrong.com/cgi-bin/bugreport.cgi?bug=465
Emacs Bug Tracking System
Contact don@donarmstrong.com with problems

[-- Attachment #2: Type: message/rfc822, Size: 2531 bytes --]

From: Thien-Thi Nguyen <ttn@gnuvola.org>
To: "ken manheimer" <ken.manheimer@gmail.com>
Cc: emacs-pretest-bug@gnu.org, emacs-devel <emacs-devel@gnu.org>
Subject: Re: pgg-gpg.el - pgg-gpg-process-region timing problem
Date: Tue, 24 Jun 2008 12:42:22 +0200
Message-ID: <87mylb2jxt.fsf@ambire.localdomain>

() "ken manheimer" <ken.manheimer@gmail.com>
() Mon, 23 Jun 2008 13:44:52 -0400

   the hang happens at the first
   '(accept-process-output process)'.

What happens if you convert
  (accept-process-output process)
to
  (accept-process-output)
?

thi



[-- Attachment #3: Type: message/rfc822, Size: 1087 bytes --]

From: Chong Yidong <cyd@stupidchicken.com>
To: 465-done@emacsbugs.donarmstrong.com
Subject: Re: pgg-gpg.el - pgg-gpg-process-region timing problem
Date: Sat, 02 Aug 2008 15:08:30 -0400
Message-ID: <873aln2q1d.fsf@stupidchicken.com>

I've checked in a fix for this (verified by Ken Manheimer via private
email).  This fix is in pgg-gpg.el, which I think was calling
accept-process-output in a slightly incorrect way.

bug#470: marked as done (pgg-gpg.el - pgg-gpg-process-region timing problem)

[Emacs bug Tracking System]

[-- Attachment #1: Type: text/plain, Size: 865 bytes --]


Your message dated Sat, 02 Aug 2008 15:08:30 -0400
with message-id <873aln2q1d.fsf@stupidchicken.com>
and subject line Re: pgg-gpg.el - pgg-gpg-process-region timing problem
has caused the Emacs bug report #465,
regarding pgg-gpg.el - pgg-gpg-process-region timing problem
to be marked as done.

This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
bug report if necessary, and/or fix the problem forthwith.

(NB: If you are a system administrator and have no idea what this
message is talking about, this may indicate a serious mail system
misconfiguration somewhere. Please contact don@donarmstrong.com
immediately.)


-- 
465: http://emacsbugs.donarmstrong.com/cgi-bin/bugreport.cgi?bug=465
Emacs Bug Tracking System
Contact don@donarmstrong.com with problems

[-- Attachment #2: Type: message/rfc822, Size: 2709 bytes --]

From: Thien-Thi Nguyen <ttn@gnuvola.org>
To: "ken manheimer" <ken.manheimer@gmail.com>
Cc: ueno@unixuser.org, wilde@sha-bang.de
Cc: emacs-devel <emacs-devel@gnu.org>, emacs-pretest-bug@gnu.org
Subject: Re: pgg-gpg.el - pgg-gpg-process-region timing problem
Date: Mon, 23 Jun 2008 10:17:51 +0200
Message-ID: <87zlpc8t00.fsf@ambire.localdomain>

() "ken manheimer" <ken.manheimer@gmail.com>
() Sun, 22 Jun 2008 13:37:11 -0400

   pggprob-patch.txt, with a single-line modification to
   pgg-gpg-process-region to work around the problem.  both
   are attached.  since the workaround is merely a timing
   delay, it's clearly not a fix!

Please find below an UNTESTED patch that addresses the
problem in two ways:
 - use a downward funarg sentinel for complete monitoring
 - reduce unnecessary data motion

Does it give good results?

(Authors added to cc.)

thi



[-- Attachment #3: Type: message/rfc822, Size: 1087 bytes --]

From: Chong Yidong <cyd@stupidchicken.com>
To: 465-done@emacsbugs.donarmstrong.com
Subject: Re: pgg-gpg.el - pgg-gpg-process-region timing problem
Date: Sat, 02 Aug 2008 15:08:30 -0400
Message-ID: <873aln2q1d.fsf@stupidchicken.com>

I've checked in a fix for this (verified by Ken Manheimer via private
email).  This fix is in pgg-gpg.el, which I think was calling
accept-process-output in a slightly incorrect way.

bug#487: pgg-gpg.el - pgg-gpg-process-region timing problem

[Chong Yidong]

> i am not sure how to proceed with this problem.  as demonstrated by
> the pggprob.el script, i consistently get encryption and decryption
> problems without the contrived delay in my pgg-gpg-process-region
> patch.  (the problem does not happen if i just do a regular
> pgg-decrypt - i think the crucial thing is that a symmetric passphrase
> is being passed into pgg-gpg-process-region.)

Here's one suggestion, though I don't know how useful it is: if it's
possible to run the same or similar version of pgg in Emacs 22, you
could and see if it appears there.  If not, we could narrow the problem
down to a change in Emacs' process code during Emacs 23 (though OTOH I
don't know of any that could lead to this).

bug#486: marked as done (pgg-gpg.el - pgg-gpg-process-region timing problem)

[Emacs bug Tracking System]

[-- Attachment #1: Type: text/plain, Size: 865 bytes --]


Your message dated Sat, 02 Aug 2008 15:08:30 -0400
with message-id <873aln2q1d.fsf@stupidchicken.com>
and subject line Re: pgg-gpg.el - pgg-gpg-process-region timing problem
has caused the Emacs bug report #465,
regarding pgg-gpg.el - pgg-gpg-process-region timing problem
to be marked as done.

This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
bug report if necessary, and/or fix the problem forthwith.

(NB: If you are a system administrator and have no idea what this
message is talking about, this may indicate a serious mail system
misconfiguration somewhere. Please contact don@donarmstrong.com
immediately.)


-- 
465: http://emacsbugs.donarmstrong.com/cgi-bin/bugreport.cgi?bug=465
Emacs Bug Tracking System
Contact don@donarmstrong.com with problems

[-- Attachment #2: Type: message/rfc822, Size: 10449 bytes --]

[-- Attachment #2.1.1: Type: text/plain, Size: 5144 bytes --]

i am not sure how to proceed with this problem.  as demonstrated by
the pggprob.el script, i consistently get encryption and decryption
problems without the contrived delay in my pgg-gpg-process-region
patch.  (the problem does not happen if i just do a regular
pgg-decrypt - i think the crucial thing is that a symmetric passphrase
is being passed into pgg-gpg-process-region.)

i have tried everything i could think of to exclude local quirks from
the problem, including starting emacs with '--no-site-file -q' and
eliminating everything from my ~/.gnupg/gpg.conf file.  i have
reproduced the problem (and workaround) consistently in kubuntu 8.04
emacs-snapshot package and a recent CVS HEAD checkout.  in fact, i
have incidentally reinstalled my kubuntu 8.04 OS recently, on both my
main 2.0 Ghz T43 thinkpad and on a 750 Mhz a20p thinkpad.  the problem
continues consistently on the t43 but does *not* occur on the a20p,
which is not contrary to the sensitivities of a timing problem.

i have also tried thien-thi nguyen's alternative approaches, but
neither worked for me, hanging in different places.

i am pretty sure this is a real problem, and something that warrants
attention.  i realize that the difficulty in repeating the problem
elsewhere increases the challenge in addressing it, and am open to
cooperating with people to try things, if that would help.  the
consistent effectiveness of my workaround should help provide clues
about the nature of the problem, at least.

any suggestions?
--
ken
http://myriadicity.net

On Sun, Jun 22, 2008 at 1:37 PM, ken manheimer <ken.manheimer@gmail.com> wrote:
> allout encryption, which depends on pgg.el &c, has been failing due to
> a weird timing problem i've narrowed down to the
> pgg-gpg-process-region function.  this is in emacs 23.0.60 built from
> a CVS HEAD checkout of an hour or so ago, and some earlier 23.0.60
> versions.  i've reproduced the problem using emacs -q and regularly
> launched emacs, etc.
>
> the problem is that pgg-gpg-process-region often returns a nil status
> despite successful decryption, unless a delay (eg, (sleep-for .3) is
> inserted between process start and later interaction with the process.
>  with the delay, the status is something like 207 or 208, while
> without it the status is nil - yet, on examination the content is
> sucessfully decrypted.
>
> the size of the delay needs to be increased when the machine is
> heavily loaded, and can be reduced (on my thinkpad t43 2 GHz laptop,
> 3Gb ram, running kubuntu 8.0.4 +) to no lower than (sleep-for .2).
> rarely, the proper status is returned without the delay, depending on
> the machine load and other things i can't identify.  considering the
> real-time sensitivity of the problem, your mileage will almost
> certainly vary!
>
> i originally thought the problem might be in gpg, but now figure it
> resides either in the emacs process mechanism, pgg's use of that, or
> somewhere in the combination of emacs and pgg.
>
> i've created a simple .el script, pggprob.el, with minimal code and
> data to produce the problem, and a patch, pggprob-patch.txt, with a
> single-line modification to pgg-gpg-process-region to work around the
> problem.  both are attached.  since the workaround is merely a timing
> delay, it's clearly not a fix!  i'm hoping this is enough to enable
> others to reproduce the problem, and investigate...
> --
> ken
> http://myriadicity.net
>
> If Emacs crashed, and you have the Emacs process in the gdb debugger,
> please include the output from the following gdb commands:
>    `bt full' and `xbacktrace'.
> If you would like to further debug the crash, please read the file
> /home/klm/src/emacs-HEAD/etc/DEBUG for instructions.
>
> In GNU Emacs 23.0.60.1 (i686-pc-linux-gnu, GTK+ Version 2.12.9)
>  of 2008-06-22 on twist
> Windowing system distributor `The X.Org Foundation', version 11.0.10400090
> Important settings:
>  value of $LC_ALL: nil
>  value of $LC_COLLATE: nil
>  value of $LC_CTYPE: nil
>  value of $LC_MESSAGES: nil
>  value of $LC_MONETARY: nil
>  value of $LC_NUMERIC: nil
>  value of $LC_TIME: nil
>  value of $LANG: en_US.UTF-8
>  value of $XMODIFIERS: nil
>  locale-coding-system: utf-8-unix
>  default-enable-multibyte-characters: t
>
> Major mode: Emacs-Lisp
>
> Minor modes in effect:
>  tooltip-mode: t
>  tool-bar-mode: t
>  mouse-wheel-mode: t
>  menu-bar-mode: t
>  file-name-shadow-mode: t
>  global-font-lock-mode: t
>  font-lock-mode: t
>  blink-cursor-mode: t
>  global-auto-composition-mode: t
>  auto-composition-mode: t
>  auto-encryption-mode: t
>  auto-compression-mode: t
>  line-number-mode: t
>  transient-mark-mode: t
>
> Recent input:
> C-x C-f <M-backspace> <M-backspace> <M-backspace> l
> i b / e m a <tab> a l l <tab> s c r <tab> p g g p <tab>
> <return> M-x e v a l - c u r <tab> <return> M-. C-g
> M-> C-x C-e M-x e m a c s <M-backspace> r e p o r t
> - e m <tab> <return>
>
> Recent messages:
> For information about GNU Emacs and the GNU system, type C-h C-a.
> Source file `/home/klm/src/emacs-HEAD/lisp/pgg-gpg.el' newer than
> byte-compiled file
> Quit
> Mark set
> nil
>



-- 
ken
http://myriadicity.net

[-- Attachment #2.1.2: pggprob.el --]
[-- Type: application/octet-stream, Size: 671 bytes --]

;; demonstrate timing problem in pgg-gpg-process-region

(setq passphrase "a"
      verifier "-----BEGIN PGP MESSAGE-----
Version: GnuPG v1.4.6 (GNU/Linux)

jA0EAwMCo72fNGB1rNxgySv+rxdkPv46nVxQ5fg4yT78ldgVcDseD7Mre4Fhjifn
VTA1/PdmetVfFb9+
=MXF6
-----END PGP MESSAGE-----"
      count 0)

(defun decrypt (passphrase text)
  (with-current-buffer (get-buffer-create "*pgg-test*")
    (erase-buffer)
    (insert text)
    (pgg-decrypt nil nil passphrase)))

;; the following yields nil w/out the patched delay, 208 with it.
;; examination of the actual results buffer shows that the verifier is
;; successfully decrypted in both cases, however.
(decrypt passphrase verifier)

[-- Attachment #2.1.3: pgg-patch.txt --]
[-- Type: text/plain, Size: 409 bytes --]

--- pgg-gpg.el.~1.27.~	2008-05-06 03:57:46.000000000 -0400
+++ pgg-gpg.el	2008-06-22 13:24:14.000000000 -0400
@@ -86,6 +86,7 @@
 	    (setq process
 		  (apply #'start-process "*GnuPG*" errors-buffer
 			 program args)))
+          (sleep-for .3)                ; klm: timing delay
 	  (set-process-sentinel process #'ignore)
 	  (when passphrase
 	    (setq passphrase-with-newline (concat passphrase "\n"))

[-- Attachment #3: Type: message/rfc822, Size: 1087 bytes --]

From: Chong Yidong <cyd@stupidchicken.com>
To: 465-done@emacsbugs.donarmstrong.com
Subject: Re: pgg-gpg.el - pgg-gpg-process-region timing problem
Date: Sat, 02 Aug 2008 15:08:30 -0400
Message-ID: <873aln2q1d.fsf@stupidchicken.com>

I've checked in a fix for this (verified by Ken Manheimer via private
email).  This fix is in pgg-gpg.el, which I think was calling
accept-process-output in a slightly incorrect way.

bug#476: marked as done (pgg-gpg.el - pgg-gpg-process-region timing problem)

[Emacs bug Tracking System]

[-- Attachment #1: Type: text/plain, Size: 865 bytes --]


Your message dated Sat, 02 Aug 2008 15:08:30 -0400
with message-id <873aln2q1d.fsf@stupidchicken.com>
and subject line Re: pgg-gpg.el - pgg-gpg-process-region timing problem
has caused the Emacs bug report #465,
regarding pgg-gpg.el - pgg-gpg-process-region timing problem
to be marked as done.

This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
bug report if necessary, and/or fix the problem forthwith.

(NB: If you are a system administrator and have no idea what this
message is talking about, this may indicate a serious mail system
misconfiguration somewhere. Please contact don@donarmstrong.com
immediately.)


-- 
465: http://emacsbugs.donarmstrong.com/cgi-bin/bugreport.cgi?bug=465
Emacs Bug Tracking System
Contact don@donarmstrong.com with problems

[-- Attachment #2: Type: message/rfc822, Size: 3203 bytes --]

From: Stefan Monnier <monnier@iro.umontreal.ca>
To: Thien-Thi Nguyen <ttn@gnuvola.org>
Cc: "ken manheimer" <ken.manheimer@gmail.com>, emacs-pretest-bug@gnu.org, emacs-devel <emacs-devel@gnu.org>
Subject: Re: pgg-gpg.el - pgg-gpg-process-region timing problem
Date: Tue, 24 Jun 2008 08:30:08 -0400
Message-ID: <jwviqvznhja.fsf-monnier+emacs@gnu.org>

>    the hang happens at the first
>    '(accept-process-output process)'.

> What happens if you convert
>   (accept-process-output process)
> to
>   (accept-process-output)
> ?

If that fixes it, it means either that `process' is not the right
process, or that we may have a bug in the C code.  So if that fixes the
bug, please don't just install such a workaround without trying to find
the root cause first.


        Stefan



[-- Attachment #3: Type: message/rfc822, Size: 1087 bytes --]

From: Chong Yidong <cyd@stupidchicken.com>
To: 465-done@emacsbugs.donarmstrong.com
Subject: Re: pgg-gpg.el - pgg-gpg-process-region timing problem
Date: Sat, 02 Aug 2008 15:08:30 -0400
Message-ID: <873aln2q1d.fsf@stupidchicken.com>

I've checked in a fix for this (verified by Ken Manheimer via private
email).  This fix is in pgg-gpg.el, which I think was calling
accept-process-output in a slightly incorrect way.

bug#487: marked as done (pgg-gpg.el - pgg-gpg-process-region timing problem)

[Emacs bug Tracking System]

[-- Attachment #1: Type: text/plain, Size: 865 bytes --]


Your message dated Sat, 02 Aug 2008 15:08:30 -0400
with message-id <873aln2q1d.fsf@stupidchicken.com>
and subject line Re: pgg-gpg.el - pgg-gpg-process-region timing problem
has caused the Emacs bug report #465,
regarding pgg-gpg.el - pgg-gpg-process-region timing problem
to be marked as done.

This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
bug report if necessary, and/or fix the problem forthwith.

(NB: If you are a system administrator and have no idea what this
message is talking about, this may indicate a serious mail system
misconfiguration somewhere. Please contact don@donarmstrong.com
immediately.)


-- 
465: http://emacsbugs.donarmstrong.com/cgi-bin/bugreport.cgi?bug=465
Emacs Bug Tracking System
Contact don@donarmstrong.com with problems

[-- Attachment #2: Type: message/rfc822, Size: 2984 bytes --]

From: Chong Yidong <cyd@stupidchicken.com>
To: "ken manheimer" <ken.manheimer@gmail.com>
Cc: emacs-devel <emacs-devel@gnu.org>, emacs-pretest-bug@gnu.org, "Thien-Thi Nguyen" <ttn@gnuvola.org>, "Stefan Monnier" <monnier@iro.umontreal.ca>, "Daiki Ueno" <ueno@unixuser.org>, "Sascha Wilde" <wilde@sha-bang.de>
Subject: Re: pgg-gpg.el - pgg-gpg-process-region timing problem
Date: Thu, 26 Jun 2008 12:32:05 -0400
Message-ID: <87fxr040oq.fsf@stupidchicken.com>

> i am not sure how to proceed with this problem.  as demonstrated by
> the pggprob.el script, i consistently get encryption and decryption
> problems without the contrived delay in my pgg-gpg-process-region
> patch.  (the problem does not happen if i just do a regular
> pgg-decrypt - i think the crucial thing is that a symmetric passphrase
> is being passed into pgg-gpg-process-region.)

Here's one suggestion, though I don't know how useful it is: if it's
possible to run the same or similar version of pgg in Emacs 22, you
could and see if it appears there.  If not, we could narrow the problem
down to a change in Emacs' process code during Emacs 23 (though OTOH I
don't know of any that could lead to this).



[-- Attachment #3: Type: message/rfc822, Size: 1087 bytes --]

From: Chong Yidong <cyd@stupidchicken.com>
To: 465-done@emacsbugs.donarmstrong.com
Subject: Re: pgg-gpg.el - pgg-gpg-process-region timing problem
Date: Sat, 02 Aug 2008 15:08:30 -0400
Message-ID: <873aln2q1d.fsf@stupidchicken.com>

I've checked in a fix for this (verified by Ken Manheimer via private
email).  This fix is in pgg-gpg.el, which I think was calling
accept-process-output in a slightly incorrect way.