From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!not-for-mail From: "Riccardo Murri" Newsgroups: gmane.emacs.devel Subject: Re: url-retrieve-synchronously randomly fails on https URLs (patch included) Date: Sun, 28 Oct 2007 14:40:11 +0200 Message-ID: <1c34ba170710280540g5b2a9983o33abfcba2843d95@mail.gmail.com> References: <20071027104716.E9BA773545@tanja.localdomain> NNTP-Posting-Host: lo.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Trace: ger.gmane.org 1193575274 27263 80.91.229.12 (28 Oct 2007 12:41:14 GMT) X-Complaints-To: usenet@ger.gmane.org NNTP-Posting-Date: Sun, 28 Oct 2007 12:41:14 +0000 (UTC) Cc: emacs-devel@gnu.org To: rms@gnu.org Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Sun Oct 28 13:41:16 2007 Return-path: Envelope-to: ged-emacs-devel@m.gmane.org Original-Received: from lists.gnu.org ([199.232.76.165]) by lo.gmane.org with esmtp (Exim 4.50) id 1Im7S4-0004XR-22 for ged-emacs-devel@m.gmane.org; Sun, 28 Oct 2007 13:41:12 +0100 Original-Received: from localhost ([127.0.0.1] helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1Im7Rv-0004ai-3G for ged-emacs-devel@m.gmane.org; Sun, 28 Oct 2007 08:41:03 -0400 Original-Received: from mailman by lists.gnu.org with tmda-scanned (Exim 4.43) id 1Im7R9-0003Xg-Ut for emacs-devel@gnu.org; Sun, 28 Oct 2007 08:40:16 -0400 Original-Received: from exim by lists.gnu.org with spam-scanned (Exim 4.43) id 1Im7R9-0003Vx-5Q for emacs-devel@gnu.org; Sun, 28 Oct 2007 08:40:15 -0400 Original-Received: from [199.232.76.173] (helo=monty-python.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1Im7R8-0003Vj-JV for emacs-devel@gnu.org; Sun, 28 Oct 2007 08:40:14 -0400 Original-Received: from wa-out-1112.google.com ([209.85.146.182]) by monty-python.gnu.org with esmtp (Exim 4.60) (envelope-from ) id 1Im7R7-00034R-K7 for emacs-devel@gnu.org; Sun, 28 Oct 2007 08:40:14 -0400 Original-Received: by wa-out-1112.google.com with SMTP id k34so1742797wah for ; Sun, 28 Oct 2007 05:40:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; bh=/4DSnnNKn19Bz1mc9jt13UUbCgSnPcJXO7KV7LuAus4=; b=otcsLKk4VcdPzzpi04QNSqDhczx33AJGeVE7INjHaFC+bBdQSdd1y0uqNDSOZ2wR0zaJ3knFINdF73rtRRN6Ague51fjNgicosbjDGTeaAfn3S4O/lRA5RzDhx8cjJbcZT+mNkUyxqz1AxsIwbxD+290zWPHO+d0P6mUonn4Dac= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=IQ8dU8xH2Yh62MK6IeSi7pG0lcrRHXhRYCgNpDmq8uJmfNWmT+B/R1FI+BLex7/r02efWGhLIj/6hbOOEaqMcIhLgRjrSkH8E4xcOQl3wnM6stRinszFWqn1vYZA4yA7jaFsqH68KBqpFESxmQroagch6usZNfE1I+Dn6Vo6GIo= Original-Received: by 10.114.148.1 with SMTP id v1mr5793317wad.1193575211579; Sun, 28 Oct 2007 05:40:11 -0700 (PDT) Original-Received: by 10.114.75.13 with HTTP; Sun, 28 Oct 2007 05:40:11 -0700 (PDT) In-Reply-To: Content-Disposition: inline X-detected-kernel: by monty-python.gnu.org: Linux 2.6 (newer, 2) X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Original-Sender: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Xref: news.gmane.org gmane.emacs.devel:81934 Archived-At: On 10/28/07, Richard Stallman wrote: > I traced down the bug to `open-tls-stream', which returns immediately > after having seen the regexp `tls-success' in the buffer. However, > both `openssl s_client' and `gnutls' print more information after that > line, thus `open-tls-stream' may occasionally return the buffer to the > caller when point is still amidst openssl/gnutls output. > > A solution is to have `open-tls-stream' wait a little more to accept > output; > [deletia] > > This is not totally reliable, because it is possible for > accept-process-output to get just part of the output. > That may be unlikely, which is why you have not seen it happen, > but it is possible. > > So here is another suggestion. Can you change the `tls-success' > regexp so that it matches all the text that openssl or gnutls > would produce? (You might need to use \(...\|...\) to match > each on separately.) > No, I don't think that is feasible: `openssl s_client' is *very* verbose and its output is meant for humans to read rather than for programs to parse, so any regexp that matches the informational messages has a good chance of eating up some of the data too. I had a look at the sources, and wrote a regexp that tries to match only the *last* informational message; the appended patch looks for that *after* `tls-success' has been matched. Unless I got it all wrong, this is much more complicated than the original three-liner; I wonder if it is worth going this way instead of a very simple patch that could still be buggy on rare occasions. -- Riccardo Murri, via Galeazzo Alessi 61, 00176 Roma --- src/emacs22/lisp/net/tls.el 2007-08-05 21:06:12.000000000 +0200 +++ emacs/lisp/tls.el 2007-10-28 13:12:11.000000000 +0100 @@ -51,6 +51,10 @@ (autoload 'format-spec "format-spec") (autoload 'format-spec-make "format-spec")) +(eval-when-compile + (require 'rx) ; for writing readable regexps + (require 'cl)) ; for `decf' + (defgroup tls nil "Transport Layer Security (TLS) parameters." :group 'comm) @@ -89,6 +93,50 @@ :type 'string :group 'tls) +(defcustom tls-end-of-info + (rx + (or + ;; `openssl s_client` regexp + (sequence + ;; see ssl/ssl_txt.c lines 219--220 + line-start + " Verify return code: " + (one-or-more not-newline) + "\n" + ;; according apps/s_client.c line 1515 this is always the last line + ;; that is printed by s_client before the real data + "---\n") + + ;; `gnutls` regexp + (sequence + ;; see src/cli.c lines 721-- + (sequence line-start "- Simple Client Mode:\n") + (zero-or-more + (or + "\n" ; ignore blank lines + ;; XXX: we have no way of knowing if the STARTTLS handshake + ;; sequence has completed successfully, because `gnutls` will + ;; only report failure. + (sequence line-start "\*\*\* Starting TLS handshake\n")))))) + "Regexp matching end of TLS session information; client data begins after this. +The default matches `openssl s_client' (version 0.9.8c) and +`gnutls-cli' (version 2.0.1) output." + :version "22.1" + :type 'regexp + :group 'tls) + +(defcustom tls-end-of-info-match-attempts 10 + "How many attempts to make at matching `tls-end-of-info'. +This is intended as a safety measure to avoid a potentially +endless loop: after this many attempts, presume `tls-end-of-info' +regexp is broken and will never be matched, so consider that +stream is already at the start of client data and hope for the best. + +Use a negative value to retry indefinitely." + :version "22.1" + :type 'integer + :group 'tls) + (defun tls-certificate-information (der) "Parse X.509 certificate in DER format into an assoc list." (let ((certificate (concat "-----BEGIN CERTIFICATE-----\n" @@ -130,6 +178,8 @@ process cmd done) (if use-temp-buffer (setq buffer (generate-new-buffer " TLS"))) + (save-excursion + (set-buffer buffer) (message "Opening TLS connection to `%s'..." host) (while (and (not done) (setq cmd (pop cmds))) (message "Opening TLS connection with `%s'..." cmd) @@ -146,19 +196,39 @@ port))))) (while (and process (memq (process-status process) '(open run)) - (save-excursion - (set-buffer buffer) ;; XXX "blue moon" nntp.el bug + (progn (goto-char (point-min)) (not (setq done (re-search-forward tls-success nil t))))) (unless (accept-process-output process 1) (sit-for 1))) (message "Opening TLS connection with `%s'...%s" cmd (if done "done" "failed")) - (if done - (setq done process) - (delete-process process)))) + (if (not done) + (delete-process process) + ;; advance point to after all informational messages that + ;; `openssl s_client' and `gnutls' print + (lexical-let ((attempts tls-end-of-info-match-attempts) + (start-of-data nil)) + (while (and + (not (= 0 (if (> attempts 0) (decf attempts) -1))) + ;; the string matching `tls-end-of-info' might come + ;; in separate chunks from `accept-process-output', + ;; so start the search always where `tls-success' ended + (not (setq start-of-data + (save-excursion + (if (re-search-forward tls-end-of-info nil t) + (match-end 0)))))) + (unless (accept-process-output process 1) + (sit-for 1))) + (if start-of-data + ;; move point to start of client data + (goto-char start-of-data) + (warn + "`open-tls-stream': Could not match `tls-end-of-info' regexp in %d attempts." + tls-end-of-info-match-attempts))) + (setq done process)))) (message "Opening TLS connection to `%s'...%s" - host (if done "done" "failed")) + host (if done "done" "failed"))) (when use-temp-buffer (if done (set-process-buffer process nil)) (kill-buffer buffer))