IMAP and Exchange 2007 - imap-fetch-safe (was: FIXMEs in imap.el and nnimap.el)

IMAP and Exchange 2007 - imap-fetch-safe (was: FIXMEs in imap.el and nnimap.el)

[Reiner Steib]

On Tue, Jan 13 2009, Dave Love wrote:

> I'm afraid there's something wrong with my fix, but it seems to be
> something like a race condition, and isn't consistently reproducible.

Bjorn Solberg reported that the fix wont work for him too:
See <news:m3bpud5mmy.fsf@famsolberg.com> on ding, or 
http://thread.gmane.org/gmane.emacs.gnus.general/68106/focus=68138
where he posted the *imap-debug*" buffer.

> It also goes away when I try to debug it.  I see the old error sometimes
> when starting Gnus afresh, and the trace says imap-fetch-safe is called
> from nnimap-find-minmax-uid.  It appears that the condition-case in
> imap-fetch-safe isn't catching the error thrown by imap-parse-message.  
>
> Now I think about it, I'm more puzzled, and obviously don't understand
> the logic.  An error in a process filter should be caught anyway, so I
> wonder why the code ever works, though it does some of the time.  Any
> ideas?

Maybe someone on emacs-devel has an idea (Cc-ed).  For emacs-devel: we
are talking about the recent addition `imap-fetch-safe' in `imap.el'
and `nnimap.el'.  AFAICS, the code in the Emacs trunk is uptodate.
Note that Dave uses Emacs 21, Bjorn uses Emacs/23.0.50 according to
his headers.

> Setting imap-enable-exchange-bug-workaround globally works, of course,

It should also be possible to set it as a server variable
(info "(gnus)Server Variables").

> and it's not clear to me what the problem is with always using the
> workaround anyhow.  Are there servers on which it's known to fail or be
> noticeably slow?

Simon?  Others?
http://thread.gmane.org/gmane.emacs.gnus.general/66635/focus=94117

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: IMAP and Exchange 2007 - imap-fetch-safe

[Simon Josefsson]

Reiner Steib <reinersteib+gmane@imap.cc> writes:

>> It also goes away when I try to debug it.  I see the old error sometimes
>> when starting Gnus afresh, and the trace says imap-fetch-safe is called
>> from nnimap-find-minmax-uid.  It appears that the condition-case in
>> imap-fetch-safe isn't catching the error thrown by imap-parse-message.  

Maybe the problem is a condition-case within another condition-case
work?  I recall problems related to this when run as an async process
filter.

Maybe it is possible to re-write the approach without using
condition-case, that would likely be easier to debug anyway.

>> and it's not clear to me what the problem is with always using the
>> workaround anyhow.  Are there servers on which it's known to fail or be
>> noticeably slow?
>
> Simon?  Others?
> http://thread.gmane.org/gmane.emacs.gnus.general/66635/focus=94117

I'd prefer to avoid sending the Exchange bug-workaround approach
("1,*:*") to any server that does not need it.  I've seen servers that
(internally) open up all e-mails in the folder and searches them, but
for the 1,* approach was able to return data quickly.  This may be old
information now, but generally I don't see why imap.el should send poor
protocol output to all servers just because Exchange is broken.

/Simon

Re: IMAP and Exchange 2007 - imap-fetch-safe

[Bjorn Solberg]

Reiner Steib writes:

> On Tue, Jan 13 2009, Dave Love wrote:
>> I'm afraid there's something wrong with my fix, but it seems to be
>> something like a race condition, and isn't consistently reproducible.

> Bjorn Solberg reported that the fix wont work for him too:
> See <news:m3bpud5mmy.fsf@famsolberg.com> on ding, or 
> http://thread.gmane.org/gmane.emacs.gnus.general/68106/focus=68138
> where he posted the *imap-debug*" buffer.

Note that it's only upon startup I am seeing trouble.  Once Gnus has
started and displays the group list, I can

(setq imap-enable-exchange-bug-workaround nil)

and everything still works fine, albeit slow.  Fetching new mail and
using nnimap-split-inbox '("INBOX") and nnimap-split-rule processes
about 2s/header or 1/2 header/sec.  Thunderbird seems to open folders
and fetch headers from Exchange-2007 faster than Gnus also.  Using Gnus
and IMAP against GMail is fast though.

Bjorn.

Re: IMAP and Exchange 2007 - imap-fetch-safe

[Dave Love]

[-- Attachment #1: Type: text/plain, Size: 1795 bytes --]

Simon Josefsson <simon@josefsson.org> writes:

> Maybe the problem is a condition-case within another condition-case
> work?  I recall problems related to this when run as an async process
> filter.

Yes -- signals from process filters are normally caught; I wasn't
thinking or looking closely enough at what the code does originally.  I
must have had debug-on-error set when testing, but I was sure I'd
actually tried it in fresh Emacs.

> Maybe it is possible to re-write the approach without using
> condition-case, that would likely be easier to debug anyway.

I'm not sure it would be easier, but see the comment in the patch below.
I made it before reading this, and at least the fix over my previous
code is just an extra binding.  It works for me in a fresh Emacs, and
isn't broken in Emacs 22.

> I'd prefer to avoid sending the Exchange bug-workaround approach
> ("1,*:*") to any server that does not need it.  I've seen servers that
> (internally) open up all e-mails in the folder and searches them, but
> for the 1,* approach was able to return data quickly.

Obviously that's a good reason.  Previous comments about efficiency that
I was referred to seemed to be about something different.

> This may be old
> information now, but generally I don't see why imap.el should send poor
> protocol output to all servers just because Exchange is broken.

[It does have workarounds for various other servers, not that I want to
defend Exchange in any way.  Exchange 2007 is doing horrible things like
messing with MIME parts, which I don't think the previous version did,
and I wish I could avoid it.]

Anyhow, per the comment in the patch, is there a good reason -- other
than simplicity? -- to use FETCH rather than UID like other clients?
I'm speaking mostly in ignorance of IMAP...


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 3149 bytes --]

2009-01-17  Dave Love  <fx@gnu.org>

	* imap.el (imap-fetch-safe): Bind debug-on-error.
	(imap-debug): Add imap-fetch-safe.

Index: imap.el
===================================================================
RCS file: /usr/local/cvsroot/gnus/lisp/imap.el,v
retrieving revision 7.52
diff -u -r7.52 imap.el
--- imap.el	8 Jan 2009 20:51:29 -0000	7.52
+++ imap.el	17 Jan 2009 19:33:31 -0000
@@ -1798,25 +1800,38 @@
 of the UIDS specification, and the cdr is the one which works with
 Exchange 2007 or, potentially, other buggy servers.
 See `imap-enable-exchange-bug-workaround'."
-  ;; We don't unconditionally use the alternative (valid) form, since
-  ;; this is said to be significantly inefficient.  The first time we
-  ;; get here for a given, we'll try the canonical form.  If we get
-  ;; the known error from the buggy server, set the flag
-  ;; buffer-locally (to account for connections to multiple servers),
-  ;; then re-try with the alternative UIDS spec.
+  ;; The first time we get here for a given, we'll try the canonical
+  ;; form.  If we get the known error from the buggy server, set the
+  ;; flag buffer-locally (to account for connections to multiple
+  ;; servers), then re-try with the alternative UIDS spec.  We don't
+  ;; unconditionally use the alternative form, since the
+  ;; currently-used alternatives are seriously inefficient with some
+  ;; servers (although they are valid).
+  ;;
+  ;; FIXME:  Maybe it would be cleaner to have a flag to not signal
+  ;; the error (which otherwise gives a message), and test
+  ;; `imap-failed-tags'.  Also, Other IMAP clients use other forms of
+  ;; request which work with Exchange, e.g. Claws does "UID FETCH 1:*
+  ;; (UID)" rather than "FETCH UID 1,*".  Is there a good reason not
+  ;; to do the same?
   (condition-case data
-      (imap-fetch (if imap-enable-exchange-bug-workaround
-		      (cdr uids)
-		    (car uids))
-		  props receive nouidfetch buffer)
+      ;; Binding `debug-on-error' allows us to get the error from
+      ;; `imap-parse-response' -- it's normally caught by Emacs around
+      ;; execution of a process filter.
+      (let ((debug-on-error t))
+	(imap-fetch (if imap-enable-exchange-bug-workaround
+			(cdr uids)
+		      (car uids))
+		    props receive nouidfetch buffer))
     (error
      (if (and (not imap-enable-exchange-bug-workaround)
-	      (string-match
-	       "The specified message set is invalid"
-	       (cadr data)))
+	      ;; This is the Exchange 2007 response.  It may be more
+	      ;; robust just to check for a BAD response to the
+	      ;; attempted fetch.
+	      (string-match "The specified message set is invalid"
+			    (cadr data)))
 	 (with-current-buffer (or buffer (current-buffer))
-	   (set (make-local-variable
-		 'imap-enable-exchange-bug-workaround)
+	   (set (make-local-variable 'imap-enable-exchange-bug-workaround)
 		t)
 	   (imap-fetch (cdr uids) props receive nouidfetch))
        (signal (car data) (cdr data))))))
@@ -3023,6 +3038,7 @@
 	  imap-list-to-message-set
 	  imap-fetch-asynch
 	  imap-fetch
+	  imap-fetch-safe
 	  imap-message-put
 	  imap-message-get
 	  imap-message-map

Re: IMAP and Exchange 2007 - imap-fetch-safe

[Dave Love]

Bjorn Solberg <bjorn_ding2@hekneby.org> writes:

> and everything still works fine, albeit slow.  Fetching new mail and
> using nnimap-split-inbox '("INBOX") and nnimap-split-rule processes
> about 2s/header or 1/2 header/sec.

It's not obviously slow for me (on a fast machine, these days), but I
haven't tried to time it against, say, Evolution.  I think there is too
much consing going on, but not necessarily in the IMAP code.  M-x
elp-instrument-package should reveal any bottleneck.

Re: IMAP and Exchange 2007 - imap-fetch-safe

[Reiner Steib]

On Sat, Jan 17 2009, Dave Love wrote:

> Simon Josefsson <simon@josefsson.org> writes:
>
>> Maybe the problem is a condition-case within another condition-case
>> work?  I recall problems related to this when run as an async process
>> filter.
>
> Yes -- signals from process filters are normally caught; I wasn't
> thinking or looking closely enough at what the code does originally.  I
> must have had debug-on-error set when testing, but I was sure I'd
> actually tried it in fresh Emacs.
>
>> Maybe it is possible to re-write the approach without using
>> condition-case, that would likely be easier to debug anyway.
>
> I'm not sure it would be easier, but see the comment in the patch below.
> I made it before reading this, and at least the fix over my previous
> code is just an extra binding.  It works for me in a fresh Emacs, and
> isn't broken in Emacs 22.
>
>> I'd prefer to avoid sending the Exchange bug-workaround approach
>> ("1,*:*") to any server that does not need it.  I've seen servers that
>> (internally) open up all e-mails in the folder and searches them, but
>> for the 1,* approach was able to return data quickly.
>
> Obviously that's a good reason.  Previous comments about efficiency that
> I was referred to seemed to be about something different.
>
>> This may be old information now, but generally I don't see why
>> imap.el should send poor protocol output to all servers just
>> because Exchange is broken.
>
> [It does have workarounds for various other servers, not that I want to
> defend Exchange in any way.  Exchange 2007 is doing horrible things like
> messing with MIME parts, which I don't think the previous version did,
> and I wish I could avoid it.]
>
> Anyhow, per the comment in the patch, 

+  ;; FIXME:  Maybe it would be cleaner to have a flag to not signal
+  ;; the error (which otherwise gives a message), and test
+  ;; `imap-failed-tags'.  Also, Other IMAP clients use other forms of
+  ;; request which work with Exchange, e.g. Claws does "UID FETCH 1:*
+  ;; (UID)" rather than "FETCH UID 1,*".  Is there a good reason not
+  ;; to do the same?

> is there a good reason -- other than simplicity? -- to use FETCH
> rather than UID like other clients?  I'm speaking mostly in
> ignorance of IMAP...

Comments?  (Simon?)

> 2009-01-17  Dave Love  <fx@gnu.org>
>
> 	* imap.el (imap-fetch-safe): Bind debug-on-error.
> 	(imap-debug): Add imap-fetch-safe.

Installed.

Bjorn (and others), does the auto-detection work for you with the
current version from CVS?

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: IMAP and Exchange 2007 - imap-fetch-safe

[Bjorn Solberg]

Reiner Steib writes:

> On Sat, Jan 17 2009, Dave Love wrote:
>> Simon Josefsson <simon@josefsson.org> writes:
>> 
>>> Maybe the problem is a condition-case within another condition-case
>>> work?  I recall problems related to this when run as an async process
>>> filter.
>> 
>> Yes -- signals from process filters are normally caught; I wasn't
>> thinking or looking closely enough at what the code does originally.  I
>> must have had debug-on-error set when testing, but I was sure I'd
>> actually tried it in fresh Emacs.
>> 
>>> Maybe it is possible to re-write the approach without using
>>> condition-case, that would likely be easier to debug anyway.
>> 
>> I'm not sure it would be easier, but see the comment in the patch below.
>> I made it before reading this, and at least the fix over my previous
>> code is just an extra binding.  It works for me in a fresh Emacs, and
>> isn't broken in Emacs 22.
>> 
>>> I'd prefer to avoid sending the Exchange bug-workaround approach
>>> ("1,*:*") to any server that does not need it.  I've seen servers that
>>> (internally) open up all e-mails in the folder and searches them, but
>>> for the 1,* approach was able to return data quickly.
>> 
>> Obviously that's a good reason.  Previous comments about efficiency that
>> I was referred to seemed to be about something different.
>> 
>>> This may be old information now, but generally I don't see why
>>> imap.el should send poor protocol output to all servers just
>>> because Exchange is broken.
>> 
>> [It does have workarounds for various other servers, not that I want to
>> defend Exchange in any way.  Exchange 2007 is doing horrible things like
>> messing with MIME parts, which I don't think the previous version did,
>> and I wish I could avoid it.]
>> 
>> Anyhow, per the comment in the patch, 

> +  ;; FIXME:  Maybe it would be cleaner to have a flag to not signal
> +  ;; the error (which otherwise gives a message), and test
> +  ;; `imap-failed-tags'.  Also, Other IMAP clients use other forms of
> +  ;; request which work with Exchange, e.g. Claws does "UID FETCH 1:*
> +  ;; (UID)" rather than "FETCH UID 1,*".  Is there a good reason not
> +  ;; to do the same?

>> is there a good reason -- other than simplicity? -- to use FETCH
>> rather than UID like other clients?  I'm speaking mostly in
>> ignorance of IMAP...

> Comments?  (Simon?)

>> 2009-01-17  Dave Love  <fx@gnu.org>
>> 
>> * imap.el (imap-fetch-safe): Bind debug-on-error.
>> (imap-debug): Add imap-fetch-safe.

> Installed.

> Bjorn (and others), does the auto-detection work for you with the
> current version from CVS?

I just did a "cvs update" in trunk.  First off, I get the following
message

byte-code: Error in ~/.gnus: Please use `smtpmail.el' bundled with Emacs.

as smtpmail.el got restored in the contrib directory when I did the cvs
update.

But other than that, Gnus now starts up fine with no need to set
imap-enable-exchange-bug-workaround to t.

Thanks,

Bjorn.

Re: IMAP and Exchange 2007 - imap-fetch-safe

[Reiner Steib]

On Sun, Feb 01 2009, Bjorn Solberg wrote:

> Reiner Steib writes:
>> Bjorn 

your From address bounced.

>> (and others), does the auto-detection work for you with the current
>> version from CVS?
>
> I just did a "cvs update" in trunk.  First off, I get the following
> message
>
> byte-code: Error in ~/.gnus: Please use `smtpmail.el' bundled with Emacs.
>
> as smtpmail.el got restored in the contrib directory when I did the cvs
> update.

,----[ contrib/README ]
| Please note that it is NOT good to just add this directory to
| `load-path': a number of files in this directory will become part of
| more recent Emacs versions, so that you might be running obsolete
| libraries with all kinds of ill effects
| (cf. `list-load-path-shadows').
| 
| The suggested method for installation is to copy those files that
| you need to a directory which is in `load-path'.
`----

> But other than that, Gnus now starts up fine with no need to set
> imap-enable-exchange-bug-workaround to t.

Good to hear.

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/