bug#34177: smtpmail.el aborts on transient errors

bug#34177: smtpmail.el aborts on transient errors

[Brian Sniffen]

smtpmail.el treats all return codes over 400 as errors, and aborts.  But RFC 821 and successors call the 400-series “transient” errors and suggest retrying immediately.  Some mail servers, including current MS Exchange, use 400-series errors for load limiting.  The user can just repeat C-c C-c until the mail goes through... or Emacs SMTPmail could loop until the server accepts it.

Retrying is a little tricky in the case of multipart commands like LOGIN, but in general it’s safe to loop for seconds on any command.

-- 
Brian Sniffen

bug#34177: smtpmail.el aborts on transient errors

[Lars Ingebrigtsen]

[-- Attachment #1: Type: text/plain, Size: 1937 bytes --]

Brian Sniffen <bts@evenmere.org> writes:

> smtpmail.el treats all return codes over 400 as errors, and aborts.
> But RFC 821 and successors call the 400-series “transient” errors and
> suggest retrying immediately.  Some mail servers, including current MS
> Exchange, use 400-series errors for load limiting.  The user can just
> repeat C-c C-c until the mail goes through... or Emacs SMTPmail could
> loop until the server accepts it.
>
> Retrying is a little tricky in the case of multipart commands like
> LOGIN, but in general it’s safe to loop for seconds on any command.

This is what RFC 5321 says:

https://tools.ietf.org/html/rfc5321#section-4.2.1

4yz  Transient Negative Completion reply
      The command was not accepted, and the requested action did not
      occur.  However, the error condition is temporary, and the action
      may be requested again.  The sender should return to the beginning
      of the command sequence (if any).  It is difficult to assign a
      meaning to "transient" when two different sites (receiver- and
      sender-SMTP agents) must agree on the interpretation.  Each reply
      in this category might have a different time value, but the SMTP
      client SHOULD try again.  A rule of thumb to determine whether a
      reply fits into the 4yz or the 5yz category (see below) is that
      replies are 4yz if they can be successful if repeated without any
      change in command form or in properties of the sender or receiver
      (that is, the command is repeated identically and the receiver
      does not put up a new implementation).

So, indeed, smtpmail SHOULD try resending when given a 4xx response
code...  but presumably not forever, either.

It's a bit difficult to test, though.  Does the following patch work for
you?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: retry.patch --]
[-- Type: text/x-diff, Size: 1528 bytes --]

diff --git a/lisp/mail/smtpmail.el b/lisp/mail/smtpmail.el
index 741c4393e6..d9930491d9 100644
--- a/lisp/mail/smtpmail.el
+++ b/lisp/mail/smtpmail.el
@@ -658,10 +658,12 @@ smtpmail-user-mail-address
 	      user-mail-address))))
 
 (defun smtpmail-via-smtp (recipient smtpmail-text-buffer
-				    &optional ask-for-password)
+				    &optional ask-for-password
+                                    send-attempts)
   (unless smtpmail-smtp-server
     (smtpmail-query-smtp-server))
   (let ((process nil)
+        (send-attempts (or send-attempts 1))
 	(host (or smtpmail-smtp-server
 		  (error "`smtpmail-smtp-server' not defined")))
 	(port smtpmail-smtp-service)
@@ -817,6 +819,20 @@ smtpmail-via-smtp
 	       ((smtpmail-ok-p (setq result (smtpmail-read-response process)))
 		;; Success.
 		)
+               ((and (numberp (car result))
+                     (<= 400 (car result) 499)
+                     (< send-attempts 10))
+                ;; Retry on getting a transient 4xx code; see
+                ;; https://tools.ietf.org/html/rfc5321#section-4.2.1
+                (ignore-errors
+		  (smtpmail-send-command process "QUIT")
+		  (smtpmail-read-response process))
+		(delete-process process)
+		(setq process nil)
+		(throw 'done
+		       (smtpmail-via-smtp recipient smtpmail-text-buffer
+                                          ask-for-password
+                                          (1+ send-attempts))))
 	       ((and auth-mechanisms
 		     (not ask-for-password)
 		     (eq (car result) 530))

bug#34177: smtpmail.el aborts on transient errors

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Date: Tue, 09 Jul 2019 20:24:19 +0200
> Cc: 34177@debbugs.gnu.org
> 
> So, indeed, smtpmail SHOULD try resending when given a 4xx response
> code...  but presumably not forever, either.
> 
> It's a bit difficult to test, though.  Does the following patch work for
> you?

IMO, this should at least say in the echo area that it's retrying,
probably with the number of retries.  Bonus points for doing that from
a separate thread or a timer.  smtpmail locks up Emacs too much
already, when the network is busy.

bug#34177: smtpmail.el aborts on transient errors

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> IMO, this should at least say in the echo area that it's retrying,
> probably with the number of retries.

Yup.  And perhaps there should be a `(sleep-for 1)' in there...

> Bonus points for doing that from a separate thread or a timer.
> smtpmail locks up Emacs too much already, when the network is busy.

Yeah, it'd be very, very nice if the entire thing was asynchronous.  I
vaguely remember somebody talking about trying to make the entire thing
run from a separate thread (in Emacs versions with support for that)?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#34177: smtpmail.el aborts on transient errors

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> It's a bit difficult to test, though.  Does the following patch work for
> you?

I got no response, but I've tried to "manually" step through it, and the
patch seems to do its thing.  It's not asynchronous, though, but it
shouldn't make things worse for the normal case, and rewriting smtpmail
to be asynchronous is a major task.

So I applied the patch to Emacs 27.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#34177: smtpmail.el aborts on transient errors

[Brian Sniffen via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Oh!  I did write back a little while ago. Let me try now from a different address. It seemed to reduce the problem greatly.  Thank you! At a limit of 20 requests, it was gone. Can this be a customizable constant?

-- 
Brian Sniffen

> On Sep 16, 2019, at 5:49 PM, Lars Ingebrigtsen <larsi@gnus.org> wrote:
> 
> Lars Ingebrigtsen <larsi@gnus.org> writes:
> 
>> It's a bit difficult to test, though.  Does the following patch work for
>> you?
> 
> I got no response, but I've tried to "manually" step through it, and the
> patch seems to do its thing.  It's not asynchronous, though, but it
> shouldn't make things worse for the normal case, and rewriting smtpmail
> to be asynchronous is a major task.
> 
> So I applied the patch to Emacs 27.
> 
> -- 
> (domestic pets only, the antidote for overdose, milk.)
>   bloggy blog: http://lars.ingebrigtsen.no
> 

bug#34177: smtpmail.el aborts on transient errors

[Lars Ingebrigtsen]

Brian Sniffen <bts@me.com> writes:

> Oh!  I did write back a little while ago. Let me try now from a
> different address. It seemed to reduce the problem greatly.  Thank
> you! At a limit of 20 requests, it was gone. Can this be a
> customizable constant?

Sure; makes sense.

I've now added smtpmail-retries to the trunk.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#34177: smtpmail.el aborts on transient errors

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Date: Tue, 17 Sep 2019 01:27:12 +0200
> Cc: 34177@debbugs.gnu.org
> 
> I've now added smtpmail-retries to the trunk.

Please mention it in NEWS.

Thanks.

bug#34177: smtpmail.el aborts on transient errors

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Lars Ingebrigtsen <larsi@gnus.org>
>> Date: Tue, 17 Sep 2019 01:27:12 +0200
>> Cc: 34177@debbugs.gnu.org
>> 
>> I've now added smtpmail-retries to the trunk.
>
> Please mention it in NEWS.

I've done so already.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no