sendmail.el bug or expected behavior?

sendmail.el bug or expected behavior?

[Rob Browning]

Emacs doesn't always check for a non-zero exit status from sendmail,
which can lead to silent mail lossage.  For example:

 emacs21 -q
 [in the *scratch* buffer]
 (setq sendmail-program "/bin/false") C-j
 M-x vm-mail RET
 check that sendmail-program has right value with C-h v
 fill in a short test message
 C-c C-c
 emacs says `Sending...done'
 (no mail has been sent, of course)

Although from reading mail-interactive's description, I wasn't sure if
this behavior was a bug or intentional:

  mail-interactive's value is nil
  Documentation: *Non-nil means when sending a message wait for and
  display errors.  nil means let mailer mail back a message to report
  errors.

If this isn't just expected behavior, then someone suggested this as a
fix:

+++ emacs21-21.3+1/lisp/mail/sendmail.el
@@ -794,7 +794,7 @@
   (require 'mail-utils)
   (let ((errbuf (if mail-interactive
 		    (generate-new-buffer " sendmail errors")
-		  0))
+		  t))
 	(tembuf (generate-new-buffer " sendmail temp"))
 	(case-fold-search nil)
 	(coding (and (local-variable-p 'buffer-file-coding-system)
@@ -972,7 +972,7 @@
 		     (args 
 		      (append (list (point-min) (point-max)
 				    program
-				    nil errbuf nil "-oi")
+				    (not mail-interactive) errbuf nil "-oi")
 			      (and mail-specify-envelope-from
 				   (list "-f" (or mail-envelope-from
 						  user-mail-address)))

but I wasn't sure that this would be OK since it looks like it would
briefly insert, and then remove the sendmail error output from the
current buffer whenever mail-interactive is nil.

(Thanks to Ian Jackson for the example, and Matt Kraai for the
suggested fix.)

-- 
Rob Browning
rlb @defaultvalue.org and @debian.org; previously @cs.utexas.edu
GPG starting 2002-11-03 = 14DD 432F AE39 534D B592  F9A0 25C8 D377 8C7E 73A4

Re: sendmail.el bug or expected behavior?

[Simon Josefsson]

Rob Browning <rlb@defaultvalue.org> writes:

> Emacs doesn't always check for a non-zero exit status from sendmail,
> which can lead to silent mail lossage.  For example:
>
>  emacs21 -q
>  [in the *scratch* buffer]
>  (setq sendmail-program "/bin/false") C-j
>  M-x vm-mail RET
>  check that sendmail-program has right value with C-h v
>  fill in a short test message
>  C-c C-c
>  emacs says `Sending...done'
>  (no mail has been sent, of course)
>
> Although from reading mail-interactive's description, I wasn't sure if
> this behavior was a bug or intentional:
>
>   mail-interactive's value is nil
>   Documentation: *Non-nil means when sending a message wait for and
>   display errors.  nil means let mailer mail back a message to report
>   errors.

It is intentional.  I'm not sure it is a good idea, but it has been
discussed before.

> If this isn't just expected behavior, then someone suggested this as a
> fix:

I think the proper fix is to change the default value of
`mail-interactive' to t instead, but I'm not sure there is agreement
to do that.  If you don't like the default behaviour of silently
ignoring exit codes, you can customize the variable as well, no need
to patch the code.

IIRC, the motivation for this is that waiting for /usr/sbin/sendmail
to exit, and thus return an exit code, can take hours with some MTAs
if there is no network connectivity.

Re: sendmail.el bug or expected behavior?

[Richard Stallman]

When mail-interactive is nil, Emacs should not wait.

Many years ago I found that waiting for sendmail to finish took so
long it was intolerable.  Maybe that has changed; if so, we could
change the default value of mail-interactive to make Emacs wait
by default.

What experiences do people have?

Re: sendmail.el bug or expected behavior?

[Kevin Rodgers]

Richard Stallman wrote:

> When mail-interactive is nil, Emacs should not wait.


I think the variable name is confusing.  From the user's perspective,
sending mail interactively should not force him/her to wait.  So when
mail-interactive is is non-nil, Emacs should not wait; but when it is
nil, it should wait.  I.e. reverse the semantics of mail-interactive,
or change the name to something like sendmail-send-it-synchronously
(since it only applies to sendmail-send-it, not smtpmail-send-it or
anything else that might be used for send-mail-function).


> Many years ago I found that waiting for sendmail to finish took so
> long it was intolerable.  Maybe that has changed; if so, we could
> change the default value of mail-interactive to make Emacs wait
> by default.
> 
> What experiences do people have?

It's a lot easier to configure smtpmail.el (in site-start.el) than sendmail
itself (in sendmail.cf), and smtpmail-send-it seems fast to me.

-- 
Kevin Rodgers

Re: sendmail.el bug or expected behavior?

[Simon Josefsson]

Richard Stallman <rms@gnu.org> writes:

> When mail-interactive is nil, Emacs should not wait.
>
> Many years ago I found that waiting for sendmail to finish took so
> long it was intolerable.  Maybe that has changed; if so, we could
> change the default value of mail-interactive to make Emacs wait
> by default.
>
> What experiences do people have?

It has not changed.  I recall (someone please correct me if I
misremember) the XEmacs version of sendmail.el actually did change the
default, just because the current situation can lead to lost mail, but
it was quickly reverted because of the lockups.  This was just a few
years (1-2) ago.

Another solution may be to make sendmail.el more asynchronous, so it
can wait for sendmail to exit in the background, and then possibly
report the error code.  But this may not work well if
/usr/lib/sendmail hangs for hours, which appears to sometimes be the
case, because the user might quit emacs before sendmail finishes.

Re: sendmail.el bug or expected behavior?

[Richard Stallman]

      But this may not work well if
    /usr/lib/sendmail hangs for hours,

Are all mailers liable to hang for hours, or is it only Sendmail that
does so?  For instance, we use Exim.  Can Exim hang for hours when you
invoke it in this way to send a message?

Re: sendmail.el bug or expected behavior?

[Simon Josefsson]

Richard Stallman <rms@gnu.org> writes:

>       But this may not work well if
>     /usr/lib/sendmail hangs for hours,
>
> Are all mailers liable to hang for hours, or is it only Sendmail that
> does so?  For instance, we use Exim.  Can Exim hang for hours when you
> invoke it in this way to send a message?

I don't know.  As far as I remember, the reports were not very
detailed, just that the change of default behaviour generated
problems.  Perhaps someone can experiment a little, although I'm not
really sure what MTA and what configuration triggers the problem.

I suspect that if the MTA is trying to deliver the message to its
final destination, then it may hang regardless of what MTA is used.

Re: sendmail.el bug or expected behavior?

[Ian Jackson]

Simon Josefsson writes ("Re: sendmail.el bug or expected behavior?"):
> I suspect that if the MTA is trying to deliver the message to its
> final destination, then it may hang regardless of what MTA is used.

IMO the correct way for a MUA to invoke sendmail (on Debian) is
  /usr/sbin/sendmail -odb -oee ...
(To be portable to non-FHS systems, use /usr/lib/sendmail)

This is supposed to
 (a) queue the message, exiting non-0 if unsuccessful
 (b) fork off a detached delivery attempt, which will
     deliver the message, leave it queued, or bounce it,
     as appropriate
 (c) exit 0, leaving the delivery attempting going

The problem that RMS mentions happens if you don't give any -odX
options: usually that means you'll get `foreground' delivery, which
typically means that the MTA will try one delivery attempt now.

This is even more unhelpful when combined with -oem (also sometimes
the default), which (AIUI[1]) exits 0 on success or if the first
delivery attempt fails but the message is queued, but exits nonzero if
the delivery failed straight away even if the message was successfully
bounced.

Emacs needs, from the MTA, the behaviour I specify above for
-odb -oee.  Ie, exit status 0 from sendmail should mean the same as
`250 OK' does after SMTP DATA final dot.

If it is difficult to get this behaviour portably (which seems
unlikely nowadays although may have been much harder in times past)
then I think it's still worth doing.  In any case any other behaviour
by the MTA will always result in at least one or more of these
symptoms:

 * Excessive delay when sending mail interactive.
 * Silently vanishing mail.  (`Mail sent' but not actually sent.)
 * Interactive report of a problem sending mail (`Mail could
   not be sent') followed _also_ by a bounce.
 * Interactive report of a problem sending mail (`Mail could
   not be sent') but in fact the mail is also later delivered.

Is there some way to configure Emacs to use -odb -oee ?  If so we
(FSF, or failing that, Debian) should change the default.

If not all MTAs support -odb -oee (or use it to mean the same thing)
then for Debian we should patch them, and probably FSF Emacs should
have a configuration option to work with broken MTAs (in this case
passing sendmail other options may be helpful).

I know that Exim does support -odb -oee (and last time I looked the
Exim docs are reasonably good about these questions).  Exim 3.35
(which I'm running here) has a bug where at least one of
stdin/stdout/stderr isn't properly closed by the asynchronous child,
which can cause things to hang if you run sendmail via ssh or userv,
but hopefully Emacs can be made to stop waiting when sendmail has
exited, even if one of the pipes to it hasn't been closed at the other
end.

Smail interprets -oee to mean the same as -oem, but with background
delivery you probably won't get too many false nonzero exits.

The copy of the Sendmail Installation and Operation Guide I have to
hand here is unhelpfully vague about the exact behaviour of the
various -oeX options.  (Search the document for the string
`ErrorMode'.)

Postfix seems to ignore -oeX and -odX completely.  I think it's
probably modern enough to do the right thing.

Ian.
(submitter of the Debian bug #114849, and long-time free software
developer and user)

Re: sendmail.el bug or expected behavior?

[Jan D.]

> Simon Josefsson writes ("Re: sendmail.el bug or expected behavior?"):
>> I suspect that if the MTA is trying to deliver the message to its
>> final destination, then it may hang regardless of what MTA is used.
>
> IMO the correct way for a MUA to invoke sendmail (on Debian) is
>   /usr/sbin/sendmail -odb -oee ...
> (To be portable to non-FHS systems, use /usr/lib/sendmail)
>
> This is supposed to
>  (a) queue the message, exiting non-0 if unsuccessful
>  (b) fork off a detached delivery attempt, which will
>      deliver the message, leave it queued, or bounce it,
>      as appropriate
>  (c) exit 0, leaving the delivery attempting going
>
> The problem that RMS mentions happens if you don't give any -odX
> options: usually that means you'll get `foreground' delivery, which
> typically means that the MTA will try one delivery attempt now.

This may not do the right thing.  It is not uncommon to not run any 
sendmail
daemon, but just forward everything to another host.  If we do it like
this, the mail will never be delivered, but sit in the queue forever.

	Jan D.

Re: sendmail.el bug or expected behavior?

[Ian Jackson]

Jan D. writes ("Re: sendmail.el bug or expected behavior?"):
> This may not do the right thing.  It is not uncommon to not run any
> sendmail daemon, but just forward everything to another host.  If we
> do it like this, the mail will never be delivered, but sit in the
> queue forever.

There is no reliable way for Emacs to prevent /usr/lib/sendmail from
choosing to sit on the mail, if that's what it's configured by the
sysadmin to do.  Consider an installation whose mail hub is
temporarily down.

System administrators can avoid this kind of grief by installing (or
configuring) an implementation of /usr/lib/sendmail which always
immediately delivers to the mail hub (by SMTP or otherwise) and has no
queue.  Requiring this is not a problem because all mail-sending
programs on the client machine need the same behaviour.

Ian.

Re: sendmail.el bug or expected behavior?

[Jan D.]

> Jan D. writes ("Re: sendmail.el bug or expected behavior?"):
>> This may not do the right thing.  It is not uncommon to not run any
>> sendmail daemon, but just forward everything to another host.  If we
>> do it like this, the mail will never be delivered, but sit in the
>> queue forever.
>
> There is no reliable way for Emacs to prevent /usr/lib/sendmail from
> choosing to sit on the mail, if that's what it's configured by the
> sysadmin to do.  Consider an installation whose mail hub is
> temporarily down.

The point is if this is changed, there will be places where Emacs
previously did deliver mail, and now no longer does.  It is an
incompatible change, not to be made lightly IMHO.

	Jan D.

Re: sendmail.el bug or expected behavior?

[Ian Jackson]

Jan D. writes ("Re: sendmail.el bug or expected behavior?"):
> The point is if this is changed, there will be places where Emacs
> previously did deliver mail, and now no longer does.  It is an
> incompatible change, not to be made lightly IMHO.

Currently there are large numbers of places where Emacs sometimes
loses mail !

Ian.

Re: sendmail.el bug or expected behavior?

[Jan D.]

> Jan D. writes ("Re: sendmail.el bug or expected behavior?"):
>> The point is if this is changed, there will be places where Emacs
>> previously did deliver mail, and now no longer does.  It is an
>> incompatible change, not to be made lightly IMHO.
>
> Currently there are large numbers of places where Emacs sometimes
> loses mail !

But since delivery is made in the foreground, you notice when that 
happens.
Is it worth it to exchange this to a situation where mail is prehaps
very quietly not delivered?

	Jan D.

Re: sendmail.el bug or expected behavior?

[Ian Jackson]

Jan D. writes ("Re: sendmail.el bug or expected behavior?"):
> > Jan D. writes ("Re: sendmail.el bug or expected behavior?"):
> >> The point is if this is changed, there will be places where Emacs
> >> previously did deliver mail, and now no longer does.  It is an
> >> incompatible change, not to be made lightly IMHO.
> >
> > Currently there are large numbers of places where Emacs sometimes
> > loses mail !
> 
> But since delivery is made in the foreground, you notice when that
> happens.  Is it worth it to exchange this to a situation where mail
> is prehaps very quietly not delivered?

No, the situation I'm complaining about is precisely silent mail loss.
I think you're mistaken about the current default: it's to deliver in
the _background_, because Emacs effectively backgrounds
/usr/lib/sendmail.

Ian.

Re: sendmail.el bug or expected behavior?

[Jan D.]

>>> Currently there are large numbers of places where Emacs sometimes
>>> loses mail !
>>
>> But since delivery is made in the foreground, you notice when that
>> happens.  Is it worth it to exchange this to a situation where mail
>> is prehaps very quietly not delivered?
>
> No, the situation I'm complaining about is precisely silent mail loss.
> I think you're mistaken about the current default: it's to deliver in
> the _background_, because Emacs effectively backgrounds
> /usr/lib/sendmail.

Actually, I have mail-interactive set to t, I forgot about that.
What about setting timeouts to a lower value, wouldn't that help?

	Jan D.

Re: sendmail.el bug or expected behavior?

[Ian Jackson]

Jan D. writes ("Re: sendmail.el bug or expected behavior?"):
> Actually, I have mail-interactive set to t, I forgot about that.
> What about setting timeouts to a lower value, wouldn't that help?

Timeouts, what timeouts ?  Does Emacs have a timeout ?

Changing the timeouts in your MTA is probably not a good idea (except
perhaps as part of configuring it as a hub-based client - see my other
mail).

Ian.

Re: sendmail.el bug or expected behavior?

[Jan D.]

> Jan D. writes ("Re: sendmail.el bug or expected behavior?"):
>> Actually, I have mail-interactive set to t, I forgot about that.
>> What about setting timeouts to a lower value, wouldn't that help?
>
> Timeouts, what timeouts ?  Does Emacs have a timeout ?

No, but sendmail has.

>
> Changing the timeouts in your MTA is probably not a good idea (except
> perhaps as part of configuring it as a hub-based client - see my other
> mail).

Since the reason not to deliver in the foreground per default is that
it might hang, changing the MTA timeout for foreground delivery sounds
like a way forward.  At least you know when things fail.

	Jan D.

Re: sendmail.el bug or expected behavior?

[Ian Jackson]

Jan D. writes ("Re: sendmail.el bug or expected behavior?"):
> Since the reason not to deliver in the foreground per default is that
> it might hang, changing the MTA timeout for foreground delivery sounds
> like a way forward.  At least you know when things fail.

RFC1123 specifies minimum timeouts for SMTP delivery.  These are
far too long to make a user wait when sending mail.  If a receiving
site is as slow as permitted by the RFC, a single message delivery
will take tens of minutes.

So as I said,
> > Changing the timeouts in your MTA is probably not a good idea (except
> > perhaps as part of configuring it as a hub-based client - see my other
> > mail).

If your MTA is a hub-based client, and you're starting with a normal
MTA, then as well as changing the timeouts you have to disable its
queue (so that -odq and -odb do the same as -odi and so that all
temporary or partial SMTP errors are treated as fatal).  If you do
this then Emacs can safely do -odb -oee as I suggested.

If you don't do this then things will definitely go wrong some of the
time, no matter what Emacs does.

Ian.

Re: sendmail.el bug or expected behavior?

[Richard Stallman]

    Currently there are large numbers of places where Emacs sometimes
    loses mail !

If the mail gets lost, it is the mailer that loses it.
That is not Emacs's fault.
Emacs uses -oem -odb by default (when mail-interactive is nil)
and that says to start another process to deliver the mail.
-oem says to send the user mail to report any failure.
If the mailer obeys those options, you won't lose mail
without a failure report.

Some seem to be suggesting that when mail-interactive is nil, Emacs
could use -oem -odb, but even so wait for sendmail to terminate.  I
think that is what we used to do; but Emacs often had to wait for a
substantial time, and I found that intolerable.  Perhaps nowadays
the wait will be smaller and acceptable.  So people might want to
try this change and see if it is fast enough.

*** sendmail.el.~1.274.~	Sat Jan  3 09:10:43 2004
--- sendmail.el	Tue Jan 27 13:04:02 2004
***************
*** 820,826 ****
    (require 'mail-utils)
    (let ((errbuf (if mail-interactive
  		    (generate-new-buffer " sendmail errors")
! 		  0))
  	(tembuf (generate-new-buffer " sendmail temp"))
  	(multibyte enable-multibyte-characters)
  	(case-fold-search nil)
--- 820,826 ----
    (require 'mail-utils)
    (let ((errbuf (if mail-interactive
  		    (generate-new-buffer " sendmail errors")
! 		  nil0))
  	(tembuf (generate-new-buffer " sendmail temp"))
  	(multibyte enable-multibyte-characters)
  	(case-fold-search nil)

Re: sendmail.el bug or expected behavior?

[Stefan Monnier]

> If the mail gets lost, it is the mailer that loses it.
> That is not Emacs's fault.

Unless Emacs did not give the mailer the tools necessary to report the loss.

> Emacs uses -oem -odb by default (when mail-interactive is nil)
> and that says to start another process to deliver the mail.
> -oem says to send the user mail to report any failure.

If sendmail is unable to deliver *any* mail, then it won't be able to
report the failure either.

> If the mailer obeys those options, you won't lose mail
> without a failure report.

Ian suggested to use -oeq which hands the message to sendmail in the
foreground (with mail-interactive set to t) so sendmail gets a chance to
complain if it has no space in the queue or if there's no queue and no hub,
or if there's no queue and the hub is not accessible.  It should still be
"quick" in that sendmail should as much as possible stash the mail in
a queue and attempt actual delivery later.  If it is too slow, then setting
mail-interactive to nil and do as we do now is the next-best-thing.

> Some seem to be suggesting that when mail-interactive is nil, Emacs
> could use -oem -odb, but even so wait for sendmail to terminate.

My understanding is instead to set mail-interactive to t but change the
args passed to sendmail so it does not immediately attempt to deliver
the message.


        Stefan

Re: sendmail.el bug or expected behavior?

[Richard Stallman]

    Ian suggested to use -oeq which hands the message to sendmail in the
    foreground (with mail-interactive set to t) so sendmail gets a chance to
    complain if it has no space in the queue or if there's no queue and no hub,
    or if there's no queue and the hub is not accessible.

When mail-interactive is t, Emacs uses -oep.  That seems right.  The
intention of mail-interactive = t is that you should get to see errors
right away, as much as possible, and this is what does that.

If people find that there is reasonable performance waiting
for sendmail to finish with -oem and -odb for mail-interactive = nil,
we could conceivably try moving further to -oeq in that case.

Re: sendmail.el bug or expected behavior?

[Richard Stallman]

    IMO the correct way for a MUA to invoke sendmail (on Debian) is
      /usr/sbin/sendmail -odb -oee ...

Perhaps that means there should be a change in this code:

			      (if mail-interactive
				  ;; These mean "report errors to terminal"
				  ;; and "deliver interactively"
				  '("-oep" "-odi")

Perhaps mail-interactive would be a good default if those options
were different.  But I am not sure.

Re: sendmail.el bug or expected behavior?

[Richard Stallman]

    I suspect that if the MTA is trying to deliver the message to its
    final destination, then it may hang regardless of what MTA is used.

I would expect people to design an MTAs not to wait very long
in the process that you created to send the mail.  If sendmail
does this badly, does that mean they all do?

Re: sendmail.el bug or expected behavior?

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Date: Fri, 23 Jan 2004 13:25:57 -0500
> 
> I would expect people to design an MTAs not to wait very long
> in the process that you created to send the mail.

I'm not an expert, but I always thought that the amount of time they
wait is determined by the TCP/IP time-out, which is typically 1 or 2
minutes.  At least for those cases where the destination doesn't
respond.

Re: sendmail.el bug or expected behavior?

[Simon Josefsson]

Richard Stallman <rms@gnu.org> writes:

>     I suspect that if the MTA is trying to deliver the message to its
>     final destination, then it may hang regardless of what MTA is used.
>
> I would expect people to design an MTAs not to wait very long
> in the process that you created to send the mail.  If sendmail
> does this badly, does that mean they all do?

I guess not.  But how do MTAs implement this, if they don't wait in
the foreground?  Simply storing the e-mail in /var/spool/mqueue and
hope that the system is polling the queue every hour or so might not
be reliable.  This approach also has the disadvantage that any errors
that CAN be detected (and reported to the user) immediately, probably
won't be, since /usr/lib/sendmail will always just push the message on
the queue, without doing any error checking or simple delivery
attempts.

Just some thoughts.

Re: sendmail.el bug or expected behavior?

[Ian Jackson]

Simon Josefsson writes ("Re: sendmail.el bug or expected behavior?"):
> Richard Stallman <rms@gnu.org> writes:
> > I would expect people to design an MTAs not to wait very long
> > in the process that you created to send the mail.  If sendmail
> > does this badly, does that mean they all do?
> 
> I guess not.  But how do MTAs implement this, if they don't wait in
> the foreground?  Simply storing the e-mail in /var/spool/mqueue and
> hope that the system is polling the queue every hour or so might not
> be reliable.

When a user asks to send a mail, there is not time to do a delivery
attempt to remote systems.  Remote systems are allowed to be down
(TCP connection times), and they're allowed to make the SMTP client
wait for around 5 minutes _per SMTP response_, so a whole delivery can
(and sometimes does) take quite a while even if it's going to succeed.

If you think the system should try to do a delivery to another site on
the Internet when the user hits `send', what do you think it should do
when the delivery attempt takes longer than the user is reasonably
prepared to wait ?  (Personally I want mail sending to be nearly
instant - pratting about waiting for SMTP is what I have computers
for.)  MTAs are not typically designed to allow the delivery attempt
to be retrospectively made noninteractive, and the whole delivery
attempt might last longer than the lifetime of the MUA, so you can't
just remember you have it going and try to tell the user later.

The fact is that delivery to remote sites, queue management, retry,
timeouts, etc. are the _functions_ of an Internet MTA !  Without those
functions being implemented somewhere you can't send mail to the
Internet (or at least, not reliably).  Ie, without a queue, and a
queue runner (somewhere), your mail is already broken.

Now, traditionally, UN*X used to have the model that every host was
its own site.  So every host has to take responsibility for
maintaining a mail queue.  In this model you should install a
full-scale Internet MTA on your system, and when the user hits `send'
the local full MTA should accept responsibility for it - putting it in
the queue, and delivering/retrying as necessary.

Nowadays people often prefer to have simpler `client' systems, many of
which run some kind of UN*X.  In this situation you may indeed not
want to try to support and maintain a full MTA on such a client
machine; instead, you have your some simpler (or more simply
configured) software locally which attempts delivery to a mail hub,
and the mail hub performs the necessary queueing etc.  In this model
when the user hits `send' the local mail software should immediately
attempt to pass the message to the mail hub and report success only
if and when the mail hub says `OK'.  When the mail hub is down no-one
can send mail, and attempts to do so can hang unpleasantly.

Both of these are sensible models.  Moreover, they have _identical_
requirements for the functionality and semantics of /usr/lib/sendmail:
in all cases, the calling program _must_ wait for sendmail to exit and
check the exit status (since on a modern UN*X, and many older ones,
system resource problems can always cause failures).  When the local
/usr/lib/sendmail exits nonzero it has taken responsibility for the
delivery of the message.  What this means in the two models is
different: with the full MTA it means that the message has been
queued; with client-and-hub it means that the message has been
transferred to the mail hub (possibly by SMTP) and the mail hub has
accepted it.

What this shows is that this is a matter of _system configuration_,
not a matter of _MUA configuration_.  Emacs shouldn't need to know
which model is in use.

Now let us consider options to /usr/lib/sendmail.  The option set was
designed for the traditional full MTA model.  With queueless client
passing mail by SMTP to a hub, most of these options don't make any
sense.

The SMTP spec says that the hub should say `OK' as soon as it has
received the message, without performing a delivery attempt, and there
is no way for a client to request specific error behaviour; if
delivery fails later the hub is supposed to generate a bounce.  So an
SMTP client implicitly gets (from the hub) the behaviour equivalent to
-odq with a mixture of reporting error codes now or bouncing later
depending on when the hub detects the error.

>From the point of view of a MUA on the client, a MUA passing `-odq'
couldn't possibly mean putting the mail in the local queue, since
there isn't one.  If the system is supposed not to have a local queue,
but passing -odb or -odq overrides this and puts the message in it,
then this is a configuration problem.

>  This approach also has the disadvantage that any errors
> that CAN be detected (and reported to the user) immediately, probably
> won't be, since /usr/lib/sendmail will always just push the message on
> the queue, without doing any error checking or simple delivery
> attempts.

Errors which depend on information from the global Internet, or
dependent on the outcome of local delivery attempts, can't in general
be detected during the time which it's reasonable to wait when sending
mail.  Simple local addressing errors can easily be detected anyway
when the mail is queued.

Of course the current Emacs default is that any errors which are
detected by the MTA are not reported anywhere.  For example, if you
make a mistake while editing the headers of your mail, and commit a
syntax error, your mail will silently vanish (depending on local MTA
and configuration).

Ian.

Re: sendmail.el bug or expected behavior?

[Simon Josefsson]

I see much discussion, but few solutions.  Here is one idea (I can't
implement this now, but since I thought about the problem briefly, I
reckoned I should write it down somewhere):

The actual problem that users actually encounter seem to be when
`sendmail-program' points to a non-existing, a non-executable, or a
non-conforming (i.e., it rejects a command line parameter or exit
immediately because of invalid configuration file) application.

I think all of these cases can be detected, without negatively
affecting proper operation, by using start-process instead of
call-process with BUFFER=0.  The code would spawn the process, poll
the process for, e.g., 1 second, and if it is still running, continue.
If it returns with a fatal error code within 1 second, report that to
the user.  One potential problem would be that if sendmail is still
running when emacs is about to quit, then the process should not be
killed.  How can you achieve that?  There is
process-kill-without-query, but a process-forget-without-query would
be needed here.

E.g., replace

(setq foo (call-process ... 0 ...))

with

(setq foo (start-process ...))
(sit-for 1)
(unless (eq 0 (process-exit-status foo))
  (error "Invoking sendmail failed..."))

Re: sendmail.el bug or expected behavior?

[Stefan Monnier]

> I see much discussion, but few solutions.

I see Ian's part of the discussion as just saying "we should always use
mail-interactive and set sendmail's arguments to queuing".  Sounds like
a good solution to me.

Your message seems to be about something else, but it also leads to the
need to run sendmail not-in-the-background, so it goes in the
same direction, i.e.: run sendmail in the foreground (telling it to queue
the message) and if there's an error analyse it and report it clearly to
the user (typically tellnig it that the delivery failed or that he should
use smtpmail.el).


        Stefan

Re: sendmail.el bug or expected behavior?

[Simon Josefsson]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> I see much discussion, but few solutions.
>
> I see Ian's part of the discussion as just saying "we should always use
> mail-interactive and set sendmail's arguments to queuing".  Sounds like
> a good solution to me.

I missed the conclusion, sorry.  Does all MTAs support that option?
Doesn't this require that the system is running a queue poll every now
and then?  It isn't uncommon to see workstations that doesn't run
sendmail -q or the equivalent (Windows comes to mind, not that this
fact alone should influence us).  I'm worried that the change will
require environmental behaviour that emacs didn't require before, thus
potentially introducing new breakage.

> Your message seems to be about something else, but it also leads to the
> need to run sendmail not-in-the-background, so it goes in the
> same direction, i.e.: run sendmail in the foreground (telling it to queue
> the message) and if there's an error analyse it and report it clearly to
> the user (typically tellnig it that the delivery failed or that he should
> use smtpmail.el).

Yes.  The approaches are not mutually exclusive, and in fact probably
can be combined with some benefits.

Re: sendmail.el bug or expected behavior?

[Stefan Monnier]

>> I see Ian's part of the discussion as just saying "we should always use
>> mail-interactive and set sendmail's arguments to queuing".  Sounds like
>> a good solution to me.

> I missed the conclusion, sorry.  Does all MTAs support that option?
> Doesn't this require that the system is running a queue poll every now
> and then?

Reread what Ian wrote: setting the args to "put in the queue" does not mean
that it will necessarily put the mail in the queue: if there's no queue, it
will deliver immediately to the mail hub (i.e. do-the-right-thing).

Basically the arg just means "place this mail in some MTA queue somewhere,
and do it fast".  If you're running with a local daemon (i.e. a local
queue), it'll put it there, otherwise it'll put it in your mail-hub's queue.

This is how mailers are meant to be used, so if it doesn't work it's not an
Emacs problem but a local config problem (that will impact other MUAs as
well).


        Stefan

Re: sendmail.el bug or expected behavior?

[Richard Stallman]

    I think all of these cases can be detected, without negatively
    affecting proper operation, by using start-process instead of
    call-process with BUFFER=0.

I expect that would be slower in the usual case, and I don't want to
pay such a price.

      The code would spawn the process, poll
    the process for, e.g., 1 second, and if it is still running, continue.

    (setq foo (start-process ...))
    (sit-for 1)

A one-second delay is absolutely unacceptable.

If you can implement this method without any appreciable slowdown,
then I have no objection to switching to it.

Re: sendmail.el bug or expected behavior?

[Rob Browning]

Rob Browning <rlb@defaultvalue.org> writes:

> Emacs doesn't always check for a non-zero exit status from sendmail,
> which can lead to silent mail lossage.  For example:

Presuming I haven't misunderstood, where do we stand here?  From what
I gather everyone would be happy if we could do something that would
make emacs always wait for the MTA to quit (checking the exit code),
but only if that doesn't introduce "too much delay" when running on a
properly configured system.  (What would be "too much delay" in
situations where something wasn't actually wrong and was legitimately
causing the delay?)

Personally, I can't imagine it being OK to leave things with potential
for silent mail loss, especially not as the default, even if fixing it
might introduce a 5 minute delay.  If there is a 5 minute delay then
that just tells me that my MTA is broken (probably misconfigured), or
perhaps my hardware is bad, etc., and that I need to look in to it.
Of course even if I can't fix the delay, *and* I don't mind the extra
risk, I can always decide to set mail-interactive to nil.  This issue
could even be a FAQ.

At the very least, I'd submit that perhaps the documentation for
mail-interactive should be adjusted to provide more warning.
i.e.

  "Setting mail-interactive to nil may cause you to lose mail under
   any number of unpredictable conditions[1] without any warning.
   Emacs will *not* check to see if the mailer process failed."

([1] Where unpredictable conditions include but are not limited to
 your partition filling up, broken libraries during or left after an
 upgrade, mailer running out of memory, PATH altered improperly, etc.)

IMO if the common mailers (sendmail/postfix/exim/etc.) can be told to
queue the mail reasonably quickly these days, then I'd prefer to see
us figure out how to have emacs to do that by default while still
making it easy for people in special circumstances to arrange for
something faster, though perhaps less safe.

Also, I'm not sure it's relevant (or even feasible with emacs current
code), and I don't use any of these systems myself, but it seems like
a lot of other MUAs just queue up the outgoing mail themselves and
either send it via SMTP, either on-demand or in the background, and
they treat messages that haven't made it out yet in much the same way
that emacs treats unsaved buffers -- if you try to quit, they ask you
about them.  For people who have a properly configured
postfix/sendmail/exim/whatever queuing arrangement, this might not be
interesting, but for others, perhaps.

-- 
Rob Browning
rlb @defaultvalue.org and @debian.org; previously @cs.utexas.edu
GPG starting 2002-11-03 = 14DD 432F AE39 534D B592  F9A0 25C8 D377 8C7E 73A4

Re: sendmail.el bug or expected behavior?

[Stefan Monnier]

> properly configured system.  (What would be "too much delay" in
> situations where something wasn't actually wrong and was legitimately
> causing the delay?)

If sendmail is called with -oeq, it should only cause delay if there's
a problem such as "your hub is down or slow".  But if you access your hub
through a 56K connection and send a 50MB attachment, then you'll suffer
a serious legitimate delay (unless you use a local queue of course).

> Personally, I can't imagine it being OK to leave things with potential
> for silent mail loss, especially not as the default, even if fixing it
> might introduce a 5 minute delay.  If there is a 5 minute delay then
> that just tells me that my MTA is broken (probably misconfigured), or
> perhaps my hardware is bad, etc., and that I need to look in to it.
> Of course even if I can't fix the delay, *and* I don't mind the extra
> risk, I can always decide to set mail-interactive to nil.  This issue
> could even be a FAQ.

I agree.  But right now the mail-interactive setting of t implies a slower
delivery than -oeq (it might try to contact the destination which will
sooner or later be unavailable).  I.e. right now you can choose between
"fast and dangerous" or "very slow and safer".
I'd like the default to be inbetween (i.e. -oeq).


        Stefan

Re: sendmail.el bug or expected behavior?

[Stefan Monnier]

> Also, I'm not sure it's relevant (or even feasible with emacs current
> code), and I don't use any of these systems myself, but it seems like
> a lot of other MUAs just queue up the outgoing mail themselves and
> either send it via SMTP, either on-demand or in the background, and
> they treat messages that haven't made it out yet in much the same way
> that emacs treats unsaved buffers -- if you try to quit, they ask you
> about them.  For people who have a properly configured
> postfix/sendmail/exim/whatever queuing arrangement, this might not be
> interesting, but for others, perhaps.

It's not really relevant since this thread is about sendmail.el
(i.e. sending mail with an external MTA) whereas what you're suggesting is
to do the mail sending ourselves, which is what smtpmail.el does
(i.e. you're suggesting to add queue management to smtpmail.el).


        Stefan

Re: sendmail.el bug or expected behavior?

[David Kastrup]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> > Also, I'm not sure it's relevant (or even feasible with emacs current
> > code), and I don't use any of these systems myself, but it seems like
> > a lot of other MUAs just queue up the outgoing mail themselves and
> > either send it via SMTP, either on-demand or in the background, and
> > they treat messages that haven't made it out yet in much the same way
> > that emacs treats unsaved buffers -- if you try to quit, they ask you
> > about them.  For people who have a properly configured
> > postfix/sendmail/exim/whatever queuing arrangement, this might not be
> > interesting, but for others, perhaps.
> 
> It's not really relevant since this thread is about sendmail.el
> (i.e. sending mail with an external MTA) whereas what you're suggesting is
> to do the mail sending ourselves, which is what smtpmail.el does
> (i.e. you're suggesting to add queue management to smtpmail.el).

Like (setq smtpmail-queue-mail t) does?

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: sendmail.el bug or expected behavior?

[Stefan Monnier]

>> It's not really relevant since this thread is about sendmail.el
>> (i.e. sending mail with an external MTA) whereas what you're suggesting is
>> to do the mail sending ourselves, which is what smtpmail.el does
>> (i.e. you're suggesting to add queue management to smtpmail.el).

> Like (setq smtpmail-queue-mail t) does?

*blush*
Well, hmm, I guess so, yes.


        Stefan

Re: sendmail.el bug or expected behavior?

[Richard Stallman]

    Personally, I can't imagine it being OK to leave things with potential
    for silent mail loss, especially not as the default, even if fixing it
    might introduce a 5 minute delay.

I set up the code the way it is because it caused a delay of a few
seconds when everything was working properly.  I found that unacceptable.

Re: sendmail.el bug or expected behavior?

[Rob Browning]

Richard Stallman <rms@gnu.org> writes:

> I set up the code the way it is because it caused a delay of a few
> seconds when everything was working properly.  I found that unacceptable.

I suspect we just disagree about the default.  As I've mentioned, I'd
favor arranging things so that mail loss is less likely by default,
even if doing so might introduce a few scond delay.  Of course we'd
also need to provide good documentation for the alternative setting
for those systems and people for whom the delay is a problem.

However, I also got the impression from the current disussion that we
might have other alternatives now.  For example, are (most) MTAs
better about queueing now, and can emacs itself handle queueing, and
if either of these things is true, then can we arrange for
alternatives that satisfy (nearly) everyone now?

-- 
Rob Browning
rlb @defaultvalue.org and @debian.org; previously @cs.utexas.edu
GPG starting 2002-11-03 = 14DD 432F AE39 534D B592  F9A0 25C8 D377 8C7E 73A4