Problems with new smtpmail

Problems with new smtpmail

[Óscar Fuentes]

After building emacs with the sources from July 1, I can't send e-mails
anymore.

NEWS contain some info, but insufficient.

This is my setup:

(custom-set-variables
 ...
 '(message-send-mail-function (quote smtpmail-send-it))
 '(send-mail-function (quote smtpmail-send-it))
 '(smtpmail-default-smtp-server "smtp.server.net")
...
 )

(load-library "smtpmail")

(setq smtpmail-auth-credentials  ; or use ~/.authinfo
       '(("smtp.server.net" 25 "mylogin" "mypass")))

First, smtpmail-default-smtp-server is ignored. I have to set it with
setq before (load-library "smtpmail"). The documentation for
smtpmail-default-smtp-server says it is only effective if setted before
smtpmail is loaded, but it would be nice to mention that on NEWS too for
the benefit of those who used Emacs' standard customization interface
for setting the variable.

Second, NEWS says that smtpmail-auth-credentials no longer exists, but

".. if it were a direct list of user names and passwords, you will be
prompted for the user name and the password instead, and they will then
be saved to ~/.authinfo."

The above makes me think that smtpmail does something with
smtpmail-auth-credentials if it is assigned, which is contrary to my
experience, but maybe the questions are made on a stage that I didn't
reach, because the problem mentioned below.

So I go ahead and add

machine smtp.server.net login mylogin password mypass

to ~/.authinfo.

but sending mail fails with this:

smtpmail-send-it: Sending failed: Server supports STARTTLS, but Emacs
does not have support for it

Emacs worked fine for sending mail through that smtp server for
years. Now it seems that something is missing. What's the problem?

In any case, I find the information about the smtpmail change in NEWS
very sparse. Please note that most people is not an expert on mail
protocols and configured Emacs following recipes, possibly long time
ago. It is desirable to provide thorough documentation for updating the
most usual configurations to the new requirements.

Re: Problems with new smtpmail

[Lars Magne Ingebrigtsen]

Óscar Fuentes <ofv@wanadoo.es> writes:

>  '(smtpmail-default-smtp-server "smtp.server.net")

[...]

> First, smtpmail-default-smtp-server is ignored. I have to set it with
> setq before (load-library "smtpmail"). The documentation for
> smtpmail-default-smtp-server says it is only effective if setted before
> smtpmail is loaded, but it would be nice to mention that on NEWS too for
> the benefit of those who used Emacs' standard customization interface
> for setting the variable.

Yes.  You shouldn't use that variable.  It should probably be marked as
obsolete.  `smtpmail-smtp-server' is the variable that's used.

> Second, NEWS says that smtpmail-auth-credentials no longer exists

That's correct.

> ".. if it were a direct list of user names and passwords, you will be
> prompted for the user name and the password instead, and they will then
> be saved to ~/.authinfo."

That's badly phrased.  I meant to convey that it's ignored, and you will
be prompted for a user name and password, if the SMTP server asks for it.

> but sending mail fails with this:
>
> smtpmail-send-it: Sending failed: Server supports STARTTLS, but Emacs
> does not have support for it
>
> Emacs worked fine for sending mail through that smtp server for
> years. Now it seems that something is missing. What's the problem?

So your server announces that it supports STARTTLS, but doesn't require
it? 

> In any case, I find the information about the smtpmail change in NEWS
> very sparse. Please note that most people is not an expert on mail
> protocols and configured Emacs following recipes, possibly long time
> ago. It is desirable to provide thorough documentation for updating the
> most usual configurations to the new requirements.

You shouldn't need to know anything about SMTP, or edit any files, or
set any variables by hand.  smtpmail.el is meant to handle all that by
querying you for the stuff that it wants to know.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Problems with new smtpmail

[Lars Magne Ingebrigtsen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

>> smtpmail-send-it: Sending failed: Server supports STARTTLS, but Emacs
>> does not have support for it
>>
>> Emacs worked fine for sending mail through that smtp server for
>> years. Now it seems that something is missing. What's the problem?
>
> So your server announces that it supports STARTTLS, but doesn't require
> it? 

You have an SMTP server that accepts AUTH over a plain network
connection?  I didn't know any of those existed, but I've now checked in
a change to network-stream that should allow that combination of
features.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Problems with new smtpmail

[Óscar Fuentes]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

[snip]

>> but sending mail fails with this:
>>
>> smtpmail-send-it: Sending failed: Server supports STARTTLS, but Emacs
>> does not have support for it
>>
>> Emacs worked fine for sending mail through that smtp server for
>> years. Now it seems that something is missing. What's the problem?
>
> So your server announces that it supports STARTTLS, but doesn't require
> it? 

Dunno. In any case, if the server announces that it supports STARTTLS,
it is no reason for bailing out. Support for STARTTLS does not mean that
other authentication methods are unsupported. At least this server works
fine with user/pass auth.

And STARTTLS is not supported by Emacs (generic) or there was something
missing at configure time that makes *my* Emacs to lack STARTTLS? From
reading NEWS again, it may seem that STARTTLS is supported but not
properly configured on my setup (missing smtpmail-starttls-credentials,
perhaps?) If that's the case, the error message is misleading. Saying
something like "STARTTLS is not properly configured, see
smtpmail-starttls-credentials variable" would be more helpful.

BTW, smtpmail-starttls-credentials is not recognized by C-h v.

>> In any case, I find the information about the smtpmail change in NEWS
>> very sparse. Please note that most people is not an expert on mail
>> protocols and configured Emacs following recipes, possibly long time
>> ago. It is desirable to provide thorough documentation for updating the
>> most usual configurations to the new requirements.
>
> You shouldn't need to know anything about SMTP, or edit any files, or
> set any variables by hand.  smtpmail.el is meant to handle all that by
> querying you for the stuff that it wants to know.

That's fine if you start from a clean setup, but my short experience
demonstrates that if you have previous settings, one ends confused.

Maybe NEWS should suggest to remove all smtpmail settings from the
configuration files and let the package DTRT?

Re: Problems with new smtpmail

[Óscar Fuentes]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
>
>>> smtpmail-send-it: Sending failed: Server supports STARTTLS, but Emacs
>>> does not have support for it
>>>
>>> Emacs worked fine for sending mail through that smtp server for
>>> years. Now it seems that something is missing. What's the problem?
>>
>> So your server announces that it supports STARTTLS, but doesn't require
>> it? 
>
> You have an SMTP server that accepts AUTH over a plain network
> connection?  I didn't know any of those existed,

It's smtp.telefonica.net, in case you are curious.

> but I've now checked in a change to network-stream that should allow
> that combination of features.

Thanks! I'll test it ASAP.

Re: Problems with new smtpmail

[Lars Magne Ingebrigtsen]

Óscar Fuentes <ofv@wanadoo.es> writes:

> Dunno. In any case, if the server announces that it supports STARTTLS,
> it is no reason for bailing out. Support for STARTTLS does not mean that
> other authentication methods are unsupported. At least this server works
> fine with user/pass auth.

Ok.

> And STARTTLS is not supported by Emacs (generic) or there was something
> missing at configure time that makes *my* Emacs to lack STARTTLS?

Yes.  You compiled your Emacs without gnutls support, and you don't have
gnutls-cli installed, either.

> From reading NEWS again, it may seem that STARTTLS is supported but
> not properly configured on my setup (missing
> smtpmail-starttls-credentials, perhaps?)

That variable doesn't exist, so, no.  :-)

> If that's the case, the error message is misleading. Saying
> something like "STARTTLS is not properly configured, see
> smtpmail-starttls-credentials variable" would be more helpful.

No, the error message means what it says.

> That's fine if you start from a clean setup, but my short experience
> demonstrates that if you have previous settings, one ends confused.

I think the only thing your example showed was that smtpmail was
over-zealous in requiring STARTTLS if the server offered it...

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Problems with new smtpmail

[Óscar Fuentes]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> You have an SMTP server that accepts AUTH over a plain network
> connection?  I didn't know any of those existed, but I've now checked in
> a change to network-stream that should allow that combination of
> features.

It works. Thanks again.

Re: Problems with new smtpmail

[Lars Magne Ingebrigtsen]

Óscar Fuentes <ofv@wanadoo.es> writes:

> It's smtp.telefonica.net, in case you are curious.

Ah, right.  I see it announces both AUTH and STARTTLS, which is somewhat
unusual, I think.

Anyway, the fail-if-STARTTLS-isn't-possible-but-announced was only a
"fail early" kind of thing, where we wouldn't try to do a MAIL FROM, and
get an error message from that, if the server had announced STARTTLS.

Hm...  But!

If the server announces STARTTLS (but not AUTH), and we don't have
STARTTLS, and we do a MAIL FROM, and it says "nope", then we could error
out with "You need STARTTLS to talk to this server" instead "The server
doesn't let you talk".

It's only a case of giving the user a better error message -- it'll fail
anyway.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Problems with new smtpmail

[Óscar Fuentes]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

[snip]

>> And STARTTLS is not supported by Emacs (generic) or there was something
>> missing at configure time that makes *my* Emacs to lack STARTTLS?
>
> Yes.  You compiled your Emacs without gnutls support, and you don't have
> gnutls-cli installed, either.

Maybe another good candidate for a NEWS mention. There is no immediate
relation among gnutls and STARTTLS for me. It is the first time that I
see gnutls-cli mentioned. It is not in NEWS.

>> From reading NEWS again, it may seem that STARTTLS is supported but
>> not properly configured on my setup (missing
>> smtpmail-starttls-credentials, perhaps?)
> 
> That variable doesn't exist, so, no.  :-)

So why is it mentioned in NEWS?

[snip]

>> That's fine if you start from a clean setup, but my short experience
>> demonstrates that if you have previous settings, one ends confused.
>
> I think the only thing your example showed was that smtpmail was
> over-zealous in requiring STARTTLS if the server offered it...

That was one half of the problem. My previous setup is now not
functional because it used variables that are not obeyed by smtpmail as
it used to (smtpmail-default-smtp-server ignored when in
custom-set-variables, smtpmail-auth-credentials gone)

Re: Problems with new smtpmail

[Lars Magne Ingebrigtsen]

Óscar Fuentes <ofv@wanadoo.es> writes:

> Maybe another good candidate for a NEWS mention. There is no immediate
> relation among gnutls and STARTTLS for me. It is the first time that I
> see gnutls-cli mentioned. It is not in NEWS.

Ok, but I'm not sure what to say.

> So why is it mentioned in NEWS?

It's mentioned that it, like `smtpmail-auth-credentials', it no longer
exists.  I've now clarified further.

> That was one half of the problem. My previous setup is now not
> functional because it used variables that are not obeyed by smtpmail as
> it used to (smtpmail-default-smtp-server ignored when in
> custom-set-variables,

`smtpmail-default-smtp-server' was always ignored if you loaded smtpmail
before setting the variable.

> smtpmail-auth-credentials gone)

Yes, that's a change.  Which you would have been prompted for.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Problems with new smtpmail

[Eli Zaretskii]

> From: Lars Magne Ingebrigtsen <larsi@gnus.org>
> Date: Wed, 06 Jul 2011 18:52:16 +0200
> 
> Óscar Fuentes <ofv@wanadoo.es> writes:
> 
> > Maybe another good candidate for a NEWS mention. There is no immediate
> > relation among gnutls and STARTTLS for me. It is the first time that I
> > see gnutls-cli mentioned. It is not in NEWS.
> 
> Ok, but I'm not sure what to say.

Say that if the SMTP server supports STARTTLS, one needs to build
Emacs with gnutls support, and maybe cite the corresponding
configure-time option.

Re: Problems with new smtpmail

[Lars Magne Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Say that if the SMTP server supports STARTTLS, one needs to build
> Emacs with gnutls support, and maybe cite the corresponding
> configure-time option.

I've removed that restriction.  So the only thing to say now is that if
the SMTP server *requires* STARTTLS, and you don't have it, then you
have a problem, but that's not exactly news.  :-)

I think.  Unless I've missed something...

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Problems with new smtpmail

[Óscar Fuentes]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Óscar Fuentes <ofv@wanadoo.es> writes:
>
>> Maybe another good candidate for a NEWS mention. There is no immediate
>> relation among gnutls and STARTTLS for me. It is the first time that I
>> see gnutls-cli mentioned. It is not in NEWS.
>
> Ok, but I'm not sure what to say.

Maybe listing the requirements for STARTTLS to work? Something like "You
need to build Emacs with gnutls support and have gnutls-cli on your
PATH". Saying that the gnutls-cli is often contained in the gnutls-bin
package on GNU systems would also be nice :-)

>> So why is it mentioned in NEWS?
>
> It's mentioned that it, like `smtpmail-auth-credentials', it no longer
> exists.  I've now clarified further.

Ah, my English parser is buggy.

>> That was one half of the problem. My previous setup is now not
>> functional because it used variables that are not obeyed by smtpmail as
>> it used to (smtpmail-default-smtp-server ignored when in
>> custom-set-variables,
>
> `smtpmail-default-smtp-server' was always ignored if you loaded smtpmail
> before setting the variable.

Yes, but now it is ignored if listed on custom-set-variables even when
smtpmail is loaded later:

(custom-set-variables
  '(smtpmail-default-smtp-server "foo"))

(load "smtpmail")

fails here (smtpmail asks for a smtp server while sending email).

But

(custom-set-variables
  '(smtpmail-smtp-server "foo"))

(load "smtpmail")

works.

[snip]

Re: Problems with new smtpmail

[Lars Magne Ingebrigtsen]

Óscar Fuentes <ofv@wanadoo.es> writes:

> Yes, but now it is ignored if listed on custom-set-variables even when
> smtpmail is loaded later:
>
> (custom-set-variables
>   '(smtpmail-default-smtp-server "foo"))
>
> (load "smtpmail")
>
> fails here (smtpmail asks for a smtp server while sending email).

The definition of these two variables hasn't changed.  It's still:

(defvar smtpmail-default-smtp-server nil
  "Specify default SMTP server.
This only has effect if you specify it before loading the smtpmail library.")

(defcustom smtpmail-smtp-server
  (or (getenv "SMTPSERVER") smtpmail-default-smtp-server)
  "The name of the host running SMTP server."
  :type '(choice (const nil) string)
  :group 'smtpmail)

The only change I've done is to make `smtpmail-default-smtp-server' a
defvar.

(custom-set-variables
  '(smtpmail-default-smtp-server "foo"))

on a defvar-ed variable still sets it, doesn't it?

So I'm wondering whether you're somehow loading smtpmail.el before you
set `smtpmail-default-smtp-server'.

Or is there something else subtle that I'm missing about going from a
defcustom to a defvar?

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Problems with new smtpmail

[Óscar Fuentes]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Óscar Fuentes <ofv@wanadoo.es> writes:
>
>> Yes, but now it is ignored if listed on custom-set-variables even when
>> smtpmail is loaded later:
>>
>> (custom-set-variables
>>   '(smtpmail-default-smtp-server "foo"))
>>
>> (load "smtpmail")
>>
>> fails here (smtpmail asks for a smtp server while sending email).
>
> The definition of these two variables hasn't changed.  It's still:
>
> (defvar smtpmail-default-smtp-server nil
>   "Specify default SMTP server.
> This only has effect if you specify it before loading the smtpmail library.")
>
> (defcustom smtpmail-smtp-server
>   (or (getenv "SMTPSERVER") smtpmail-default-smtp-server)
>   "The name of the host running SMTP server."
>   :type '(choice (const nil) string)
>   :group 'smtpmail)
>
> The only change I've done is to make `smtpmail-default-smtp-server' a
> defvar.
>
> (custom-set-variables
>   '(smtpmail-default-smtp-server "foo"))
>
> on a defvar-ed variable still sets it, doesn't it?
>
> So I'm wondering whether you're somehow loading smtpmail.el before you
> set `smtpmail-default-smtp-server'.
>
> Or is there something else subtle that I'm missing about going from a
> defcustom to a defvar?

With .emacs containing

(custom-set-variables
  '(smtpmail-default-smtp-server "foo"))

starting Emacs and M-x eval-expression smtpmail-default-smtp-server

Debugger entered--Lisp error: (void-variable smtpmail-default-smtp-server)
...

but

(custom-set-variables
 '(smtpmail-default-smtp-server "smtp.telefonica.net" NOW))

works. So yes, there is something about custom-set-variables and
defvar/defcustom.

Re: Problems with new smtpmail

[Sam Steingold]

> * Lars Magne Ingebrigtsen <ynefv@tahf.bet> [2011-07-06 21:48:50 +0200]:
>
> (custom-set-variables
>   '(smtpmail-default-smtp-server "foo"))
>
> on a defvar-ed variable still sets it, doesn't it?

only if it is already set.
e.g.
(custom-set-variables '(foo 1))
keeps 'foo unbound.

-- 
Sam Steingold (http://sds.podval.org/) on CentOS release 5.6 (Final) X 11.0.60900031
http://iris.org.il http://openvotingconsortium.org http://mideasttruth.com
http://truepeace.org http://honestreporting.com
Don't use force -- get a bigger hammer.

Re: Problems with new smtpmail

[Sam Steingold]

> * Lars Magne Ingebrigtsen <ynefv@tahf.bet> [2011-07-06 17:03:03 +0200]:
>
>> Second, NEWS says that smtpmail-auth-credentials no longer exists
>
> That's correct.

This means that I can no longer set it to netrc-file and have to resort
to symlinking .authinfo to .netrc.

yuk!

-- 
Sam Steingold (http://sds.podval.org/) on CentOS release 5.6 (Final) X 11.0.60900031
http://jihadwatch.org http://truepeace.org http://pmw.org.il
http://dhimmi.com http://thereligionofpeace.com http://palestinefacts.org
You do not have time or money to sue anyone rich enough to be worth suing.

Re: Problems with new smtpmail

[Lars Magne Ingebrigtsen]

Sam Steingold <sds@gnu.org> writes:

> only if it is already set.
> e.g.
> (custom-set-variables '(foo 1))
> keeps 'foo unbound.

Ah, right.

I've now changed it back to a defcustom again.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Problems with new smtpmail

[Lars Magne Ingebrigtsen]

Sam Steingold <sds@gnu.org> writes:

> This means that I can no longer set it to netrc-file and have to resort
> to symlinking .authinfo to .netrc.

No, ~/.netrc is read by default by auth-source.  :-)

If you want to customise this, change `auth-sources'.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Problems with new smtpmail

[Sam Steingold]

> * Lars Magne Ingebrigtsen <ynefv@tahf.bet> [2011-07-06 23:34:56 +0200]:
>
> Sam Steingold <sds@gnu.org> writes:
>
>> only if it is already set.
>> e.g.
>> (custom-set-variables '(foo 1))
>> keeps 'foo unbound.
>
> Ah, right.
>
> I've now changed it back to a defcustom again.

honestly, I don't see any reason for the default variable given that the
actual variable is customizable.

-- 
Sam Steingold (http://sds.podval.org/) on CentOS release 5.6 (Final) X 11.0.60900031
http://palestinefacts.org http://openvotingconsortium.org http://iris.org.il
http://pmw.org.il http://www.memritv.org
We're too busy mopping the floor to turn off the faucet.

Re: Problems with new smtpmail

[Lars Magne Ingebrigtsen]

Sam Steingold <sds@gnu.org> writes:

> honestly, I don't see any reason for the default variable given that the
> actual variable is customizable.

Having two variables like that, where one inherits from the other, might
make sense in some cases (although at the moment, the sense of it
escapes my mind).

But backwards compatibility at this point is important, so if some
people have customised the -default- variable, for whatever reason, it
should probably be respected.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Problems with new smtpmail

[Eli Zaretskii]

> From: Lars Magne Ingebrigtsen <larsi@gnus.org>
> Cc: emacs-devel@gnu.org
> Date: Wed, 06 Jul 2011 20:08:03 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Say that if the SMTP server supports STARTTLS, one needs to build
> > Emacs with gnutls support, and maybe cite the corresponding
> > configure-time option.
> 
> I've removed that restriction.  So the only thing to say now is that if
> the SMTP server *requires* STARTTLS, and you don't have it, then you
> have a problem, but that's not exactly news.  :-)

Nevertheless, since Emacs can now be built with gnutls support, it
would be useful to mention that where you describe smtpmail changes.