Outgoing mail defaults

Outgoing mail defaults

[Lars Magne Ingebrigtsen]

I'm wonder whether the defaults when sending email should change.

I recently got a new laptop and installed Ubuntu on it.  I fired up
Emacs and tried sending mail, as one does, and it failed.

`message-send-mail-function' returned `message-send-mail-with-sendmail',
since Ubuntu does come with a functioning sendmail program, but it
refused to actually send any mail.

So I'm wondering whether Emacs should default to using smtpmail.  The
nice thing about using smtpmail is that Emacs has explicit control of
how the mail is sent, which is more important now than it used to be,
what with all the closed port 25s that ISP like to give their customers
these days.

I'm thinking:

1) change the defaults to use smtpmail

2) change smtpmail so that it queries the user for SMTP host name (and
possibly port number)

3) use customize to save the user's reply

In addition, smtpmail should be altered to use auth-source instead of
the slightly confusing use-both-auth-source-and-netrc setup it's using
now, and use `open-protocol-stream' to get opportunistic STARTTLS
upgrades.

Change 1) is something that will annoy people who have a perfectly good
local sendmail-like setup, so I'm not absolutely sure that this is the
way to go.  On the other hand, in the long run (as my Ubuntu experience
shows), I think it might be the right thing to do, and is what most
other modern MUAs (like Thunderbird) do.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Outgoing mail defaults

[Ted Zlatanov]

On Thu, 17 Mar 2011 18:04:39 +0100 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> 1) change the defaults to use smtpmail

Yes.  The landscape is definitely changed since the days when sendmail
was all you needed.

LMI> 2) change smtpmail so that it queries the user for SMTP host name (and
LMI> possibly port number)

LMI> 3) use customize to save the user's reply

LMI> In addition, smtpmail should be altered to use auth-source instead of
LMI> the slightly confusing use-both-auth-source-and-netrc setup it's using
LMI> now, and use `open-protocol-stream' to get opportunistic STARTTLS
LMI> upgrades.

smtpmail can get auth-source to do 1+2 better that Customize IMO through
`auth-source-creation-prompts' and `auth-source-creation-defaults'.

Perhaps auth-source itself should be smarter when running for the first
time, asking the user if he'd like to use the Secrets API, or (on Mac OS
X) the upcoming keychain support, or an unencrypted file on Windows, or
some other auth-source backend.  The default works for those who use
EPA/EPG and cache or agentize their passphrases, but otherwise it's
annoying.

Ted

Re: Outgoing mail defaults

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> smtpmail can get auth-source to do 1+2 better that Customize IMO through
> `auth-source-creation-prompts' and `auth-source-creation-defaults'.

Hm...  but we still have to set what the smtp host is somewhere?  Or
were you thinking that smtpmail could just ask auth-source for "give me
an smtp host from your backends", which would, for most people, be a
line from their ~/.authinfo file saying

machine smtp.gmail.com port smtp

?

That would work...  hm...

> Perhaps auth-source itself should be smarter when running for the first
> time, asking the user if he'd like to use the Secrets API, or (on Mac OS
> X) the upcoming keychain support, or an unencrypted file on Windows, or
> some other auth-source backend.  The default works for those who use
> EPA/EPG and cache or agentize their passphrases, but otherwise it's
> annoying.

Yeah, auth-source could be querying the user about this stuff the first
time it runs.  (And it could determine whether it's the first time just
by trying all the sources, and see whether any of them exist?)

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Outgoing mail defaults

[Ted Zlatanov]

On Thu, 17 Mar 2011 18:31:58 +0100 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Ted Zlatanov <tzz@lifelogs.com> writes:
>> smtpmail can get auth-source to do 1+2 better that Customize IMO through
>> `auth-source-creation-prompts' and `auth-source-creation-defaults'.

LMI> Hm...  but we still have to set what the smtp host is somewhere?  Or
LMI> were you thinking that smtpmail could just ask auth-source for "give me
LMI> an smtp host from your backends", which would, for most people, be a
LMI> line from their ~/.authinfo file saying

LMI> machine smtp.gmail.com port smtp

Right, it would be the first SMTP host found or none if they don't have
one.  I think that's the right thing either way for most new cases.

>> Perhaps auth-source itself should be smarter when running for the first
>> time, asking the user if he'd like to use the Secrets API, or (on Mac OS
>> X) the upcoming keychain support, or an unencrypted file on Windows, or
>> some other auth-source backend.  The default works for those who use
>> EPA/EPG and cache or agentize their passphrases, but otherwise it's
>> annoying.

LMI> Yeah, auth-source could be querying the user about this stuff the first
LMI> time it runs.  (And it could determine whether it's the first time just
LMI> by trying all the sources, and see whether any of them exist?)

Hmm, what a pain.  Yes, it makes sense to build the auth-sources more
intelligently.  It shouldn't stop the smtpmail.el changes you're
planning.  I'll put it on my TODO list.

Ted

Re: Outgoing mail defaults

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> LMI> machine smtp.gmail.com port smtp
>
> Right, it would be the first SMTP host found or none if they don't have
> one.  I think that's the right thing either way for most new cases.

The complicating factor here is that auth-source writes to
~/.authinfo.gpg by default, which probably isn't appropriate if you just
want to store the SMTP server name.  Hm.  Perhaps auth-source could
write to ~/.authinfo if there's nothing secret in the credentials? 

But then it'd have to move stuff from ~/.authinfo to ~/.authinfo.gpg if
the user later is prompted for a password from the SMTP server.

Sounds overly complicated.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Outgoing mail defaults

[Ted Zlatanov]

On Thu, 17 Mar 2011 19:33:22 +0100 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Ted Zlatanov <tzz@lifelogs.com> writes:
LMI> machine smtp.gmail.com port smtp
>> 
>> Right, it would be the first SMTP host found or none if they don't have
>> one.  I think that's the right thing either way for most new cases.

LMI> The complicating factor here is that auth-source writes to
LMI> ~/.authinfo.gpg by default, which probably isn't appropriate if you just
LMI> want to store the SMTP server name.  Hm.  Perhaps auth-source could
LMI> write to ~/.authinfo if there's nothing secret in the credentials? 

LMI> But then it'd have to move stuff from ~/.authinfo to ~/.authinfo.gpg if
LMI> the user later is prompted for a password from the SMTP server.

Why isn't it appropriate to encrypt authentication tokens and connection
parameters by default?  All we need to "bootstrap" `auth-source-search'
is the port in this case and we can add the server to the query if the
user specifies it:

(let* ((auth-source-creation-prompts
        '((user  . "SMTP user at %h: ")
          (host . "SMTP host: ")
          (port . "SMTP port (25 or 993 or smtp typically): ")
          (secret . "SMTP password for %u@%h: ")))
       (search '(:max 1 :create t
                      :require '(:user :host :secret :foo)
                      :port '("smtp" "25" "993")))
       (search (if smtpmail-server
                   (append search (list :host smtpmail-server))
                 search))
       (cred (nth 0 (apply 'auth-source-search search)))
       (saver (plist-get cred :save-function))
       (host (plist-get cred :host))
       (user (plist-get cred :user))
       (secret (plist-get cred :secret))
       (secret (if (functionp secret)
                   (funcall secret)
                 secret)))
  (when (debug user host secret) ; do connection here
    (funcall saver))) ; save result if the entry worked (does nothing if the entry is not newly created)

LMI> Sounds overly complicated.  :-)

Compared to doing the same setup in Customize it's not much more,
especially since choosing the auth-source backend is a one-time task.

But you can prepend the step where we configure smtpmail-server by some
external means, like a defcustom.

On Thu, 17 Mar 2011 15:02:08 -0400 David Reitter <david.reitter@gmail.com> wrote: 

DR> On Mar 17, 2011, at 1:43 PM, Lars Magne Ingebrigtsen wrote:

>> And the open SMTP server could also listen on an alternative port to
>> avoid firewall issues slightly more.

DR> Making it listen on port 80 would be easier than installing a
DR> server-side script taking a POST request on port 80.

Most firewalls will do packet inspection so this is not as useful as it
used to be.  It sucks but it's what people consider "normal" security I
guess...

DR> OTOH, Ted's suggestion about the hashcash seems very attractive,
DR> especially if the SMTP server would forward bug reports to other
DR> packages in the process.

Yay! :)

Ted

Re: Outgoing mail defaults

[David Reitter]

On Mar 17, 2011, at 1:04 PM, Lars Magne Ingebrigtsen wrote:
> 
> So I'm wondering whether Emacs should default to using smtpmail.  The
> nice thing about using smtpmail is that Emacs has explicit control of
> how the mail is sent, which is more important now than it used to be,
> what with all the closed port 25s that ISP like to give their customers
> these days.

> 2) change smtpmail so that it queries the user for SMTP host name (and
> possibly port number)

Yes.  
In addition, you'll want to ask for the Sender name and address.

The `mailclient' package was written to address this problem, but the resulting behavior is not ideal as it passes the message to another mail client, which usually displays it before it is sent.  `Mailclient' is used on OS X as a default.

It may be sensible to do what you say, but use a different configuration, by default, for the bug reporter.   When no SMTP server is configured, I would suggest that the bug reporter attempts to go through a server-side script running on Savannah that files the bug report (or sends the message to a specific address).  One could also configure an open SMTP server that only accepts e-mail to the specific bug reporting addresses, which is then used as a default for bug reporting.

Otherwise, non-configuration will make bug-reporting cumbersome and keep some users from reporting their bugs.

Re: Outgoing mail defaults

[Lars Magne Ingebrigtsen]

David Reitter <david.reitter@gmail.com> writes:

> One could also configure an open SMTP server that only accepts e-mail
> to the specific bug reporting addresses, which is then used as a
> default for bug reporting.

Many ISPs block port 25 for outgoing traffic, so it's probably better
not to rely on that working.  But it could be used as a fallback if
smtpmail hasn't been configured, or if it fails when sending the bug
report, perhaps?

And the open SMTP server could also listen on an alternative port to
avoid firewall issues slightly more.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Outgoing mail defaults

[Ted Zlatanov]

On Thu, 17 Mar 2011 18:43:45 +0100 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> David Reitter <david.reitter@gmail.com> writes:
>> One could also configure an open SMTP server that only accepts e-mail
>> to the specific bug reporting addresses, which is then used as a
>> default for bug reporting.

LMI> Many ISPs block port 25 for outgoing traffic, so it's probably better
LMI> not to rely on that working.  But it could be used as a fallback if
LMI> smtpmail hasn't been configured, or if it fails when sending the bug
LMI> report, perhaps?

LMI> And the open SMTP server could also listen on an alternative port to
LMI> avoid firewall issues slightly more.

It's got to be a web form.  Nowadays most people are firewalled off for
outbound mail on any port.  I'd use hashcash to verify the sender's good
intentions and to ensure the form submission is from an Emacs client.
This would benefit Emacs in general...

Ted

Re: Outgoing mail defaults

[John Sullivan]

Ted Zlatanov <tzz@lifelogs.com> writes:

> On Thu, 17 Mar 2011 18:43:45 +0100 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 
>
> LMI> David Reitter <david.reitter@gmail.com> writes:
>>> One could also configure an open SMTP server that only accepts e-mail
>>> to the specific bug reporting addresses, which is then used as a
>>> default for bug reporting.
>
> LMI> Many ISPs block port 25 for outgoing traffic, so it's probably better
> LMI> not to rely on that working.  But it could be used as a fallback if
> LMI> smtpmail hasn't been configured, or if it fails when sending the bug
> LMI> report, perhaps?
>
> LMI> And the open SMTP server could also listen on an alternative port to
> LMI> avoid firewall issues slightly more.
>
> It's got to be a web form.  Nowadays most people are firewalled off for
> outbound mail on any port.  I'd use hashcash to verify the sender's good
> intentions and to ensure the form submission is from an Emacs client.
> This would benefit Emacs in general...

I switched to port 587 and haven't had a problem since, on any ISP or
hotel/mobile wifi. 

Re: Outgoing mail defaults

[David Reitter]

On Mar 17, 2011, at 1:43 PM, Lars Magne Ingebrigtsen wrote:

> And the open SMTP server could also listen on an alternative port to
> avoid firewall issues slightly more.

Making it listen on port 80 would be easier than installing a server-side script taking a POST request on port 80.
OTOH, Ted's suggestion about the hashcash seems very attractive, especially if the SMTP server would forward bug reports to other packages in the process.

Re: Outgoing mail defaults

[chad]

[-- Attachment #1: Type: text/plain, Size: 722 bytes --]

On Mar 17, 2011, at 3:02 PM, David Reitter wrote:
> 
> Making it listen on port 80 would be easier than installing a server-side script taking a POST request on port 80.
> OTOH, Ted's suggestion about the hashcash seems very attractive, especially if the SMTP server would forward bug reports to other packages in the process.

Do you think we can expect hashcash to reasonably work in the case were the user hasn't configured emacs, it's not working, and sending mail doesn't work? I don't know the details of the hashcash implementation, but a quick internet search suggest that emacs would need to successfully communicate with a subprocess to a binary that isn't likely to be installed.

Hope that helps,
*Chad

[-- Attachment #2: Type: text/html, Size: 1024 bytes --]

Re: Outgoing mail defaults

[Ted Zlatanov]

On Thu, 17 Mar 2011 18:27:06 -0400 chad <chadpbrown@gmail.com> wrote: 

c> On Mar 17, 2011, at 3:02 PM, David Reitter wrote:
>> 
>> Making it listen on port 80 would be easier than installing a
>> server-side script taking a POST request on port 80.  OTOH, Ted's
>> suggestion about the hashcash seems very attractive, especially if
>> the SMTP server would forward bug reports to other packages in the
>> process.

c> Do you think we can expect hashcash to reasonably work in the case
c> were the user hasn't configured emacs, it's not working, and sending
c> mail doesn't work? 

Yes, since it can be implemented in Emacs Lisp.  The idea of "proving
some work has been done" is what's important.  Plus it's not a
well-known algorithm so there's a low chance of random spammers
implementing it.

c> I don't know the details of the hashcash implementation, but a quick
c> internet search suggest that emacs would need to successfully
c> communicate with a subprocess to a binary that isn't likely to be
c> installed.

That's how hashcash.el (now removed from Gnus) operated IIRC.  But
there's no reason we need to do it that way.  See
http://www.gnosis.cx/download/gnosis/util/hashcash.py for an example
implementation.

Ted

Re: Outgoing mail defaults

[chad]

On Mar 17, 2011, at 10:38 PM, Ted Zlatanov wrote:
> 
> Yes, since it can be implemented in Emacs Lisp.

Excellent; this is the piece that I was missing. Thanks.

*Chad

Re: Outgoing mail defaults

[Ted Zlatanov]

On Mon, 21 Mar 2011 20:46:51 +0100 asjo@koldfront.dk (Adam Sjøgren) wrote: 

AS> On Thu, 17 Mar 2011 21:38:44 -0500, Ted wrote:
>> That's how hashcash.el (now removed from Gnus) operated IIRC.

AS> When was hashcash.el removed from Gnus?

AS>  * http://git.gnus.org/cgit/gnus.git/tree/lisp/hashcash.el ?

I remembered it incorrectly, sorry.  But the point was, we can't rely on
it because smtpmail.el should work on all platforms.  We need a pure
Lisp version.

Ted

Re: Outgoing mail defaults

[Stefan Monnier]

> 1) change the defaults to use smtpmail

Sadly, it does seem to be the case, yes.


        Stefan

Re: Outgoing mail defaults

[Chong Yidong]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> 1) change the defaults to use smtpmail
>
> 2) change smtpmail so that it queries the user for SMTP host name (and
> possibly port number)
>
> 3) use customize to save the user's reply

These sound like the right thing to do, indeed.  Anyone feel like
implementing it?

Re: Outgoing mail defaults

[Ted Zlatanov]

On Sat, 19 Mar 2011 15:11:45 -0400 Chong Yidong <cyd@stupidchicken.com> wrote: 

CY> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
>> 1) change the defaults to use smtpmail
>> 
>> 2) change smtpmail so that it queries the user for SMTP host name (and
>> possibly port number)
>> 
>> 3) use customize to save the user's reply

CY> These sound like the right thing to do, indeed.  Anyone feel like
CY> implementing it?

(CC-ed to the Gnus mailing list also)

I offered to implement it and gave example code of how auth-source could
be used to do it (so it would get and store the user name, password,
port number, and host name).  Can you look at that earlier in the
thread?  I can change it if you think it needs to be improved in any
way, especially as far as the prompts go.

Lars was concerned this was too complicated for the case where the user
only wants to give a host name and everything else is unnecessary.  In
that case it seems indeed that a simple defcustom is an easier solution,
but I'm concerned that next will come "oh can we just have the port
customizable too... and the user name... and the password... and several
sets of connection parameters depending on the location" and again we're
back where we were before auth-source: data in Emacs Lisp (defcustoms or
not), all over the place, unencrypted, with ad hoc syntax.  So instead
I'd like to make it possible to have the line

machine mysmtphost port smtp

in the authinfo.gpg file.  I would do this by changing the example code
to start with "does your SMTP host need authentication?" and if no, do
the simple setup; if yes, do the full setup.

The other idea I had was that some people just don't want an authinfo
file.  They want to store everything in Lisp.  So for those cases I can
make a "data" auth-source backend which reads/writes using variables
that are saved with the Customize facilities.  First-time auth-source
users can then get the choice between the Secrets API (most secure,
least portable); a authinfo file (fairly secure if EPA/EPG is available,
portable); or the "data" backend (least secure, most portable).

If you don't want to do this and would prefer to just do a defcustom I
can write that too, as Lars' proposal went originally.

Let me know what you think.  Thanks.
Ted

Re: Outgoing mail defaults

[Stefan Monnier]

> I'd like to make it possible to have the line
> machine mysmtphost port smtp
> in the authinfo.gpg file.  I would do this by changing the example code

I'm not sure I understand what you're suggesting: AFAIK auto-source
maintains a database of how to connect to a given host, not which host
to use for which kind of service.
Or are you suggesting the smtpmail should default to using "whichever
host name appears in .authinfo where the associated port is `smtp'"?


        Stefan

Re: Outgoing mail defaults

[Ted Zlatanov]

On Sat, 19 Mar 2011 23:06:25 -0400 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

>> I'd like to make it possible to have the line
>> machine mysmtphost port smtp
>> in the authinfo.gpg file.  I would do this by changing the example code

SM> I'm not sure I understand what you're suggesting: AFAIK auto-source
SM> maintains a database of how to connect to a given host, not which host
SM> to use for which kind of service.
SM> Or are you suggesting the smtpmail should default to using "whichever
SM> host name appears in .authinfo where the associated port is `smtp'"?

The first such host name, yes.  There are really two cases: SMTP with
and without passwords.  Because I think the former should try to use
auth-source I'm trying to make the latter use it as well.  I can
understand if it seems like overkill.

How about this approach for first-time setup if people don't like the
above, which avoids the "first found host" solution:

1) ask the user about the SMTP server name if `smtpmail-auth-source' is
nil (the default)

2) ask the user if the server connection info should be saved through
auth-source

3) if yes, do the auth-source creation prompts for user, port, and
password

4) use Customize to save `smtpmail-host', `smtpmail-port', and
`smtpmail-auth-source' to t or 'never according to (2).  From now on the
user will either just use `smtpmail-host' and `smtpmail-port' or call
`auth-source-search' with :host ,smtpmail-host and (if the port is not
nil) :port ,smtpmail-port

Ted

Re: Outgoing mail defaults

[Stefan Monnier]

> How about this approach for first-time setup if people don't like the
> above, which avoids the "first found host" solution:

Yes, I'm not completely happy about this "first found" solution.

> 1) ask the user about the SMTP server name if `smtpmail-auth-source' is
> nil (the default)

> 2) ask the user if the server connection info should be saved through
> auth-source

I guess such a question might sound unclear to an unsuspecting user;
but if you phrase it as "do you need to use authentication" or something
like that, it would be fine.

> 3) if yes, do the auth-source creation prompts for user, port, and
> password

> 4) use Customize to save `smtpmail-host', `smtpmail-port', and
> `smtpmail-auth-source' to t or 'never according to (2).  From now on the
> user will either just use `smtpmail-host' and `smtpmail-port' or call
> `auth-source-search' with :host ,smtpmail-host and (if the port is not
> nil) :port ,smtpmail-port

Sounds OK.


        Stefan

Re: Outgoing mail defaults

[Ted Zlatanov]

[-- Attachment #1: Type: text/plain, Size: 1812 bytes --]

On Mon, 21 Mar 2011 10:20:22 -0400 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

>> 1) ask the user about the SMTP server name if `smtpmail-auth-source' is
>> nil (the default)

>> 2) ask the user if the server connection info should be saved through
>> auth-source

SM> I guess such a question might sound unclear to an unsuspecting user;
SM> but if you phrase it as "do you need to use authentication" or something
SM> like that, it would be fine.

>> 3) if yes, do the auth-source creation prompts for user, port, and
>> password

>> 4) use Customize to save `smtpmail-host', `smtpmail-port', and
>> `smtpmail-auth-source' to t or 'never according to (2).  From now on the
>> user will either just use `smtpmail-host' and `smtpmail-port' or call
>> `auth-source-search' with :host ,smtpmail-host and (if the port is not
>> nil) :port ,smtpmail-port

This is implemented with `smtpmail-use-auth-source' set to 'ask by
default and then either t or nil as the user may desire.

`smtpmail-smtp-server' and `smtpmail-smtp-service' are obtained with
`read-string' and then we call `customize-save-variable' on them.  All
this happens in `smtpmail-via-smtp' and not `smtpmail-try-auth-methods'.

The :save-function is called on a successful auth exchange.

The patch is pretty simple overall.  You'll see that the creation
prompts are just for the user and the password, since the port and host
are guaranteed to be defined from earlier.

I'd also like to remove the netrc-* function calls if no one objects.
It would break people that expect ~/.authinfo and ~/.netrc to work.
Those are the usual defaults of `netrc-file' but we strongly encourage
~/.authinfo.gpg now, so I think the damage is justified as it may get
former netrc.el users to consider secure alternatives like EPA/EPG or
the Secrets API.

Ted


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: smtpmail-create-auth.patch --]
[-- Type: text/x-diff, Size: 4105 bytes --]

=== modified file 'lisp/mail/smtpmail.el'
--- lisp/mail/smtpmail.el	2011-02-12 17:51:02 +0000
+++ lisp/mail/smtpmail.el	2011-03-21 19:40:38 +0000
@@ -103,6 +103,13 @@
   :type '(choice (integer :tag "Port") (string :tag "Service"))
   :group 'smtpmail)
 
+(defcustom smtpmail-use-auth-source 'ask
+  "Whether `auth-sources' should be consulted for username and password."
+  :type '(choice (const :tag "Ask" ask)
+                 (const :tag "Don't use auth-source" nil)
+                 (const :tag "Use auth-source" t))
+  :group 'smtpmail)
+
 (defcustom smtpmail-local-domain nil
   "Local domain name without a host name.
 If the function `system-name' returns the full internet address,
@@ -480,6 +487,9 @@
 (defsubst smtpmail-cred-passwd (cred)
   (nth 3 cred))
 
+(defsubst smtpmail-cred-saver-function (cred)
+  (nth 4 cred))
+
 (defun smtpmail-find-credentials (cred server port)
   (catch 'done
     (let ((l cred) el)
@@ -536,18 +546,33 @@
 (declare-function password-cache-add "password-cache" (key password))
 
 (defun smtpmail-try-auth-methods (process supported-extensions host port)
+
+  ;; Find out if auth-source should be consulted
+  (when (eq smtpmail-use-auth-source 'ask)
+    (customize-save-variable
+     'smtpmail-use-auth-source
+     (y-or-n-p "Do you need authentication for SMTP? ")))
+
   (let* ((mechs (cdr-safe (assoc 'auth supported-extensions)))
 	 (mech (car (smtpmail-intersection mechs smtpmail-auth-supported)))
-         (auth-info (auth-source-search :max 1
-                                        :host host
-                                        :port (or port "smtp")))
-         (auth-user (plist-get (nth 0 auth-info) :user))
-         (auth-pass (plist-get (nth 0 auth-info) :secret))
+         (auth-source-creation-prompts
+          '((user . "SMTP user at %h: ")
+            (secret . "SMTP password for %u@%h: ")))
+         (auth-results (and smtpmail-use-auth-source
+                            (auth-source-search :max 1
+                                                :create t
+                                                :host host
+                                                :port (or port "smtp"))))
+         (auth-info (nth 0 auth-results))
+         (auth-user (plist-get auth-info :user))
+         (auth-pass (plist-get auth-info :secret))
          (auth-pass (if (functionp auth-pass)
                         (funcall auth-pass)
                       auth-pass))
+         (auth-saver (plist-get auth-info :save-function))
 	 (cred (if (and auth-user auth-pass) ; try user-auth-* before netrc-*
-		   (list host port auth-user auth-pass)
+                   ;; remember the :save-function for later
+		   (list host port auth-user auth-pass auth-saver)
 		 ;; else, if auth-source didn't return them...
 		 (if (stringp smtpmail-auth-credentials)
 		     (let* ((netrc (netrc-parse smtpmail-auth-credentials))
@@ -636,9 +661,11 @@
 
        (t
 	(error "Mechanism %s not implemented" mech)))
-      ;; Remember the password.
-      (when (null (smtpmail-cred-passwd cred))
-	(password-cache-add prompt passwd)))))
+      ;; Remember the password or call the auth-source :save-function.
+      (cond ((null (smtpmail-cred-passwd cred))
+             (password-cache-add prompt passwd))
+            ((functionp (smtpmail-cred-saver-function cred))
+             (funcall (smtpmail-cred-saver-function cred)))))))
 
 (defun smtpmail-via-smtp (recipient smtpmail-text-buffer)
   (let ((process nil)
@@ -655,6 +682,17 @@
 	greeting
 	process-buffer
 	(supported-extensions '()))
+    ;; If the host is nil, read host and port and save them in Customize.
+    (unless host
+      (setq host (customize-save-variable
+                  'smtpmail-smtp-server
+                  (read-string "SMTP server: ")))
+      (setq port (customize-save-variable
+                  'smtpmail-smtp-service
+                  (read-string
+                   "SMTP port number or service name: "
+                   nil nil (or port smtpmail-smtp-service)))))
+
     (unwind-protect
 	(catch 'done
 	  ;; get or create the trace buffer

Re: Outgoing mail defaults

[Stefan Monnier]

> I'd also like to remove the netrc-* function calls if no one objects.
> It would break people that expect ~/.authinfo and ~/.netrc to work.

I haven't looked at the patch yet, but I don't think this is OK, tho I'm
not sure exactly how bad this is.

> Those are the usual defaults of `netrc-file' but we strongly encourage
> ~/.authinfo.gpg now, so I think the damage is justified as it may get
> former netrc.el users to consider secure alternatives like EPA/EPG or
> the Secrets API.

Encouraging is fine, but preventing the use of an unencrypted file
doesn't sound right.


        Stefan

Re: Outgoing mail defaults

[Ted Zlatanov]

On Mon, 21 Mar 2011 18:14:56 -0400 Stefan Monnier <monnier@IRO.UMontreal.CA> wrote: 

>> I'd also like to remove the netrc-* function calls if no one objects.
>> It would break people that expect ~/.authinfo and ~/.netrc to work.

SM> I haven't looked at the patch yet, but I don't think this is OK, tho I'm
SM> not sure exactly how bad this is.

>> Those are the usual defaults of `netrc-file' but we strongly encourage
>> ~/.authinfo.gpg now, so I think the damage is justified as it may get
>> former netrc.el users to consider secure alternatives like EPA/EPG or
>> the Secrets API.

SM> Encouraging is fine, but preventing the use of an unencrypted file
SM> doesn't sound right.

How about testing (file-exists-p netrc-file) and if it exists, asking
the user if he'd like to put netrc-file at the head of `auth-sources'?

netrc.el is entirely replaced by auth-source.el at this point so I'd
really like to remove netrc-* calls whenever possible.

Ted

Re: Outgoing mail defaults

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> Lars was concerned this was too complicated for the case where the user
> only wants to give a host name and everything else is unnecessary.

Yes, I think that would be the common case.

Even though this means duplicating data, somewhat, I think we should
keep normal application defaults separate from secret credentials.

If the user has said that they want to use smtp.gmail.com as the
outgoing host, that's an application default, and not a credential.  If
the user then says "yes, and use this password", then we're in
credential territory, and we have to store the smtp.gmail.com token in
~/.authinfo{.gpg}, too.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Application resource storage (was: Outgoing mail defaults)

[Lars Magne Ingebrigtsen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Even though this means duplicating data, somewhat, I think we should
> keep normal application defaults separate from secret credentials.

Which reminds me of something we discussed half a year ago: Storing
responses to questions.

Today, if Emacs prompts the user for something, and it's something that
should be remembered always by Emacs, Emacs uses custom to save the
value.  This works well for things that map nicely to one single
variable.

But if the question is of the kind "do you want to use port 587 when
connecting to smtp.gmail.com?", then that doesn't really map
straightforwardly to a simple variable.  It might map to an alist of
some kind, but the user may set that alist somewhere else, which makes
all this kind of tricky.

Wouldn't it be nice if Emacs had a way to store these values in a
simpler manner without encroaching on the variable space?

That is, I'm kinda thinking of something like

(store-value "smtpmail-smtp-server-smtp.gmail.com-port" 587)

The first is a token uniquely identifying the resource, and the second
is the value.

These could be stored in ~/.emacs.d/resources, which could then be
queried by the applications.

Does this exist already?  Stupid idea?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Application resource storage

[Ted Zlatanov]

On Tue, 29 Mar 2011 21:34:05 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
>> Even though this means duplicating data, somewhat, I think we should
>> keep normal application defaults separate from secret credentials.

LMI> Which reminds me of something we discussed half a year ago: Storing
LMI> responses to questions.

LMI> Today, if Emacs prompts the user for something, and it's something that
LMI> should be remembered always by Emacs, Emacs uses custom to save the
LMI> value.  This works well for things that map nicely to one single
LMI> variable.

LMI> But if the question is of the kind "do you want to use port 587 when
LMI> connecting to smtp.gmail.com?", then that doesn't really map
LMI> straightforwardly to a simple variable.  It might map to an alist of
LMI> some kind, but the user may set that alist somewhere else, which makes
LMI> all this kind of tricky.

I think you're describing an assistant.el walkthrough.  Right now the
assistants store their return in an artificial data structure but the
walkthrough with the user decisions and answers could *be* the data
structure (it's a decision tree, right?)

Also assistant.el is currently a full-buffer application but it could be
implemented as a prompting interface as well to make these queries less
distracting.

LMI> Wouldn't it be nice if Emacs had a way to store these values in a
LMI> simpler manner without encroaching on the variable space?

LMI> That is, I'm kinda thinking of something like

LMI> (store-value "smtpmail-smtp-server-smtp.gmail.com-port" 587)

LMI> The first is a token uniquely identifying the resource, and the second
LMI> is the value.

LMI> These could be stored in ~/.emacs.d/resources, which could then be
LMI> queried by the applications.

LMI> Does this exist already?  Stupid idea?

It seems like an application-specific thing.  I would implement it as an
assistant.el facility to save the walkthrough and to query nodes in it.

As a global Emacs facility it would be easy to make all these strings
part of a big global-registry alist, which can then be customized and
saved independently.  But I think it would squash the decision tree from
assistant.el into unwieldy string representations of the nodes and edges.

Ted

Re: Application resource storage

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> I think you're describing an assistant.el walkthrough.  Right now the
> assistants store their return in an artificial data structure but the
> walkthrough with the user decisions and answers could *be* the data
> structure (it's a decision tree, right?)

Yes, that would be a use case.  There are other things that this could
be used for that I've been missing -- for instance, now when nnimap
connects to a server, it tries both the imap and the imaps port.  If
nnimap could stash what the successful port was, and try that first the
next time, the next startup would be slightly faster.

Not particularly essential, but there's a lot of stuff like that that
could be done trivially if there was a place to stash this data...

> As a global Emacs facility it would be easy to make all these strings
> part of a big global-registry alist, which can then be customized and
> saved independently.  But I think it would squash the decision tree from
> assistant.el into unwieldy string representations of the nodes and edges.

True.  But it's basically what Firefox does with its data, if you look
under about:config.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Application resource storage

[Ted Zlatanov]

On Tue, 29 Mar 2011 22:14:11 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Ted Zlatanov <tzz@lifelogs.com> writes:
>> I think you're describing an assistant.el walkthrough.  Right now the
>> assistants store their return in an artificial data structure but the
>> walkthrough with the user decisions and answers could *be* the data
>> structure (it's a decision tree, right?)

LMI> Yes, that would be a use case.  There are other things that this could
LMI> be used for that I've been missing -- for instance, now when nnimap
LMI> connects to a server, it tries both the imap and the imaps port.  If
LMI> nnimap could stash what the successful port was, and try that first the
LMI> next time, the next startup would be slightly faster.

LMI> Not particularly essential, but there's a lot of stuff like that that
LMI> could be done trivially if there was a place to stash this data...

Something like:

(preferred (imap (servers ("serverA" :port 587)
                          ("serverB" :port 1957))))

would be a nice way to remember this data.  The trick is to make this
data also the one that an assistant.el walkthrough would generate.

>> As a global Emacs facility it would be easy to make all these strings
>> part of a big global-registry alist, which can then be customized and
>> saved independently.  But I think it would squash the decision tree from
>> assistant.el into unwieldy string representations of the nodes and edges.

LMI> True.  But it's basically what Firefox does with its data, if you look
LMI> under about:config.

I think that would be a step backwards considering the available Lisp
facilities.  It would be pretty easy to generate these strings
dynamically and let the user edit them:

Prefs -> IMAP -> Servers -> [serverA] -> [port 587]
                         -> [serverB] -> [port 1957]
                         [+]

or some such interface.  But going backwards, where we decide on these
magical strings to express "the port and name for the Gnus IMAP serverA"
and only store the strings, is harder and less flexible because you have
to write the printer and the reader+parser.

Ted

Re: Application resource storage

[chad]

An idea:

On Tue, 29 Mar 2011 21:34:05 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> But if the question is of the kind "do you want to use port 587 when
LMI> connecting to smtp.gmail.com?", then that doesn't really map
LMI> straightforwardly to a simple variable.  It might map to an alist of
LMI> some kind, but the user may set that alist somewhere else, which 
LMI> makes all this kind of tricky.

I wonder if this isn't the key insight -- for all that I'm not a big fan, custom does a decent job of trying to do all of the user-input interactions. The trickiness seems to come in when something is set both directly and via custom. If the custom machinery were to add a flag or slot to a variable claiming it as custom-set, then we could at least warn the user that a custom setting is being overridden, potentially confirming the change, offering to push the change into custom, etc.

If we added such a check to custom and to user-facing variable setters, then the performance hit should be negligible. In other words, things like set-variable would check to see if its argument was `customized', and calling M-x custom would check to see if any of `its' variables had been changed.

Worth the trouble?
*Chad

Re: Outgoing mail defaults

[Stefan Monnier]

> Even though this means duplicating data, somewhat, I think we should
> keep normal application defaults separate from secret credentials.

Agreed.


        Stefan

Re: Outgoing mail defaults

[Simon Josefsson]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> I'm wonder whether the defaults when sending email should change.

It should.  Most other MUAs default to SMTP/Submission for sending
e-mails these days.

> I'm thinking:
>
> 1) change the defaults to use smtpmail

Yes, once we got 2)+3) working.

> 2) change smtpmail so that it queries the user for SMTP host name (and
> possibly port number)

I believe port number is crucial: port 25 is often not available, so it
has to go through 587 which actually is the Right Thing to use.

There should be some probing: ask for the hostname and try to contact it
on port 587, if that works, ask for port number and offer 587 as the
default.  If 587 doesn't work, probe port 25 and if that works, offer it
as the default.  If probing doesn't work, just prompt for the port.  The
prompts should ideally contain information about whether the probes
worked or not.

> 3) use customize to save the user's reply

Yes, smtpmail-smtp-server and smtpmail-smtp-service.

> In addition, smtpmail should be altered to use auth-source instead of
> the slightly confusing use-both-auth-source-and-netrc setup it's using
> now, and use `open-protocol-stream' to get opportunistic STARTTLS
> upgrades.

Yep.

> Change 1) is something that will annoy people who have a perfectly good
> local sendmail-like setup, so I'm not absolutely sure that this is the
> way to go.  On the other hand, in the long run (as my Ubuntu experience
> shows), I think it might be the right thing to do, and is what most
> other modern MUAs (like Thunderbird) do.

It may be good to complete 2)+3) first and get user feedback on it.
Maybe smtpmail could be the default in Emacs master to get early
adopters to test it.

/Simon

Re: Outgoing mail defaults

[Lars Magne Ingebrigtsen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> I'm thinking:
>
> 1) change the defaults to use smtpmail
>
> 2) change smtpmail so that it queries the user for SMTP host name (and
> possibly port number)
>
> 3) use customize to save the user's reply

I was thinking about starting to implement this stuff (with 1) last :-),
and one problem occurred to me:  Should Emacs do something with mail
delivery failures?

I'm thinking mainly about transient failures and "failures" like
greylisting.  Currently (on Linux), since the mail defaults to using the
local MTA, it handles this quite well.  (Well, if you have a local MTA,
which modern Linux installations don't have, by default.)

So would it make sense to put in a simple MTA layer in Emacs itself?
That is, an mta.el library that's on top of smtpmail.el, which checks
for transient failures (4xx, I think), queues stuff up, and tries
resending in the background?

Or is that too brittle?  If you're not running Emacs, your message will
just be sitting in the queue.  Currently, if Message gets a "450"
(greylisting) error, it'll report the error, and you have to try again
later manually.

So I'm not sure what would be best here.  Opinions?

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Outgoing mail defaults

[Lars Magne Ingebrigtsen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Currently, if Message gets a "450" (greylisting) error, it'll report
> the error, and you have to try again later manually.

Uhm, I think it does that.  I haven't actually tried.  :-)

I mean, smtpmail.el is talking to an MTA.  That MTA won't be giving you
a 450.  So the 450 will only happen when your MTA is talking to the next
MTA.  So transient failures won't happen at the smtpmail.el stage.

So, er, never mind.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Outgoing mail defaults

[Ted Zlatanov]

On Sat, 16 Apr 2011 18:45:05 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> I was thinking about starting to implement this stuff (with 1) last :-),
LMI> and one problem occurred to me:  Should Emacs do something with mail
LMI> delivery failures?
...
LMI> So would it make sense to put in a simple MTA layer in Emacs itself?
LMI> That is, an mta.el library that's on top of smtpmail.el, which checks
LMI> for transient failures (4xx, I think), queues stuff up, and tries
LMI> resending in the background?

It may be useful on W32 but on Unix I would not need it.

LMI> Or is that too brittle?  If you're not running Emacs, your message will
LMI> just be sitting in the queue.  Currently, if Message gets a "450"
LMI> (greylisting) error, it'll report the error, and you have to try again
LMI> later manually.

The MUA should retry later if it's doing SMTP on its own.  Gnus could
use the Gnus Agent queue, for instance.  I think it's a MUA function,
though, not something for Emacs in general.

Ted