more on starttls, gnutls-cli and using tls for mail

more on starttls, gnutls-cli and using tls for mail

[T. V. Raman]

Savannah git has gotten a few more updates, and I've gotten close
to getting smtpmail working without hacks. The remaining problem:
(and this bites if you are using starttls and not gnutls-cli)

In function 
(defun network-stream-open-starttls (name buffer host service
parameters)
the following let binding forces gnutls-cli -- even though
starttls-use-gnutls has been set to nil earlier because
gnutls-cli ws not found on the system.
(let* ((starttls-use-gnutls t)

Also, if you ask smtpmail to save the security settings, it
creates a world-readable .authinfo with the password stored in
the clear --- looks like a bad idea on all counts.

-- 
Best Regards,
--raman

Re: more on starttls, gnutls-cli and using tls for mail

[Karl Fogel]

"T. V. Raman" <tv.raman.tv@gmail.com> writes:
>Also, if you ask smtpmail to save the security settings, it
>creates a world-readable .authinfo with the password stored in
>the clear --- looks like a bad idea on all counts.

I've been thinking that lately too.

First, the fact that .authinfo is created world-readable just seems like
a clear bug.  Also easy to fix (sorry, I don't have patch, but I could
come up with one if we all agree this is a straight bug).

Second: I think it was a mistake that we fully deprecated
`smtpmail-auth-credentials' in favor of ~/.authinfo, instead of, say,
just making the latter override the former when the latter is present.
It's good to have an entirely off-disk option for passing credentials;
maybe most users won't use it, but some will, and it's good in principle
to offer it.  (For example, search for that variable in [1] to see how I
was using it.)

I'd like to know how people feel about the above assertions, before I
start patching anything, though.

-Karl

[1] http://svn.red-bean.com/repos/kfogel/trunk/.emacs

Re: more on starttls, gnutls-cli and using tls for mail

[Leo]

On 2011-08-14 09:26 +0800, Karl Fogel wrote:
> Second: I think it was a mistake that we fully deprecated
> `smtpmail-auth-credentials' in favor of ~/.authinfo, instead of, say,
> just making the latter override the former when the latter is present.
> It's good to have an entirely off-disk option for passing credentials;
> maybe most users won't use it, but some will, and it's good in principle
> to offer it.  (For example, search for that variable in [1] to see how I
> was using it.)

I use smtpmail-auth-credentials to pass different user names for the
gmail smtps I am using. With the new smtpmail.el I haven't found a way
to do that.

Leo

Re: more on starttls, gnutls-cli and using tls for mail

[Karl Fogel]

Leo <sdl.web@gmail.com> writes:
>I use smtpmail-auth-credentials to pass different user names for the
>gmail smtps I am using. With the new smtpmail.el I haven't found a way
>to do that.

Yes, I'm in that situation too now -- which is why I think it's so
unfortunate that `smtpmail-auth-credentials' went away :-(.

I've found a way to do it, using the new smtpmail.el, but it's ugly.
I have `message-send-hook' set up ~/.authinfo for every mail message,
and then I remove the file afterwards in `message-sent-hook'.  Search
for "kf-set-up-authinfo" in [1] if you want the code.

Naturally I hope we'll restore the lost functionality to smtpmail.el, so
this kluge will become unnecessary!  The ideal behavior, I think, would
be to pay attention to ~/.authinfo when it is present, but fall back to
trying `smtpmail-auth-credentials' when it's not.

-Karl

[1] http://svn.red-bean.com/repos/kfogel/trunk/.emacs

Re: more on starttls, gnutls-cli and using tls for mail

[Vijay Lakshminarayanan]

Karl Fogel <kfogel@red-bean.com> writes:

> Leo <sdl.web@gmail.com> writes:
>>I use smtpmail-auth-credentials to pass different user names for the
>>gmail smtps I am using. With the new smtpmail.el I haven't found a way
>>to do that.
>
> Yes, I'm in that situation too now -- which is why I think it's so
> unfortunate that `smtpmail-auth-credentials' went away :-(.
>
> I've found a way to do it, using the new smtpmail.el, but it's ugly.
> I have `message-send-hook' set up ~/.authinfo for every mail message,
> and then I remove the file afterwards in `message-sent-hook'.  Search
> for "kf-set-up-authinfo" in [1] if you want the code.
>
> Naturally I hope we'll restore the lost functionality to smtpmail.el, so
> this kluge will become unnecessary!  The ideal behavior, I think, would
> be to pay attention to ~/.authinfo when it is present, but fall back to
> trying `smtpmail-auth-credentials' when it's not.

I have multiple GMail accounts and I use a hook to use the correct
authentication depending upon which account I'm using.

The hook function is

(defun change-smtp ()
  "Change the SMTP server according to the current from line."
  (save-excursion
    (let* ((username-fn
            (lambda (from)
              (when (string-match "\\<\\([A-Za-z.]*\\)@" from)
                (setq from (match-string 1 from))
                ;; Gmail addresses can have dots in them, so sending
                ;; an email to abcd@gmail.com or a.b.c.d@gmail.com
                ;; will go to the same destination.  So credentials
                ;; for both addresses are stored under the symbol
                ;; `abcd'.
                (setq from (replace-regexp-in-string "\\." "" from))
                (intern from))))
           (from (save-restriction
                   (message-narrow-to-headers)
                   (message-fetch-field "from")))
           (username (funcall username-fn from))
           (credentials (cdr (assoc username *gmail-auth-credentials*))))
      (if credentials
          (setq smtpmail-starttls-credentials credentials
                smtpmail-auth-credentials credentials)
        (error "Could not find auth credentials for %s" from)))))

(add-hook 'message-send-hook 'change-smtp)

where the variable *gmail-auth-credentials* maintains all my user
account info as an alist in the form:

((account1 ("smtp.gmail.com" 587 "account1@gmail.com"  "password1"))
 (account2 ("smtp.gmail.com" 587 "account2@gmail.com"  "password2"))
 (account3 ("smtp.gmail.com" 587 "account3@gmail.com"  "password3"))
 (account4 ("smtp.gmail.com" 587 "acc.ount4@gmail.com" "password4")))

I save it in a file ~/.gmails.gpg and in my .gnus I have 

(eval-when-compile
  (load "~/.gmails.gpg"))

Takes care of authentication.  Of course, now that this is out, someone
could possibly get my email account information by convincing me to
download their cool emacs package.  But I don't think I'm that important :-)

Hope this code helps someone.  As with other Free Software licenses,
this comes with NO WARRANTY.

> -Karl
>
> [1] http://svn.red-bean.com/repos/kfogel/trunk/.emacs
>

-- 
Cheers
~vijay

Gnus should be more complicated.

Re: more on starttls, gnutls-cli and using tls for mail

[Leo]

On 2011-08-14 20:02 +0800, Vijay Lakshminarayanan wrote:
>           (setq smtpmail-starttls-credentials credentials
>                 smtpmail-auth-credentials credentials)

Note: both variables are no more in the new smtpmail.el.

Leo

Re: more on starttls, gnutls-cli and using tls for mail

[Vijay Lakshminarayanan]

Leo <sdl.web@gmail.com> writes:

> On 2011-08-14 20:02 +0800, Vijay Lakshminarayanan wrote:
>>           (setq smtpmail-starttls-credentials credentials
>>                 smtpmail-auth-credentials credentials)
>
> Note: both variables are no more in the new smtpmail.el.

The smtpmail.el in my Emacs describes them.

,-----
| (defcustom smtpmail-auth-credentials "~/.authinfo"
|   "Specify username and password for servers, directly or via .netrc file.
| This variable can either be a filename pointing to a file in netrc(5)
| format, or list of four-element lists that contain, in order,
| `servername' (a string), `port' (an integer), `user' (a string) and
| `password' (a string, or nil to query the user when needed).  If you
| need to enter a `realm' too, add it to the user string, so that it
| looks like `user@realm'."
|   :type '(choice file
| 		 (repeat (list (string  :tag "Server")
|                                (integer :tag "Port")
|                                (string  :tag "Username")
|                                (choice (const :tag "Query when needed" nil)
| 				       (string  :tag "Password")))))
|   :version "22.1"
|   :group 'smtpmail)
`-----

,-----
| (defcustom smtpmail-starttls-credentials '(("" 25 "" ""))
|   "Specify STARTTLS keys and certificates for servers.
| This is a list of four-element list with `servername' (a string),
| `port' (an integer), `key' (a filename) and `certificate' (a
| filename).
| If you do not have a certificate/key pair, leave the `key' and
| `certificate' fields as `nil'.  A key/certificate pair is only
| needed if you want to use X.509 client authenticated
| connections."
|   :type '(repeat (list (string  :tag "Server")
| 		       (integer :tag "Port")
| 		       (file    :tag "Key")
| 		       (file    :tag "Certificate")))
|   :version "21.1"
|   :group 'smtpmail)
`-----

It's possible I don't have the latest smtpmail.el.  The file doesn't
specify its version number.  I'm using the smtpmail.el that ships with

GNU Emacs 24.0.50.1 (i386-mingw-nt6.0.6002) of 2011-06-20 on 3249CTO

Am I missing something?

> Leo

-- 
Cheers
~vijay

Gnus should be more complicated.

Re: more on starttls, gnutls-cli and using tls for mail

[Leo]

On 2011-08-15 11:53 +0800, Vijay Lakshminarayanan wrote:
> It's possible I don't have the latest smtpmail.el.  The file doesn't
> specify its version number.  I'm using the smtpmail.el that ships with
>
> GNU Emacs 24.0.50.1 (i386-mingw-nt6.0.6002) of 2011-06-20 on 3249CTO
>
> Am I missing something?

You need the latest version to see the problem. See:
http://repo.or.cz/w/emacs.git/blob/HEAD:/lisp/mail/smtpmail.el

Leo

Re: more on starttls, gnutls-cli and using tls for mail

[Vijay Lakshminarayanan]

Leo <sdl.web@gmail.com> writes:

> On 2011-08-15 11:53 +0800, Vijay Lakshminarayanan wrote:
>> It's possible I don't have the latest smtpmail.el.  The file doesn't
>> specify its version number.  I'm using the smtpmail.el that ships with
>>
>> GNU Emacs 24.0.50.1 (i386-mingw-nt6.0.6002) of 2011-06-20 on 3249CTO
>>
>> Am I missing something?
>
> You need the latest version to see the problem. See:
> http://repo.or.cz/w/emacs.git/blob/HEAD:/lisp/mail/smtpmail.el

I reread this thread and see what you mean.  Why were these variables
removed from the latest smtpmail?  It seems you too don't have a
solution to the problem.  Is there any discussion around why it's been
removed and what the new alternatives are?

> Leo

-- 
Cheers
~vijay

Gnus should be more complicated.

Re: more on starttls, gnutls-cli and using tls for mail

[Tim Cross]

On Sun, Aug 14, 2011 at 10:02 PM, Vijay Lakshminarayanan
<laksvij@gmail.com> wrote:
> Karl Fogel <kfogel@red-bean.com> writes:
>
>> Leo <sdl.web@gmail.com> writes:
>>>I use smtpmail-auth-credentials to pass different user names for the
>>>gmail smtps I am using. With the new smtpmail.el I haven't found a way
>>>to do that.
>>
>> Yes, I'm in that situation too now -- which is why I think it's so
>> unfortunate that `smtpmail-auth-credentials' went away :-(.
>>
>> I've found a way to do it, using the new smtpmail.el, but it's ugly.
>> I have `message-send-hook' set up ~/.authinfo for every mail message,
>> and then I remove the file afterwards in `message-sent-hook'.  Search
>> for "kf-set-up-authinfo" in [1] if you want the code.
>>
>> Naturally I hope we'll restore the lost functionality to smtpmail.el, so
>> this kluge will become unnecessary!  The ideal behavior, I think, would
>> be to pay attention to ~/.authinfo when it is present, but fall back to
>> trying `smtpmail-auth-credentials' when it's not.
>
> I have multiple GMail accounts and I use a hook to use the correct
> authentication depending upon which account I'm using.
>
> The hook function is
>
> (defun change-smtp ()
>  "Change the SMTP server according to the current from line."
>  (save-excursion
>    (let* ((username-fn
>            (lambda (from)
>              (when (string-match "\\<\\([A-Za-z.]*\\)@" from)
>                (setq from (match-string 1 from))
>                ;; Gmail addresses can have dots in them, so sending
>                ;; an email to abcd@gmail.com or a.b.c.d@gmail.com
>                ;; will go to the same destination.  So credentials
>                ;; for both addresses are stored under the symbol
>                ;; `abcd'.
>                (setq from (replace-regexp-in-string "\\." "" from))
>                (intern from))))
>           (from (save-restriction
>                   (message-narrow-to-headers)
>                   (message-fetch-field "from")))
>           (username (funcall username-fn from))
>           (credentials (cdr (assoc username *gmail-auth-credentials*))))
>      (if credentials
>          (setq smtpmail-starttls-credentials credentials
>                smtpmail-auth-credentials credentials)
>        (error "Could not find auth credentials for %s" from)))))
>
> (add-hook 'message-send-hook 'change-smtp)
>
> where the variable *gmail-auth-credentials* maintains all my user
> account info as an alist in the form:
>
> ((account1 ("smtp.gmail.com" 587 "account1@gmail.com"  "password1"))
>  (account2 ("smtp.gmail.com" 587 "account2@gmail.com"  "password2"))
>  (account3 ("smtp.gmail.com" 587 "account3@gmail.com"  "password3"))
>  (account4 ("smtp.gmail.com" 587 "acc.ount4@gmail.com" "password4")))
>
> I save it in a file ~/.gmails.gpg and in my .gnus I have
>
> (eval-when-compile
>  (load "~/.gmails.gpg"))
>
> Takes care of authentication.  Of course, now that this is out, someone
> could possibly get my email account information by convincing me to
> download their cool emacs package.  But I don't think I'm that important :-)
>
> Hope this code helps someone.  As with other Free Software licenses,
> this comes with NO WARRANTY.
>
>> -Karl
>>

You might be able to clarify something for me. Your the second person
I've come across in as many months who changes smtp server based on
the from address. Your process is even more of puzzling and I'd like
to understand what the reasons are.

For example, if your already authenticated with gmail's smtp server,
why re-authenticate with different credentials just to send a message
with a different from/return address? As far as Iknow, this is not
required and it seems to be adding a lot more complexity for no
apparent reason that I am aware of.

Is there any technical reason that requires this? I frequently use
authenticated smtp, but just auithenticate as one user and send email
with from/return addresses of different users with no problems.

I'm interested  knowing what the use case is for doing this as it
seems unnecessary and something which is making things needlessly
complicated. If there is a good technical reason to do it, I would
like to know so that I can be prepared should I need to modify my
setup and because I sometimes assist in maintaining a mail client and
like to be familiar with the various use cases.

thanks,

Tim

Re: more on starttls, gnutls-cli and using tls for mail

[Vijay Lakshminarayanan]

Tim Cross <theophilusx@gmail.com> writes:

> On Sun, Aug 14, 2011 at 10:02 PM, Vijay Lakshminarayanan
> <laksvij@gmail.com> wrote:
>> Karl Fogel <kfogel@red-bean.com> writes:
>>
>>> Leo <sdl.web@gmail.com> writes:
>>>>I use smtpmail-auth-credentials to pass different user names for the
>>>>gmail smtps I am using. With the new smtpmail.el I haven't found a way
>>>>to do that.
>>>
>>> Yes, I'm in that situation too now -- which is why I think it's so
>>> unfortunate that `smtpmail-auth-credentials' went away :-(.
>>>
>>> I've found a way to do it, using the new smtpmail.el, but it's ugly.
>>> I have `message-send-hook' set up ~/.authinfo for every mail message,
>>> and then I remove the file afterwards in `message-sent-hook'.  Search
>>> for "kf-set-up-authinfo" in [1] if you want the code.
>>>
>>> Naturally I hope we'll restore the lost functionality to smtpmail.el, so
>>> this kluge will become unnecessary!  The ideal behavior, I think, would
>>> be to pay attention to ~/.authinfo when it is present, but fall back to
>>> trying `smtpmail-auth-credentials' when it's not.
>>
>> I have multiple GMail accounts and I use a hook to use the correct
>> authentication depending upon which account I'm using.
>>
>> The hook function is
>>
>> (defun change-smtp ()
>>  "Change the SMTP server according to the current from line."
>>  (save-excursion
>>    (let* ((username-fn
>>            (lambda (from)
>>              (when (string-match "\\<\\([A-Za-z.]*\\)@" from)
>>                (setq from (match-string 1 from))
>>                ;; Gmail addresses can have dots in them, so sending
>>                ;; an email to abcd@gmail.com or a.b.c.d@gmail.com
>>                ;; will go to the same destination.  So credentials
>>                ;; for both addresses are stored under the symbol
>>                ;; `abcd'.
>>                (setq from (replace-regexp-in-string "\\." "" from))
>>                (intern from))))
>>           (from (save-restriction
>>                   (message-narrow-to-headers)
>>                   (message-fetch-field "from")))
>>           (username (funcall username-fn from))
>>           (credentials (cdr (assoc username *gmail-auth-credentials*))))
>>      (if credentials
>>          (setq smtpmail-starttls-credentials credentials
>>                smtpmail-auth-credentials credentials)
>>        (error "Could not find auth credentials for %s" from)))))
>>
>> (add-hook 'message-send-hook 'change-smtp)
>>
>> where the variable *gmail-auth-credentials* maintains all my user
>> account info as an alist in the form:
>>
>> ((account1 ("smtp.gmail.com" 587 "account1@gmail.com"  "password1"))
>>  (account2 ("smtp.gmail.com" 587 "account2@gmail.com"  "password2"))
>>  (account3 ("smtp.gmail.com" 587 "account3@gmail.com"  "password3"))
>>  (account4 ("smtp.gmail.com" 587 "acc.ount4@gmail.com" "password4")))
>>
>> I save it in a file ~/.gmails.gpg and in my .gnus I have
>>
>> (eval-when-compile
>>  (load "~/.gmails.gpg"))
>>
>> Takes care of authentication.  Of course, now that this is out, someone
>> could possibly get my email account information by convincing me to
>> download their cool emacs package.  But I don't think I'm that important :-)
>>
>> Hope this code helps someone.  As with other Free Software licenses,
>> this comes with NO WARRANTY.
>>
>>> -Karl
>>>
>
> You might be able to clarify something for me. Your the second person
> I've come across in as many months who changes smtp server based on
> the from address. Your process is even more of puzzling and I'd like
> to understand what the reasons are.
>
> For example, if your already authenticated with gmail's smtp server,
> why re-authenticate with different credentials just to send a message
> with a different from/return address? As far as Iknow, this is not
> required and it seems to be adding a lot more complexity for no
> apparent reason that I am aware of.
>
> Is there any technical reason that requires this? I frequently use
> authenticated smtp, but just auithenticate as one user and send email
> with from/return addresses of different users with no problems.
>
> I'm interested  knowing what the use case is for doing this as it
> seems unnecessary and something which is making things needlessly
> complicated. If there is a good technical reason to do it, I would
> like to know so that I can be prepared should I need to modify my
> setup and because I sometimes assist in maintaining a mail client and
> like to be familiar with the various use cases.

Originally, I used your technique too but recently I got a warning on
one of my gmail accounts regarding "suspicious activity on the account"
and had to change my password, receive an authentication code to my
phone to confirm the account etc., after which I switched to my hook.

Now, I don't know if changing the from address while being authenticated
to another /caused/ the issue but I haven't faced this issue after
switching to my current scheme which, admittedly complicated, isn't so
hard either.

On the browser, gmail allows you to explicitly change your from address
when replying but it first requires confirmation that you control the
other address.  I have not linked my accounts with each other this way
and I don't want to.

> thanks,
>
> Tim

-- 
Cheers
~vijay

Gnus should be more complicated.

Re: more on starttls, gnutls-cli and using tls for mail

[Tim Cross]

On Mon, Aug 15, 2011 at 5:38 PM, Vijay Lakshminarayanan
<laksvij@gmail.com> wrote:
> Tim Cross <theophilusx@gmail.com> writes:
>
>> On Sun, Aug 14, 2011 at 10:02 PM, Vijay Lakshminarayanan
>> <laksvij@gmail.com> wrote:
>>> Karl Fogel <kfogel@red-bean.com> writes:
>>>
>>>> Leo <sdl.web@gmail.com> writes:
>>>>>I use smtpmail-auth-credentials to pass different user names for the
>>>>>gmail smtps I am using. With the new smtpmail.el I haven't found a way
>>>>>to do that.
>>>>
>>>> Yes, I'm in that situation too now -- which is why I think it's so
>>>> unfortunate that `smtpmail-auth-credentials' went away :-(.
>>>>
>>>> I've found a way to do it, using the new smtpmail.el, but it's ugly.
>>>> I have `message-send-hook' set up ~/.authinfo for every mail message,
>>>> and then I remove the file afterwards in `message-sent-hook'.  Search
>>>> for "kf-set-up-authinfo" in [1] if you want the code.
>>>>
>>>> Naturally I hope we'll restore the lost functionality to smtpmail.el, so
>>>> this kluge will become unnecessary!  The ideal behavior, I think, would
>>>> be to pay attention to ~/.authinfo when it is present, but fall back to
>>>> trying `smtpmail-auth-credentials' when it's not.
>>>
>>> I have multiple GMail accounts and I use a hook to use the correct
>>> authentication depending upon which account I'm using.
>>>
>>> The hook function is
>>>
>>> (defun change-smtp ()
>>>  "Change the SMTP server according to the current from line."
>>>  (save-excursion
>>>    (let* ((username-fn
>>>            (lambda (from)
>>>              (when (string-match "\\<\\([A-Za-z.]*\\)@" from)
>>>                (setq from (match-string 1 from))
>>>                ;; Gmail addresses can have dots in them, so sending
>>>                ;; an email to abcd@gmail.com or a.b.c.d@gmail.com
>>>                ;; will go to the same destination.  So credentials
>>>                ;; for both addresses are stored under the symbol
>>>                ;; `abcd'.
>>>                (setq from (replace-regexp-in-string "\\." "" from))
>>>                (intern from))))
>>>           (from (save-restriction
>>>                   (message-narrow-to-headers)
>>>                   (message-fetch-field "from")))
>>>           (username (funcall username-fn from))
>>>           (credentials (cdr (assoc username *gmail-auth-credentials*))))
>>>      (if credentials
>>>          (setq smtpmail-starttls-credentials credentials
>>>                smtpmail-auth-credentials credentials)
>>>        (error "Could not find auth credentials for %s" from)))))
>>>
>>> (add-hook 'message-send-hook 'change-smtp)
>>>
>>> where the variable *gmail-auth-credentials* maintains all my user
>>> account info as an alist in the form:
>>>
>>> ((account1 ("smtp.gmail.com" 587 "account1@gmail.com"  "password1"))
>>>  (account2 ("smtp.gmail.com" 587 "account2@gmail.com"  "password2"))
>>>  (account3 ("smtp.gmail.com" 587 "account3@gmail.com"  "password3"))
>>>  (account4 ("smtp.gmail.com" 587 "acc.ount4@gmail.com" "password4")))
>>>
>>> I save it in a file ~/.gmails.gpg and in my .gnus I have
>>>
>>> (eval-when-compile
>>>  (load "~/.gmails.gpg"))
>>>
>>> Takes care of authentication.  Of course, now that this is out, someone
>>> could possibly get my email account information by convincing me to
>>> download their cool emacs package.  But I don't think I'm that important :-)
>>>
>>> Hope this code helps someone.  As with other Free Software licenses,
>>> this comes with NO WARRANTY.
>>>
>>>> -Karl
>>>>
>>
>> You might be able to clarify something for me. Your the second person
>> I've come across in as many months who changes smtp server based on
>> the from address. Your process is even more of puzzling and I'd like
>> to understand what the reasons are.
>>
>> For example, if your already authenticated with gmail's smtp server,
>> why re-authenticate with different credentials just to send a message
>> with a different from/return address? As far as Iknow, this is not
>> required and it seems to be adding a lot more complexity for no
>> apparent reason that I am aware of.
>>
>> Is there any technical reason that requires this? I frequently use
>> authenticated smtp, but just auithenticate as one user and send email
>> with from/return addresses of different users with no problems.
>>
>> I'm interested  knowing what the use case is for doing this as it
>> seems unnecessary and something which is making things needlessly
>> complicated. If there is a good technical reason to do it, I would
>> like to know so that I can be prepared should I need to modify my
>> setup and because I sometimes assist in maintaining a mail client and
>> like to be familiar with the various use cases.
>
> Originally, I used your technique too but recently I got a warning on
> one of my gmail accounts regarding "suspicious activity on the account"
> and had to change my password, receive an authentication code to my
> phone to confirm the account etc., after which I switched to my hook.
>
> Now, I don't know if changing the from address while being authenticated
> to another /caused/ the issue but I haven't faced this issue after
> switching to my current scheme which, admittedly complicated, isn't so
> hard either.
>
> On the browser, gmail allows you to explicitly change your from address
> when replying but it first requires confirmation that you control the
> other address.  I have not linked my accounts with each other this way
> and I don't want to.
>
>> thanks,
>>
>> Tim
>
> --
> Cheers
> ~vijay
>
> Gnus should be more complicated.
>

OK, thanks Jijay. So, it would seem the use case is possibly something
specific google has done to detect possible abuse of an email account.
I've not run into this myself, but at least this gives one possible
data point on why this additional complexity may be required.

Tim

Re: more on starttls, gnutls-cli and using tls for mail

[Karl Fogel]

Tim Cross <theophilusx@gmail.com> writes:
>OK, thanks Jijay. So, it would seem the use case is possibly something
>specific google has done to detect possible abuse of an email account.
>I've not run into this myself, but at least this gives one possible
>data point on why this additional complexity may be required.

There's another issue too:

Although "smtp.gmail.com" is the One True Server for sending mails
through Google [1], the account you log in with there could be a regular
Google/Gmail account (call this type "A") or it could be a Google Apps
For Your Domain (type "B") account.

These are rather different beasts.  While you could set up your "A"
account such that it's able to send mail with a From address that is
really of type "B", I had some experiences -- clean reproduction recipes
are hard to create, unfortunately -- that indicated there could be authn
problems or being-blocked-as-spam problems if I sent from type "A"
repeatedly with a From address of type "B".  But these problems go away
if I authenticate to smtp.gmail.com as user "B" whenever sending mail
with an address of type "B".

Again, in practice it's impossible to get definitive answers to what the
rules are.  Running experiments is very time consuming, it's hard to
isolate variables, and Google doesn't have the staff to answer in-depth
technical questions (though random members of the public sometimes do --
see [2]).

Essentially, running one's own email server has become too hard
nowadays, but if one uses an email server hosted by an organization
large enough to deal competently with the running an email server in the
modern era, then by definition that organization will be too large to
answer individual questions.  The problem is systemic; I'm not blaming
Google -- I'm glad they offer the smtp.gmail.com service.

So I have to get Emacs to switch sender authn creds on an email-by-email
basis.  Recent smtpmail.el changes have made this harder, not easier.

-Karl

[1] The reason I send mail through Google is because if I use my own
    server, it can be difficult to avoid the server being blacklisted
    even when that server is not the origin of any spam.  See
    http://www.rants.org/2010/05/26/email-blacklisting-considered-harmful/

[2] http://www.google.com/support/forum/p/gmail/thread?tid=45781946ea84651e

Re: more on starttls, gnutls-cli and using tls for mail

[Tim Cross]

On Thu, Aug 18, 2011 at 12:28 AM, Karl Fogel <kfogel@red-bean.com> wrote:
> Tim Cross <theophilusx@gmail.com> writes:
>>OK, thanks Jijay. So, it would seem the use case is possibly something
>>specific google has done to detect possible abuse of an email account.
>>I've not run into this myself, but at least this gives one possible
>>data point on why this additional complexity may be required.
>
> There's another issue too:
>
> Although "smtp.gmail.com" is the One True Server for sending mails
> through Google [1], the account you log in with there could be a regular
> Google/Gmail account (call this type "A") or it could be a Google Apps
> For Your Domain (type "B") account.
>
> These are rather different beasts.  While you could set up your "A"
> account such that it's able to send mail with a From address that is
> really of type "B", I had some experiences -- clean reproduction recipes
> are hard to create, unfortunately -- that indicated there could be authn
> problems or being-blocked-as-spam problems if I sent from type "A"
> repeatedly with a From address of type "B".  But these problems go away
> if I authenticate to smtp.gmail.com as user "B" whenever sending mail
> with an address of type "B".
>
> Again, in practice it's impossible to get definitive answers to what the
> rules are.  Running experiments is very time consuming, it's hard to
> isolate variables, and Google doesn't have the staff to answer in-depth
> technical questions (though random members of the public sometimes do --
> see [2]).
>
> Essentially, running one's own email server has become too hard
> nowadays, but if one uses an email server hosted by an organization
> large enough to deal competently with the running an email server in the
> modern era, then by definition that organization will be too large to
> answer individual questions.  The problem is systemic; I'm not blaming
> Google -- I'm glad they offer the smtp.gmail.com service.
>
> So I have to get Emacs to switch sender authn creds on an email-by-email
> basis.  Recent smtpmail.el changes have made this harder, not easier.
>
> -Karl
>
> [1] The reason I send mail through Google is because if I use my own
>    server, it can be difficult to avoid the server being blacklisted
>    even when that server is not the origin of any spam.  See
>    http://www.rants.org/2010/05/26/email-blacklisting-considered-harmful/
>
> [2] http://www.google.com/support/forum/p/gmail/thread?tid=45781946ea84651e
>

Thanks Karl. It seems there are use cases for using different
authenticated users based on the from/reply address being used.
However, it should be noted that this is not due to any requirement or
limitation of smtp - this is because of addtional requirements imposed
by providers, such as google's gmail, which adds additional
restrictions that are not standard smtp behavior.

Tim

Re: more on starttls, gnutls-cli and using tls for mail

[chad]

On Aug 17, 2011, at 3:48 PM, Tim Cross wrote:
> However, it should be noted that this is not due to any requirement or
> limitation of smtp - this is because of addtional requirements imposed
> by providers, such as google's gmail, which adds additional
> restrictions that are not standard smtp behavior.

There is no `standard' for smtp behavior in this area - although there
are several Standards one might variously attempt to follow. To be clear, 
there are `standard smtp systems' that also require people who want to 
use distinct `from addresses' to authenticate separately for each; gmail 
is just the largest fish in the email waters, so there's no need to enumerate 
the edges beyond the majority.

To put it another way, once you know that the oceans are wet, do you 
really need to test the lakes, rivers, and streams?

*Chad

P.S. Not so long ago I rearranged my mail handling of the ~7 addresses
I use regularly to avoid problems like these, but I still have a lot of
sympathy for people who can't use my approach.

Re: more on starttls, gnutls-cli and using tls for mail

[Stephen J. Turnbull]

Tim Cross writes:

 > Thanks Karl. It seems there are use cases for using different
 > authenticated users based on the from/reply address being used.
 > However, it should be noted that this is not due to any requirement
 > or limitation of smtp

Lack of a standard authentication method *is* the limitation of
email-as-we-know-it.  As Chad points out, there are various standards
available, but SMTP itself knows about none of them, and therefore
none are reliably available.

There is a fundamental requirement of email-as-we-know-it, that it be
a way for any dog on the Internet to get in touch with you.  (This is
why Karl and Chad have so many addresses: "kfogel@red-bean" means
nothing to most latent correspondents, while "kfogel@civiccommons"
does, to some fraction that Karl cares about.)  On the other hand, the
fact that among the dogs is Dogbert (aka Canter/Seigel et al, not to
mention even less lovable folk such as stalkers) means that private
mailboxes are widely desired.

Lack of a standard authentication method *at the receiving end* means
that there's no single way to identify mail from expected senders at
your *private* mailbox.  Lack of a standard authentication method *at
the sending end* means there's no way to guarantee you'll be
recognized by the recipient's private mailbox.  So there's no way to
implement reliable private mailboxes.  Not even security-via-obscurity
works because your ISP may filter, *must filter*, based on something
other than sender credentials.

It should be obvious that users will evolve complex, *idiosyncratic*
methods to deal with this complex environment, as recipients and
senders implement a variety of partially coordinated solutions to the
problem of protecting mailbox privacy where desired.

I don't know whether this means that smtp-auth-credentials is needed
to implement such methods (presumably not, Are We Not Hackers?), but
I'm a bit surprised that a project sufficiently conservative about
email that RMail is its default MUA didn't follow the usual process of
obsolete'ing the variable before, uh, jerking the rug out from under
people's .emacs'es.

Re: more on starttls, gnutls-cli and using tls for mail

[Vijay Lakshminarayanan]

Tim Cross <theophilusx@gmail.com> writes:

> On Mon, Aug 15, 2011 at 5:38 PM, Vijay Lakshminarayanan
> <laksvij@gmail.com> wrote:
>> Tim Cross <theophilusx@gmail.com> writes:

[big snip]

> OK, thanks Jijay. So, it would seem the use case is possibly something
> specific google has done to detect possible abuse of an email account.
> I've not run into this myself, but at least this gives one possible
> data point on why this additional complexity may be required.

And, as Richard Riley pointed out:

,-----
| Secondly reliable and correct storage of the sent email in the gmail
| archives.
`-----

> Tim

-- 
Cheers
~vijay

Gnus should be more complicated.

Re: more on starttls, gnutls-cli and using tls for mail

[Richard Riley]

Tim Cross <theophilusx@gmail.com> writes:

> You might be able to clarify something for me. Your the second person
> I've come across in as many months who changes smtp server based on
> the from address. Your process is even more of puzzling and I'd like
> to understand what the reasons are.

Firstly gmail wont let you send with a different from unless they are
friend acocunts in which case the other email addresses are buried in
the header. Secondly reliable and correct storage of the sent email in
the gmail archives. Thirdly some companies have all sorts of silly spam
rules which will detect "strange" header setups such as "friend account
smtp usage" and block your email.

It can only be cleaner and better to use the correct smtp server for the
correct email sender and is a must imo.

Re: more on starttls, gnutls-cli and using tls for mail

[David Engster]

Tim Cross writes:
> Is there any technical reason that requires this? I frequently use
> authenticated smtp, but just auithenticate as one user and send email
> with from/return addresses of different users with no problems.

Then you've been lucky. See

http://en.wikipedia.org/wiki/Sender_Policy_Framework

-David

Re: more on starttls, gnutls-cli and using tls for mail

[Tim Cross]

On Mon, Aug 15, 2011 at 7:21 PM, David Engster <deng@randomsample.de> wrote:
> Tim Cross writes:
>> Is there any technical reason that requires this? I frequently use
>> authenticated smtp, but just auithenticate as one user and send email
>> with from/return addresses of different users with no problems.
>
> Then you've been lucky. See
>
> http://en.wikipedia.org/wiki/Sender_Policy_Framework
>
> -David
>
>

As I understood it, SPF is domain/host based, not individual user
based. All the implementations I've seen use the domain/IP address of
the host, not what is reported in the from line of the address. If yo
think about  it, using the host as reported inthe from address would
completely break the whole framework. Therefore, using different from
addresses is irrelevant.

This does not mean there isn't a case, just that SPF is not it. A
couple of other reasons, mostly specific to gmail and possibly other
web based mail services have been suggested that do seem reasonable.
However, there is nothing at the smtp level I've seen so far.

Tim

Re: more on starttls, gnutls-cli and using tls for mail

[David Engster]

Tim Cross writes:
> On Mon, Aug 15, 2011 at 7:21 PM, David Engster <deng@randomsample.de> wrote:
>> Tim Cross writes:
>>> Is there any technical reason that requires this? I frequently use
>>> authenticated smtp, but just auithenticate as one user and send email
>
>>> with from/return addresses of different users with no problems.
>>
>> Then you've been lucky. See
>>
>> http://en.wikipedia.org/wiki/Sender_Policy_Framework
>>
>> -David
>>
>>
>
> As I understood it, SPF is domain/host based, not individual user
> based.

Correct. I thought we were talking about the general case why it makes
sense to choose different SMTP hosts dependending on the FROM
address. As long as you're using the same domain, SPF is not an issue,
indeed.

-David

Re: more on starttls, gnutls-cli and using tls for mail

[Dimitri Fontaine]

Tim Cross <theophilusx@gmail.com> writes:
> You might be able to clarify something for me. Your the second person
> I've come across in as many months who changes smtp server based on
> the from address. Your process is even more of puzzling and I'd like
> to understand what the reasons are.

I just now realize I'm doing the same thing here, but instead of
changing things at the gnus level, I have gnus always connect to the
smtp server at localhost, which will then find the right next relay.

  http://tapoueh.org/blog/2010/09/23-postfix-sender_dependent_relayhost_maps.html

Regards,
-- 
dim

Re: more on starttls, gnutls-cli and using tls for mail

[Leo]

On 2011-08-16 00:25 +0800, Dimitri Fontaine wrote:
> I just now realize I'm doing the same thing here, but instead of
> changing things at the gnus level, I have gnus always connect to the
> smtp server at localhost, which will then find the right next relay.
>
>   http://tapoueh.org/blog/2010/09/23-postfix-sender_dependent_relayhost_maps.html

Thanks for that. There are other alternatives such as using msmtp. But
they are a nuisance because they require installing and configuring
another tool for something that Emacs is capable of doing.

The point is there is one key feature that has been accidentally deleted
during the rewrite of smtpmail. This thread is about whether and how
should we bring it back.

Leo

Re: more on starttls, gnutls-cli and using tls for mail

[Dimitri Fontaine]

Leo <sdl.web@gmail.com> writes:
>>   http://tapoueh.org/blog/2010/09/23-postfix-sender_dependent_relayhost_maps.html
>
> Thanks for that. There are other alternatives such as using msmtp. But
> they are a nuisance because they require installing and configuring
> another tool for something that Emacs is capable of doing.

The problem with msmtp is both that it hangs emacs while sending the
email, and that on delivery error you have to handle it yourself.  The
reason why I so much prefer postfix here is that it will queue the
outgoing mail then deliver it in the background, and it will manage the
delivery queue and retry if necessary.

See also my mailq mode that allows me to quickly see if I have to force
a send retry and to flush my postfix queue all from Emacs.

  http://tapoueh.org/blog/2011/05/05-mailq-modeline-display.html
  http://tapoueh.org/emacs/mailq.html

> The point is there is one key feature that has been accidentally deleted
> during the rewrite of smtpmail. This thread is about whether and how
> should we bring it back.

That's a whole other problem here, agreed.

Regards,
-- 
dim

Re: more on starttls, gnutls-cli and using tls for mail

[Tim Cross]

On Tue, Aug 16, 2011 at 8:12 PM, Dimitri Fontaine <dim@tapoueh.org> wrote:
> Leo <sdl.web@gmail.com> writes:
>>>   http://tapoueh.org/blog/2010/09/23-postfix-sender_dependent_relayhost_maps.html
>>
>> Thanks for that. There are other alternatives such as using msmtp. But
>> they are a nuisance because they require installing and configuring
>> another tool for something that Emacs is capable of doing.
>
> The problem with msmtp is both that it hangs emacs while sending the
> email, and that on delivery error you have to handle it yourself.  The
> reason why I so much prefer postfix here is that it will queue the
> outgoing mail then deliver it in the background, and it will manage the
> delivery queue and retry if necessary.
>

same issue with smtpmail as well.

> See also my mailq mode that allows me to quickly see if I have to force
> a send retry and to flush my postfix queue all from Emacs.
>
>  http://tapoueh.org/blog/2011/05/05-mailq-modeline-display.html
>  http://tapoueh.org/emacs/mailq.html
>
>> The point is there is one key feature that has been accidentally deleted
>> during the rewrite of smtpmail. This thread is about whether and how
>> should we bring it back.
>
> That's a whole other problem here, agreed.
>

Agree.

Re: more on starttls, gnutls-cli and using tls for mail

[Richard Riley]

Leo <sdl.web@gmail.com> writes:

> On 2011-08-16 00:25 +0800, Dimitri Fontaine wrote:
>> I just now realize I'm doing the same thing here, but instead of
>> changing things at the gnus level, I have gnus always connect to the
>> smtp server at localhost, which will then find the right next relay.
>>
>>   http://tapoueh.org/blog/2010/09/23-postfix-sender_dependent_relayhost_maps.html
>
> Thanks for that. There are other alternatives such as using msmtp. But
> they are a nuisance because they require installing and configuring
> another tool for something that Emacs is capable of doing.

Also its not async. Much better to configure it at the postfix/exim level.

>
> The point is there is one key feature that has been accidentally deleted
> during the rewrite of smtpmail. This thread is about whether and how
> should we bring it back.

Sender specific smtp auth credentials are a must as many people dont run
their own mta.

Re: more on starttls, gnutls-cli and using tls for mail

[Roland Winkler]

On Sat, Aug 13 2011, Karl Fogel wrote: 
> "T. V. Raman" <tv.raman.tv@gmail.com> writes: 
>>Also, if you ask smtpmail to save the security settings, it 
>>creates a world-readable .authinfo with the password stored in 
>>the clear --- looks like a bad idea on all counts. 
> 
> I've been thinking that lately too. 
> 
> First, the fact that .authinfo is created world-readable just 
> seems like a clear bug.  Also easy to fix (sorry, I don't have 
> patch, but I could come up with one if we all agree this is a 
> straight bug). 
 
See bug #9113. So yes, I agree that this is a bug.   See also bug 
#7487 where some issues related to .authinfo were discussed: Under 
certain circumstances Gnus needed to repeatedly decrypt
~/.authinfo.gpg, which requires the gpg passphrase. Yet I do not find it
justified to make an unencrypted ~/.authinfo the default because of such
a nuisance. If at all, I believe it should be the other way round: the
default should be ~/.authinfo.gpg. If someone doesn't like that for
whatever reason, he or she can change that in the init file.

Roland

Re: more on starttls, gnutls-cli and using tls for mail

[Roland Winkler]

On Sun, Aug 14 2011, Roland Winkler wrote:
> See bug #9113. So yes, I agree that this is a bug.

I should add:

I think we should include Ted in this thread, the author of
auth-source.el.  Unfortunately he has been off-line recently.
(I am including him in this message.)

Roland

Re: more on starttls, gnutls-cli and using tls for mail

[Karl Fogel]

Roland Winkler <winkler@gnu.org> writes:
>> I've been thinking that lately too. 
>>
>> First, the fact that .authinfo is created world-readable just seems
>> like a clear bug.  Also easy to fix (sorry, I don't have patch, but
>> I could come up with one if we all agree this is a straight bug). 
>
>See bug #9113. So yes, I agree that this is a bug.   See also bug
>#7487 where some issues related to .authinfo were discussed: Under
>certain circumstances Gnus needed to repeatedly decrypt
>~/.authinfo.gpg, which requires the gpg passphrase. Yet I do not find it
>justified to make an unencrypted ~/.authinfo the default because of such
>a nuisance. If at all, I believe it should be the other way round: the
>default should be ~/.authinfo.gpg. If someone doesn't like that for
>whatever reason, he or she can change that in the init file.

Bug #9113 is slightly different from what T.V. and I were saying.  #9113
suggests solving the exposure problem through encryption, and then #7487
has a long discussion about what kind of encryption it should be --
public key or symmetric -- how the user interface should work, etc.

But I think T.V. and I are just saying: "In the plaintext case, let's at
least make the file non-world-readable!"

Offering encryption is great, but it's also very complex and error-prone
(as the bug reports show).  There will always be a plaintext case, since
users cannot be required to have GPG-like software installed.  In the
plaintext case, we could behave better than we do.

But it sounds like we probably agree on this too, and I should just make
the change :-).

Separately, I think it's bad that we removed the Elisp-based API for
passing this authn information, since some people (like me) are already
using Elisp to fetch the auth creds securely from elsewhere, and having
to dynamically construct a ~/.authinfo file as a means of passing that
information *to other Elisp* is, shall we say, a really poor API.

There's no reason we can't have both `smtpmail-auth-credentials' and
~/.authinfo (or ~/.authinfo.foo), and simply fall try the former when
the latter is unavailable.

However, that's a larger change, or semi-reversion.  I don't know if it
would be accepted; I guess it belongs in a distinct thread.

-K

Re: more on starttls, gnutls-cli and using tls for mail

[Roland Winkler]

On Sun Aug 14 2011 Karl Fogel wrote:
> Offering encryption is great, but it's also very complex and
> error-prone (as the bug reports show). There will always be a
> plaintext case, since users cannot be required to have GPG-like
> software installed. In the plaintext case, we could behave better
> than we do.

I see, yes, your perspective is yet different but equally valid!

> But it sounds like we probably agree on this too, and I should
> just make the change :-).

What kind of solution you have in mind? Just changing the file mode
bits?  Could it also help to use ~/.emacs.d?

Roland

Re: more on starttls, gnutls-cli and using tls for mail

[Tim Cross]

On Tue, Aug 16, 2011 at 1:21 AM, Roland Winkler <winkler@gnu.org> wrote:
> On Sun Aug 14 2011 Karl Fogel wrote:
>> Offering encryption is great, but it's also very complex and
>> error-prone (as the bug reports show). There will always be a
>> plaintext case, since users cannot be required to have GPG-like
>> software installed. In the plaintext case, we could behave better
>> than we do.
>
> I see, yes, your perspective is yet different but equally valid!
>
>> But it sounds like we probably agree on this too, and I should
>> just make the change :-).
>
> What kind of solution you have in mind? Just changing the file mode
> bits?  Could it also help to use ~/.emacs.d?
>
> Roland
>
>

Changing the mode of the file would seem like a minimal first
necessary step. Programs like ssh and gpg make sure that files they
manage have good/sound access permissions and emacs should do the same
with potentially sensitive data like this. Things like ssh even go so
far as to refuse to operate if certain files don't have a resrictive
(user only) access mode.

As to whether these files should go into .emacs.d, I'm not sure. I
guess it does reduce clutter within the home directory, but I don't
think it aids much to security.

It probably wouldn't be a bad idea if emacs, when detecting a palin
text authinfo file, did perhaps suggest using encryption and provide a
link to more information. However, how to do this and not end up with
something which continually nags people who have made the concious
decision not to encrypt is possibly mroe effort than its worth.

Tim

Re: more on starttls, gnutls-cli and using tls for mail

[Ted Zlatanov]

On Sun, 14 Aug 2011 12:23:09 -0400 Karl Fogel <kfogel@red-bean.com> wrote: 

KF> Separately, I think it's bad that we removed the Elisp-based API for
KF> passing this authn information, since some people (like me) are already
KF> using Elisp to fetch the auth creds securely from elsewhere, and having
KF> to dynamically construct a ~/.authinfo file as a means of passing that
KF> information *to other Elisp* is, shall we say, a really poor API.

KF> There's no reason we can't have both `smtpmail-auth-credentials' and
KF> ~/.authinfo (or ~/.authinfo.foo), and simply fall try the former when
KF> the latter is unavailable.

KF> However, that's a larger change, or semi-reversion.  I don't know if it
KF> would be accepted; I guess it belongs in a distinct thread.

`smtpmail-auth-credentials' is not a good interface for many reasons.
It was very, very well discussed in the past.  Everything it can do,
should be possible with `auth-source-search'.

`auth-source' supports multiple backends.  There's no reason we can't
provide a backend that does the dynamic fetching you want and does not
use a authinfo/netrc file.  The authinfo/netrc backend supports entry
creation and can share the file with other consumers such as libcurl;
this is the main reason why it's the default now.

You should also note that you can configure `auth-sources' to use any
combination of backends.  So your custom ELisp backend could be first,
then you'd hit the Secrets API, then the authinfo/netrc backend.  This
is simply impossible with `smtpmail-auth-credentials'.

Ted

Re: more on starttls, gnutls-cli and using tls for mail

[Karl Fogel]

Ted Zlatanov <tzz@lifelogs.com> writes:
>`smtpmail-auth-credentials' is not a good interface for many reasons.
>It was very, very well discussed in the past.  Everything it can do,
>should be possible with `auth-source-search'.
>
>`auth-source' supports multiple backends.  There's no reason we can't
>provide a backend that does the dynamic fetching you want and does not
>use a authinfo/netrc file.  The authinfo/netrc backend supports entry
>creation and can share the file with other consumers such as libcurl;
>this is the main reason why it's the default now.
>
>You should also note that you can configure `auth-sources' to use any
>combination of backends.  So your custom ELisp backend could be first,
>then you'd hit the Secrets API, then the authinfo/netrc backend. [...]

Thanks.  This all sounds good in principle.  As a user (even as an
Elisp-literate user) I have no idea yet how to take advantage of the
functionality you describe above, but I assume that's just a matter of
reading the documentation.  From what I've read so far, auth-source
seems to be a superset of everything anyone could ever want.

It may be that after I set up something dynamic, it will be
contributable back to Emacs, either as code or as examples for the
auth-source Info manual.  I'll try to keep that in mind.

For now, I'm dynamically constructing ~/.authinfo and then destroying it
after the mail is sent, because I got that working and its undeniable
awkwardness is a mere annoyance, not a showstopper.

-K

Re: more on starttls, gnutls-cli and using tls for mail

[Ted Zlatanov]

On Sun, 25 Sep 2011 13:26:08 -0400 Karl Fogel <kfogel@red-bean.com> wrote: 

KF> Ted Zlatanov <tzz@lifelogs.com> writes:
>> `smtpmail-auth-credentials' is not a good interface for many reasons.
>> It was very, very well discussed in the past.  Everything it can do,
>> should be possible with `auth-source-search'.
>> 
>> `auth-source' supports multiple backends.  There's no reason we can't
>> provide a backend that does the dynamic fetching you want and does not
>> use a authinfo/netrc file.  The authinfo/netrc backend supports entry
>> creation and can share the file with other consumers such as libcurl;
>> this is the main reason why it's the default now.
>> 
>> You should also note that you can configure `auth-sources' to use any
>> combination of backends.  So your custom ELisp backend could be first,
>> then you'd hit the Secrets API, then the authinfo/netrc backend. [...]

KF> Thanks.  This all sounds good in principle.  As a user (even as an
KF> Elisp-literate user) I have no idea yet how to take advantage of the
KF> functionality you describe above, but I assume that's just a matter of
KF> reading the documentation.  From what I've read so far, auth-source
KF> seems to be a superset of everything anyone could ever want.

KF> It may be that after I set up something dynamic, it will be
KF> contributable back to Emacs, either as code or as examples for the
KF> auth-source Info manual.  I'll try to keep that in mind.

KF> For now, I'm dynamically constructing ~/.authinfo and then destroying it
KF> after the mail is sent, because I got that working and its undeniable
KF> awkwardness is a mere annoyance, not a showstopper.

Heheh.  OK.  You really, really want a dynamic backend then.  No
problem.

First, look at the definition of `auth-sources'.  You need to augment
the list of backends:

                 (choice
                  (string :tag "Just a file")
                  (const :tag "Default Secrets API Collection" 'default)
                  (const :tag "Login Secrets API Collection" "secrets:Login")
                  (const :tag "Temp Secrets API Collection" "secrets:session")
+                  (const :tag "Karl's Dynamic Backend" 'dynamic-data)

Then look at `auth-source-backend-parse'.  Add your new backend to the
cond statement so it's parsed properly.  Here you can parse a string
prefix like "dynamic-data:karl-dynamic-variable" (as the Sessions API does) so
your users can point to a variable easily without customizing
`auth-sources' too much.  They would just have to add a string.  The
prefix doesn't have to match the backend name.

The last step is to create your backend instance when needed and return
it:

       (auth-source-backend
        (plist-get entry :source)
        :source (plist-get entry :source)
        :type 'dynamic-data
        :search-function 'auth-source-dynamic-data-search
        :create-function 'auth-source-dynamic-data-create)))

Your create function can be a stub, that's not a problem.

Finally your search function...  Copy the `auth-source-netrc-search'
template:

(defun* auth-source-netrc-search (&rest
                                  spec
                                  &key backend require create delete
                                  type max host user port
                                  &allow-other-keys)

1) the backend is the backend you created earlier, you'll need the
"source" slot which in your case is e.g. karl-dynamic-variable.  You'd
map that to a symbol name and manipulate the value, obviously.

2) require is a list of required keys

3) max is the maximum number of results you should return

4) create can be ignored if your create-function is a stub

5) delete can be ignored if your backend doesn't delete entries,
otherwise delete everything you found up to max

6) host, user, port are the only search criteria accepted by the
netrc/authinfo backend; yours could take more

7) type is the backend type, you should return nothing if it's not
dynamic-data (the `auth-source-search' caller may ask for this).

If you want to allow creation, look at `auth-source-netrc-create'.
There's a lot of code to deal with prompting that should IMO be factored
out but I haven't had the time.  It sounds like you'd be OK with letting
the user modify the data externally though.

Daiki Ueno went through this with his plstore backend so you're the
second one to possibly write a custom backend.  If it goes well for you
I'll put these instructions in the auth-source texinfo pages.

If you think this is too complicated or you're busy, I'll do it.  Please
let me know.

Thanks
Ted

Re: more on starttls, gnutls-cli and using tls for mail

[Karl Fogel]

Ted Zlatanov <tzz@lifelogs.com> writes:
>Heheh.  OK.  You really, really want a dynamic backend then.  No
>problem.
>
>First, look at the definition of `auth-sources'.  You need to augment
>the list of backends:
>
>[...]

Ted, thanks for this amazingly detailed mail!  It'll be my starting
point.  I wish I could start right away, but unfortunately can't, due to
time constraints.  Once I do, I'll try to make something re-usable, and
leave the Info pages in better shape than I found them, to at least "pay
forward" for your valuable guidance.

-K

>                 (choice
>                  (string :tag "Just a file")
>                  (const :tag "Default Secrets API Collection" 'default)
>                  (const :tag "Login Secrets API Collection" "secrets:Login")
>                  (const :tag "Temp Secrets API Collection" "secrets:session")
>+                  (const :tag "Karl's Dynamic Backend" 'dynamic-data)
>
>Then look at `auth-source-backend-parse'.  Add your new backend to the
>cond statement so it's parsed properly.  Here you can parse a string
>prefix like "dynamic-data:karl-dynamic-variable" (as the Sessions API does) so
>your users can point to a variable easily without customizing
>`auth-sources' too much.  They would just have to add a string.  The
>prefix doesn't have to match the backend name.
>
>The last step is to create your backend instance when needed and return
>it:
>
>       (auth-source-backend
>        (plist-get entry :source)
>        :source (plist-get entry :source)
>        :type 'dynamic-data
>        :search-function 'auth-source-dynamic-data-search
>        :create-function 'auth-source-dynamic-data-create)))
>
>Your create function can be a stub, that's not a problem.
>
>Finally your search function...  Copy the `auth-source-netrc-search'
>template:
>
>(defun* auth-source-netrc-search (&rest
>                                  spec
>                                  &key backend require create delete
>                                  type max host user port
>                                  &allow-other-keys)
>
>1) the backend is the backend you created earlier, you'll need the
>"source" slot which in your case is e.g. karl-dynamic-variable.  You'd
>map that to a symbol name and manipulate the value, obviously.
>
>2) require is a list of required keys
>
>3) max is the maximum number of results you should return
>
>4) create can be ignored if your create-function is a stub
>
>5) delete can be ignored if your backend doesn't delete entries,
>otherwise delete everything you found up to max
>
>6) host, user, port are the only search criteria accepted by the
>netrc/authinfo backend; yours could take more
>
>7) type is the backend type, you should return nothing if it's not
>dynamic-data (the `auth-source-search' caller may ask for this).
>
>If you want to allow creation, look at `auth-source-netrc-create'.
>There's a lot of code to deal with prompting that should IMO be factored
>out but I haven't had the time.  It sounds like you'd be OK with letting
>the user modify the data externally though.
>
>Daiki Ueno went through this with his plstore backend so you're the
>second one to possibly write a custom backend.  If it goes well for you
>I'll put these instructions in the auth-source texinfo pages.
>
>If you think this is too complicated or you're busy, I'll do it.  Please
>let me know.
>
>Thanks
>Ted

Multiple SMTP accounts with smtpmail.el (was: more on starttls, gnutls-cli and using tls for mail)

[Lars Magne Ingebrigtsen]

Karl Fogel <kfogel@red-bean.com> writes:

> Second: I think it was a mistake that we fully deprecated
> `smtpmail-auth-credentials' in favor of ~/.authinfo, instead of, say,
> just making the latter override the former when the latter is present.
> It's good to have an entirely off-disk option for passing credentials;
> maybe most users won't use it, but some will, and it's good in principle
> to offer it.  (For example, search for that variable in [1] to see how I
> was using it.)

I think it would be nice to find an auth-source based solution for the
multiple-account smtpmail case.  Which shouldn't be difficult, I think.
How about just adding a `smtpmail-user' variable, that when bound or set
would make smtpmail.el feed that user name to auth-source, and then get
the correct credentials?

`smtpmail-user' would be set from `message-send-hook' or the like, of
course.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Multiple SMTP accounts with smtpmail.el

[Leo]

On 2011-08-18 05:06 +0800, Lars Magne Ingebrigtsen wrote:
[snipped 10 lines]
> I think it would be nice to find an auth-source based solution for the
> multiple-account smtpmail case.  Which shouldn't be difficult, I think.
> How about just adding a `smtpmail-user' variable, that when bound or set
> would make smtpmail.el feed that user name to auth-source, and then get
> the correct credentials?
>
> `smtpmail-user' would be set from `message-send-hook' or the like, of
> course.

How about smtpmail-auth-spec (for the SPEC arg of auth-source-search)?

Leo

Re: Multiple SMTP accounts with smtpmail.el

[Karl Fogel]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
>I think it would be nice to find an auth-source based solution for the
>multiple-account smtpmail case.  Which shouldn't be difficult, I think.
>How about just adding a `smtpmail-user' variable, that when bound or set
>would make smtpmail.el feed that user name to auth-source, and then get
>the correct credentials?

I'm not expert enough in Emacs's smtp authn system to fully understand
this proposal -- I don't know exactly what "auth-source" means here.  It
sounds like you mean something very specific by it?  (If the variable is
`smtpmail-user', then what variable(s) hold server/port/password
information?)

Overall, I'm just saying:

There should be a purely in-elisp way to pass smtp authentication data
when sending email.  There should be no need to create or modify a file
on disk containing authentication data (although such a file could
certainly be an option, since it might be easier for some users).

We used to have an entirely in-elisp way, but then it got removed.  If
the same way came back, that would be fine; if a different way were
implemented, that would also be fine.  As long as the NEWS file
describes the situation accurately, people will eventually recover.

I have a mild preference for the old way (`smtpmail-auth-credentials')
just because it's familiar, but the emphasis is on "mild".

-Karl

Re: Multiple SMTP accounts with smtpmail.el

[Vijay Lakshminarayanan]

Karl Fogel <kfogel@red-bean.com> writes:

> We used to have an entirely in-elisp way, but then it got removed.  If
> the same way came back, that would be fine; if a different way were
> implemented, that would also be fine.  As long as the NEWS file
> describes the situation accurately, people will eventually recover.

I see a lot of you discussing the removal of these variables from the
latest-greatest smtpmail.el and I even see some references to it being
accidental.  I could not find any comments in the file that explained
why the old behavior was changed but it seems that everyone is unhappy
with the change.  (Everyone is probably too strong but I haven't seen
anyone defend these changes.)

Since all changes are discussed on this mailing list, could someone give
me a reference to why the changes occurred and what's wrong with the
scheme as it was?

> I have a mild preference for the old way (`smtpmail-auth-credentials')
> just because it's familiar, but the emphasis is on "mild".

I'm still on the "old" scheme that's available with "GNU Emacs 24.0.50.1
(i386-mingw-nt6.0.6002) of 2011-06-20 on 3249CTO" and I *really like
it*, I have a hook for selecting authentication that actually works and
so on.  So any pointers would be greatly appreciated.

> -Karl

-- 
Cheers
~vijay

Gnus should be more complicated.

Re: Multiple SMTP accounts with smtpmail.el

[Lars Magne Ingebrigtsen]

Karl Fogel <kfogel@red-bean.com> writes:

> I'm not expert enough in Emacs's smtp authn system to fully understand
> this proposal -- I don't know exactly what "auth-source" means here.  It
> sounds like you mean something very specific by it?

auth-source is the new infrastructure for credentials.

> There should be a purely in-elisp way to pass smtp authentication data
> when sending email.

Sure.  Just add another source for credentials to `auth-source'.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Multiple SMTP accounts with smtpmail.el

[Karl Fogel]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
>Karl Fogel <kfogel@red-bean.com> writes:
>
>> I'm not expert enough in Emacs's smtp authn system to fully understand
>> this proposal -- I don't know exactly what "auth-source" means here.  It
>> sounds like you mean something very specific by it?
>
>auth-source is the new infrastructure for credentials.
>
>> There should be a purely in-elisp way to pass smtp authentication data
>> when sending email.
>
>Sure.  Just add another source for credentials to `auth-source'.

Gotcha.  

I just found the (presumably new) auth-source Info pages, so I'll start
there.  Thanks.

-Karl

Re: Multiple SMTP accounts with smtpmail.el

[Lars Magne Ingebrigtsen]

I've now added a `smtpmail-smtp-user' variable used when searching for
credentials.  This should allow people to hack up their own
message-sending auth mechanisms, as in older versions of Emacs.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Multiple SMTP accounts with smtpmail.el

[Glenn Morris]

Lars Magne Ingebrigtsen wrote:

> I've now added a `smtpmail-smtp-user' variable used when searching for
> credentials.

Please give new (and changed) defcustoms an appropriate "version:" tag;
thanks.

Re: Multiple SMTP accounts with smtpmail.el

[Ted Zlatanov]

On Wed, 17 Aug 2011 23:06:29 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> I think it would be nice to find an auth-source based solution for the
LMI> multiple-account smtpmail case.  Which shouldn't be difficult, I think.
LMI> How about just adding a `smtpmail-user' variable, that when bound or set
LMI> would make smtpmail.el feed that user name to auth-source, and then get
LMI> the correct credentials?

LMI> `smtpmail-user' would be set from `message-send-hook' or the like, of
LMI> course.

It sounds like the search spec passed to `auth-source-search' simply
needs to be modified to use `smtpmail-user' as you suggest.  I see no
problem with that and it should be a trivial change.  Do you want me to
make it or will you?

Thanks
Ted

Re: Multiple SMTP accounts with smtpmail.el

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> It sounds like the search spec passed to `auth-source-search' simply
> needs to be modified to use `smtpmail-user' as you suggest.  I see no
> problem with that and it should be a trivial change.  Do you want me to
> make it or will you?

I already did that.  :-)  It's called `smtpmail-smtp-user', though, to
fit the pattern of the other variable names.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Multiple SMTP accounts with smtpmail.el

[Ted Zlatanov]

On Mon, 26 Sep 2011 20:06:24 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Ted Zlatanov <tzz@lifelogs.com> writes:
>> It sounds like the search spec passed to `auth-source-search' simply
>> needs to be modified to use `smtpmail-user' as you suggest.  I see no
>> problem with that and it should be a trivial change.  Do you want me to
>> make it or will you?

LMI> I already did that.  :-)  It's called `smtpmail-smtp-user', though, to
LMI> fit the pattern of the other variable names.

Thanks!  I stupidly grepped for smtpmail-user so I didn't see your change.

Ted

Re: Multiple SMTP accounts with smtpmail.el

[Rasmus]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
> I think it would be nice to find an auth-source based solution for the
> multiple-account smtpmail case.  Which shouldn't be difficult, I think.
> How about just adding a `smtpmail-user' variable, that when bound or set
> would make smtpmail.el feed that user name to auth-source, and then get
> the correct credentials?
>
> `smtpmail-user' would be set from `message-send-hook' or the like, of
> course.

For me MultipleSMTPAccounts¹ works quite well in Emacs-bzr. SMTP-info is
determined based on From.  From in turn is set explicitly or to the
expected values in e.g. replies.  Psk is fetched from .authinfo.gpg.  At
the moment I have four SMTP servers and six accounts.

Anyhow, I guess a more integrated way of handling SMTP servers would
probably be nice.  I know that very issue kept me away from Gnus for
some years.

–Rasmus


Footnotes: 
¹  http://www.emacswiki.org/emacs/MultipleSMTPAccounts

-- 
Sent from my Emacs

Re: more on starttls, gnutls-cli and using tls for mail

[Chong Yidong]

"T. V. Raman" <tv.raman.tv@gmail.com> writes:

> In function
> (defun network-stream-open-starttls (name buffer host service
> parameters)
> the following let binding forces gnutls-cli -- even though
> starttls-use-gnutls has been set to nil earlier because
> gnutls-cli ws not found on the system.
> (let* ((starttls-use-gnutls t)

Could you elaborate?  That code branch should not be called unless
builtin gnutls is not available:

  (let* (...
	 (builtin-starttls (and (fboundp 'gnutls-available-p)
				(gnutls-available-p)))
      ...
      (unless builtin-starttls
	(delete-process stream)
	(setq start (with-current-buffer buffer (point-max)))
	(let* ((starttls-use-gnutls t)
          ...

> Also, if you ask smtpmail to save the security settings, it
> creates a world-readable .authinfo with the password stored in
> the clear --- looks like a bad idea on all counts.

Yes, this should be fixed.

Re: more on starttls, gnutls-cli and using tls for mail

[Lars Magne Ingebrigtsen]

"T. V. Raman" <tv.raman.tv@gmail.com> writes:

> In function 
> (defun network-stream-open-starttls (name buffer host service
> parameters)
> the following let binding forces gnutls-cli -- even though
> starttls-use-gnutls has been set to nil earlier because
> gnutls-cli ws not found on the system.
> (let* ((starttls-use-gnutls t)

This has now been fixed.

> Also, if you ask smtpmail to save the security settings, it
> creates a world-readable .authinfo with the password stored in
> the clear --- looks like a bad idea on all counts.

Oops.  I've now changed the permissions to #o600.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

more on starttls, gnutls-cli and using tls for mail

[raman]

Karl,

Iagree with you. Also, would be nice to encrypt .authinfo using
epg for extra strennnngth.
-- 
Best Regards,
--raman

-- 
Best Regards,
--raman