Emacs RPC

Emacs RPC

[Lars Magne Ingebrigtsen]

I'm running a lot of Emacs-based servers here and there, and I
communicate with them via "emacsclient --eval".  But it just occurred to
me that it would be much more elegant to have an `emacs-client-rpc'
function, so that I wouldn't have to parse the output from emacsclient
in a shell.

I've peeked through the info files and I couldn't find anything.  And it
looks like it would be a simple enough thing to add to server.el.
Anybody mind if I add a function like, er,

(server-eval-at HOST COMMAND)

to server.el?

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Emacs RPC

[T.V. Raman]

1+, I'd love to see such a function, and can think of many places
where I would use it --- given that emacs is my primary (err
only)user environment for doing *everything*.
-- 
Best Regards,
--raman

-- 
Best Regards,
--raman


On 4/23/11, Lars Magne Ingebrigtsen <larsi@gnus.org> wrote:
> I'm running a lot of Emacs-based servers here and there, and I
> communicate with them via "emacsclient --eval".  But it just occurred to
> me that it would be much more elegant to have an `emacs-client-rpc'
> function, so that I wouldn't have to parse the output from emacsclient
> in a shell.
>
> I've peeked through the info files and I couldn't find anything.  And it
> looks like it would be a simple enough thing to add to server.el.
> Anybody mind if I add a function like, er,
>
> (server-eval-at HOST COMMAND)
>
> to server.el?
>
> --
> (domestic pets only, the antidote for overdose, milk.)
>   bloggy blog http://lars.ingebrigtsen.no/
>
>
>

Re: Emacs RPC

[Richard Stallman]

If we implement this, its documentation should inform users that
intimate communication between Emacs and other code might imply that
they are one program and that the the other program must be covered by
GPLv3.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org, www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.

Re: Emacs RPC

[Lars Magne Ingebrigtsen]

Richard Stallman <rms@gnu.org> writes:

> If we implement this, its documentation should inform users that
> intimate communication between Emacs and other code might imply that
> they are one program and that the the other program must be covered by
> GPLv3.

The other program will be (a different) Emacs, so I'm not sure I
understand.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Emacs RPC

[Richard Stallman]

    > If we implement this, its documentation should inform users that
    > intimate communication between Emacs and other code might imply that
    > they are one program and that the the other program must be covered by
    > GPLv3.

    The other program will be (a different) Emacs, so I'm not sure I
    understand.

I guess I don't understand either.  I thought you were talking about
RPC through emacsclient.  Are you saying one Emacs process would
run emacsclient to execute something in another Emacs process?


-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org, www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.

Re: Emacs RPC

[Lars Magne Ingebrigtsen]

Richard Stallman <rms@gnu.org> writes:

> I guess I don't understand either.  I thought you were talking about
> RPC through emacsclient.  Are you saying one Emacs process would
> run emacsclient to execute something in another Emacs process?

No, I'm talking about having an in-Emacs `server-eval-on' (or something)
function that would work (in principle) the same way as forking
emacsclient, but would have a bit more machinery around it so that you
could, basically, exchange any instance of `eval' with `server-eval-on'
and expect it to work.

So it would not provide anything new in principle, but it would be more
convenient to use.

I'll get around to implementing it one of these months, unless somebody
else does it first.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Emacs RPC

[Lars Magne Ingebrigtsen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> I'll get around to implementing it one of these months, unless somebody
> else does it first.

The months came pretty quickly.

(server-eval-at "rocket-sam" '(+ 1 2))
=> 3

I've just implemented it in a simple reverse-engineered "emacsclient
-eval" fashion, and works fine in my use cases.

But it needs some thought on how to handle non-ASCII stuff, since

(server-eval-at "rocket-sam" '(format "héllo"))

does not give satisfactory results, but I'm not quite sure what the
right approach here would be...

Perhaps it needs to send over

(encode-coding-string (format "%S" (eval '(format "héllo"))) 'utf-8)

and decode it the same way to be sure that both sides agree what the
coding system is?  Hm...  would that even work?

Anyway, I've checked it in, so feel free to make it better.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Emacs RPC

[Chong Yidong]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> (server-eval-at "rocket-sam" '(+ 1 2))
> => 3
>
> I've just implemented it in a simple reverse-engineered "emacsclient
> -eval" fashion, and works fine in my use cases.

Please add a NEWS entry.  Thanks.

Re: Emacs RPC

[Lars Magne Ingebrigtsen]

Chong Yidong <cyd@stupidchicken.com> writes:

> Please add a NEWS entry.  Thanks.

Ok, done.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Emacs RPC

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 1427 bytes --]

On 4/24/11 1:04 PM, Richard Stallman wrote:
> If we implement this, its documentation should inform users that
> intimate communication between Emacs and other code might imply that
> they are one program and that the the other program must be covered by
> GPLv3.

I realize it's difficult to ask this question without expecting to spark
controversy is a bit like shooting an Archduke in the Balkans without
expecting war, but:

Doesn't the "derivative work" boundary end at the IPC level? If I have a
GPLv3ed web server, clients don't become bound by the GPLv3.  If I have
a GPLv3 web *service* that provides services over SOAP (a form of RPC),
users of that web service are not automatically bound by the GPLv3 ---
that's why we have the AGPL, after all. Linux programs aren't bound by
that kernel's GPLv2 license, nor are programs that interact with the
GPLv3d Samba.

So, why would it be the case that clients of some Emacs RPC become bound
by the GPL? I realize that a degree of "intimacy" can bring a system
over the license event horizon, but it's not clear to me that there's
any bright line we know we've crossed. Like an astrophysical event
horizon, it seems like you can only know you've crossed it after you've
tried to escape and failed.

On a more practical basis, some programs, I imagine, use emacsclient's
eval feature to run elisp today. Is every such user today bound the GPLv3?


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Emacs RPC

[Richard Stallman]

    Doesn't the "derivative work" boundary end at the IPC level?

In the common cases, we think it does, but that is not an ironclad
rule.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org, www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.

Re: Emacs RPC

[Chong Yidong]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> I'm running a lot of Emacs-based servers here and there, and I
> communicate with them via "emacsclient --eval".  But it just occurred to
> me that it would be much more elegant to have an `emacs-client-rpc'
> function, so that I wouldn't have to parse the output from emacsclient
> in a shell.
>
> I've peeked through the info files and I couldn't find anything.  And it
> looks like it would be a simple enough thing to add to server.el.
> Anybody mind if I add a function like, er,
>
> (server-eval-at HOST COMMAND)
>
> to server.el?

Sounds fine in principle, but could you provide more details by posting
the docstring for the proposed function?  For instance, is COMMAND
supposed to be a Lisp form, or a string?

Re: Emacs RPC

[Lars Magne Ingebrigtsen]

Chong Yidong <cyd@stupidchicken.com> writes:

> Sounds fine in principle, but could you provide more details by posting
> the docstring for the proposed function?  For instance, is COMMAND
> supposed to be a Lisp form, or a string?

Originally I was thinking a string (since that's what emacsclient does),
but I now feel that a Lisp form would be more useful and RPC-ey, and I
think the return value should also be a Lisp form.

That is, you'd say

(server-eval-at "foo" '(+ 1 2))

and get back

3

That is, `server-eval-at' will do a `read-from-string' on the output it
gets back from the server, basically.

Perhaps `eval-at' would be a better name, though?

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Emacs RPC

[Chong Yidong]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Originally I was thinking a string (since that's what emacsclient does),
> but I now feel that a Lisp form would be more useful and RPC-ey, and I
> think the return value should also be a Lisp form.

Sounds reasonable.

> Perhaps `eval-at' would be a better name, though?

No, we want to maintain the prefix conventions.  Maybe something like
server-remote-eval.

Re: Emacs RPC

[Ted Zlatanov]

On Sun, 24 Apr 2011 20:00:55 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Originally I was thinking a string (since that's what emacsclient does),
LMI> but I now feel that a Lisp form would be more useful and RPC-ey, and I
LMI> think the return value should also be a Lisp form.

LMI> That is, you'd say

LMI> (server-eval-at "foo" '(+ 1 2))

LMI> and get back

LMI> 3

LMI> That is, `server-eval-at' will do a `read-from-string' on the output it
LMI> gets back from the server, basically.

LMI> Perhaps `eval-at' would be a better name, though?

Please, please implement this securely from the start.  emacsclient is
terribly insecure and we don't need to repeat that.

The communication itself doesn't have to be secure, only signed.  So the
signature could be as simple as a MD5 hash of the data concatenated with
a secret, or a full-blown GPG signature.

You could also use the GnuTLS server facilities (very similar to the
existing client facilities) to check the certificates mutually and
encrypt the connection.  Then you don't need signatures on the content.
The client has to have a client-side SSL certificate to present to the
server, and the server's certificate is checked by the client as well.

Whether you choose to use GnuTLS or something simpler, I hope you agree
there should be something better than "just eval some code and trust
everyone is good" for this facility.

Ted

Re: Emacs RPC

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> Please, please implement this securely from the start.  emacsclient is
> terribly insecure and we don't need to repeat that.

The network connection isn't encrypted, but how is emacsclient insecure
otherwise?  You have to pass the (shared) secret to the server to get it
to do anything.

In any case, it's rather an orthogonal issue.  I see no reason why the
in-Emacs RPC should be more secure than the command line RPC.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Emacs RPC

[Ted Zlatanov]

On Mon, 25 Apr 2011 03:26:46 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Ted Zlatanov <tzz@lifelogs.com> writes:
>> Please, please implement this securely from the start.  emacsclient is
>> terribly insecure and we don't need to repeat that.

LMI> The network connection isn't encrypted, but how is emacsclient insecure
LMI> otherwise?  You have to pass the (shared) secret to the server to get it
LMI> to do anything.

The server executes anything passed to it.  There's no verification that
the content came from a trusted client beyond the shared secret.  I'm
suggesting more granular permissions that work across operating systems
and don't require NFS mounts (have you ever set up a NFS mount to a W32
machine?  it's painful... and expensive...)

The permissions should be applied at the procedure level, so different
procedures can be authorized and authenticated differently.

But encryption would be nice too, hence my GnuTLS suggestion.  90% of
the code is already in Emacs.

LMI> In any case, it's rather an orthogonal issue.  I see no reason why the
LMI> in-Emacs RPC should be more secure than the command line RPC.

...because the command-line RPC can then use the in-Emacs RPC and
improve security for everyone.  It would (IMHO) make emacsclient and
in-Emacs RPC real, viable RPC mechanisms.  Right now they can't be.

Ted

Re: Emacs RPC

[Stefan Monnier]

> Please, please implement this securely from the start.  emacsclient is
> terribly insecure and we don't need to repeat that.

Lars's proposal has nothing to do with the network communication level.
So, please take this security issue to another thread.


        Stefan

Re: Emacs RPC

[Stefan Monnier]

Sounds fine by me.  It will end up duplicating emacsclient.c's code in
server.el, which is too bad, but largely unavoidable a long as we don't
want to use `emacs' as the standard client.


        Stefan

Emacs RPC security (was: Emacs RPC)

[Ted Zlatanov]

On Sat, 23 Apr 2011 20:54:28 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> I'm running a lot of Emacs-based servers here and there, and I
LMI> communicate with them via "emacsclient --eval".  But it just occurred to
LMI> me that it would be much more elegant to have an `emacs-client-rpc'
LMI> function, so that I wouldn't have to parse the output from emacsclient
LMI> in a shell.

On Mon, 25 Apr 2011 09:57:15 -0300 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

>> Please, please implement this securely from the start.  emacsclient is
>> terribly insecure and we don't need to repeat that.

SM> Lars's proposal has nothing to do with the network communication level.

If we're going to provide *RPC*, we should worry about security at all
levels, not just at the transport level.  Otherwise it's just "run any
code remotely on an Emacs instance" which doesn't sound as fun, right?

SM> So, please take this security issue to another thread.

I've changed the subject as requested.

I think RPC support needs two things:

1) authentication: the server should be able to verify the client's
identity and the client should be able to verify the server's identity.
This can be accomplished with SSL certificates and GnuTLS or by signing
each message.

2) authorization: the server should be able to associate each client
identity with only certain functions it can invoke directly.

The above should work across platforms and between any two networked
machines with as few external dependencies as possible.  I think the
GnuTLS support can handle (1), plus it can encrypt the network traffic
which improves security.  (2) needs to happen in the server's eval loop.

Ted

Re: Emacs RPC security

[Stefan Monnier]

>>> Please, please implement this securely from the start.  emacsclient is
>>> terribly insecure and we don't need to repeat that.
SM> Lars's proposal has nothing to do with the network communication level.
> If we're going to provide *RPC*, we should worry about security at all
> levels, not just at the transport level.  Otherwise it's just "run any
> code remotely on an Emacs instance" which doesn't sound as fun, right?

Still unrelated to Lars's proposal.  The corresponding security problem
already exists since Emacs-22.

> 1) authentication: the server should be able to verify the client's
> identity and the client should be able to verify the server's identity.
> This can be accomplished with SSL certificates and GnuTLS or by signing
> each message.

We currently have that via xauth-style cookies for TCP and via
Unix-based access rights for Unix sockets.

Using GnuTLS for the TCP connections could be a good idea as well:
patches welcome.

> 2) authorization: the server should be able to associate each client
> identity with only certain functions it can invoke directly.

When such a need will arise, we will think about it.  In all the cases
I've seen until now, the Emacs server is only used by the same user as
the client, so there's not much point making the security structure
so complicated, right now.


        Stefan

Re: Emacs RPC security

[Ted Zlatanov]

On Mon, 25 Apr 2011 14:35:49 -0300 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 
SM> Ted wrote:
>> 1) authentication: the server should be able to verify the client's
>> identity and the client should be able to verify the server's identity.
>> This can be accomplished with SSL certificates and GnuTLS or by signing
>> each message.

SM> Using GnuTLS for the TCP connections could be a good idea as well:
SM> patches welcome.

I will put server GnuTLS support in Emacs on my TODO list, but it will
take a while.  I hope you consider it important.

>> 2) authorization: the server should be able to associate each client
>> identity with only certain functions it can invoke directly.

SM> When such a need will arise, we will think about it.  In all the cases
SM> I've seen until now, the Emacs server is only used by the same user as
SM> the client, so there's not much point making the security structure
SM> so complicated, right now.

Of course, since the security is so weak right now, no one is using it
outside a limited one-user so you haven't seen any unusual cases.  I
would use it personally as a remote password server so all my
auth-source data doesn't live on all the machines I use.  I would also
use it to implement a remote synchronization facility for Gnus and BBDB.

Ted

Re: Emacs RPC security

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 1666 bytes --]

On 4/25/11 11:02 AM, Ted Zlatanov wrote:
> On Mon, 25 Apr 2011 14:35:49 -0300 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 
> SM> Ted wrote:
>>> 1) authentication: the server should be able to verify the client's
>>> identity and the client should be able to verify the server's identity.
>>> This can be accomplished with SSL certificates and GnuTLS or by signing
>>> each message.
> 
> SM> Using GnuTLS for the TCP connections could be a good idea as well:
> SM> patches welcome.
> 
> I will put server GnuTLS support in Emacs on my TODO list, but it will
> take a while.  I hope you consider it important.
> 
>>> 2) authorization: the server should be able to associate each client
>>> identity with only certain functions it can invoke directly.
> 
> SM> When such a need will arise, we will think about it.  In all the cases
> SM> I've seen until now, the Emacs server is only used by the same user as
> SM> the client, so there's not much point making the security structure
> SM> so complicated, right now.
> 
> Of course, since the security is so weak right now, no one is using it
> outside a limited one-user so you haven't seen any unusual cases.  I
> would use it personally as a remote password server so all my
> auth-source data doesn't live on all the machines I use.  I would also
> use it to implement a remote synchronization facility for Gnus and BBDB.

That's a fine goal, but you don't need to implement the requisite
security in Emacs proper.  stunnel will give you a secure channel and,
with client certificates, can authenticate both parties.  I'd prefer not
to have a GnuTLS server in Emacs right now.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Emacs RPC security

[Ted Zlatanov]

On Mon, 25 Apr 2011 11:17:02 -0700 Daniel Colascione <dan.colascione@gmail.com> wrote: 

DC> On 4/25/11 11:02 AM, Ted Zlatanov wrote:

>> Of course, since the security is so weak right now, no one is using it
>> outside a limited one-user so you haven't seen any unusual cases.  I
>> would use it personally as a remote password server so all my
>> auth-source data doesn't live on all the machines I use.  I would also
>> use it to implement a remote synchronization facility for Gnus and BBDB.

DC> That's a fine goal, but you don't need to implement the requisite
DC> security in Emacs proper.  stunnel will give you a secure channel and,
DC> with client certificates, can authenticate both parties.  

I'd rather not rely on stunnel or any other external utilities.  My
experience with supporting them with Gnus, especially for W32 users, has
been painful.

DC> I'd prefer not to have a GnuTLS server in Emacs right now.

Even if stunnel works for some cases, I don't see why you're against a
built-in GnuTLS server now.  Are you concerned about performance and
memory usage, code bloat and maintenance cost, security issues,
documentation, user confusion, or something else?  Or do you mean you
want to delay the functionality until something else is done?

Thanks
Ted

Re: Emacs RPC security

[Stefan Monnier]

> Of course, since the security is so weak right now, no one is using it
> outside a limited one-user so you haven't seen any unusual cases.  I
> would use it personally as a remote password server so all my
> auth-source data doesn't live on all the machines I use.  I would also
> use it to implement a remote synchronization facility for Gnus and BBDB.

Still sound like "single user" cases to me.


        Stefan

Re: Emacs RPC security

[Ted Zlatanov]

On Mon, 25 Apr 2011 15:38:44 -0300 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

>> Of course, since the security is so weak right now, no one is using it
>> outside a limited one-user so you haven't seen any unusual cases.  I
>> would use it personally as a remote password server so all my
>> auth-source data doesn't live on all the machines I use.  I would also
>> use it to implement a remote synchronization facility for Gnus and BBDB.

SM> Still sound like "single user" cases to me.

The sync facility will be for general use by multiple users, not personal.

Ted

Re: Emacs RPC security

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> I will put server GnuTLS support in Emacs on my TODO list, but it will
> take a while.

Having a GnuTLS server in Emacs would be nice.  (But, like Stefan said,
pretty much orthogonal to my emacs-server wrapper thing, which would be
used to communicate with, for instance, Emacs 22 instances.)

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Emacs RPC security

[Lars Magne Ingebrigtsen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Having a GnuTLS server in Emacs would be nice.

I just had a horrible idea.

I converted pop3.el to use opportunistic STARTTLS upgrades now (one less
thing on my imaginary todo list -- only googleplex more to go), and it
occurred to me that the Emacs Server could use STARTTLS too.

Today you just send the shared secret and then the command, but we could
easily implement a CAPABILITY command, and offer STARTTLS and thereby
allow forward-and-backward compatibility between encrypted and
non-encrypted clients and servers.  :-)

Anyway, I'm not going to tackle that, but just an idea.

Hm...  perhaps I should convert smtpmail.el to use opportunistic
STARTTLS while I'm at it.  Is that the only (major) network library that
has escaped opportunist encryption so far?

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Opportunistic STARTTLS in smtpmail.el (was: Emacs RPC security)

[Lars Magne Ingebrigtsen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Hm...  perhaps I should convert smtpmail.el to use opportunistic
> STARTTLS while I'm at it. 

Oh, now I remember why I didn't do the smtpmail.el `open-network-stream'
conversion the last time.

smtpmail.el provides an option to pass TLS credentials to the server via
switches like 

"--x509keyfile" "--x509certfile"

to gnutlc-cli.  `open-network-stream' has no concept of these things,
and I'm not sure gnutls.c has, either.  Ted?

If gnutls.c has, I can extend `open-network-stream' to take keywords for
the keyfile and the certfile, if that is the way we want to go.  Or
perhaps add a global variable like `smtpmail-starttls-credentials',

;;(setq smtpmail-starttls-credentials
;;      '(("YOUR SMTP HOST" 25 "~/.my_smtp_tls.key" "~/.my_smtp_tls.cert")))

but call it `network-tls-credentials', and have `open-network-stream'
deal with this stuff itself -- if it's doing a STARTTLS or a TLS
connection, is can consult the `network-tls-credential' variable, see if
it finds a match, and then feed the required data to
starttls.el/tls.el/gnutls.c.  (*Phew*.)

But I'm wondering -- does anybody use this credential stuff for talking
to their SMTP servers?

I'd rather just delete `smtpmail-starttls-credentials' and pretend that
I've never heard about it.

Opinions, please...

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Opportunistic STARTTLS in smtpmail.el

[James Cloos]

>>>>> "LMI" == Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

LMI> But I'm wondering -- does anybody use this credential stuff for
LMI> talking to their SMTP servers?

It is likely that users in some corporate settings need to if they
want to send mail from Emacs.

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 1024D/ED7DAEA6

Re: Opportunistic STARTTLS in smtpmail.el

[Ted Zlatanov]

On Mon, 02 May 2011 00:19:18 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
>> Hm...  perhaps I should convert smtpmail.el to use opportunistic
>> STARTTLS while I'm at it. 

LMI> Oh, now I remember why I didn't do the smtpmail.el `open-network-stream'
LMI> conversion the last time.

LMI> smtpmail.el provides an option to pass TLS credentials to the server via
LMI> switches like 

LMI> "--x509keyfile" "--x509certfile"

LMI> to gnutlc-cli.  `open-network-stream' has no concept of these things,
LMI> and I'm not sure gnutls.c has, either.  Ted?

Yes, definitely, with the :keyfiles and :trustfiles parameters to
`gnutls-boot'.  But I haven't tested that much (the functionality is on
the GnuTLS side in any case).  So you'd need some dynamically-scoped
global variables like `gnutls-keyfiles' and `gnutls-trustfiles' to hold
these, so they can be overridden as needed.

LMI> If gnutls.c has, I can extend `open-network-stream' to take keywords for
LMI> the keyfile and the certfile, if that is the way we want to go.  Or
LMI> perhaps add a global variable like `smtpmail-starttls-credentials',

LMI> ;;(setq smtpmail-starttls-credentials
LMI> ;;      '(("YOUR SMTP HOST" 25 "~/.my_smtp_tls.key" "~/.my_smtp_tls.cert")))

LMI> but call it `network-tls-credentials', and have `open-network-stream'
LMI> deal with this stuff itself -- if it's doing a STARTTLS or a TLS
LMI> connection, is can consult the `network-tls-credential' variable, see if
LMI> it finds a match, and then feed the required data to
LMI> starttls.el/tls.el/gnutls.c.  (*Phew*.)

This is all nasty, nasty for the user.  The whole
`smtpmail-starttls-credentials' structure can be replaced with
`auth-source-search' calls for all possible use cases.  The user can
say, for instance:

machine mysmtpserver.com login tzz password mypassword keyfile "~/.keyfile"

LMI> But I'm wondering -- does anybody use this credential stuff for talking
LMI> to their SMTP servers?

LMI> I'd rather just delete `smtpmail-starttls-credentials' and pretend that
LMI> I've never heard about it.

Credentials are useful.  Move them to auth-source.

Ted

Re: Opportunistic STARTTLS in smtpmail.el

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> LMI> "--x509keyfile" "--x509certfile"
>
> LMI> to gnutlc-cli.  `open-network-stream' has no concept of these things,
> LMI> and I'm not sure gnutls.c has, either.  Ted?
>
> Yes, definitely, with the :keyfiles and :trustfiles parameters to
> `gnutls-boot'.

Right.  Would "--x509keyfile" correspond to :keyfiles and
"--x509certfile" to :trustfiles?

> This is all nasty, nasty for the user.  The whole
> `smtpmail-starttls-credentials' structure can be replaced with
> `auth-source-search' calls for all possible use cases.  The user can
> say, for instance:
>
> machine mysmtpserver.com login tzz password mypassword keyfile "~/.keyfile"

Yes, that makes a whole lot more sense.  Hm...  but on what level would
this be checked?  `open-network-stream' could do that, but if the auth
file is a .gpg file, it'll have to ask for a password just to check
whether there is a keyfile, which, in 99.99% of the cases there won't
be.

Uhm.  How did that discussion about non-secret credentials go?  :-)

It wouldn't be backwards-compatible in any case, though -- anybody mind
if I break that for smtpmail.el?

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Opportunistic STARTTLS in smtpmail.el

[Ted Zlatanov]

On Mon, 02 May 2011 20:59:18 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Ted Zlatanov <tzz@lifelogs.com> writes:
LMI> "--x509keyfile" "--x509certfile"
>> 
LMI> to gnutlc-cli.  `open-network-stream' has no concept of these things,
LMI> and I'm not sure gnutls.c has, either.  Ted?
>> 
>> Yes, definitely, with the :keyfiles and :trustfiles parameters to
>> `gnutls-boot'.

LMI> Right.  Would "--x509keyfile" correspond to :keyfiles and
LMI> "--x509certfile" to :trustfiles?

Oh wait, I think I'm wrong.  The key+cert files (client-side SSL certs)
are not the same as the trust files (which verify the server's SSL
cert).  Let me take a look, this may require another parameter or I'm
missing something.

>> This is all nasty, nasty for the user.  The whole
>> `smtpmail-starttls-credentials' structure can be replaced with
>> `auth-source-search' calls for all possible use cases.  The user can
>> say, for instance:
>> 
>> machine mysmtpserver.com login tzz password mypassword keyfile "~/.keyfile"

LMI> Yes, that makes a whole lot more sense.  Hm...  but on what level would
LMI> this be checked?  `open-network-stream' could do that, but if the auth
LMI> file is a .gpg file, it'll have to ask for a password just to check
LMI> whether there is a keyfile, which, in 99.99% of the cases there won't
LMI> be.

There's no problem with specifying an unencrypted authinfo file for a
specific server+port+user (or any subset) combination, see
`auth-sources'.  So the authinfo line would look like this:

machine mysmtpserver.com login tzz password mypassword keyfile "~/.keyfile" certfile "~/.certfile"

LMI> Uhm.  How did that discussion about non-secret credentials go?  :-)

Look!  It's Elvis! (runs away)

Ted

Re: Opportunistic STARTTLS in smtpmail.el

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> There's no problem with specifying an unencrypted authinfo file for a
> specific server+port+user (or any subset) combination, see
> `auth-sources'.  So the authinfo line would look like this:
>
> machine mysmtpserver.com login tzz password mypassword keyfile "~/.keyfile" certfile "~/.certfile"

But if you do have a ~/.authinfo.gpg, then auth-source will be opening
it just to check whether there are any mysmtpserver.com entries in it.

> LMI> Uhm.  How did that discussion about non-secret credentials go?  :-)
>
> Look!  It's Elvis! (runs away)

That's how I thought it went.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Opportunistic STARTTLS in smtpmail.el

[Ted Zlatanov]

On Tue, 03 May 2011 01:36:47 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Ted Zlatanov <tzz@lifelogs.com> writes:
>> There's no problem with specifying an unencrypted authinfo file for a
>> specific server+port+user (or any subset) combination, see
>> `auth-sources'.  So the authinfo line would look like this:
>> 
>> machine mysmtpserver.com login tzz password mypassword keyfile "~/.keyfile" certfile "~/.certfile"

LMI> But if you do have a ~/.authinfo.gpg, then auth-source will be opening
LMI> it just to check whether there are any mysmtpserver.com entries in
LMI> it.

Only once, then it's cached.  Plus you can specify an unencrypted file
at any point.  I really don't think this is a big deal, no more than
what happens for IMAP and NNTP.

If you insist on avoiding this file check, we could have a "Lisp data"
backend in addition to the file and Secrets backends.  That would be
pretty trivial to implement and would mirror the existing netrc parse
results structurally.

Ted

Re: Opportunistic STARTTLS in smtpmail.el

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> If you insist on avoiding this file check, we could have a "Lisp data"
> backend in addition to the file and Secrets backends.  That would be
> pretty trivial to implement and would mirror the existing netrc parse
> results structurally.

I think the idea of putting this stuff in auth-source is good, but I'm
just wondering whether we could have a meaningful separation of secret
credentials (i.e., passwords and user names) and non-secret credentials
(key files to be used, in this instance).

I think putting stuff like key files into ~/.authinfo is fine.  But if
the user has a ~/.authinfo.gpg file (for IMAP use, for instance), and
smtpmail wants to see whether a key file is to be used, there should be
a way for smtpmail to get at this information without typing the .gpg
password.

I don't really see how that would work in any convenient way with our
current interfaces.  smtpmail really wants to say "check whether this
exists, but don't try too hard", sort of.

I think the previous discussion along these lines led us to the idea of
having two files: ~/.authinfo for all the non-secret data, and
~/.authinfo.gpg for the passwords themselves.  I don't think anybody
really were much in favour of that split, since it meant duplicating
data a lot.

But I don't think anybody has really had a better idea, so perhaps we
should just go with that idea, anyway.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Opportunistic STARTTLS in smtpmail.el

[Ted Zlatanov]

On Tue, 03 May 2011 03:01:40 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Ted Zlatanov <tzz@lifelogs.com> writes:
>> If you insist on avoiding this file check, we could have a "Lisp data"
>> backend in addition to the file and Secrets backends.  That would be
>> pretty trivial to implement and would mirror the existing netrc parse
>> results structurally.

LMI> I think the idea of putting this stuff in auth-source is good, but I'm
LMI> just wondering whether we could have a meaningful separation of secret
LMI> credentials (i.e., passwords and user names) and non-secret credentials
LMI> (key files to be used, in this instance).

LMI> I think putting stuff like key files into ~/.authinfo is fine.  But if
LMI> the user has a ~/.authinfo.gpg file (for IMAP use, for instance), and
LMI> smtpmail wants to see whether a key file is to be used, there should be
LMI> a way for smtpmail to get at this information without typing the .gpg
LMI> password.

LMI> I don't really see how that would work in any convenient way with our
LMI> current interfaces.  smtpmail really wants to say "check whether this
LMI> exists, but don't try too hard", sort of.

What's wrong with a pure data backend?  It would be specified in
`auth-sources', basically an inlined netrc parse which could be
generated by an assistant.el walk-through!

Any non-secret data can be specified inline.  Then we won't need any
file splitting, though the users can choose to put non-secret data in
a file just the same.

For example:

(setq auth-sources '((:source (:user tzz :keyfile "mykeyfile" :host "myhost" :port 587))
                     "~/.authinfo.gpg"))

I think that's cleaner since the inlined data maps nicely to the netrc format.

Ted

Re: Opportunistic STARTTLS in smtpmail.el

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> For example:
>
> (setq auth-sources '((:source (:user tzz :keyfile "mykeyfile" :host "myhost" :port 587))
>                      "~/.authinfo.gpg"))
>
> I think that's cleaner since the inlined data maps nicely to the netrc format.

Won't this still require opening the ~/.authinfo.gpg file, or does it
stop searching after you've find the first match?

Anyway, I don't really like having long, complicated user-exposed
variables.  Users usually mess them up.  Putting stuff like this in a
file seems like a nice feature.  

Another idea occurred to me based on the /etc/passwd + /etc/secret
split, plus the password in-memory obfuscation code.  :-)

That is, if we allow lines like

machine smtp.mail.host login foo password .secrets.gpg:smtp1 port smtp keyfile mykeyfile

in ~/.authinfo and then have a ~/.secrets.gpg file with

smtp1 password bar

we could allow mixing the queries for open and secret credentials.

Let me explain.

The typical usage will be

(auth-source-search :host "smtp.mail.hos" :port "smtp")

which would return an auth-source object, but will not read
~/.secrets.gpg.  If we look at elements like :keyfile, we'll find the
:keyfile element.  If, however, we try to access the :password element,
auth-source.el will *then* open ~/.secrets.gpg, read it, and return the
password.

So we defer reading the ~/.secrets.gpg file to the very last possible
moment -- which is when we know that we actually need it.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Opportunistic STARTTLS in smtpmail.el

[Ted Zlatanov]

On Wed, 04 May 2011 00:04:06 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Ted Zlatanov <tzz@lifelogs.com> writes:
>> For example:
>> 
>> (setq auth-sources '((:source (:user tzz :keyfile "mykeyfile" :host "myhost" :port 587))
>> "~/.authinfo.gpg"))
>> 
>> I think that's cleaner since the inlined data maps nicely to the netrc format.

LMI> Won't this still require opening the ~/.authinfo.gpg file, or does it
LMI> stop searching after you've find the first match?

With :max 1 it will stop after the first match.  This is why I usually
specify :max 1 when I use `auth-search'.

LMI> Anyway, I don't really like having long, complicated user-exposed
LMI> variables.  Users usually mess them up.  Putting stuff like this in a
LMI> file seems like a nice feature.  

OK.

LMI> That is, if we allow lines like

LMI> machine smtp.mail.host login foo password .secrets.gpg:smtp1 port smtp keyfile mykeyfile

LMI> in ~/.authinfo and then have a ~/.secrets.gpg file with

LMI> smtp1 password bar

LMI> we could allow mixing the queries for open and secret credentials.

I see.  I think we can avoid this kind of complication:

LMI> The typical usage will be

LMI> (auth-source-search :host "smtp.mail.hos" :port "smtp")

LMI> which would return an auth-source object, but will not read
LMI> ~/.secrets.gpg.  If we look at elements like :keyfile, we'll find the
LMI> :keyfile element.  If, however, we try to access the :password element,
LMI> auth-source.el will *then* open ~/.secrets.gpg, read it, and return the
LMI> password.

LMI> So we defer reading the ~/.secrets.gpg file to the very last possible
LMI> moment -- which is when we know that we actually need it.

Let the user choose.  The query is:

(auth-source-search :host "smtp.mail.hos" :port "smtp" :keyfile t :max 1)

to find the first entry that has a keyfile and

(auth-source-search :host "smtp.mail.hos" :port "smtp" :secret t :max 1)

to find the first entry that has a secret.  So these two lines:

machine smtp.mail.hos port smtp keyfile xyz
machine smtp.mail.hos port smtp password mypass login myuser

can be separated into a plain file and an encrypted file, or combined:

machine smtp.mail.hos port smtp password mypass login myuser keyfile xyz

which as a single line can live in a plain file or an encrypted file.

Ted

Re: Opportunistic STARTTLS in smtpmail.el

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> Let the user choose.  The query is:
>
> (auth-source-search :host "smtp.mail.hos" :port "smtp" :keyfile t :max 1)
>
> to find the first entry that has a keyfile and
>
> (auth-source-search :host "smtp.mail.hos" :port "smtp" :secret t :max 1)
>
> to find the first entry that has a secret.  So these two lines:
>
> machine smtp.mail.hos port smtp keyfile xyz
> machine smtp.mail.hos port smtp password mypass login myuser
>
> can be separated into a plain file and an encrypted file, or combined:

I want this to be non-annoying for the typical user without the user
having to do anything.  :-)

If the user has a ~/.authinfo.gpg file, then querying for the SMTP
credentials (which 99.9% of people don't have) will require them to type
a password, which is something that they didn't have to do before.

So I want a way to ask auth-info "is there even something that vaguely
applies to me here?" without having to type a password.  Because I know
that for almost everybody, there won't be anything there.

But perhaps that can be an smtpmail.el configuration thing.  But I'd
hoped to avoid that, but perhaps that isn't in the cards.  SMTP is kinda
special amongst all the protocols, since all the other ones are pretty
much guaranteed to need credentials, while SMTP rarely uses it.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Opportunistic STARTTLS in smtpmail.el

[Robert Pluim]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

>
> But perhaps that can be an smtpmail.el configuration thing.  But I'd
> hoped to avoid that, but perhaps that isn't in the cards.  SMTP is kinda
> special amongst all the protocols, since all the other ones are pretty
> much guaranteed to need credentials, while SMTP rarely uses it.

Where have you been whilst spam has been conquering the world? ;)

I have access to 3 separate SMTP servers, all of which require
authentication, and it has been normal in every company I've worked in
for the last half decade at least.

Robert

Re: Opportunistic STARTTLS in smtpmail.el

[Lars Magne Ingebrigtsen]

Robert Pluim <rpluim@gmail.com> writes:

> Where have you been whilst spam has been conquering the world? ;)

I should have been on Hawaii...

> I have access to 3 separate SMTP servers, all of which require
> authentication, and it has been normal in every company I've worked in
> for the last half decade at least.

My ISP at home doesn't require auth for SMTP -- it's all IP based.  So
is the work SMTP server...

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Opportunistic STARTTLS in smtpmail.el

[Robert Pluim]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Robert Pluim <rpluim@gmail.com> writes:
>
>> Where have you been whilst spam has been conquering the world? ;)
>
> I should have been on Hawaii...
>
>> I have access to 3 separate SMTP servers, all of which require
>> authentication, and it has been normal in every company I've worked in
>> for the last half decade at least.
>
> My ISP at home doesn't require auth for SMTP -- it's all IP based.  So
> is the work SMTP server...

You Northern Europeans. So trusting... :) (or maybe my admins have been
paranoid)

Seriously, I think specifying the username/password for smtp access
needs to be fairly painless, and potentially automatic: it's not going
to decrease in use. I'd prefer it to stay in .authinfo if possible,
forcing people to .authinfo.gpg is too much.

Robert

Re: Opportunistic STARTTLS in smtpmail.el

[Stefan Monnier]

>> But perhaps that can be an smtpmail.el configuration thing.  But I'd
>> hoped to avoid that, but perhaps that isn't in the cards.  SMTP is kinda
>> special amongst all the protocols, since all the other ones are pretty
>> much guaranteed to need credentials, while SMTP rarely uses it.
> Where have you been whilst spam has been conquering the world? ;)
> I have access to 3 separate SMTP servers, all of which require
> authentication, and it has been normal in every company I've worked in
> for the last half decade at least.

It's still very common for SMTP servers to use the IP address instead of
authentication credentials, and having to enter a password each time in
order for smtpmail to discover that no authentication was needed is
*really* annoying.


        Stefan

Re: Opportunistic STARTTLS in smtpmail.el

[Lars Magne Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> It's still very common for SMTP servers to use the IP address instead of
> authentication credentials, and having to enter a password each time in
> order for smtpmail to discover that no authentication was needed is
> *really* annoying.

Yup.

A yucky work-around (which (some versions of) Thunderbird uses, I think)
is that if the SMTP server announces that it accepts auth, then
Thunderbird prompts the user for the auth.

However, some SMTP servers announce auth even though they don't require
auth (from all users), so it's icky.

So I'd really prefer a way to query whether credentials exists without
having to enter a password.  Which I think means that an /etc/passwd +
/etc/secrets-type split is unavoidable.

Since the pre-release date is fast approaching, we should try to hash
this out in a timely fashion.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Opportunistic STARTTLS in smtpmail.el

[Lars Magne Ingebrigtsen]

Here's my concrete suggestion for how auth-source would deal with this
stuff.

The design requirements are:

1) if users want it, credentials should be encrypted

2) the credentials should be stored in a file that can be edited by
hand, if necessary

5) it should be possible to check whether credentials exist without
giving a password, even if the credentials are encrypted

My solution to all this is to allow putting encrypted stuff into the
~/.authinfo file.

It's currently a one-credential-per-line file like this, and this would
still be perfectly valid:

machine news.foo.org force yes port nntp login bar password zot

However, if auth-info.el prompts somebody for a password, auth-info.el
will also prompt them for whether the credentials should be stored
encrypted.  If the user says yes, then auth-info.el will write the
following to the file:

machine news.foo.org force yes port nntp secret bG9naW4AYmFyAHBhc3N3b3JkAHpvdA

The secret is simply a base64-encoded gpg-encoded string made something
like this:

(base64-encode-string (gpg-encode-string "login^@bar^@password^@zot"
                       (read-string "Password? ")))

We can add some padding and entropy to make things l33tly secure, like

(base64-encode-string
 (gpg-encode-string
  (format "login^@bar^@password^@zot^@pad^@%s" (random 42))
  (read-string "Password? ")))

When decoding, we don't have to decode anything until we actually know
that we need the password.
  
People who think this is too insecure can use ~/.authinfo.gpg files,
just like now.  That's up to them.  

And people that think that using no encryption at all can do that, too.


-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Opportunistic STARTTLS in smtpmail.el

[Robert Pluim]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Here's my concrete suggestion for how auth-source would deal with this
> stuff.
>
> The design requirements are:
>
> 1) if users want it, credentials should be encrypted
>
> 2) the credentials should be stored in a file that can be edited by
> hand, if necessary
>
> 5) it should be possible to check whether credentials exist without
> giving a password, even if the credentials are encrypted
>
> My solution to all this is to allow putting encrypted stuff into the
> ~/.authinfo file.
>
> It's currently a one-credential-per-line file like this, and this would
> still be perfectly valid:
>
> machine news.foo.org force yes port nntp login bar password zot
>
> However, if auth-info.el prompts somebody for a password, auth-info.el
> will also prompt them for whether the credentials should be stored
> encrypted.

Could this be a tri-state question: yes/no/never? (I'd be in the
never camp, but you'd need to store that response somewhere...)

Actually, there would be two questions: 
- do you want to store? y/n/never
- do you want to encrypt? y/n/never

with the 'never' modes being per-entry.

Robert

Re: Opportunistic STARTTLS in smtpmail.el

[Ted Zlatanov]

On Tue, 31 May 2011 01:10:04 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> The design requirements are:

LMI> 1) if users want it, credentials should be encrypted

LMI> 2) the credentials should be stored in a file that can be edited by
LMI> hand, if necessary

LMI> 5) it should be possible to check whether credentials exist without
LMI> giving a password, even if the credentials are encrypted

LMI> My solution to all this is to allow putting encrypted stuff into the
LMI> ~/.authinfo file.

LMI> It's currently a one-credential-per-line file like this, and this would
LMI> still be perfectly valid:

LMI> machine news.foo.org force yes port nntp login bar password zot

LMI> However, if auth-info.el prompts somebody for a password, auth-info.el
LMI> will also prompt them for whether the credentials should be stored
LMI> encrypted.  If the user says yes, then auth-info.el will write the
LMI> following to the file:

LMI> machine news.foo.org force yes port nntp secret bG9naW4AYmFyAHBhc3N3b3JkAHpvdA

LMI> The secret is simply a base64-encoded gpg-encoded string made something
LMI> like this:

LMI> (base64-encode-string (gpg-encode-string "login^@bar^@password^@zot"
LMI>                        (read-string "Password? ")))

LMI> We can add some padding and entropy to make things l33tly secure, like

LMI> (base64-encode-string
LMI>  (gpg-encode-string
LMI>   (format "login^@bar^@password^@zot^@pad^@%s" (random 42))
LMI>   (read-string "Password? ")))

LMI> When decoding, we don't have to decode anything until we actually know
LMI> that we need the password.
  
LMI> People who think this is too insecure can use ~/.authinfo.gpg files,
LMI> just like now.  That's up to them.  

LMI> And people that think that using no encryption at all can do that, too.

s/auth-info/auth-source/g right?  Or are you making a new library?

The above works for me, mostly.  The creation prompts can set a
dynamically scoped `auth-source-encrypted-p' variable to tell all the
creation functions that they are looking at an encrypted file or not
(since double-encrypting is not necessary).  But I would encrypt each
token separately, not several at once, so `auth-source-search' knows if
the token is present.

IOW rather than your "secret" token, let's keep the existing tokens but
the netrc backend of auth-source will know that when it sees "xyz
gpg:<hex data>" it needs to decode that hex data.

We should provide a general mode that can show the file with all the
gpg:<hex data> locations replaced, showing the decrypted data with text
overlays and different colors.  The mode could also edit the encrypted
data inline.  This would be very useful for all of Emacs, not just
auth-source.  Sort of a scratch pad with arbitrary encryption intervals.
With such a mode, a lot less direct auth-source support will be needed
for these encrypted tokens.  The netrc backend would simply use the
general mode.

Ted

Re: Opportunistic STARTTLS in smtpmail.el

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> s/auth-info/auth-source/g right?

Yes.  :-)

> IOW rather than your "secret" token, let's keep the existing tokens but
> the netrc backend of auth-source will know that when it sees "xyz
> gpg:<hex data>" it needs to decode that hex data.

I don't know how gpg works here.  Does gpg-encrypting the same string
give you identical results, or does gpg auto-salt things?  The idea with
putting several tokens into the secret part was to 1) make it more
difficult to brute-force, and 2) make it possible to salt the string, so
that if you have two services with the same user-name/password, the
secret tokens would not be identical.

> We should provide a general mode that can show the file with all the
> gpg:<hex data> locations replaced, showing the decrypted data with text
> overlays and different colors.  The mode could also edit the encrypted
> data inline.  This would be very useful for all of Emacs, not just
> auth-source.  Sort of a scratch pad with arbitrary encryption intervals.
> With such a mode, a lot less direct auth-source support will be needed
> for these encrypted tokens.  The netrc backend would simply use the
> general mode.

Sounds way too complicated, I think.  The usage at hand is the netrc
file format, and I don't think it would have much utility beyond that.

Besides, adding this to netrc would be really trivial.  Making it
general would be difficult.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Opportunistic STARTTLS in smtpmail.el

[Ted Zlatanov]

On Tue, 31 May 2011 20:19:54 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Ted Zlatanov <tzz@lifelogs.com> writes:
>> s/auth-info/auth-source/g right?

LMI> Yes.  :-)

>> IOW rather than your "secret" token, let's keep the existing tokens but
>> the netrc backend of auth-source will know that when it sees "xyz
>> gpg:<hex data>" it needs to decode that hex data.

LMI> I don't know how gpg works here.  Does gpg-encrypting the same string
LMI> give you identical results, or does gpg auto-salt things?  The idea with
LMI> putting several tokens into the secret part was to 1) make it more
LMI> difficult to brute-force, and 2) make it possible to salt the string, so
LMI> that if you have two services with the same user-name/password, the
LMI> secret tokens would not be identical.

That's all fine, we can still auto-salt and bundle random data in the
binary.  The gpg:<hex data> format should be an alist with 'data as
the key we want; any other keys can be ignored.  But it's important to
support it everywhere, not just for passwords.

I propose the hex data be the alist printed in the UTF-8 encoding, then
converted to the unibyte conversion, encrypted, and hex-encoded.  The
next non-hex character (usually space or newline) ends the data.  If we
fail to decode it, we print a warning message.

>> We should provide a general mode that can show the file with all the
>> gpg:<hex data> locations replaced, showing the decrypted data with text
>> overlays and different colors.  The mode could also edit the encrypted
>> data inline.  This would be very useful for all of Emacs, not just
>> auth-source.  Sort of a scratch pad with arbitrary encryption intervals.
>> With such a mode, a lot less direct auth-source support will be needed
>> for these encrypted tokens.  The netrc backend would simply use the
>> general mode.

LMI> Sounds way too complicated, I think.  The usage at hand is the netrc
LMI> file format, and I don't think it would have much utility beyond that.

I disagree about the utility, but...

LMI> Besides, adding this to netrc would be really trivial.  Making it
LMI> general would be difficult.

I agree about this.  For your purpose, let's get the gpg tokens working.
Then we can overengineer the general solution into a minor mode, an
EmacsWiki page, and a long discussion about the aesthetics of hex data.

What do you need from me to get the above done, if you agree about the
implementation?

Ted

Re: Opportunistic STARTTLS in smtpmail.el

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> I propose the hex data be the alist printed in the UTF-8 encoding, then
> converted to the unibyte conversion, encrypted, and hex-encoded.  The
> next non-hex character (usually space or newline) ends the data.  If we
> fail to decode it, we print a warning message.

I think it would be nice if it was as short as possible, too, because
this will be a blob of stuff in a file that people would be editing.

Which is why I kinda like the

secret gpg:<base 64> idea.

If we put all the secrets into the same blob, we can say stuff like

pfoo^@ubar^@ssalt

that is, make the token names really short (one character), and use NUL
as the separator, so that we can put random other characters (including
space) into the passwords...

> I agree about this.  For your purpose, let's get the gpg tokens working.
> Then we can overengineer the general solution into a minor mode, an
> EmacsWiki page, and a long discussion about the aesthetics of hex data.

:-)

> What do you need from me to get the above done, if you agree about the
> implementation?

Well, I'd hoped that you'd implement this.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Opportunistic STARTTLS in smtpmail.el

[Ted Zlatanov]

On Tue, 31 May 2011 22:32:47 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Ted Zlatanov <tzz@lifelogs.com> writes:
>> I propose the hex data be the alist printed in the UTF-8 encoding, then
>> converted to the unibyte conversion, encrypted, and hex-encoded.  The
>> next non-hex character (usually space or newline) ends the data.  If we
>> fail to decode it, we print a warning message.

LMI> I think it would be nice if it was as short as possible, too, because
LMI> this will be a blob of stuff in a file that people would be editing.

LMI> Which is why I kinda like the

LMI> secret gpg:<base 64> idea.

LMI> If we put all the secrets into the same blob, we can say stuff like

LMI> pfoo^@ubar^@ssalt

LMI> that is, make the token names really short (one character), and use NUL
LMI> as the separator, so that we can put random other characters (including
LMI> space) into the passwords...

I understand.  But it sucks from the `auth-source-search' perspective
because now every secret blob has to be decoded to find out if it has
tokens X or Y when the search spec requires X or Y.  So I'm against it.

From the user's perspective, it's no good either because looking at the
netrc file is not enough to tell what it contains, and this new format
can't be used by any other programs besides Emacs.  My format has the
nice property that it degrades into a normal netrc file gracefully.
It's trivial to write a bidirectional converter function, too.

(Just to be clear: my proposed format is
"login joe password gpg:ABCD123456" where the gpg: data decodes to
((data "mysecret") (salt "mysalt")) and no other values besides the
data are used outside; a gpg: value can only yield one piece of
data and only needs to be decoded when you need the actual data.)

>> What do you need from me to get the above done, if you agree about the
>> implementation?

LMI> Well, I'd hoped that you'd implement this.  :-)

I hope we will agree on the format.  I'll implement it when you give up
arguing with me ;)

Ted

Re: Opportunistic STARTTLS in smtpmail.el

[Stefan Monnier]

> (Just to be clear: my proposed format is
> "login joe password gpg:ABCD123456" where the gpg: data decodes to
> ((data "mysecret") (salt "mysalt")) and no other values besides the
> data are used outside; a gpg: value can only yield one piece of
> data and only needs to be decoded when you need the actual data.)

I have a question about this: does the Gnome keychain tool (as well as
comparable tools for other systems) offer the possibility to know if
a password exists without having first granted access to that password?
If not, then we will need an smtpmail-use-auth variable anyway, so the
above gymnastic will be unnecessary.


        Stefan

Re: Opportunistic STARTTLS in smtpmail.el

[Ted Zlatanov]

On Tue, 31 May 2011 22:29:51 -0300 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

>> (Just to be clear: my proposed format is
>> "login joe password gpg:ABCD123456" where the gpg: data decodes to
>> ((data "mysecret") (salt "mysalt")) and no other values besides the
>> data are used outside; a gpg: value can only yield one piece of
>> data and only needs to be decoded when you need the actual data.)

SM> I have a question about this: does the Gnome keychain tool (as well as
SM> comparable tools for other systems) offer the possibility to know if
SM> a password exists without having first granted access to that
SM> password?

Yes, you can usually search without retrieving the secret.  But why does
it matter what the Gnome tools do?  The netrc format is not connected to
the Secrets API or any other keychain-style backends at all.

SM> If not, then we will need an smtpmail-use-auth variable anyway, so the
SM> above gymnastic will be unnecessary.

I think it's necessary no matter what.  We've had several suggestions
(from me, Lars, and Daiki Ueno) for something like what I'm proposing.
It's definitely useful.

Speaking of which, I think in addition to gpg: tokens we should support
crypt: tokens (using the native OS crypt call) and MD4 or some other
symmetric cipher simple enough to implement in ELisp.  GPG is not
necessarily available or wanted.

Ted

Re: Opportunistic STARTTLS in smtpmail.el

[Stefan Monnier]

> I think it's necessary no matter what.  We've had several suggestions
> (from me, Lars, and Daiki Ueno) for something like what I'm proposing.
> It's definitely useful.

> Speaking of which, I think in addition to gpg: tokens we should support
> crypt: tokens (using the native OS crypt call) and MD4 or some other
> symmetric cipher simple enough to implement in ELisp.  GPG is not
> necessarily available or wanted.

One more thing: a user which has a ~/.authinfo.gpg but no unencrypted
~/.netrc nor ~/.authinfo should not be prompted for a password (since
that would be very annoying, if in the end she doesn't need
authentication).
I really think that trying to avoid smtpmail-use-auth is ill-advised.


        Stefan

Re: Opportunistic STARTTLS in smtpmail.el

[Ted Zlatanov]

On Wed, 01 Jun 2011 09:37:43 -0300 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

>> I think it's necessary no matter what.  We've had several suggestions
>> (from me, Lars, and Daiki Ueno) for something like what I'm proposing.
>> It's definitely useful.

>> Speaking of which, I think in addition to gpg: tokens we should support
>> crypt: tokens (using the native OS crypt call) and MD4 or some other
>> symmetric cipher simple enough to implement in ELisp.  GPG is not
>> necessarily available or wanted.

SM> One more thing: a user which has a ~/.authinfo.gpg but no unencrypted
SM> ~/.netrc nor ~/.authinfo should not be prompted for a password (since
SM> that would be very annoying, if in the end she doesn't need
SM> authentication).

If the SMTP server requires authentication, how do we know
~/.authinfo.gpg does NOT have the password we need?  If it does not
require authentication, `auth-source-search' should not be called.

If you find the password prompt for ~/.authinfo.gpg annoying, don't use
that file in your `auth-sources' or use proper GPG key encryption, which
should not prompt you after you load your keyring.

SM> I really think that trying to avoid smtpmail-use-auth is ill-advised.

I'm not trying to avoid anything.  My goal is to provide a netrc format
that freely mixes encrypted and unencrypted data, so you can do a search
without decrypting data and for other benefits.  Do you see any issues
with the format I've proposed?  If the format works and everyone likes
it, I can make ~/.authinfo the default `auth-sources' backend instead of
~/.authinfo.gpg.

You and Lars can do what you like with smtpmail.el; I only need to get
involved when you call `auth-source-search'.

Ted

Re: Opportunistic STARTTLS in smtpmail.el

[Stefan Monnier]

SM> One more thing: a user which has a ~/.authinfo.gpg but no unencrypted
SM> ~/.netrc nor ~/.authinfo should not be prompted for a password (since
SM> that would be very annoying, if in the end she doesn't need
SM> authentication).
> If the SMTP server requires authentication, how do we know
> ~/.authinfo.gpg does NOT have the password we need?

Exactly: we can only know if a var like smtpmail-use-auth tells us.

SM> I really think that trying to avoid smtpmail-use-auth is ill-advised.
> I'm not trying to avoid anything.

AFAICT, the main drive (in this discussion) to introduce
field-encryption within the unencrypted .netrc file is to avoid
introducing a smtpmail-use-auth customization.

And my point is that this customization is a necessary thing anyway.

Maybe there are still other cases where field-encryption in the
unencrypted file is worthwhile (I don't know of any), but that's
a separate discussion.

So can we add this smtpmail-use-auth, make smtpmail.el use opportunistic
STARTTLS and move on?


        Stefan

Re: Opportunistic STARTTLS in smtpmail.el

[Ted Zlatanov]

On Wed, 01 Jun 2011 11:39:59 -0300 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

SM> One more thing: a user which has a ~/.authinfo.gpg but no unencrypted
SM> ~/.netrc nor ~/.authinfo should not be prompted for a password (since
SM> that would be very annoying, if in the end she doesn't need
SM> authentication).
>> If the SMTP server requires authentication, how do we know
>> ~/.authinfo.gpg does NOT have the password we need?

SM> Exactly: we can only know if a var like smtpmail-use-auth tells us.

...or if the user's ~/.authinfo* is already cached and (for .gpg files)
decrypted in memory and we can look inside quickly.  But OK, you want to
avoid any passphrases or other prompts, I understand.  It makes the user
experience better.

SM> AFAICT, the main drive (in this discussion) to introduce
SM> field-encryption within the unencrypted .netrc file is to avoid
SM> introducing a smtpmail-use-auth customization.

From my perspective the chief benefit is that any `auth-source-search'
call against an unencrypted file will not require a passphrase until the
password is actually needed, and yet the password will be stored
securely.  This is good for everyone, not just smtpmail.el.  It will
reduce prompts just like you have requested.  So I want this improvement
regardless of what you and Lars do with smtpmail.el.

SM> And my point is that this customization is a necessary thing anyway.
SM> So can we add this smtpmail-use-auth, make smtpmail.el use
SM> opportunistic STARTTLS and move on?

As I said, I will stay out of the way and you and Lars can do what you
like.  I may have to tweak the `auth-source-search' calls afterwards,
but I will not change the defaults you choose without asking.

Ted

Re: Opportunistic STARTTLS in smtpmail.el

[Stefan Monnier]

> From my perspective the chief benefit is that any `auth-source-search'
> call against an unencrypted file will not require a passphrase until the
> password is actually needed, and yet the password will be stored
> securely.

Sounds OK.  But only if you push if further and deprecate
authinfo.gpg.


        Stefan

Re: Opportunistic STARTTLS in smtpmail.el

[Robert Pluim]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> From my perspective the chief benefit is that any `auth-source-search'
>> call against an unencrypted file will not require a passphrase until the
>> password is actually needed, and yet the password will be stored
>> securely.
>
> Sounds OK.  But only if you push if further and deprecate
> authinfo.gpg.

I'm not clear on why you'd want that.  I can imagine someone wanting to
hide username & server identities from inspection, not just the
associated passwords. ie I distinguish 3 cases

1) everything unencrypted
2) passwords encrypted only
3) everything encrypted

Robert

Re: Opportunistic STARTTLS in smtpmail.el

[Daiki Ueno]

Robert Pluim <rpluim@gmail.com> writes:

> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
>>> From my perspective the chief benefit is that any `auth-source-search'
>>> call against an unencrypted file will not require a passphrase until the
>>> password is actually needed, and yet the password will be stored
>>> securely.
>>
>> Sounds OK.  But only if you push if further and deprecate
>> authinfo.gpg.
>
> I'm not clear on why you'd want that.  I can imagine someone wanting to
> hide username & server identities from inspection, not just the
> associated passwords. ie I distinguish 3 cases
>
> 1) everything unencrypted
> 2) passwords encrypted only
> 3) everything encrypted

That reminds me of Lars' post on the Gnus list:

http://article.gmane.org/gmane.emacs.gnus.general/77172

in the discussion:

http://thread.gmane.org/gmane.emacs.gnus.general/77009/focus=77143

where I suggested to have a separate file to store unencrypted
information along with authinfo.gpg.  I can't remember what was the
conclusion :)

Regards,
-- 
Daiki Ueno

Re: Opportunistic STARTTLS in smtpmail.el

[Stefan Monnier]

>>> From my perspective the chief benefit is that any `auth-source-search'
>>> call against an unencrypted file will not require a passphrase until the
>>> password is actually needed, and yet the password will be stored
>>> securely.
>> Sounds OK.  But only if you push if further and deprecate
>> authinfo.gpg.
> I'm not clear on why you'd want that.

I'm not convinced it would be a good thing, but as long as we encourage
users to use authinfo.gpg, then any code using auth-source.el will want
to decide whether authentication is needed *before* calling auth-source
since that call may prompt the user for a password.
And if the code has to make that decision before calling auth-source,
there's no real benefit to having field-encryption.


        Stefan

Re: Opportunistic STARTTLS in smtpmail.el

[Ted Zlatanov]

On Thu, 02 Jun 2011 09:24:38 -0300 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

SM> I'm not convinced it would be a good thing, but as long as we encourage
SM> users to use authinfo.gpg, then any code using auth-source.el will want
SM> to decide whether authentication is needed *before* calling auth-source
SM> since that call may prompt the user for a password.
SM> And if the code has to make that decision before calling auth-source,
SM> there's no real benefit to having field-encryption.

What you described does not require us to deprecate authinfo.gpg, only
demote it.  I think making it the second choice in the `auth-sources'
list, as I suggested, is sufficient, and will let users who have already
set it up keep using it without changing the default.

On Thu, 02 Jun 2011 20:45:40 +0900 Daiki Ueno <ueno@unixuser.org> wrote: 

DU> That reminds me of Lars' post on the Gnus list:

DU> http://article.gmane.org/gmane.emacs.gnus.general/77172

DU> in the discussion:

DU> http://thread.gmane.org/gmane.emacs.gnus.general/77009/focus=77143

DU> where I suggested to have a separate file to store unencrypted
DU> information along with authinfo.gpg.  I can't remember what was the
DU> conclusion :)

The discussion petered out, and your proposal was .  I think my current
proposal is a continuation of
http://thread.gmane.org/gmane.emacs.gnus.general/77009/focus=77208 where
I propose something similar.  The key difference is Lars' idea of
encrypting only pieces of an otherwise unencrypted file.

On Thu, 02 Jun 2011 22:44:59 +0900 Daiki Ueno <ueno@unixuser.org> wrote: 

DU> Ted Zlatanov <tzz@lifelogs.com> writes:

>> It will be less necessary as the first `auth-sources' choice, but still
>> useful, as Robert noted (I see case 2 as "encrypted tokens" since any
>> token can be encrypted in my proposal).

DU> Could you define the word "token"?  I have thought that it is an opaque
DU> value which connects unencrypted data to encrypted data.  However, it
DU> seems not.

In netrc files, tokens are any keys or values.  Unfortunately the word
is overused and confusing in our context.  Sorry.  I'll use "netrc field
encryption" from now on.

>> I'll simply make `auth-sources' ("~/.authinfo" "~/.authinfo.gpg")
>> 
>> which as a default will work fine.  Creation prompts will target the
>> first one.  The users can put insecure or token-encrypted data in the
>> first one and use the second one for more secure storage.

DU> Though I can't say anything without the detail of your proposal, I feel
DU> you are oversimplifying.

Why?  Have you tried it?  It will work immediately, without any code
changes.  The netrc field encryption is an add-on that doesn't affect
the general `auth-sources' and `auth-source-search' interaction.

DU> Anyway I will be happy if:

DU> - Gnus does not ask password when connecting to password-less services

It doesn't.  EPA/EPG may ask for a passphrase if `auth-sources' points
to a GPG-encrypted file, when and if `auth-source-search' hits that
file.  Usually :max 1 is specified for `auth-source-search' so it will
stop after the first match.  So, assuming my proposed change to
`auth-sources' above is installed, if ~/.authinfo is unencrypted
and the user puts parameters there, there should be no prompting at all
because `auth-source-search' won't hit ~/.authinfo.gpg.

DU> - Gnus does not store my passwords in plain text files by default

That's a personal preference.  It's inevitable, for instance, to use the
~/.netrc unencrypted file if you want libcurl to authenticate HTTP.  So
if you are forced into that situation (e.g. because you must use Git),
you may as well use ~/.netrc as an `auth-sources' entry.

The netrc field encryption will alleviate this problem but IMO it must
be at least the user's decision and we can't pick a default that will
make everyone happy.

DU> - Gnus can hide server or user names (or other attributes) if necessary

The netrc field encryption will work on any value, not just passwords.
You could even use it on keys, though that seems silly.

DU> Otherwise, no persistent password store is much better for me...

You could also use the Secrets API or propose a new `auth-sources'
backend that fits your needs (your expertise is greatly appreciated).  

I like the idea of a pure-data backend, for instance, where all the data
is in the `auth-sources' entry directly.  I would implement it if people
needed it, but it seems everyone prefers data formats they can share
with programs outside Emacs.

Ted

Re: Opportunistic STARTTLS in smtpmail.el

[Daiki Ueno]

Ted Zlatanov <tzz@lifelogs.com> writes:

> The discussion petered out, and your proposal was .

(was what?)

> I think my current proposal is a continuation of
> http://thread.gmane.org/gmane.emacs.gnus.general/77009/focus=77208
> where I propose something similar.  The key difference is Lars' idea
> of encrypting only pieces of an otherwise unencrypted file.

How those pieces of encrypted data can share the same passphrase?
Otherwise a user will be asked passphrase every time when decrypting
those pieces IIUC - that would be more painful than the current
situation or the unencrypted file solution.

> DU> Anyway I will be happy if:
>
> DU> - Gnus does not ask password when connecting to password-less services
>
> It doesn't.

Why it doesn't?  Do you think the current behavior is reasonable enough?
On my modern GNOME desktop environment, NetworkManager etc. do not asks
passwords for password-less services.

> You could also use the Secrets API or propose a new `auth-sources'
> backend that fits your needs (your expertise is greatly appreciated).  

Yes, I know, but are you using it daily?

> I like the idea of a pure-data backend, for instance, where all the data
> is in the `auth-sources' entry directly.  I would implement it if people
> needed it, but it seems everyone prefers data formats they can share
> with programs outside Emacs.

I don't understand what you are talking about - sounds like a sales talk
by researchers in software engineering academia ;)

Regards,
-- 
Daiki Ueno

Re: Opportunistic STARTTLS in smtpmail.el

[Ted Zlatanov]

On Fri, 03 Jun 2011 00:03:18 +0900 Daiki Ueno <ueno@unixuser.org> wrote: 

DU> Ted Zlatanov <tzz@lifelogs.com> writes:
>> The discussion petered out, and your proposal was .

DU> (was what?)

Sorry, I jumped ahead there.  Your proposal was not implemented.

>> I think my current proposal is a continuation of
>> http://thread.gmane.org/gmane.emacs.gnus.general/77009/focus=77208
>> where I propose something similar.  The key difference is Lars' idea
>> of encrypting only pieces of an otherwise unencrypted file.

DU> How those pieces of encrypted data can share the same passphrase?
DU> Otherwise a user will be asked passphrase every time when decrypting
DU> those pieces IIUC - that would be more painful than the current
DU> situation or the unencrypted file solution.

We'd associate a passphrase with the file, not each piece.  It should be
just like using a fully encrypted file, with the same passphrase caching
mechanism if symmetric encryption is used.

DU> Anyway I will be happy if:
>> 
DU> - Gnus does not ask password when connecting to password-less services
>> 
>> It doesn't.

DU> Why it doesn't?  Do you think the current behavior is reasonable enough?
DU> On my modern GNOME desktop environment, NetworkManager etc. do not asks
DU> passwords for password-less services.

EPA/EPG is actually the one asking for passphrases if an `auth-sources'
entry needs to be decrypted.  So I was just correcting the "Gnus" part,
not saying the behaviour is reasonable.  Partly, this thread is about
correcting the current behavior.

>> You could also use the Secrets API or propose a new `auth-sources'
>> backend that fits your needs (your expertise is greatly appreciated).  

DU> Yes, I know, but are you using it daily?

Are you asking about the Secrets API backend?  I'm personally not able
to use it because I need my ~/.authinfo.gpg on several platforms.  But
it works: I've tested it and used it.  It doesn't support creation or
deletion yet.

>> I like the idea of a pure-data backend, for instance, where all the data
>> is in the `auth-sources' entry directly.  I would implement it if people
>> needed it, but it seems everyone prefers data formats they can share
>> with programs outside Emacs.

DU> I don't understand what you are talking about - sounds like a sales talk
DU> by researchers in software engineering academia ;)

In the first sentence, I mean that `auth-sources' could look like this:

'((data ((:host "x" :user "y" :secret "z")))
  "~/.authinfo")

and then (auth-source-search :host "x" :user "y" :max 1)  would return
that first entry directly.  ~/.authinfo would not be examined.

The second sentence means that whenever I've proposed this, I seem to be
the only one that likes the idea.  So I haven't implemented it.  People
seem to like the netrc format because it's understood by other programs,
e.g. libcurl-based software like Git and curl.  But this idea seems to
me a good way to specify connection parameters and it could even support
the "gpg:<hex data>" format we are discussing for netrc field
encryption.

Ted

Re: Opportunistic STARTTLS in smtpmail.el

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> We'd associate a passphrase with the file, not each piece.  It should be
> just like using a fully encrypted file, with the same passphrase caching
> mechanism if symmetric encryption is used.

Yeah, that was what I was thinking, too.  If you've given the GPG
password to decrypt the IMAP password, it wouldn't be necessary to
repeat that to get the NNTP password.

So there would be the same amount of GPG passwords with plain-text gpg:
tokens as with the fully-encrypted .authinfo.gpg file.  The main
functional difference is that with the plain-text file you don't have to
give the GPG password immediately to see whether you even have a token
that matches your search criteria.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

netrc field encryption in auth-source (was: Opportunistic STARTTLS in smtpmail.el)

[Ted Zlatanov]

(xposted to the Gnus mailing list and thread subject changed, finally)

On Fri, 03 Jun 2011 23:54:18 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Ted Zlatanov <tzz@lifelogs.com> writes:
>> We'd associate a passphrase with the file, not each piece.  It should be
>> just like using a fully encrypted file, with the same passphrase caching
>> mechanism if symmetric encryption is used.

LMI> Yeah, that was what I was thinking, too.  If you've given the GPG
LMI> password to decrypt the IMAP password, it wouldn't be necessary to
LMI> repeat that to get the NNTP password.

LMI> So there would be the same amount of GPG passwords with plain-text gpg:
LMI> tokens as with the fully-encrypted .authinfo.gpg file.  The main
LMI> functional difference is that with the plain-text file you don't have to
LMI> give the GPG password immediately to see whether you even have a token
LMI> that matches your search criteria.

On Fri, 03 Jun 2011 23:50:11 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Ted Zlatanov <tzz@lifelogs.com> writes:

>> I understand.  But it sucks from the `auth-source-search' perspective
>> because now every secret blob has to be decoded to find out if it has
>> tokens X or Y when the search spec requires X or Y.  So I'm against it.

LMI> True, I didn't think of that.  Then I think your idea for this makes
LMI> most sense, and please go ahead and implement it.  :-)

OK, I will implement it like so in the netrc backend:

key1 val1 key2 gpg:hexdata key3 gpg:hexdata

Where hexdata encodes "((secret "thesecret") (salt "thesalt"))"

The decoding will happen late, probably in the funcall to obtain the
secret (and it will set some scoped variables to cache the data).  That
may be a little surprising to the user but it's the most secure
approach, I think.

If a user puts gpg: tokens in a .gpg file, I'll let them.

The creation process will by default create plaintext data, but maybe I
will add a y/n prompt for "secret" and "password" tokens to save them
encrypted.  I'm not sure yet, I need to try the process.

Is there a decent cipher that's built into Emacs, as a fallback if GPG
is not installed and usable?  I don't see one.

Thanks
Ted

Re: netrc field encryption in auth-source

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> Is there a decent cipher that's built into Emacs, as a fallback if GPG
> is not installed and usable?  I don't see one.

It's kinda odd that Emacs doesn't have `gpg-encode-string' built in,
isn't it?  I mean, it has sha1 and stuff.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

GPGME (was: netrc field encryption in auth-source)

[Ted Zlatanov]

On Sun, 26 Jun 2011 12:09:42 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Ted Zlatanov <tzz@lifelogs.com> writes:
>> Is there a decent cipher that's built into Emacs, as a fallback if GPG
>> is not installed and usable?  I don't see one.

LMI> It's kinda odd that Emacs doesn't have `gpg-encode-string' built in,
LMI> isn't it?  I mean, it has sha1 and stuff.  :-)

There's `epg-encrypt-string' but it still uses the GPG program under the
covers.

I hope Daiki Ueno could consider augmenting EPA/EPG with
http://www.gnupg.org/related_software/gpgme/index.en.html, which is a
library specifically designed so we can talk to GPG directly and
securely through C.

I would love to see that happen.  I would contribute to the work in any
way I can, either in EPA/EPG or as a separate effort if Daiki Ueno is
not able or interested to do it.

Ted

Re: GPGME

[Daiki Ueno]

Ted Zlatanov <tzz@lifelogs.com> writes:

> I hope Daiki Ueno could consider augmenting EPA/EPG with
> http://www.gnupg.org/related_software/gpgme/index.en.html, which is a
> library specifically designed so we can talk to GPG directly and
> securely through C.

AFAIK, GPGME is a wrapper library which just calls gpg command
internally, and actually EPG is initially designed as a port of GPGME in
elisp (see the info).  I don't expect much differences if the caller
part is written in C.

Regards,
-- 
Daiki Ueno

Re: GPGME

[Ted Zlatanov]

On Tue, 28 Jun 2011 06:47:09 +0900 Daiki Ueno <ueno@unixuser.org> wrote: 

DU> Ted Zlatanov <tzz@lifelogs.com> writes:
>> I hope Daiki Ueno could consider augmenting EPA/EPG with
>> http://www.gnupg.org/related_software/gpgme/index.en.html, which is a
>> library specifically designed so we can talk to GPG directly and
>> securely through C.

DU> AFAIK, GPGME is a wrapper library which just calls gpg command
DU> internally, and actually EPG is initially designed as a port of GPGME in
DU> elisp (see the info).  I don't expect much differences if the caller
DU> part is written in C.

Oh, I see.  Thank you for explaining.  That's too bad!

Are there any alternatives?  Maybe you remember our discussion years ago
about encrypt.el, where I proposed a neutral API with at least some
symmetric ciphers implemented in ELisp and C in the Emacs core
(essentially what Lars was requesting).  Could something like that work
within the EPA/EPG structure, so some special invocation of
`epg-encrypt-string' could bypass the external callout to GPG?

Thank you
Ted

Re: GPGME

[Daiki Ueno]

Ted Zlatanov <tzz@lifelogs.com> writes:

> Are there any alternatives?  Maybe you remember our discussion years ago
> about encrypt.el, where I proposed a neutral API with at least some
> symmetric ciphers implemented in ELisp and C in the Emacs core
> (essentially what Lars was requesting).

I remember that the problem of encrypt.el was that the data format is
not interoperable and the algorithm used is not interchangeable though
the API might be neutral.  I guess you need a minimal encryption
function which employs the standard GPG message format (RFC4880).

> Could something like that work
> within the EPA/EPG structure, so some special invocation of
> `epg-encrypt-string' could bypass the external callout to GPG?

If your statement in <87wrh0fh4g.fsf_-_@lifelogs.com>:

  The decoding will happen late, probably in the funcall to obtain the
  secret (and it will set some scoped variables to cache the data)

is true, epg-encrypt-string is not necessarily to be optimized in that
way, I think.  How about implementing your side first and profiling
before the optimization?

One suggestion to reduce the number of calls to epg-encrypt-string is
that, I would suggest encrypt the key name as well.  For example,

  key1 val1 encrypted hexdata

where hexdata is decrypted to:

  key2 val2 key3 val3

Regards,
-- 
Daiki Ueno

secure plist store

[Daiki Ueno]

[-- Attachment #1: Type: text/plain, Size: 2163 bytes --]

Daiki Ueno <ueno@unixuser.org> writes:

> If your statement in <87wrh0fh4g.fsf_-_@lifelogs.com>:
>
>   The decoding will happen late, probably in the funcall to obtain the
>   secret (and it will set some scoped variables to cache the data)
>
> is true, epg-encrypt-string is not necessarily to be optimized in that
> way, I think.  How about implementing your side first and profiling
> before the optimization?

I didn't notice that the field encryption code is already checked in.
However, it does not work for me at all and looks too complicated - also
it apparently does not benefit from GPG2 passphrase caching (see "(auth)
GnuPG and EasyPG Assistant Configuration").

I don't want to see that the Gnus password-caching feature becomes
harder and harder to use daily... Is it not feasible to stop reusing
netrc pieces and employ a new format, which is more GPG friendly?  Yeah,
I'm reluctant to repeat the same discussion - here is a
proof-of-concept implementation of searchable, partially encrypted,
persistent plist store, called plstore.el.

Creation:

(setq store (plstore-open (expand-file-name "~/.emacs.d/auth")))
(plstore-put store "foo" '(:host "foo.example.org" :user "test") nil)
(plstore-save store)

;; mark :user property as secret
(plstore-put store "bar" '(:host "bar.example.org") '(:user "test"))
(plstore-put store "baz" '(:host "baz.example.org") '(:user "test"))
(plstore-save store) ;<= will ask passphrase via GPG

Search:

(setq store (plstore-open (expand-file-name "auth.el" user-emacs-directory)))
(plstore-find store '(:host "foo.example.org"))
(plstore-find store '(:host "bar.example.org")) ;<= will ask passphrase via GPG

The file format of ~/.emacs.d/auth:

--8<---------------cut here---------------start------------->8---
(("baz" :secret-user t :host "baz.example.org")
 ("bar" :secret-user t :host "bar.example.org")
 ("foo" :host "foo.example.org" :port 80))
"-----BEGIN PGP MESSAGE-----
Version: GnuPG v2.0.17 (GNU/Linux)

jA0EAwMCXQZhP/0Se0DUyTQcC17GCo0CdT+RfFFskWp4aNYW/aOT/qbv24M1vPfx
TFi9AR7iVc6qlg+9cA3f3buYBGvp
=UEHH
-----END PGP MESSAGE-----
"
--8<---------------cut here---------------end--------------->8---


[-- Attachment #2: plstore.el --]
[-- Type: text/plain, Size: 9251 bytes --]

;;; plstore.el --- searchable, partially encrypted, persistent plist store
;; Copyright (C) 2011 Free Software Foundation, Inc.

;; Author: Daiki Ueno <ueno@unixuser.org>
;; Keywords: PGP, GnuPG

;; This file is part of GNU Emacs.

;; GNU Emacs is free software: you can redistribute it and/or modify
;; it under the terms of the GNU General Public License as published by
;; the Free Software Foundation, either version 3 of the License, or
;; (at your option) any later version.

;; GNU Emacs is distributed in the hope that it will be useful,
;; but WITHOUT ANY WARRANTY; without even the implied warranty of
;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
;; GNU General Public License for more details.

;; You should have received a copy of the GNU General Public License
;; along with GNU Emacs.  If not, see <http://www.gnu.org/licenses/>.

;;; Commentary

;; Creating:
;;
;; (setq store (plstore-open (expand-file-name "auth" user-emacs-directory)))
;; (plstore-put store "foo" '(:host "foo.example.org" :port 80) nil)
;; (plstore-save store)
;; ;; :user property is secret
;; (plstore-put store "bar" '(:host "bar.example.org") '(:user "test"))
;; (plstore-put store "baz" '(:host "baz.example.org") '(:user "test"))
;; (plstore-save store) ;<= will ask passphrase via GPG
;;
;; Searching:
;;
;; (setq store (plstore-open (expand-file-name "auth.el" user-emacs-directory)))
;; (plstore-find store '(:host "foo.example.org"))
;; (plstore-find store '(:host "bar.example.org")) ;<= will ask passphrase via GPG
;; 

;;; Code:

(require 'epg)
(require 'epa)				;epa-passphrase-callback-function

(defvar plstore-cache-passphrase-for-symmetric-encryption nil)
(defvar plstore-passphrase-alist nil)

(defun plstore-passphrase-callback-function (context key-id plstore)
  (if (and plstore-cache-passphrase-for-symmetric-encryption
	   (eq key-id 'SYM))
      (progn
	(let* ((file (file-truename (plstore-get-file plstore)))
	       (entry (assoc file plstore-passphrase-alist))
	      passphrase)
	  (or (copy-sequence (cdr entry))
	      (progn
		(unless entry
		  (setq entry (list file)
			plstore-passphrase-alist
			(cons entry
			      plstore-passphrase-alist)))
		(setq passphrase (epa-passphrase-callback-function context
								   key-id
								   file))
		(setcdr entry (copy-sequence passphrase))
		passphrase))))
    (epa-passphrase-callback-function context key-id
				      (plstore-get-file plstore))))

(defun plstore-get-file (this)
  (aref this 0))

(defun plstore-get-alist (this)
  (aref this 1))

(defun plstore-get-encrypted-data (this)
  (aref this 2))

(defun plstore-get-secret-alist (this)
  (aref this 3))

(defun plstore-get-merged-alist (this)
  (aref this 4))

(defun plstore-get-decrypted (this)
  (aref this 5))

(defun plstore-set-file (this file)
  (aset this 0 file))

(defun plstore-set-alist (this plist)
  (aset this 1 plist))

(defun plstore-set-encrypted-data (this encrypted-data)
  (aset this 2 encrypted-data))

(defun plstore-set-secret-alist (this secret-alist)
  (aset this 3 secret-alist))

(defun plstore-set-merged-alist (this merged-alist)
  (aset this 4 merged-alist))

(defun plstore-set-decrypted (this decrypted)
  (aset this 5 decrypted))

;;;###autoload
(defun plstore-open (file)
  "Create a plstore instance associated with FILE."
  (let ((store (vector
		file
		nil		     ;plist (plist)
		nil		     ;encrypted data (string)
		nil		     ;secret plist (plist)
		nil		     ;merged plist (plist)
		nil		     ;decrypted (bool)
		)))
    (condition-case nil
	(with-temp-buffer
	  (insert-file-contents (plstore-get-file store))
	  (goto-char (point-min))
	  (plstore-set-alist store (read (point-marker)))
	  (forward-sexp)
	  (plstore-set-encrypted-data store (read (point-marker)))
	  ;; merged plist initially contains only unencrypted plist
	  (plstore-set-merged-alist store (plstore-get-alist store)))
      (error))
    store))

(defun plstore--merge-secret (plstore)
  (let ((alist (plstore-get-secret-alist plstore))
	(modified-alist (plstore-get-merged-alist plstore))
	modified-plist
	modified-entry
	entry
	plist
	placeholder)
    (while alist
      (setq entry (car alist)
	    alist (cdr alist)
	    plist (cdr entry)
	    modified-entry (assoc (car entry) modified-alist)
	    modified-plist (cdr modified-entry))
      (while plist
	(setq placeholder
	      (plist-member
	       modified-plist
	       (intern (concat ":secret-"
			       (substring (symbol-name (car plist)) 1)))))
	(if placeholder
	    (setcar placeholder (car plist)))
	(setq modified-plist
	      (plist-put modified-plist (car plist) (car (cdr plist))))
	(setq plist (nthcdr 2 plist)))
      (setcdr modified-entry modified-plist))))

(defun plstore--decrypt (plstore)
  (if (and (not (plstore-get-decrypted plstore))
	   (plstore-get-encrypted-data plstore))
      (let ((context (epg-make-context 'OpenPGP))
	    plain)
	(epg-context-set-passphrase-callback
	 context
	 (cons #'plstore-passphrase-callback-function
	       plstore))
	(setq plain
	      (epg-decrypt-string context
				  (plstore-get-encrypted-data plstore)))
	(plstore-set-secret-alist plstore (car (read-from-string plain)))
	(plstore--merge-secret plstore)
	(plstore-set-decrypted plstore t))))

(defun plstore--match (entry keys skip-if-secret-found)
  (let ((result t) key-name key-value prop-value secret-name)
    (while keys
      (setq key-name (car keys)
	    key-value (car (cdr keys))
	    prop-value (plist-get (cdr entry) key-name))
	(unless (equal prop-value key-value)
	  (if skip-if-secret-found
	      (progn
		(setq secret-name
		      (intern (concat ":secret-"
				      (substring (symbol-name key-name) 1))))
		(if (plist-member (cdr entry) secret-name)
		    (setq result 'secret)
		  (setq result nil
			keys nil)))
	    (setq result nil
		  keys nil)))
	(setq keys (nthcdr 2 keys)))
    result))

(defun plstore-find (plstore keys)
  "Perform search on PLSTORE with KEYS.
KEYS is a plist."
  (let (entries alist entry match decrypt plist)
    ;; First, go through the merged plist alist and collect entries
    ;; matched with keys.
    (setq alist (plstore-get-merged-alist plstore))
    (while alist
      (setq entry (car alist)
	    alist (cdr alist)
	    match (plstore--match entry keys t))
      (if (eq match 'secret)
	  (setq decrypt t)
	(when match
	  (setq plist (cdr entry))
	  (while plist
	    (if (string-match "\\`:secret-" (symbol-name (car plist)))
		(setq decrypt t
		      plist nil))
	    (setq plist (nthcdr 2 plist)))
	  (setq entries (cons entry entries)))))
    ;; Second, decrypt the encrypted plist and try again.
    (when decrypt
      (setq entries nil)
      (plstore--decrypt plstore)
      (setq alist (plstore-get-merged-alist plstore))
      (while alist
	(setq entry (car alist)
	      alist (cdr alist)
	      match (plstore--match entry keys nil))
	(if match
	    (setq entries (cons entry entries)))))
    (nreverse entries)))

(defun plstore-put (plstore name keys secret-keys)
  "Put an entry with NAME in PLSTORE.
KEYS is a plist containing non-secret data.
SECRET-KEYS is a plist containing secret data."
  (let (entry
	plist
	secret-plist
	merged-plist
	symbol)
    (while secret-keys
      (setq symbol
	    (intern (concat ":secret-"
			    (substring (symbol-name (car secret-keys)) 1))))
      (setq plist (plist-put plist symbol t)
	    secret-plist (plist-put secret-plist
				    (car secret-keys) (car (cdr secret-keys)))
	    merged-plist (plist-put merged-plist
				    (car secret-keys) (car (cdr secret-keys)))
	    secret-keys (nthcdr 2 secret-keys)))
    (while keys
      (setq symbol
	    (intern (concat ":secret-"
			    (substring (symbol-name (car keys)) 1))))
      (setq plist (plist-put plist (car keys) (car (cdr keys)))
	    merged-plist (plist-put merged-plist (car keys) (car (cdr keys)))
	    keys (nthcdr 2 keys)))
    (setq entry (assoc name (plstore-get-alist plstore)))
    (if entry
	(setcdr entry plist)
      (plstore-set-alist
       plstore
       (cons (cons name plist) (plstore-get-alist plstore))))
    (when secret-plist
      (setq entry (assoc name (plstore-get-secret-alist plstore)))
      (if entry
	  (setcdr entry secret-plist)
	(plstore-set-secret-alist
	 plstore
	 (cons (cons name secret-plist) (plstore-get-secret-alist plstore)))))
    (setq entry (assoc name (plstore-get-merged-alist plstore)))
    (if entry
	(setcdr entry merged-plist)
      (plstore-set-merged-alist
       plstore
       (cons (cons name merged-plist) (plstore-get-merged-alist plstore))))))

(defun plstore-save (plstore)
  "Save the contents of PLSTORE associated with a FILE."
  (with-temp-buffer
    (insert (pp-to-string (plstore-get-alist plstore)))
    (if (plstore-get-secret-alist plstore)
	(let ((context (epg-make-context 'OpenPGP))
	      (pp-escape-newlines nil)
	      cipher)
	  (epg-context-set-armor context t)
	  (epg-context-set-passphrase-callback
	   context
	   (cons #'plstore-passphrase-callback-function
		 plstore))
	  (setq cipher (epg-encrypt-string context
					   (pp-to-string
					    (plstore-get-secret-alist plstore))
					   nil))
	  (insert (pp-to-string cipher))))
    (write-region (point-min) (point-max) (plstore-get-file plstore))))

(provide 'plstore)

;;; plstore.el ends here

[-- Attachment #3: Type: text/plain, Size: 275 bytes --]


As you see, secret properties are prefixed with ":secret-" and the value
is hidden, and the real properties are encrypted together in the GPG
data at the end.  If you decrypt the GPG data, you will see:

(("baz" :user "test")
 ("bar" :user "test"))

Regards,
-- 
Daiki Ueno

Re: secure plist store

[Lars Magne Ingebrigtsen]

Daiki Ueno <ueno@unixuser.org> writes:

> I didn't notice that the field encryption code is already checked in.
> However, it does not work for me at all and looks too complicated - also
> it apparently does not benefit from GPG2 passphrase caching (see "(auth)
> GnuPG and EasyPG Assistant Configuration").

Can't it be altered to support passphrase caching?

[...]

> --8<---------------cut here---------------start------------->8---
> (("baz" :secret-user t :host "baz.example.org")
>  ("bar" :secret-user t :host "bar.example.org")
>  ("foo" :host "foo.example.org" :port 80))
> "-----BEGIN PGP MESSAGE-----
> Version: GnuPG v2.0.17 (GNU/Linux)
>
> jA0EAwMCXQZhP/0Se0DUyTQcC17GCo0CdT+RfFFskWp4aNYW/aOT/qbv24M1vPfx
> TFi9AR7iVc6qlg+9cA3f3buYBGvp
> =UEHH
> -----END PGP MESSAGE-----

The nice thing about the netrc format is that people can edit it
themselves.  This looks more fragile.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: secure plist store

[Daiki Ueno]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

>> I didn't notice that the field encryption code is already checked in.
>> However, it does not work for me at all and looks too complicated - also
>> it apparently does not benefit from GPG2 passphrase caching (see "(auth)
>> GnuPG and EasyPG Assistant Configuration").
>
> Can't it be altered to support passphrase caching?

Not really - GPG2 passphrase caching is smarter than elisp level caching
as it uses unique ID embedded in GPG data, so it allows user to share
passphrases even among multiple Emacs processes.

>> --8<---------------cut here---------------start------------->8---
>> (("baz" :secret-user t :host "baz.example.org")
>>  ("bar" :secret-user t :host "bar.example.org")
>>  ("foo" :host "foo.example.org" :port 80))
>> "-----BEGIN PGP MESSAGE-----
>> Version: GnuPG v2.0.17 (GNU/Linux)
>>
>> jA0EAwMCXQZhP/0Se0DUyTQcC17GCo0CdT+RfFFskWp4aNYW/aOT/qbv24M1vPfx
>> TFi9AR7iVc6qlg+9cA3f3buYBGvp
>> =UEHH
>> -----END PGP MESSAGE-----
>
> The nice thing about the netrc format is that people can edit it
> themselves.  This looks more fragile.

The above format is tentative and could be improved.  Anyway, as the
encrypted fields in netrc is also not easily editable and given that the
people editing netrc are kind of power user, how about making netrc
files as fallback and read-only from Gnus?

Regards,
-- 
Daiki Ueno

Re: secure plist store

[Ted Zlatanov]

On Wed, 29 Jun 2011 18:05:36 +0900 Daiki Ueno <ueno@unixuser.org> wrote: 

DU> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
>>> I didn't notice that the field encryption code is already checked in.
>>> However, it does not work for me at all and looks too complicated - also
>>> it apparently does not benefit from GPG2 passphrase caching (see "(auth)
>>> GnuPG and EasyPG Assistant Configuration").
>> 
Lars> Can't it be altered to support passphrase caching?

DU> Not really - GPG2 passphrase caching is smarter than elisp level caching
DU> as it uses unique ID embedded in GPG data, so it allows user to share
DU> passphrases even among multiple Emacs processes.

...so you're saying we don't benefit from a feature we can't use?  What
are we supposed to change or improve?

>>> --8<---------------cut here---------------start------------->8---
>>> (("baz" :secret-user t :host "baz.example.org")
>>> ("bar" :secret-user t :host "bar.example.org")
>>> ("foo" :host "foo.example.org" :port 80))

Lars> The nice thing about the netrc format is that people can edit it
Lars> themselves.  This looks more fragile.

DU> The above format is tentative and could be improved.  

The nicest thing about the netrc format, IMHO, is that other programs
understand it.  Your proposal is no better than a binary store as far as
other programs are concerned.  The GPG tokens we currently have are
backwards compatible, meaning that they can be mixed with unencrypted
lines and tokens, and that's the reason we did them that way.

DU> Anyway, as the encrypted fields in netrc is also not easily editable
DU> and given that the people editing netrc are kind of power user, how
DU> about making netrc files as fallback and read-only from Gnus?

The encrypted fields are not supposed to be editable, though that's not
hard to provide.

Editing the netrc directly is not a power user feature.  They are very
easy to read and understand.  I have shown dozens of people with various
skill levels how to use them and the only question they tend to ask is
"what about spaces in the password?"

Ted

Re: secure plist store

[Daiki Ueno]

Ted Zlatanov <tzz@lifelogs.com> writes:

> DU> Not really - GPG2 passphrase caching is smarter than elisp level caching
> DU> as it uses unique ID embedded in GPG data, so it allows user to share
> DU> passphrases even among multiple Emacs processes.
>
> ...so you're saying we don't benefit from a feature we can't use?  What
> are we supposed to change or improve?

OK, honestly, I would say that it won't work with GPG2 since GPG2 does
always do the password operation in the agent.  Have you tried that?

> The nicest thing about the netrc format, IMHO, is that other programs
> understand it.

What other programs use GPG encrypted netrc?  What other programs writes
passwords automatically into that file?  IMHO, these are very ad-hoc
approaches and causing unnecessary complexities.

> Editing the netrc directly is not a power user feature.  They are very
> easy to read and understand.  I have shown dozens of people with various
> skill levels how to use them and the only question they tend to ask is
> "what about spaces in the password?"

I guess that file is edited when a user is accessing to some machines
frequently with legacy clients (like ~/.rhosts).  I really hope that
Gnus does the password caching in more suckless way, as modern clients
like Thunderbird do, at least by default.

For my case, I have never edited netrc by hand.  After upgrading to Gnus
in Emacs 24, it started asking with confusing multiple-choice question
to save the password, and I answered the question with "y" without
reading the help carefully.  Then, from the next time, it started asking
passwords unwanted timing - really annoying, and it shouldn't be the
default behavior for new users.

Regards,
-- 
Daiki Ueno

Re: secure plist store

[Ted Zlatanov]

On Wed, 29 Jun 2011 20:30:34 +0900 Daiki Ueno <ueno@unixuser.org> wrote: 

DU> Ted Zlatanov <tzz@lifelogs.com> writes:
DU> Not really - GPG2 passphrase caching is smarter than elisp level caching
DU> as it uses unique ID embedded in GPG data, so it allows user to share
DU> passphrases even among multiple Emacs processes.
>> 
>> ...so you're saying we don't benefit from a feature we can't use?  What
>> are we supposed to change or improve?

DU> OK, honestly, I would say that it won't work with GPG2 since GPG2 does
DU> always do the password operation in the agent.  Have you tried that?

I'm not sure what you're asking, unfortunately.  What do you want me to try?

>> The nicest thing about the netrc format, IMHO, is that other programs
>> understand it.

DU> What other programs use GPG encrypted netrc?

You mean fully encrypted (authinfo.gpg)?  None; that format, however, is
the only one available that offers full encryption.  The field
encryption (GPG tokens) is backwards compatible and no other programs
support decrypting those tokens yet (although I would push for it in
libcurl, for example).

DU> What other programs writes passwords automatically into that file?

Why does that matter?  It's a convenience we offer.  Most other programs
that use it fail silently if the credentials are not in there; we ask to
save instead.  That seems a good choice to me but I want to listen and
change things if there are better ways to do it.  So far Stefan Monnier
and you have complained about the *prompting* and I've tried to fix the
prompting issues that Stefan noted in a long bug thread.  But no one has
complained about the *functionality* or asked to change it.

DU> IMHO, these are very ad-hoc approaches and causing unnecessary
DU> complexities.

Perhaps you could recommend a better way.  Besides the Secrets API,
which does not work across platforms, I'm not aware of one.

>> Editing the netrc directly is not a power user feature.  They are very
>> easy to read and understand.  I have shown dozens of people with various
>> skill levels how to use them and the only question they tend to ask is
>> "what about spaces in the password?"

DU> I guess that file is edited when a user is accessing to some machines
DU> frequently with legacy clients (like ~/.rhosts).

Git+libcurl use the netrc file, for instance.  It's the only way to
provide passwords for Git over HTTP AFAIK.

DU> I really hope that Gnus does the password caching in more suckless
DU> way, as modern clients like Thunderbird do, at least by default.

What are you talking about when you say "password caching"?  There are
at least 3 pieces:

- searching for credentials and using them (host, port, user name, secret)

- saving credentials for the session

- saving credentials permanently

Not to mention the EPA/EPG interaction with the .gpg files, where it
sometimes needs to ask for the passphrase.

DU> For my case, I have never edited netrc by hand.  After upgrading to Gnus
DU> in Emacs 24, it started asking with confusing multiple-choice question
DU> to save the password, and I answered the question with "y" without
DU> reading the help carefully.  Then, from the next time, it started asking
DU> passwords unwanted timing - really annoying, and it shouldn't be the
DU> default behavior for new users.

I'm trying to understand the problem of "unwanted timing."  Do you mean
you're getting prompted for your credentials repeatedly?  What Gnus
backend are you talking about?  Set `auth-source-debug' to 'trivia and
check the *Messages* buffer to see what `auth-source-search' calls are
failing; that's a good first step to understand what's wrong.

In general, if you don't think we should be asking for passwords, how
would you suggest we behave when passwords are needed, e.g. for IMAP?
Would you rather see something like assistant.el employed to do a
multi-step configuration, and then when credentials are missing we
simply say "sorry but you have to redo the setup for service X" or ask
for the credentials immediately?  I think that's what most other MUAs
do, with some small variations.  They usually save the credentials to a
place that only works for that MUA.

Ted

Re: secure plist store

[Daiki Ueno]

Ted Zlatanov <tzz@lifelogs.com> writes:

> DU> OK, honestly, I would say that it won't work with GPG2 since GPG2 does
> DU> always do the password operation in the agent.  Have you tried that?
>
> I'm not sure what you're asking, unfortunately.  What do you want me to try?

I mean, simply, have you tested your code with GPG2?

Regards,
-- 
Daiki Ueno

Re: secure plist store

[Ted Zlatanov]

On Wed, 29 Jun 2011 17:07:57 +0900 Daiki Ueno <ueno@unixuser.org> wrote: 

DU> I didn't notice that the field encryption code is already checked in.
DU> However, it does not work for me at all and looks too complicated

I would appreciate it if you were more specific about "does not work" by
either filing a bug or explaining.  Obviously it works for me and Lars,
who tested it.  Similarly, since you wrote EPA/EPG, your advice on
reducing complexity when we use it is greatly appreciated but you don't
give us any here.  Should we dig through plstore.el and figure out how
to use the pieces in auth-source.el?  I don't think that was your
intent...

DU> I don't want to see that the Gnus password-caching feature becomes
DU> harder and harder to use daily...

I don't think we've done anything that makes it harder to use.  The GPG
token functionality is off by default right now.

DU> Yeah, I'm reluctant to repeat the same discussion - here is a
DU> proof-of-concept implementation of searchable, partially encrypted,
DU> persistent plist store, called plstore.el.
...
DU> As you see, secret properties are prefixed with ":secret-" and the value
DU> is hidden, and the real properties are encrypted together in the GPG
DU> data at the end.

So it's not line-based.  I think that's a minus: people expect to be
able to copy a line out of the netrc file, and it makes managing such
files easier.

Ted

Re: secure plist store

[Daiki Ueno]

Ted Zlatanov <tzz@lifelogs.com> writes:

> I would appreciate it if you were more specific about "does not work" by
> either filing a bug or explaining.  Obviously it works for me and Lars,
> who tested it.  Similarly, since you wrote EPA/EPG, your advice on
> reducing complexity when we use it is greatly appreciated but you don't
> give us any here.

I have too many minor comments to list here, but at least duplicate code
regarding stashfile should be simplified by supplying a custom
passphrase-callback and calling epg-{encrypt,decrypt}-string.  Currently
it's really hard to understand what the code does.

> DU> I don't want to see that the Gnus password-caching feature becomes
> DU> harder and harder to use daily...
>
> I don't think we've done anything that makes it harder to use.  The GPG
> token functionality is off by default right now.

Yes, and I hope that it won't be on by default.

> So it's not line-based.  I think that's a minus: people expect to be
> able to copy a line out of the netrc file, and it makes managing such
> files easier.

I think typical users don't want to edit the auto-saved passwords file
itself, as long as it saves their passwords and serves it to services as
needed.

Regards,
-- 
Daiki Ueno

Re: secure plist store

[Ted Zlatanov]

On Wed, 29 Jun 2011 20:59:10 +0900 Daiki Ueno <ueno@unixuser.org> wrote: 

DU> Ted Zlatanov <tzz@lifelogs.com> writes:
>> I would appreciate it if you were more specific about "does not work" by
>> either filing a bug or explaining.  Obviously it works for me and Lars,
>> who tested it.  Similarly, since you wrote EPA/EPG, your advice on
>> reducing complexity when we use it is greatly appreciated but you don't
>> give us any here.

DU> I have too many minor comments to list here, but at least duplicate code
DU> regarding stashfile should be simplified by supplying a custom
DU> passphrase-callback and calling epg-{encrypt,decrypt}-string.  

Can I use the way you have it in plstore.el?  That seems a good approach
but I want to be sure you agree before I rewrite the relevant
auth-source.el code.

Feel free to send me your comments as a patch or otherwise.  I
appreciate your time and attention.

DU> Currently it's really hard to understand what the code does.

Sorry about that.  I will try to simplify it.  It was written in a hurry
and I don't know the EPA/EPG interfaces well.

DU> I don't want to see that the Gnus password-caching feature becomes
DU> harder and harder to use daily...
>> 
>> I don't think we've done anything that makes it harder to use.  The GPG
>> token functionality is off by default right now.

DU> Yes, and I hope that it won't be on by default.

I think it's a useful feature, but your feedback matters and I will not
turn it on by default without discussing it further.

>> So it's not line-based.  I think that's a minus: people expect to be
>> able to copy a line out of the netrc file, and it makes managing such
>> files easier.

DU> I think typical users don't want to edit the auto-saved passwords file
DU> itself, as long as it saves their passwords and serves it to services as
DU> needed.

I guess we've observed different user patterns and see different needs.
I still think a simple line-based format is better than a multi-line
format, if the netrc file is going to be portable and extensible.  For
instance I could (and intend to) submit a patch to libcurl to handle
gpg: tokens, but I don't think I could do it for plstore-formatted
files.

Thanks
Ted

Re: secure plist store

[Ted Zlatanov]

On Wed, 29 Jun 2011 20:59:10 +0900 Daiki Ueno <ueno@unixuser.org> wrote: 

DU> Ted Zlatanov <tzz@lifelogs.com> writes:
>> I would appreciate it if you were more specific about "does not work" by
>> either filing a bug or explaining.  Obviously it works for me and Lars,
>> who tested it.  Similarly, since you wrote EPA/EPG, your advice on
>> reducing complexity when we use it is greatly appreciated but you don't
>> give us any here.

DU> I have too many minor comments to list here, but at least duplicate code
DU> regarding stashfile should be simplified by supplying a custom
DU> passphrase-callback and calling epg-{encrypt,decrypt}-string.  Currently
DU> it's really hard to understand what the code does.

I am attaching an auth-source.el patch based on the plstore.el code.  It
simplifies things greatly.

I have one suggestion, that `epa-passphrase-callback-function' should
take an extra detail string, as shown in my patch:

(defun auth-source-passphrase-callback-function (context key-id handback
                                                         &optional sym-detail)
"Exactly like `epa-passphrase-callback-function' but takes an
extra SYM-DETAIL parameter which will be printed at the end of
the symmetric passphrase prompt."
...

That makes it possible to prompt for "passphrase for file xyz tokens:"
instead of just "passphrase for file xyz:" (I think the latter is
confusing to the user).

Obviously that's not strictly needed since I can write my own function
as you see.

If you could review the new code, I would appreciate any comments you
may have.

Ted

Re: secure plist store

[Daiki Ueno]

Ted Zlatanov <tzz@lifelogs.com> writes:

> I have one suggestion, that `epa-passphrase-callback-function' should
> take an extra detail string, as shown in my patch:

[...]

> Obviously that's not strictly needed since I can write my own function
> as you see.

epa-passphrase-callback-function is not an API, but an example
implementation of passphrase callback.  So it is right to redefine it --
you won't need (require 'epa) then.

Also, if you use only symmetric encryption, you can remove the
condition Y: (if (eq key-id 'SYM) X Y)

Regards,
-- 
Daiki Ueno

Re: secure plist store

[Ted Zlatanov]

On Thu, 30 Jun 2011 03:31:03 +0900 Daiki Ueno <ueno@unixuser.org> wrote: 

DU> Ted Zlatanov <tzz@lifelogs.com> writes:
>> I have one suggestion, that `epa-passphrase-callback-function' should
>> take an extra detail string, as shown in my patch:

DU> [...]

>> Obviously that's not strictly needed since I can write my own function
>> as you see.

DU> epa-passphrase-callback-function is not an API, but an example
DU> implementation of passphrase callback.  So it is right to redefine it --
DU> you won't need (require 'epa) then.

DU> Also, if you use only symmetric encryption, you can remove the
DU> condition Y: (if (eq key-id 'SYM) X Y)

Thanks, I'll do that.

Ted

Re: secure plist store

[Daiki Ueno]

[-- Attachment #1: Type: text/plain, Size: 432 bytes --]

Ted Zlatanov <tzz@lifelogs.com> writes:

> DU> Also, if you use only symmetric encryption, you can remove the
> DU> condition Y: (if (eq key-id 'SYM) X Y)
>
> Thanks, I'll do that.

You could simplify more; patch attached:

$ bzr diff --diff-options=-w | diffstat
 auth-source.el |   30 +++++-------------------------
 1 file changed, 5 insertions(+), 25 deletions(-)

BTW, I think you should adjust indentation of the whole file.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: auth-source.el.diff --]
[-- Type: text/x-diff, Size: 2436 bytes --]

=== modified file 'lisp/gnus/auth-source.el'
--- lisp/gnus/auth-source.el	2011-06-30 14:25:27 +0000
+++ lisp/gnus/auth-source.el	2011-06-30 23:02:41 +0000
@@ -43,7 +43,6 @@
 (require 'mm-util)
 (require 'gnus-util)
 (require 'assoc)
-(require 'epa)
 (require 'epg)
 
 (eval-when-compile (require 'cl))
@@ -984,25 +983,7 @@
 
 (defvar auth-source-passphrase-alist nil)
 
-(defun auth-source-passphrase-callback-function (context key-id handback
-                                                         &optional sym-detail)
-  "Exactly like `epa-passphrase-callback-function' but takes an
-extra SYM-DETAIL parameter which will be printed at the end of
-the symmetric passphrase prompt, and assumes symmetric
-encryption."
-  (read-passwd
-   (format "Passphrase for symmetric encryption%s%s: "
-           ;; Add the file name to the prompt, if any.
-           (if (stringp handback)
-               (format " for %s" handback)
-             "")
-           (if (stringp sym-detail)
-               sym-detail
-             ""))
-   (eq (epg-context-operation context) 'encrypt)))
-
 (defun auth-source-token-passphrase-callback-function (context key-id file)
-  (if (eq key-id 'SYM)
       (let* ((file (file-truename file))
              (entry (assoc file auth-source-passphrase-alist))
              passphrase)
@@ -1014,14 +995,13 @@
               (unless entry
                 (setq entry (list file))
                 (push entry auth-source-passphrase-alist))
-              (setq passphrase (auth-source-passphrase-callback-function context
-                                                                 key-id
-                                                                 file
-                                                                 " tokens"))
+	  (setq passphrase
+		(read-passwd
+		 (format "Passphrase for for %s token: " file)
+		 t))
               (setcdr entry (lexical-let ((p (copy-sequence passphrase)))
                               (lambda () p)))
-              passphrase)))
-    (epa-passphrase-callback-function context key-id file)))
+	  passphrase))))
 
 ;; (auth-source-epa-extract-gpg-token "gpg:LS0tLS1CRUdJTiBQR1AgTUVTU0FHRS0tLS0tClZlcnNpb246IEdudVBHIHYxLjQuMTEgKEdOVS9MaW51eCkKCmpBMEVBd01DT25qMjB1ak9rZnRneVI3K21iNm9aZWhuLzRad3cySkdlbnVaKzRpeEswWDY5di9icDI1U1dsQT0KPS9yc2wKLS0tLS1FTkQgUEdQIE1FU1NBR0UtLS0tLQo=" "~/.netrc")
 (defun auth-source-epa-extract-gpg-token (secret file)


[-- Attachment #3: Type: text/plain, Size: 25 bytes --]


Regards,
-- 
Daiki Ueno

Re: secure plist store

[Ted Zlatanov]

On Fri, 01 Jul 2011 08:10:21 +0900 Daiki Ueno <ueno@unixuser.org> wrote: 

DU> Ted Zlatanov <tzz@lifelogs.com> writes:

DU> Also, if you use only symmetric encryption, you can remove the
DU> condition Y: (if (eq key-id 'SYM) X Y)
>> 
>> Thanks, I'll do that.

DU> You could simplify more; patch attached:

Thanks.  I committed that in your name plus I reindented.

Ted

Re: secure plist store

[Ted Zlatanov]

[-- Attachment #1: Type: text/plain, Size: 50 bytes --]

Of course, I forgot to attach the patch.  Sorry!


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: auth-source-epg-direct-calls.patch --]
[-- Type: text/x-diff, Size: 10661 bytes --]

diff --git a/lisp/auth-source.el b/lisp/auth-source.el
index 146db11..4087675 100644
--- a/lisp/auth-source.el
+++ b/lisp/auth-source.el
@@ -43,6 +43,9 @@
 (require 'mm-util)
 (require 'gnus-util)
 (require 'assoc)
+(require 'epa)
+(require 'epg)
+
 (eval-when-compile (require 'cl))
 (eval-and-compile
   (or (ignore-errors (require 'eieio))
@@ -972,56 +975,86 @@ Note that the MAX parameter is used so we can exit the parse early."
 
           (nreverse result))))))
 
-(defmacro with-auth-source-epa-overrides (&rest body)
-  `(let ((file-name-handler-alist
-          ',(if (boundp 'epa-file-handler)
-                (remove (symbol-value 'epa-file-handler)
-                        file-name-handler-alist)
-              file-name-handler-alist))
-         (,(if (boundp 'find-file-hook) 'find-file-hook 'find-file-hooks)
-          ',(remove
-             'epa-file-find-file-hook
-             (if (boundp 'find-file-hook)
-		 (symbol-value 'find-file-hook)
-	       (symbol-value 'find-file-hooks))))
-         (auto-mode-alist
-          ',(if (boundp 'epa-file-auto-mode-alist-entry)
-                (remove (symbol-value 'epa-file-auto-mode-alist-entry)
-                        auto-mode-alist)
-              auto-mode-alist)))
-     ,@body))
-
+(defvar auth-source-passphrase-alist nil)
+
+(defun auth-source-passphrase-callback-function (context key-id handback
+                                                         &optional sym-detail)
+"Exactly like `epa-passphrase-callback-function' but takes an
+extra SYM-DETAIL parameter which will be printed at the end of
+the symmetric passphrase prompt."
+  (if (eq key-id 'SYM)
+      (read-passwd
+       (format "Passphrase for symmetric encryption%s%s: "
+	       ;; Add the file name to the prompt, if any.
+	       (if (stringp handback)
+		   (format " for %s" handback)
+		 "")
+	       (if (stringp sym-detail)
+		   sym-detail
+		 ""))
+       (eq (epg-context-operation context) 'encrypt))
+    (read-passwd
+     (if (eq key-id 'PIN)
+	"Passphrase for PIN: "
+       (let ((entry (assoc key-id epg-user-id-alist)))
+	 (if entry
+	     (format "Passphrase for %s %s: " key-id (cdr entry))
+	   (format "Passphrase for %s: " key-id)))))))
+
+(defun auth-source-token-passphrase-callback-function (context key-id file)
+  (if (eq key-id 'SYM)
+      (let* ((file (file-truename file))
+             (entry (assoc file auth-source-passphrase-alist))
+             passphrase)
+        ;; return the saved passphrase, calling a function if needed
+        (or (copy-sequence (if (functionp (cdr entry))
+                               (funcall (cdr entry))
+                             (cdr entry)))
+            (progn
+              (unless entry
+                (setq entry (list file))
+                (push entry auth-source-passphrase-alist))
+              (setq passphrase (auth-source-passphrase-callback-function context
+                                                                 key-id
+                                                                 file
+                                                                 " tokens"))
+              (setcdr entry (lexical-let ((p (copy-sequence passphrase)))
+                              (lambda () p)))
+              passphrase)))
+    (auth-source-passphrase-callback-function context key-id file " tokens")))
+
+
+;; (auth-source-epa-extract-gpg-token "gpg:LS0tLS1CRUdJTiBQR1AgTUVTU0FHRS0tLS0tClZlcnNpb246IEdudVBHIHYxLjQuMTEgKEdOVS9MaW51eCkKCmpBMEVBd01DT25qMjB1ak9rZnRneVI3K21iNm9aZWhuLzRad3cySkdlbnVaKzRpeEswWDY5di9icDI1U1dsQT0KPS9yc2wKLS0tLS1FTkQgUEdQIE1FU1NBR0UtLS0tLQo=" "~/.netrc")
+(defun auth-source-epa-extract-gpg-token (secret file)
+  "Pass either the decoded SECRET or the gpg:BASE64DATA version.
+FILE is the file from which we obtained this token."
+  (when (string-match "^gpg:\\(.+\\)" secret)
+    (setq secret (base64-decode-string (match-string 1 secret))))
+  (let ((context (epg-make-context 'OpenPGP))
+        plain)
+    (epg-context-set-passphrase-callback
+     context
+     (cons #'auth-source-token-passphrase-callback-function
+           file))
+    (epg-decrypt-string context secret)))
+
+;; (insert (auth-source-epa-make-gpg-token "mysecret" "~/.netrc"))
 (defun auth-source-epa-make-gpg-token (secret file)
-  (require 'epa nil t)
-  (unless (featurep 'epa)
-    (error "EPA could not be loaded."))
-  (let* ((base (file-name-sans-extension file))
-         (passkey (format "gpg:-%s" base))
-         (stash (concat base ".gpg"))
-         ;; temporarily disable EPA
-         (stashfile
-          (with-auth-source-epa-overrides
-           (make-temp-file "gpg-token" nil
-                           stash)))
-         (epa-file-passphrase-alist
-          `((,stashfile
-             . ,(password-read
-                 (format
-                  "token pass for %s? "
-                  file)
-                 passkey)))))
-    (write-region secret nil stashfile)
-    ;; temporarily disable EPA
-    (unwind-protect
-        (with-auth-source-epa-overrides
-         (with-temp-buffer
-           (insert-file-contents stashfile)
-           (base64-encode-region (point-min) (point-max) t)
-           (concat "gpg:"
-                   (buffer-substring-no-properties
-                    (point-min)
-                    (point-max)))))
-      (delete-file stashfile))))
+  (let ((context (epg-make-context 'OpenPGP))
+        (pp-escape-newlines nil)
+        cipher)
+    (epg-context-set-armor context t)
+    (epg-context-set-passphrase-callback
+     context
+     (cons #'auth-source-token-passphrase-callback-function
+           file))
+    (setq cipher (epg-encrypt-string context secret nil))
+    (with-temp-buffer
+      (insert cipher)
+      (base64-encode-region (point-min) (point-max) t)
+      (concat "gpg:" (buffer-substring-no-properties
+                      (point-min)
+                      (point-max))))))
 
 (defun auth-source-netrc-normalize (alist filename)
   (mapcar (lambda (entry)
@@ -1039,60 +1072,22 @@ Note that the MAX parameter is used so we can exit the parse early."
 
                   ;; send back the secret in a function (lexical binding)
                   (when (equal k "secret")
-                    (setq v (lexical-let ((v v)
-                                          (filename filename)
-                                          (base (file-name-nondirectory
-                                                 filename))
-                                          (token-decoder nil)
-                                          (gpgdata nil)
-                                          (stash nil))
-                              (setq stash (concat base ".gpg"))
-                              (when (string-match "gpg:\\(.+\\)" v)
-                                (require 'epa nil t)
-                                (unless (featurep 'epa)
-                                  (error "EPA could not be loaded."))
-                                (setq gpgdata (base64-decode-string
-                                               (match-string 1 v)))
-                                ;; it's a GPG token
-                                (setq
-                                 token-decoder
-                                 (lambda (gpgdata)
-;;; FIXME: this relies on .gpg files being handled by EPA/EPG
-                                   (let* ((passkey (format "gpg:-%s" base))
-                                          ;; temporarily disable EPA
-                                          (stashfile
-                                           (with-auth-source-epa-overrides
-                                            (make-temp-file "gpg-token" nil
-                                                            stash)))
-                                          (epa-file-passphrase-alist
-                                           `((,stashfile
-                                              . ,(password-read
-                                                  (format
-                                                   "token pass for %s? "
-                                                   filename)
-                                                  passkey)))))
-                                     (unwind-protect
-                                         (progn
-                                           ;; temporarily disable EPA
-                                           (with-auth-source-epa-overrides
-                                            (write-region gpgdata
-                                                          nil
-                                                          stashfile))
-                                           (setq
-                                            v
-                                            (with-temp-buffer
-                                              (insert-file-contents stashfile)
-                                              (buffer-substring-no-properties
-                                               (point-min)
-                                               (point-max)))))
-                                       (delete-file stashfile)))
-                                   ;; clear out the decoder at end
-                                   (setq token-decoder nil
-                                         gpgdata nil))))
-                          (lambda ()
-                            (when token-decoder
-                              (funcall token-decoder gpgdata))
-                            v))))
+                    (setq v (lexical-let ((lexv v)
+                                          (token-decoder nil))
+                              (when (string-match "^gpg:" lexv)
+                                ;; it's a GPG token: create a token decoder
+                                ;; which unsets itself once
+                                (setq token-decoder
+                                      (lambda (val)
+                                        (prog1
+                                            (auth-source-epa-extract-gpg-token
+                                             val
+                                             filename)
+                                          (setq token-decoder nil)))))
+                              (lambda ()
+                                (when token-decoder
+                                  (setq lexv (funcall token-decoder lexv)))
+                                lexv))))
                 (setq ret (plist-put ret
                                      (intern (concat ":" k))
                                      v))))

Re: secure plist store

[Ted Zlatanov]

On Wed, 29 Jun 2011 17:07:57 +0900 Daiki Ueno <ueno@unixuser.org> wrote: 

DU> I don't want to see that the Gnus password-caching feature becomes
DU> harder and harder to use daily... Is it not feasible to stop reusing
DU> netrc pieces and employ a new format, which is more GPG friendly?  Yeah,
DU> I'm reluctant to repeat the same discussion - here is a
DU> proof-of-concept implementation of searchable, partially encrypted,
DU> persistent plist store, called plstore.el.

Regardless of the other discussion about netrc files, do you want
plstore.el to be an auth-source backend?  The
create/search/delete/modify behavior can be defined as you see fit and
does not have to work like the netrc backend.

Thanks
Ted

Re: secure plist store

[Daiki Ueno]

Ted Zlatanov <tzz@lifelogs.com> writes:

> Regardless of the other discussion about netrc files, do you want
> plstore.el to be an auth-source backend?  The
> create/search/delete/modify behavior can be defined as you see fit and
> does not have to work like the netrc backend.

I hope that you don't mind, I've just checked in the plstore backend of
auth-source.  It works pretty well for me.  In summary:

PROS:
- it works with GPG 2 (unlike netrc field encryption)
- it does not run GPG until the secret is really needed (unlike
  ~/.authinfo.gpg)
- it writes secrets in encrypted form (unlike ~/.authinfo)
- the encrypted form can be easily decrypted using M-x
  epa-decrypt-region (unlike netrc field encryption)

CONS:
- the file format is not easily editable
- the code is not mature (delete is not supported)

Anyway, if you want to try, set:

(setq auth-sources '("~/.emacs.d/auth.plist"))

Regards,
-- 
Daiki Ueno

Re: secure plist store

[Ted Zlatanov]

On Thu, 30 Jun 2011 16:43:33 +0900 Daiki Ueno <ueno@unixuser.org> wrote: 

DU> Ted Zlatanov <tzz@lifelogs.com> writes:
>> Regardless of the other discussion about netrc files, do you want
>> plstore.el to be an auth-source backend?  The
>> create/search/delete/modify behavior can be defined as you see fit and
>> does not have to work like the netrc backend.

DU> I hope that you don't mind, I've just checked in the plstore backend of
DU> auth-source.  It works pretty well for me.

I don't mind, but it looks like you've copied and pasted a lot of the
netrc code.  This is expedient but we should extract the common
functionality into functions or macros, so it doesn't become a
maintenance nightmare later.

Also you added a generic "arg" parameter to the backend.  The other
parameters are named: source, host, port, user, type.  Can the name be
more specific, so we don't have to guess what it means?  In your case,
it's set to

(plstore-open (plist-get entry :source))

which is, I think, the plstore instance, a defstruct-like vector.  So
maybe the parameter should be called "data" or "instance" or
"internal-data"?  WDYT?

DU> In summary:

DU> PROS:
DU> - it works with GPG 2 (unlike netrc field encryption)
DU> - it does not run GPG until the secret is really needed (unlike
DU>   ~/.authinfo.gpg)
DU> - it writes secrets in encrypted form (unlike ~/.authinfo)
DU> - the encrypted form can be easily decrypted using M-x
DU>   epa-decrypt-region (unlike netrc field encryption)

DU> CONS:
DU> - the file format is not easily editable
DU> - the code is not mature (delete is not supported)

It would be great if we had a table explaining all this in the
auth-source manual.  That's very useful information.

DU> Anyway, if you want to try, set:

DU> (setq auth-sources '("~/.emacs.d/auth.plist"))

Thank you for doing this work!  It's great to have varied auth-source
backends that can fit every user's security needs.

Ted

Re: secure plist store

[Daiki Ueno]

Ted Zlatanov <tzz@lifelogs.com> writes:

> Also you added a generic "arg" parameter to the backend.  The other
> parameters are named: source, host, port, user, type.  Can the name be
> more specific, so we don't have to guess what it means?  In your case,
> it's set to
>
> (plstore-open (plist-get entry :source))
>
> which is, I think, the plstore instance, a defstruct-like vector.  So
> maybe the parameter should be called "data" or "instance" or
> "internal-data"?  WDYT?

I don't care about that - maybe host/port/user (currently unused?) can
also be "internal-data".  Frankly I don't see any reason to use defclass
here - why not using CLOS inheritance if you want to define members
specific to some class derived from auth-source-backend (I mean
host/port/user are only valuable for Secrets API backend)?

Regards,
-- 
Daiki Ueno

Re: secure plist store

[Ted Zlatanov]

On Thu, 30 Jun 2011 22:42:31 +0900 Daiki Ueno <ueno@unixuser.org> wrote: 

DU> Ted Zlatanov <tzz@lifelogs.com> writes:
>> Also you added a generic "arg" parameter to the backend.  The other
>> parameters are named: source, host, port, user, type.  Can the name be
>> more specific, so we don't have to guess what it means?  In your case,
>> it's set to
>> 
>> (plstore-open (plist-get entry :source))
>> 
>> which is, I think, the plstore instance, a defstruct-like vector.  So
>> maybe the parameter should be called "data" or "instance" or
>> "internal-data"?  WDYT?

DU> I don't care about that - maybe host/port/user (currently unused?) can
DU> also be "internal-data".

They are used.  See the backend filtering in `auth-source-search'
coupled with `auth-source-backend-parse-parameters'.  When you specify a
simple string as an auth-source, the host, port, and user are nil (so
that backend instance applies to every host, port, and user).

Since you don't care, I've renamed "arg" to "data".  There is no
functional difference.

DU> Frankly I don't see any reason to use defclass here - why not using
DU> CLOS inheritance if you want to define members specific to some
DU> class derived from auth-source-backend (I mean host/port/user are
DU> only valuable for Secrets API backend)?

For this case, EIEIO worked better for me.  I've found some issues with
it, but overall it's a nice system that covers a lot of ground that
CLOS does not.

Ted

Re: secure plist store

[Daiki Ueno]

Ted Zlatanov <tzz@lifelogs.com> writes:

> They are used.  See the backend filtering in `auth-source-search'
> coupled with `auth-source-backend-parse-parameters'.  When you specify a
> simple string as an auth-source, the host, port, and user are nil (so
> that backend instance applies to every host, port, and user).

I'm now confused about that, while those members are set in
auth-source-backend-parse-parameters, they are not referred from any
code in auth-source.el.  Am I missing something (I tried M-x occur oref
and slot-value)?

> DU> Frankly I don't see any reason to use defclass here - why not using
> DU> CLOS inheritance if you want to define members specific to some
> DU> class derived from auth-source-backend (I mean host/port/user are
> DU> only valuable for Secrets API backend)?
>
> For this case, EIEIO worked better for me.  I've found some issues with
> it, but overall it's a nice system that covers a lot of ground that
> CLOS does not.

I meant EIEIO with "CLOS".  From the EIEIO doc:

  EIEIO ("Enhanced Implementation of Emacs Interpreted Objects") is a
  CLOS (Common Lisp Object System) compatibility layer for Emacs Lisp.

I was really confused about auth-source.el usage of EIEIO, where
defclass is only used as defstruct.  It looks simply overkill such a
purpose and misleading.

Regards,
-- 
Daiki Ueno

Re: secure plist store

[Ted Zlatanov]

On Fri, 01 Jul 2011 07:18:23 +0900 Daiki Ueno <ueno@unixuser.org> wrote: 

DU> Ted Zlatanov <tzz@lifelogs.com> writes:
>> They are used.  See the backend filtering in `auth-source-search'
>> coupled with `auth-source-backend-parse-parameters'.  When you specify a
>> simple string as an auth-source, the host, port, and user are nil (so
>> that backend instance applies to every host, port, and user).

DU> I'm now confused about that, while those members are set in
DU> auth-source-backend-parse-parameters, they are not referred from any
DU> code in auth-source.el.  Am I missing something (I tried M-x occur oref
DU> and slot-value)?

`auth-source-search' will filter the backends with any keys that are
applicable before it actually searches each backend:

#+begin_src lisp
      (setq filtered-backends (copy-sequence backends))
      (dolist (backend backends)
        (dolist (key keys)
          ;; ignore invalid slots
          (condition-case signal
              (unless (eval `(auth-source-search-collection
                              (plist-get spec key)
                              (oref backend ,key)))
                (setq filtered-backends (delq backend filtered-backends))
                (return))
            (invalid-slot-name))))
#+end_src

So for instance a backend with host "x" will match a search spec of
:host '("x" "y") or :host "x".

DU> Frankly I don't see any reason to use defclass here - why not using
DU> CLOS inheritance if you want to define members specific to some
DU> class derived from auth-source-backend (I mean host/port/user are
DU> only valuable for Secrets API backend)?
...
DU> I was really confused about auth-source.el usage of EIEIO, where
DU> defclass is only used as defstruct.  It looks simply overkill such a
DU> purpose and misleading.

For now, yes, since we only had netrc and Secrets API backends, which
are too different, and the Secrets API didn't need most of the prompting
and modification functionality.  But notice how much code is shared
between plstore and netrc.  I'll take a look at using the EIEIO
inheritance to abstract the prompting and modification to a generic
file-based class from which the plstore and netrc classes will derive.
That was the original intent with the class hierarchy, at least.

I don't think, in any case, that it's overkill or misleading to use
EIEIO instead of defstruct.  In many ways it's a nicer API and even as a
simple defstruct stand-in it's better.  Perhaps it's not suitable for
large collections of objects, but `auth-sources' has at most a few
entries.

Ted

Re: secure plist store

[Daiki Ueno]

Ted Zlatanov <tzz@lifelogs.com> writes:

> `auth-source-search' will filter the backends with any keys that are
> applicable before it actually searches each backend:
>
>                               (oref backend ,key)))

Ah, I missed that - thanks for the explanation.  I got it, but I think
the slot names should be more descriptive (like :supported-keys), or at
least have some comments around the defclass?

> I'll take a look at using the EIEIO inheritance to abstract the
> prompting and modification to a generic file-based class from which
> the plstore and netrc classes will derive.  That was the original
> intent with the class hierarchy, at least.

Good to hear that.  I would also suggest to make :search-function and
:create-function virtual methods.  That is a standard OO programming
style, I expected to see when I found defclass in auth-source.el ;-)

Regards,
-- 
Daiki Ueno

Re: secure plist store

[Ted Zlatanov]

On Fri, 01 Jul 2011 11:28:01 +0900 Daiki Ueno <ueno@unixuser.org> wrote: 

DU> Ted Zlatanov <tzz@lifelogs.com> writes:
>> `auth-source-search' will filter the backends with any keys that are
>> applicable before it actually searches each backend:
>> 
>> (oref backend ,key)))

DU> Ah, I missed that - thanks for the explanation.  I got it, but I think
DU> the slot names should be more descriptive (like :supported-keys), or at
DU> least have some comments around the defclass?

OK, I'll add comments.

>> I'll take a look at using the EIEIO inheritance to abstract the
>> prompting and modification to a generic file-based class from which
>> the plstore and netrc classes will derive.  That was the original
>> intent with the class hierarchy, at least.

DU> Good to hear that.  I would also suggest to make :search-function and
DU> :create-function virtual methods.  That is a standard OO programming
DU> style, I expected to see when I found defclass in auth-source.el ;-)

Yeah, none of that was necessary so far but plstore.el has created the
necessity (which is a good thing :)

Unfortunately we got your new code right before the feature freeze and
refactoring the common code would require a fairly large rewrite, which
is not really a bug fix.  I wonder if refactoring is OK under the
feature freeze?

Thanks
Ted

Re: secure plist store

[Daiki Ueno]

Ted Zlatanov <tzz@lifelogs.com> writes:

> DU> Good to hear that.  I would also suggest to make :search-function and
> DU> :create-function virtual methods.  That is a standard OO programming
> DU> style, I expected to see when I found defclass in auth-source.el ;-)
>
> Yeah, none of that was necessary so far but plstore.el has created the
> necessity

I don't think so.  First, it is not a requirement but a preferable
style.  Second, it is helpful for understanding the code to follow the
standard OO style whenever you use the OO framework, even if there is
only a few classes (I mean the base class, the netrc backend class, and
the Secrets API class).  Otherwise you should document like "we use
defclass as a rich-defstruct so far" somewhere.

> I wonder if refactoring is OK under the feature freeze?

I would say we should definitely wait for the next release cycle.

Regards,
-- 
Daiki Ueno

Re: GPGME

[Ted Zlatanov]

On Wed, 29 Jun 2011 05:36:02 +0900 Daiki Ueno <ueno@unixuser.org> wrote: 

DU> Ted Zlatanov <tzz@lifelogs.com> writes:
>> Are there any alternatives?  Maybe you remember our discussion years ago
>> about encrypt.el, where I proposed a neutral API with at least some
>> symmetric ciphers implemented in ELisp and C in the Emacs core
>> (essentially what Lars was requesting).

DU> I remember that the problem of encrypt.el was that the data format is
DU> not interoperable and the algorithm used is not interchangeable though
DU> the API might be neutral.

Is that a problem, if the intent is to provide an Emacs facility?

DU> I guess you need a minimal encryption function which employs the
DU> standard GPG message format (RFC4880).

I'm not sure that would benefit any Emacs users since EPA/EPG already
provide so much functionality for this and the GPG message format
doesn't seem to have any obvious benefits for simple data encryption.

>> Could something like that work
>> within the EPA/EPG structure, so some special invocation of
>> `epg-encrypt-string' could bypass the external callout to GPG?

DU> If your statement in <87wrh0fh4g.fsf_-_@lifelogs.com>:

DU>   The decoding will happen late, probably in the funcall to obtain the
DU>   secret (and it will set some scoped variables to cache the data)

DU> is true, epg-encrypt-string is not necessarily to be optimized in that
DU> way, I think.  How about implementing your side first and profiling
DU> before the optimization?

That's not a performance optimization.  We decode late to avoid
prompting the user for a passphrase before the password is actually
needed.

I'm asking if, instead of a new package, we can use `epg-encrypt-string'
to provide symmetric encryption without calling GPG externally.  It can
provide it in any format and with any symmetric cipher you think would
make sense.  But if you don't think so, then we need a new package to
provide that functionality.

DU> One suggestion to reduce the number of calls to epg-encrypt-string is
DU> that, I would suggest encrypt the key name as well.  For example,

DU>   key1 val1 encrypted hexdata

DU> where hexdata is decrypted to:

DU>   key2 val2 key3 val3

But if we do that, we have to decrypt the hexdata in order to know it
has key2 and key3.  The benefit of the GPG tokens for netrc field
encryption is that you know all the keys, so you can search for "must
have key X" with confidence, without prompting the user, and without
extra decryption overhead at search time.  Obviosly if you search for
"must have key X with value Y" that doesn't work, but typically we don't
encrypt things that we search for.  Right now we only encrypt the
password in any case.

Ted

Re: GPGME

[Daiki Ueno]

Ted Zlatanov <tzz@lifelogs.com> writes:

> DU> I remember that the problem of encrypt.el was that the data format is
> DU> not interoperable and the algorithm used is not interchangeable though
> DU> the API might be neutral.
>
> Is that a problem, if the intent is to provide an Emacs facility?

The benefit using the standard GPG format is that it embeds what
algorithm is used in data.

If Emacs provides bare cipher algorithms, someone may write encryption
routines with their own format which won't be interoperable - that is a
problem I think.  For my case, I had used GPass
http://gpass.sourceforge.net as my password manager (currently
unmaintained).  It turned out that it used bare Blowfish internally and
I had to write a script to salvage my password data.

> I'm asking if, instead of a new package, we can use `epg-encrypt-string'
> to provide symmetric encryption without calling GPG externally.  It can
> provide it in any format and with any symmetric cipher you think would
> make sense.  But if you don't think so, then we need a new package to
> provide that functionality.

I don't care about it.

Regards,
-- 
Daiki Ueno

Re: GPGME

[Ted Zlatanov]

On Wed, 29 Jun 2011 22:15:02 +0900 Daiki Ueno <ueno@unixuser.org> wrote: 

DU> Ted Zlatanov <tzz@lifelogs.com> writes:
DU> I remember that the problem of encrypt.el was that the data format is
DU> not interoperable and the algorithm used is not interchangeable though
DU> the API might be neutral.
>> 
>> Is that a problem, if the intent is to provide an Emacs facility?

DU> The benefit using the standard GPG format is that it embeds what
DU> algorithm is used in data.

I don't see why it's not better to implement a storage format that works
well with ELisp.  For instance the format could be a simple plist:

(:cipher xyz :cipher-version 1.2 :data BASE64DATA :other-cipher-parameter abc :checksum SUM)

The standard OpenPGP format described in the 90-page RFC is extremely
flexible and powerful (it's packet based and has more options than GNU
ls); I think it would be a lot of work to implement it even for just a
few symmetric ciphers.  OTOH the plist format above can be done in a few
hours and has no external dependencies.  It would only be usable by
Emacs, but I honestly don't see the harm in that.

Ted

Re: GPGME

[Daiki Ueno]

Ted Zlatanov <tzz@lifelogs.com> writes:

> The standard OpenPGP format described in the 90-page RFC is extremely
> flexible and powerful (it's packet based and has more options than GNU
> ls); I think it would be a lot of work to implement it even for just a
> few symmetric ciphers.

It is not that complex - for symmetric encryption, only limited part is
used.  Try:

  $ gpg --list-packets aaa.txt.gpg
  :symkey enc packet: version 4, cipher 3, s2k 3, hash 2
  	salt 071379daac57c0c1, count 65536 (96)
  gpg: CAST5 encrypted data
  :encrypted data packet:
  	length: 32
  gpg: encrypted with 1 passphrase
  :compressed packet: algo=1
  :literal data packet:
  	mode b (62), created 1309372527, name="aaa.txt",
  	raw data: 4 bytes
  gpg: WARNING: message was not integrity protected

So there is actually four packets.  Also, you could drop the support of
compression.

Regards,
-- 
Daiki Ueno

Re: GPGME

[Ted Zlatanov]

On Thu, 30 Jun 2011 03:41:52 +0900 Daiki Ueno <ueno@unixuser.org> wrote: 

DU> Ted Zlatanov <tzz@lifelogs.com> writes:
>> The standard OpenPGP format described in the 90-page RFC is extremely
>> flexible and powerful (it's packet based and has more options than GNU
>> ls); I think it would be a lot of work to implement it even for just a
>> few symmetric ciphers.

DU> It is not that complex - for symmetric encryption, only limited part is
DU> used.  Try:

DU>   $ gpg --list-packets aaa.txt.gpg
DU>   :symkey enc packet: version 4, cipher 3, s2k 3, hash 2
DU>   	salt 071379daac57c0c1, count 65536 (96)
DU>   gpg: CAST5 encrypted data
DU>   :encrypted data packet:
DU>   	length: 32
DU>   gpg: encrypted with 1 passphrase
DU>   :compressed packet: algo=1
DU>   :literal data packet:
DU>   	mode b (62), created 1309372527, name="aaa.txt",
DU>   	raw data: 4 bytes
DU>   gpg: WARNING: message was not integrity protected

DU> So there is actually four packets.  Also, you could drop the support of
DU> compression.

Compression is specifically recommended by the RFC, so it's hard to
justify skipping it.

I'm really not eager to write a pile of C to implement this, even if
it's just 4 packets.  The math is not trivial and getting any of the
details wrong is too easy for me--my knowledge of cryptography is too
shallow.  I agree it would be nice and *many* parts of Emacs could use
the OpenPGP message format if internal support were available, but I
would rather use a library that write new code.

So, are there any libraries that will do what GPGME does, but as a real
library and not a shim, creating a stream of OpenPGP packets and
optionally ASCII-armoring them?

Ted

Re: Opportunistic STARTTLS in smtpmail.el

[Ted Zlatanov]

On Thu, 02 Jun 2011 01:09:36 -0300 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

>> From my perspective the chief benefit is that any `auth-source-search'
>> call against an unencrypted file will not require a passphrase until the
>> password is actually needed, and yet the password will be stored
>> securely.

SM> Sounds OK.  But only if you push if further and deprecate
SM> authinfo.gpg.

On Thu, 02 Jun 2011 10:57:41 +0200 Robert Pluim <rpluim@gmail.com> wrote: 

RP> I'm not clear on why you'd want that.  I can imagine someone wanting to
RP> hide username & server identities from inspection, not just the
RP> associated passwords. ie I distinguish 3 cases

RP> 1) everything unencrypted
RP> 2) passwords encrypted only
RP> 3) everything encrypted

It will be less necessary as the first `auth-sources' choice, but still
useful, as Robert noted (I see case 2 as "encrypted tokens" since any
token can be encrypted in my proposal).

I'll simply make `auth-sources' ("~/.authinfo" "~/.authinfo.gpg")

which as a default will work fine.  Creation prompts will target the
first one.  The users can put insecure or token-encrypted data in the
first one and use the second one for more secure storage.

Ted

Re: Opportunistic STARTTLS in smtpmail.el

[Daiki Ueno]

Ted Zlatanov <tzz@lifelogs.com> writes:

> It will be less necessary as the first `auth-sources' choice, but still
> useful, as Robert noted (I see case 2 as "encrypted tokens" since any
> token can be encrypted in my proposal).

Could you define the word "token"?  I have thought that it is an opaque
value which connects unencrypted data to encrypted data.  However, it
seems not.

> I'll simply make `auth-sources' ("~/.authinfo" "~/.authinfo.gpg")
>
> which as a default will work fine.  Creation prompts will target the
> first one.  The users can put insecure or token-encrypted data in the
> first one and use the second one for more secure storage.

Though I can't say anything without the detail of your proposal, I feel
you are oversimplifying.

Anyway I will be happy if:

- Gnus does not ask password when connecting to password-less services
- Gnus does not store my passwords in plain text files by default
- Gnus can hide server or user names (or other attributes) if necessary

Otherwise, no persistent password store is much better for me...

Regards,
-- 
Daiki Ueno

Re: Opportunistic STARTTLS in smtpmail.el

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> I understand.  But it sucks from the `auth-source-search' perspective
> because now every secret blob has to be decoded to find out if it has
> tokens X or Y when the search spec requires X or Y.  So I'm against it.

True, I didn't think of that.  Then I think your idea for this makes
most sense, and please go ahead and implement it.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Opportunistic STARTTLS in smtpmail.el

[Stefan Monnier]

> So I'd really prefer a way to query whether credentials exists without
> having to enter a password.  Which I think means that an /etc/passwd +
> /etc/secrets-type split is unavoidable.

I'd rather add an smtpmail-use-ssl boolean.  After all, most other MUAs
have such a toggle-box.


        Stefan

Re: Opportunistic STARTTLS in smtpmail.el

[Lars Magne Ingebrigtsen]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

>> So I'd really prefer a way to query whether credentials exists without
>> having to enter a password.  Which I think means that an /etc/passwd +
>> /etc/secrets-type split is unavoidable.
>
> I'd rather add an smtpmail-use-ssl boolean.  After all, most other MUAs
> have such a toggle-box.

This isn't about SSL, but about auth.  Which are almost, but not
totally, orthogonal issues.  :-)

(The non-orthogonality is because I wanted to allow stashing certificate
pointers into the .authinfo file in addition to the user name/password,
but that's a detail...)

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Opportunistic STARTTLS in smtpmail.el

[Stefan Monnier]

>>> So I'd really prefer a way to query whether credentials exists without
>>> having to enter a password.  Which I think means that an /etc/passwd +
>>> /etc/secrets-type split is unavoidable.
>> I'd rather add an smtpmail-use-ssl boolean.  After all, most other MUAs
>> have such a toggle-box.
> This isn't about SSL, but about auth.  Which are almost, but not
> totally, orthogonal issues.  :-)

Oh, right, it should be smtpmail-use-auth.


        Stefan

Re: Opportunistic STARTTLS in smtpmail.el

[Lars Magne Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Oh, right, it should be smtpmail-use-auth.

I don't think that's necessary.

Here's my vision of how this would work for a person using Emacs 24 to
send an email for the first time.

1) smtpmail would see that there's no host setup, so it would prompt the
user for the SMTP host and store it via customize.

2) smtpmail connects to the server, and possibly does STARTTLS

III) It sees whether the server says that it accepts AUTH or not.  If it
does, smtpmail then asks auth-source for credentials.  There will be
none there, probably

d) smtpmail does MAIL FROM, RCPT TO and DATA, and sees whether that was
successful.  If it is, we're done.

5) If we get a 5xx relay denied, we then ask auth-source to query the
user for user name and password, and then gives the server the AUTH, and
try again.  If it's successful, we're done.  If not, we give up.

Subsequent mails will be similar, only that we will know that we have
auth, so we do AUTH immediately.

I think we can pretty much get away with zero conf here.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Opportunistic STARTTLS in smtpmail.el

[Ted Zlatanov]

On Fri, 03 Jun 2011 23:48:50 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> 1) smtpmail would see that there's no host setup, so it would prompt the
LMI> user for the SMTP host and store it via customize.

What if I want to put the host and port in my ~/.authinfo?

LMI> 2) smtpmail connects to the server, and possibly does STARTTLS

LMI> III) It sees whether the server says that it accepts AUTH or not.  If it
LMI> does, smtpmail then asks auth-source for credentials.  There will be
LMI> none there, probably

LMI> d) smtpmail does MAIL FROM, RCPT TO and DATA, and sees whether that was
LMI> successful.  If it is, we're done.

LMI> 5) If we get a 5xx relay denied, we then ask auth-source to query the
LMI> user for user name and password, and then gives the server the AUTH, and
LMI> try again.  If it's successful, we're done.  If not, we give up.

LMI> Subsequent mails will be similar, only that we will know that we have
LMI> auth, so we do AUTH immediately.

LMI> I think we can pretty much get away with zero conf here.

...as long as there's a way to avoid steps tres through пет with some
external setting; I would like it if I could explicitly say "go ahead
and query auth-source for everything from the start" because I store the
login, password, and port for my SMTP hosts.  Then the 5xx errors can be
avoided.

Ted

Re: Opportunistic STARTTLS in smtpmail.el

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> LMI> 1) smtpmail would see that there's no host setup, so it would prompt the
> LMI> user for the SMTP host and store it via customize.
>
> What if I want to put the host and port in my ~/.authinfo?

I think the ~/.authinfo file can be consulted for possible defaults to
use, but the file may have several SMTP entries, so it kinda seems...
slightly sloppy.

And even if there is only one entry there, the user may want to use a
different host anyway, so I'm not sure it really helps.

Not to mention in a multi-SMTP-server setting, where you're sending out
via different SMTP servers depending on what your From address is.

Anyway, I think that can be hashed out after implementing it.  My
problem is that I'm away both this weekend and the next weekend, and
since there should be no new features after July 1st, time is getting
rather short.

I'll do my best to implement the new smtpmail.el stuff on Wednesday,
which looks like it's going to be free.  Is there any possibility of
getting the gpg: tokens done before that?  :-)  

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Opportunistic STARTTLS in smtpmail.el

[Ted Zlatanov]

On Thu, 09 Jun 2011 20:02:02 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> I'll do my best to implement the new smtpmail.el stuff on Wednesday,
LMI> which looks like it's going to be free.  Is there any possibility of
LMI> getting the gpg: tokens done before that?  :-)  

Yes.

Ted

netrc field encryption in auth-source (was: Opportunistic STARTTLS in smtpmail.el)

[Ted Zlatanov]

[-- Attachment #1: Type: text/plain, Size: 1690 bytes --]

On Thu, 09 Jun 2011 20:02:02 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> I'll do my best to implement the new smtpmail.el stuff on Wednesday,
LMI> which looks like it's going to be free.  Is there any possibility of
LMI> getting the gpg: tokens done before that?  :-)  

See the attached patch.  I really hope the nasty code in there will
inspire you or Daiki Ueno to tell me how to do it better.  It depends on
the EPA being active and tries to load it opportunistically.  It also
depends on symmetric encryption to be enabled.

The GPG token decoder is activated if the secret matches
"gpg:base64data=="; otherwise the secret lambda just returns the
secret.  So the token decoder does the following, given a filename and
the decoded GPG token data:

1) make a .gpg temp stashfile associated with the original netrc
filename

2) set the symmetric passphrase for that stashfile to something we
`password-read' from the user, again associating the password with the
original netrc filename.  This is done with a nasty dynamic scope
override of `epa-file-passphrase-alist'.

3) write the decoded GPG token into the stashfile outside of EPA

4) read the decrypted secret from the stashfile using EPA and set the
secret lambda to return it from then on

5) clear out the token decoder lambda so there's no chance it will get
called again

Test it: put this in a netrc file

machine supertest password gpg:jA0EAwMCdmeQLC3gFEpgyR3UxXKPoS5Yzzjl4+v/iaGPAVzwrIGOYVC+XCKcnA==

and then do

(let ((auth-sources '("your-netrc-file")))
  (auth-source-forget-all-cached) 
  (funcall (plist-get (nth 0 (auth-source-search :host "supertest")) :secret)))

Let me know what you think.

Ted


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: auth-source-gpg-tokens.patch --]
[-- Type: text/x-diff, Size: 5604 bytes --]

diff --git a/lisp/auth-source.el b/lisp/auth-source.el
index ce483e4..18eeb35 100644
--- a/lisp/auth-source.el
+++ b/lisp/auth-source.el
@@ -908,7 +908,7 @@ Note that the MAX parameter is used so we can exit the parse early."
                         (null require)
                         ;; every element of require is in the normalized list
                         (let ((normalized (nth 0 (auth-source-netrc-normalize
-                                                 (list alist)))))
+                                                 (list alist) file))))
                           (loop for req in require
                                 always (plist-get normalized req)))))
               (decf max)
@@ -944,7 +944,16 @@ Note that the MAX parameter is used so we can exit the parse early."
 
           (nreverse result))))))
 
-(defun auth-source-netrc-normalize (alist)
+(defmacro with-auth-source-epa-overrides (&rest body)
+  `(let ((file-name-handler-alist
+          ',(remove epa-file-handler file-name-handler-alist))
+         (find-file-hook
+          ',(remove 'epa-file-find-file-hook find-file-hook))
+         (auto-mode-alist
+          ',(remove epa-file-auto-mode-alist-entry auto-mode-alist)))
+     ,@body))
+
+(defun auth-source-netrc-normalize (alist filename)
   (mapcar (lambda (entry)
             (let (ret item)
               (while (setq item (pop entry))
@@ -960,13 +969,59 @@ Note that the MAX parameter is used so we can exit the parse early."
 
                   ;; send back the secret in a function (lexical binding)
                   (when (equal k "secret")
-                    (setq v (lexical-let ((v v))
-                              (lambda () v))))
-
+                    (setq v (lexical-let ((v v)
+                                          (filename filename)
+                                          (base (file-name-nondirectory
+                                                 filename))
+                                          (token-decoder nil)
+                                          (gpgdata nil)
+                                          (stash nil))
+                              (setq stash (concat base ".gpg"))
+                              (when (string-match "gpg:\\(.+==\\)" v)
+                                (require 'epa nil t)
+                                (unless (featurep 'epa)
+                                  (error "EPA could not be loaded."))
+                                (setq gpgdata (base64-decode-string
+                                               (match-string 1 v)))
+                                ;; it's a GPG token
+                                (setq token-decoder
+                                      (lambda (gpgdata)
+;;; FIXME: this relies on .gpg files being handled by EPA/EPG
+                                        (let* ((passkey (format "gpg:-%s" base))
+                                               ;; temporarily disable EPA
+                                               (stashfile
+                                                (with-auth-source-epa-overrides
+                                                 (make-temp-file "gpg-token" nil
+                                                                 stash)))
+                                               (epa-file-passphrase-alist
+                                                `((,stashfile
+                                                   . ,(password-read
+                                                       (format
+                                                       "token pass for %s? "
+                                                       filename)
+                                                      passkey)))))
+                                          ;; temporarily disable EPA
+                                          (with-auth-source-epa-overrides
+                                           (write-region gpgdata
+                                                         nil
+                                                         stashfile))
+                                          (setq
+                                           v
+                                           (with-temp-buffer
+                                             (insert-file-contents stashfile)
+                                             (buffer-substring-no-properties
+                                              (point-min)
+                                              (point-max))))
+                                        ;; clear out the decoder at end
+                                        (setq token-decoder nil
+                                              gpgdata nil)))))
+                              (lambda ()
+                                (when token-decoder
+                                  (funcall token-decoder gpgdata))
+                                v))))
                   (setq ret (plist-put ret
                                        (intern (concat ":" k))
-                                       v))
-                  ))
+                                       v))))
               ret))
           alist))
 
@@ -992,7 +1047,8 @@ See `auth-source-search' for details on SPEC."
                    :file (oref backend source)
                    :host (or host t)
                    :user (or user t)
-                   :port (or port t)))))
+                   :port (or port t))
+                  (oref backend source))))
 
     ;; if we need to create an entry AND none were found to match
     (when (and create

Re: netrc field encryption in auth-source

[Ted Zlatanov]

On Fri, 10 Jun 2011 11:05:57 -0500 Ted Zlatanov <tzz@lifelogs.com> wrote: 

TZ> On Thu, 09 Jun 2011 20:02:02 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 
LMI> I'll do my best to implement the new smtpmail.el stuff on Wednesday,
LMI> which looks like it's going to be free.  Is there any possibility of
LMI> getting the gpg: tokens done before that?  :-)  

TZ> See the attached patch.  I really hope the nasty code in there will
TZ> inspire you or Daiki Ueno to tell me how to do it better.  It depends on
TZ> the EPA being active and tries to load it opportunistically.  It also
TZ> depends on symmetric encryption to be enabled.

If you want this done by Wednesday (including the creation code), please
let me know if my code looked OK and if you find the interface
acceptable.  I won't be able to do it in time otherwise.

Thanks
Ted

Re: netrc field encryption in auth-source

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> If you want this done by Wednesday (including the creation code), please
> let me know if my code looked OK and if you find the interface
> acceptable.  I won't be able to do it in time otherwise.

I'm not back from holidaying yet. so I haven't looked at it im detail,
but it looked fine to me.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: netrc field encryption in auth-source

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> If you want this done by Wednesday (including the creation code), please
> let me know if my code looked OK and if you find the interface
> acceptable.  I won't be able to do it in time otherwise.

So please apply and I'll get started with the smtpmail.el stuff.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: netrc field encryption in auth-source

[Lars Magne Ingebrigtsen]

[-- Attachment #1: Type: text/plain, Size: 504 bytes --]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> So please apply and I'll get started with the smtpmail.el stuff.  :-)

I'm basically done with the smtpmail.el STARTTLS/AUTH/startup things,
and I've tested it with all the error cases I could think of (and beefed
up the error reporting significantly), so I think we're good to go.

But I'll wait until Tuesday to check this in.

It's a pretty big patch, but the good news is that the result is about
50 lines shorter than it was when I started.  :-)


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: smtpmail.diff --]
[-- Type: text/x-diff, Size: 28848 bytes --]

=== modified file 'lisp/mail/smtpmail.el'
*** lisp/mail/smtpmail.el	2011-05-30 17:23:47 +0000
--- lisp/mail/smtpmail.el	2011-06-15 20:33:56 +0000
***************
*** 34,47 ****
  ;;
  ;;(setq send-mail-function 'smtpmail-send-it) ; if you use `mail'
  ;;(setq message-send-mail-function 'smtpmail-send-it) ; if you use message/Gnus
! ;;(setq smtpmail-default-smtp-server "YOUR SMTP HOST")
  ;;(setq smtpmail-local-domain "YOUR DOMAIN NAME")
  ;;(setq smtpmail-sendto-domain "YOUR DOMAIN NAME")
  ;;(setq smtpmail-debug-info t) ; only to debug problems
  ;;(setq smtpmail-auth-credentials  ; or use ~/.authinfo
  ;;      '(("YOUR SMTP HOST" 25 "username" "password")))
- ;;(setq smtpmail-starttls-credentials
- ;;      '(("YOUR SMTP HOST" 25 "~/.my_smtp_tls.key" "~/.my_smtp_tls.cert")))
  ;; Where the 25 equals the value of `smtpmail-smtp-service', it can be an
  ;; integer or a string, just as long as they match (eq).
  
--- 34,45 ----
  ;;
  ;;(setq send-mail-function 'smtpmail-send-it) ; if you use `mail'
  ;;(setq message-send-mail-function 'smtpmail-send-it) ; if you use message/Gnus
! ;;(setq smtpmail-smtp-server "YOUR SMTP HOST")
  ;;(setq smtpmail-local-domain "YOUR DOMAIN NAME")
  ;;(setq smtpmail-sendto-domain "YOUR DOMAIN NAME")
  ;;(setq smtpmail-debug-info t) ; only to debug problems
  ;;(setq smtpmail-auth-credentials  ; or use ~/.authinfo
  ;;      '(("YOUR SMTP HOST" 25 "username" "password")))
  ;; Where the 25 equals the value of `smtpmail-smtp-service', it can be an
  ;; integer or a string, just as long as they match (eq).
  
***************
*** 58,74 ****
  ;; Authentication by the AUTH mechanism.
  ;; See http://www.ietf.org/rfc/rfc2554.txt
  
- ;; Modified by Simon Josefsson <simon@josefsson.org>, 2000-10-07, to support
- ;; STARTTLS.  Requires external program
- ;; ftp://ftp.opaopa.org/pub/elisp/starttls-*.tar.gz.
- ;; See http://www.ietf.org/rfc/rfc2246.txt, http://www.ietf.org/rfc/rfc2487.txt
- 
  ;;; Code:
  
  (require 'sendmail)
- (autoload 'starttls-any-program-available "starttls")
- (autoload 'starttls-open-stream "starttls")
- (autoload 'starttls-negotiate "starttls")
  (autoload 'mail-strip-quoted-names "mail-utils")
  (autoload 'message-make-date "message")
  (autoload 'message-make-message-id "message")
--- 56,64 ----
***************
*** 85,95 ****
    :group 'mail)
  
  
! (defcustom smtpmail-default-smtp-server nil
    "Specify default SMTP server.
! This only has effect if you specify it before loading the smtpmail library."
!   :type '(choice (const nil) string)
!   :group 'smtpmail)
  
  (defcustom smtpmail-smtp-server
    (or (getenv "SMTPSERVER") smtpmail-default-smtp-server)
--- 75,83 ----
    :group 'mail)
  
  
! (defvar smtpmail-default-smtp-server nil
    "Specify default SMTP server.
! This only has effect if you specify it before loading the smtpmail library.")
  
  (defcustom smtpmail-smtp-server
    (or (getenv "SMTPSERVER") smtpmail-default-smtp-server)
***************
*** 110,115 ****
--- 98,113 ----
    :type '(choice (const nil) string)
    :group 'smtpmail)
  
+ (defcustom smtpmail-stream-type nil
+   "Connection type SMTP connections.
+ This may be either nil (plain connection) or `starttls' (use the
+ starttls mechanism to turn on TLS security after opening the
+ stream)."
+   :version "24.1"
+   :group 'smtpmail
+   :type '(choice (const :tag "Plain" nil)
+ 		 (const starttls)))
+ 
  (defcustom smtpmail-sendto-domain nil
    "Local domain name without a host name.
  This is appended (with an @-sign) to any specified recipients which do
***************
*** 174,195 ****
    :version "22.1"
    :group 'smtpmail)
  
- (defcustom smtpmail-starttls-credentials '(("" 25 "" ""))
-   "Specify STARTTLS keys and certificates for servers.
- This is a list of four-element list with `servername' (a string),
- `port' (an integer), `key' (a filename) and `certificate' (a
- filename).
- If you do not have a certificate/key pair, leave the `key' and
- `certificate' fields as `nil'.  A key/certificate pair is only
- needed if you want to use X.509 client authenticated
- connections."
-   :type '(repeat (list (string  :tag "Server")
- 		       (integer :tag "Port")
- 		       (file    :tag "Key")
- 		       (file    :tag "Certificate")))
-   :version "21.1"
-   :group 'smtpmail)
- 
  (defcustom smtpmail-warn-about-unknown-extensions nil
    "If set, print warnings about unknown SMTP extensions.
  This is mainly useful for development purposes, to learn about
--- 172,177 ----
***************
*** 230,235 ****
--- 212,218 ----
  	(tembuf (generate-new-buffer " smtpmail temp"))
  	(case-fold-search nil)
  	delimline
+ 	result
  	(mailbuf (current-buffer))
          ;; Examine this variable now, so that
  	;; local binding in the mail buffer will take effect.
***************
*** 373,381 ****
            ;; Send or queue
  	  (if (not smtpmail-queue-mail)
  	      (if (not (null smtpmail-recipient-address-list))
! 		  (if (not (smtpmail-via-smtp
! 			    smtpmail-recipient-address-list tembuf))
! 		      (error "Sending failed; SMTP protocol error"))
  		(error "Sending failed; no recipients"))
  	    (let* ((file-data
  		    (expand-file-name
--- 356,365 ----
            ;; Send or queue
  	  (if (not smtpmail-queue-mail)
  	      (if (not (null smtpmail-recipient-address-list))
! 		  (when (setq result
! 			      (smtpmail-via-smtp
! 			       smtpmail-recipient-address-list tembuf))
! 		    (error "Sending failed: %s" result))
  		(error "Sending failed; no recipients"))
  	    (let* ((file-data
  		    (expand-file-name
***************
*** 432,438 ****
      ;; mail, send it, etc...
      (let ((file-msg "")
            (qfile (expand-file-name smtpmail-queue-index-file
!                                    smtpmail-queue-dir)))
        (insert-file-contents qfile)
        (goto-char (point-min))
        (while (not (eobp))
--- 416,423 ----
      ;; mail, send it, etc...
      (let ((file-msg "")
            (qfile (expand-file-name smtpmail-queue-index-file
!                                    smtpmail-queue-dir))
! 	  result)
        (insert-file-contents qfile)
        (goto-char (point-min))
        (while (not (eobp))
***************
*** 448,464 ****
                   (or (and mail-specify-envelope-from (mail-envelope-from))
                       user-mail-address)))
              (if (not (null smtpmail-recipient-address-list))
!                 (if (not (smtpmail-via-smtp smtpmail-recipient-address-list
!                                             (current-buffer)))
!                     (error "Sending failed; SMTP protocol error"))
                (error "Sending failed; no recipients"))))
  	(delete-file file-msg)
  	(delete-file (concat file-msg ".el"))
  	(delete-region (point-at-bol) (point-at-bol 2)))
        (write-region (point-min) (point-max) qfile))))
  
- ;; (defun smtpmail-via-smtp (host,port,sender,destination,smtpmail-text-buffer)
- 
  (defun smtpmail-fqdn ()
    (if smtpmail-local-domain
        (concat (system-name) "." smtpmail-local-domain)
--- 433,448 ----
                   (or (and mail-specify-envelope-from (mail-envelope-from))
                       user-mail-address)))
              (if (not (null smtpmail-recipient-address-list))
!                 (when (setq result (smtpmail-via-smtp
! 				    smtpmail-recipient-address-list
! 				    (current-buffer)))
! 		  (error "Sending failed: %s" result))
                (error "Sending failed; no recipients"))))
  	(delete-file file-msg)
  	(delete-file (concat file-msg ".el"))
  	(delete-region (point-at-bol) (point-at-bol 2)))
        (write-region (point-min) (point-max) qfile))))
  
  (defun smtpmail-fqdn ()
    (if smtpmail-local-domain
        (concat (system-name) "." smtpmail-local-domain)
***************
*** 503,548 ****
  	(push el2 result)))
      (nreverse result)))
  
- (defvar starttls-extra-args)
- (defvar starttls-extra-arguments)
- 
- (defun smtpmail-open-stream (process-buffer host port)
-   (let ((cred (smtpmail-find-credentials
- 	       smtpmail-starttls-credentials host port)))
-     (if (null (and cred (starttls-any-program-available)))
- 	;; The normal case.
- 	(open-network-stream "SMTP" process-buffer host port)
-       (let* ((cred-key (smtpmail-cred-key cred))
- 	     (cred-cert (smtpmail-cred-cert cred))
- 	     (starttls-extra-args
- 	      (append
- 	       starttls-extra-args
- 	       (when (and (stringp cred-key) (stringp cred-cert)
- 			  (file-regular-p
- 			   (setq cred-key (expand-file-name cred-key)))
- 			  (file-regular-p
- 			   (setq cred-cert (expand-file-name cred-cert))))
- 		 (list "--key-file" cred-key "--cert-file" cred-cert))))
- 	     (starttls-extra-arguments
- 	      (append
- 	       starttls-extra-arguments
- 	       (when (and (stringp cred-key) (stringp cred-cert)
- 			  (file-regular-p
- 			   (setq cred-key (expand-file-name cred-key)))
- 			  (file-regular-p
- 			   (setq cred-cert (expand-file-name cred-cert))))
- 		 (list "--x509keyfile" cred-key "--x509certfile" cred-cert)))))
- 	(starttls-open-stream "SMTP" process-buffer host port)))))
- 
  ;; `password-read' autoloads password-cache.
  (declare-function password-cache-add "password-cache" (key password))
  
! (defun smtpmail-try-auth-methods (process supported-extensions host port)
    (let* ((mechs (cdr-safe (assoc 'auth supported-extensions)))
  	 (mech (car (smtpmail-intersection mechs smtpmail-auth-supported)))
           (auth-info (auth-source-search :max 1
                                          :host host
!                                         :port (or port "smtp")))
           (auth-user (plist-get (nth 0 auth-info) :user))
           (auth-pass (plist-get (nth 0 auth-info) :secret))
           (auth-pass (if (functionp auth-pass)
--- 487,511 ----
  	(push el2 result)))
      (nreverse result)))
  
  ;; `password-read' autoloads password-cache.
  (declare-function password-cache-add "password-cache" (key password))
  
! (defun smtpmail-command-or-throw (process string &optional code)
!   (let (ret)
!     (smtpmail-send-command process string)
!     (unless (smtpmail-ok-p (setq ret (smtpmail-read-response process))
! 			   code)
!       (throw 'done (smtpmail-response-text ret)))
!     ret))
! 
! (defun smtpmail-try-auth-methods (process supported-extensions host port
! 					  &optional ask-for-password)
    (let* ((mechs (cdr-safe (assoc 'auth supported-extensions)))
  	 (mech (car (smtpmail-intersection mechs smtpmail-auth-supported)))
           (auth-info (auth-source-search :max 1
                                          :host host
!                                         :port (or port "smtp")
! 					:create ask-for-password))
           (auth-user (plist-get (nth 0 auth-info) :user))
           (auth-pass (plist-get (nth 0 auth-info) :secret))
           (auth-pass (if (functionp auth-pass)
***************
*** 571,584 ****
  		   (or (smtpmail-cred-passwd cred)
  		       (password-read prompt prompt))))
  	 ret)
!     (when (and cred mech)
        (cond
         ((eq mech 'cram-md5)
! 	(smtpmail-send-command process (upcase (format "AUTH %s" mech)))
! 	(if (or (null (car (setq ret (smtpmail-read-response process))))
! 		(not (integerp (car ret)))
! 		(>= (car ret) 400))
! 	    (throw 'done nil))
  	(when (eq (car ret) 334)
  	  (let* ((challenge (substring (cadr ret) 4))
  		 (decoded (base64-decode-string challenge))
--- 534,545 ----
  		   (or (smtpmail-cred-passwd cred)
  		       (password-read prompt prompt))))
  	 ret)
!     (if (not (and cred mech))
! 	mech
        (cond
         ((eq mech 'cram-md5)
! 	(setq ret (smtpmail-command-or-throw
! 		   process (format "AUTH %s" (upcase mech))))
  	(when (eq (car ret) 334)
  	  (let* ((challenge (substring (cadr ret) 4))
  		 (decoded (base64-decode-string challenge))
***************
*** 596,648 ****
  		 ;; are taken as a response to the server, and the
  		 ;; authentication fails.
  		 (encoded (base64-encode-string response t)))
! 	    (smtpmail-send-command process (format "%s" encoded))
! 	    (if (or (null (car (setq ret (smtpmail-read-response process))))
! 		    (not (integerp (car ret)))
! 		    (>= (car ret) 400))
! 		(throw 'done nil)))))
         ((eq mech 'login)
! 	(smtpmail-send-command process "AUTH LOGIN")
! 	(if (or (null (car (setq ret (smtpmail-read-response process))))
! 		(not (integerp (car ret)))
! 		(>= (car ret) 400))
! 	    (throw 'done nil))
! 	(smtpmail-send-command
  	 process (base64-encode-string (smtpmail-cred-user cred) t))
! 	(if (or (null (car (setq ret (smtpmail-read-response process))))
! 		(not (integerp (car ret)))
! 		(>= (car ret) 400))
! 	    (throw 'done nil))
! 	(smtpmail-send-command process (base64-encode-string passwd t))
! 	(if (or (null (car (setq ret (smtpmail-read-response process))))
! 		(not (integerp (car ret)))
! 		(>= (car ret) 400))
! 	    (throw 'done nil)))
         ((eq mech 'plain)
  	;; We used to send an empty initial request, and wait for an
  	;; empty response, and then send the password, but this
  	;; violate a SHOULD in RFC 2222 paragraph 5.1.  Note that this
  	;; is not sent if the server did not advertise AUTH PLAIN in
  	;; the EHLO response.  See RFC 2554 for more info.
! 	(smtpmail-send-command process
! 			       (concat "AUTH PLAIN "
! 				       (base64-encode-string
! 					(concat "\0"
! 						(smtpmail-cred-user cred)
! 						"\0"
! 						passwd) t)))
! 	(if (or (null (car (setq ret (smtpmail-read-response process))))
! 		(not (integerp (car ret)))
! 		(not (equal (car ret) 235)))
! 	    (throw 'done nil)))
! 
         (t
  	(error "Mechanism %s not implemented" mech)))
        ;; Remember the password.
        (when (null (smtpmail-cred-passwd cred))
! 	(password-cache-add prompt passwd)))))
  
! (defun smtpmail-via-smtp (recipient smtpmail-text-buffer)
    (let ((process nil)
  	(host (or smtpmail-smtp-server
  		  (error "`smtpmail-smtp-server' not defined")))
--- 557,610 ----
  		 ;; are taken as a response to the server, and the
  		 ;; authentication fails.
  		 (encoded (base64-encode-string response t)))
! 	    (smtpmail-command-or-throw process encoded))))
         ((eq mech 'login)
! 	(smtpmail-command-or-throw process "AUTH LOGIN")
! 	(smtpmail-command-or-throw
  	 process (base64-encode-string (smtpmail-cred-user cred) t))
! 	(smtpmail-command-or-throw process (base64-encode-string passwd t)))
         ((eq mech 'plain)
  	;; We used to send an empty initial request, and wait for an
  	;; empty response, and then send the password, but this
  	;; violate a SHOULD in RFC 2222 paragraph 5.1.  Note that this
  	;; is not sent if the server did not advertise AUTH PLAIN in
  	;; the EHLO response.  See RFC 2554 for more info.
! 	(smtpmail-command-or-throw
! 	 process
! 	 (concat "AUTH PLAIN "
! 		 (base64-encode-string
! 		  (concat "\0"
! 			  (smtpmail-cred-user cred)
! 			  "\0"
! 			  passwd) t))
! 	 235))
         (t
  	(error "Mechanism %s not implemented" mech)))
        ;; Remember the password.
        (when (null (smtpmail-cred-passwd cred))
! 	(password-cache-add prompt passwd))
!       nil)))
! 
! (defun smtpmail-response-code (string)
!   (when string
!     (with-temp-buffer
!       (insert string)
!       (goto-char (point-min))
!       (and (re-search-forward "^\\([0-9]+\\) " nil t)
! 	   (string-to-number (match-string 1))))))
! 
! (defun smtpmail-ok-p (response &optional code)
!   (and (car response)
!        (integerp (car response))
!        (< (car response) 400)
!        (or (null code)
! 	   (= code (car response)))))
! 
! (defun smtpmail-response-text (response)
!   (mapconcat 'identity (cdr response) "\n"))
  
! (defun smtpmail-via-smtp (recipient smtpmail-text-buffer
! 				    &optional ask-for-password)
    (let ((process nil)
  	(host (or smtpmail-smtp-server
  		  (error "`smtpmail-smtp-server' not defined")))
***************
*** 654,667 ****
                                  (mail-envelope-from))
                             user-mail-address))
  	response-code
- 	greeting
  	process-buffer
  	(supported-extensions '()))
      (unwind-protect
  	(catch 'done
  	  ;; get or create the trace buffer
  	  (setq process-buffer
! 		(get-buffer-create (format "*trace of SMTP session to %s*" host)))
  
  	  ;; clear the trace buffer of old output
  	  (with-current-buffer process-buffer
--- 616,631 ----
                                  (mail-envelope-from))
                             user-mail-address))
  	response-code
  	process-buffer
+ 	result
+ 	auth-mechanisms
  	(supported-extensions '()))
      (unwind-protect
  	(catch 'done
  	  ;; get or create the trace buffer
  	  (setq process-buffer
! 		(get-buffer-create
! 		 (format "*trace of SMTP session to %s*" host)))
  
  	  ;; clear the trace buffer of old output
  	  (with-current-buffer process-buffer
***************
*** 669,773 ****
  	    (erase-buffer))
  
  	  ;; open the connection to the server
! 	  (setq process (smtpmail-open-stream process-buffer host port))
! 	  (and (null process) (throw 'done nil))
  
  	  ;; set the send-filter
  	  (set-process-filter process 'smtpmail-process-filter)
  
  	  (with-current-buffer process-buffer
  	    (set-buffer-process-coding-system 'raw-text-unix 'raw-text-unix)
  	    (make-local-variable 'smtpmail-read-point)
  	    (setq smtpmail-read-point (point-min))
  
! 
! 	    (if (or (null (car (setq greeting (smtpmail-read-response process))))
! 		    (not (integerp (car greeting)))
! 		    (>= (car greeting) 400))
! 		(throw 'done nil))
! 
! 	    (let ((do-ehlo t)
! 		  (do-starttls t))
! 	      (while do-ehlo
!                 ;; EHLO
!                 (smtpmail-send-command process (format "EHLO %s" (smtpmail-fqdn)))
! 
!                 (if (or (null (car (setq response-code
!                                          (smtpmail-read-response process))))
!                         (not (integerp (car response-code)))
!                         (>= (car response-code) 400))
!                     (progn
!                       ;; HELO
!                       (smtpmail-send-command
!                        process (format "HELO %s" (smtpmail-fqdn)))
! 
!                       (if (or (null (car (setq response-code
!                                                (smtpmail-read-response process))))
!                               (not (integerp (car response-code)))
!                               (>= (car response-code) 400))
!                           (throw 'done nil)))
!                   (dolist (line (cdr (cdr response-code)))
!                     (let ((name
!                            (with-case-table ascii-case-table
!                              (mapcar (lambda (s) (intern (downcase s)))
!                                      (split-string (substring line 4) "[ ]")))))
!                       (and (eq (length name) 1)
!                            (setq name (car name)))
!                       (and name
!                            (cond ((memq (if (consp name) (car name) name)
!                                         '(verb xvrb 8bitmime onex xone
!                                                expn size dsn etrn
!                                                enhancedstatuscodes
!                                                help xusr
!                                                auth=login auth starttls))
!                                   (setq supported-extensions
!                                         (cons name supported-extensions)))
!                                  (smtpmail-warn-about-unknown-extensions
!                                   (message "Unknown extension %s" name)))))))
! 
!                 (if (and do-starttls
!                          (smtpmail-find-credentials smtpmail-starttls-credentials host port)
!                          (member 'starttls supported-extensions)
!                          (numberp (process-id process)))
!                     (progn
!                       (smtpmail-send-command process (format "STARTTLS"))
!                       (if (or (null (car (setq response-code (smtpmail-read-response process))))
!                               (not (integerp (car response-code)))
!                               (>= (car response-code) 400))
!                           (throw 'done nil))
!                       (starttls-negotiate process)
!                       (setq do-starttls nil))
!                   (setq do-ehlo nil))))
! 
! 	    (smtpmail-try-auth-methods process supported-extensions host port)
! 
! 	    (if (or (member 'onex supported-extensions)
! 		    (member 'xone supported-extensions))
! 		(progn
! 		  (smtpmail-send-command process (format "ONEX"))
! 		  (if (or (null (car (setq response-code (smtpmail-read-response process))))
! 			  (not (integerp (car response-code)))
! 			  (>= (car response-code) 400))
! 		      (throw 'done nil))))
! 
! 	    (if (and smtpmail-debug-verb
! 		     (or (member 'verb supported-extensions)
! 			 (member 'xvrb supported-extensions)))
! 		(progn
! 		  (smtpmail-send-command process (format "VERB"))
! 		  (if (or (null (car (setq response-code (smtpmail-read-response process))))
! 			  (not (integerp (car response-code)))
! 			  (>= (car response-code) 400))
! 		      (throw 'done nil))))
! 
! 	    (if (member 'xusr supported-extensions)
! 		(progn
! 		  (smtpmail-send-command process (format "XUSR"))
! 		  (if (or (null (car (setq response-code (smtpmail-read-response process))))
! 			  (not (integerp (car response-code)))
! 			  (>= (car response-code) 400))
! 		      (throw 'done nil))))
! 
  	    ;; MAIL FROM:<sender>
  	    (let ((size-part
  		   (if (or (member 'size supported-extensions)
--- 633,719 ----
  	    (erase-buffer))
  
  	  ;; open the connection to the server
! 	  (setq result
! 		(open-network-stream
! 		 "smtpmail" process-buffer host port
! 		 :type smtpmail-stream-type
! 		 :return-list t
! 		 :capability-command (format "EHLO %s\r\n" (smtpmail-fqdn))
! 		 :end-of-command "^[0-9]+ .*\r\n"
! 		 :success "^2.*\n"
! 		 :always-query-capabilities t
! 		 :starttls-function
! 		 (lambda (capabilities)
! 		   (and (string-match "-STARTTLS" capabilities)
! 			"STARTTLS\r\n"))))
! 
! 	  ;; If we couldn't access the server at all, we give up.
! 	  (unless (setq process (car result))
! 	    (throw 'done "Unable to contact server"))
  
  	  ;; set the send-filter
  	  (set-process-filter process 'smtpmail-process-filter)
  
+ 	  (let* ((greeting (plist-get (cdr result) :greeting))
+ 		 (code (smtpmail-response-code greeting)))
+ 	    (unless code
+ 	      (throw 'done (format "No greeting: %s" greeting)))
+ 	    (when (>= code 400)
+ 	      (throw 'done (format "Connection not allowed: %s" greeting))))
+ 	  
  	  (with-current-buffer process-buffer
  	    (set-buffer-process-coding-system 'raw-text-unix 'raw-text-unix)
  	    (make-local-variable 'smtpmail-read-point)
  	    (setq smtpmail-read-point (point-min))
  
! 	    (let* ((capabilities (plist-get (cdr result) :capabilities))
! 		   (code (smtpmail-response-code capabilities)))
! 	      (if (or (null code)
! 		      (>= code 400))
! 		  ;; The server didn't accept EHLO, so we fall back on HELO.
! 		  (smtpmail-command-or-throw
! 		   process (format "HELO %s" (smtpmail-fqdn)))
! 		;; EHLO was successful, so we parse the extensions.
! 		(dolist (line (delete
! 			       ""
! 			       (split-string
! 				(plist-get (cdr result) :capabilities)
! 				"\r\n")))
! 		  (let ((name
! 			 (with-case-table ascii-case-table
! 			   (mapcar (lambda (s) (intern (downcase s)))
! 				   (split-string (substring line 4) "[ ]")))))
! 		    (when (= (length name) 1)
! 		      (setq name (car name)))
! 		    (when name
! 		      (cond ((memq (if (consp name) (car name) name)
! 				   '(verb xvrb 8bitmime onex xone
! 					  expn size dsn etrn
! 					  enhancedstatuscodes
! 					  help xusr
! 					  auth=login auth starttls))
! 			     (setq supported-extensions
! 				   (cons name supported-extensions)))
! 			    (smtpmail-warn-about-unknown-extensions
! 			     (message "Unknown extension %s" name))))))))
! 
! 	    (setq auth-mechanisms
! 		  (smtpmail-try-auth-methods
! 		   process supported-extensions host port
! 		   ask-for-password))
! 
! 	    (when (or (member 'onex supported-extensions)
! 		      (member 'xone supported-extensions))
! 	      (smtpmail-command-or-throw process (format "ONEX")))
! 
! 	    (when (and smtpmail-debug-verb
! 		       (or (member 'verb supported-extensions)
! 			   (member 'xvrb supported-extensions)))
! 	      (smtpmail-command-or-throw process (format "VERB")))
! 
! 	    (when (member 'xusr supported-extensions)
! 	      (smtpmail-command-or-throw process (format "XUSR")))
! 	    
  	    ;; MAIL FROM:<sender>
  	    (let ((size-part
  		   (if (or (member 'size supported-extensions)
***************
*** 797,861 ****
  			   " BODY=8BITMIME"
  			 "")
  		     "")))
!               ;; (smtpmail-send-command process (format "MAIL FROM:%s@%s" (user-login-name) (smtpmail-fqdn)))
! 	      (smtpmail-send-command process (format "MAIL FROM:<%s>%s%s"
!                                                      envelope-from
! 						     size-part
! 						     body-part))
! 
! 	      (if (or (null (car (setq response-code (smtpmail-read-response process))))
! 		      (not (integerp (car response-code)))
! 		      (>= (car response-code) 400))
! 		  (throw 'done nil)))
  
  	    ;; RCPT TO:<recipient>
  	    (let ((n 0))
  	      (while (not (null (nth n recipient)))
! 		(smtpmail-send-command process (format "RCPT TO:<%s>" (smtpmail-maybe-append-domain (nth n recipient))))
! 		(setq n (1+ n))
! 
! 		(setq response-code (smtpmail-read-response process))
! 		(if (or (null (car response-code))
! 			(not (integerp (car response-code)))
! 			(>= (car response-code) 400))
! 		    (throw 'done nil))))
! 
! 	    ;; DATA
! 	    (smtpmail-send-command process "DATA")
! 
! 	    (if (or (null (car (setq response-code (smtpmail-read-response process))))
! 		    (not (integerp (car response-code)))
! 		    (>= (car response-code) 400))
! 		(throw 'done nil))
  
! 	    ;; Mail contents
  	    (smtpmail-send-data process smtpmail-text-buffer)
- 
  	    ;; DATA end "."
! 	    (smtpmail-send-command process ".")
! 
! 	    (if (or (null (car (setq response-code (smtpmail-read-response process))))
! 		    (not (integerp (car response-code)))
! 		    (>= (car response-code) 400))
! 		(throw 'done nil))
! 
! 	    ;; QUIT
!             ;; (smtpmail-send-command process "QUIT")
!             ;; (and (null (car (smtpmail-read-response process)))
!             ;;      (throw 'done nil))
! 	    t))
!       (if process
! 	  (with-current-buffer (process-buffer process)
! 	    (smtpmail-send-command process "QUIT")
! 	    (smtpmail-read-response process)
! 
!             ;; (if (or (null (car (setq response-code (smtpmail-read-response process))))
!             ;;         (not (integerp (car response-code)))
!             ;;         (>= (car response-code) 400))
!             ;;	   (throw 'done nil))
! 	    (delete-process process)
! 	    (unless smtpmail-debug-info
! 	      (kill-buffer process-buffer)))))))
  
  
  (defun smtpmail-process-filter (process output)
--- 743,795 ----
  			   " BODY=8BITMIME"
  			 "")
  		     "")))
! 	      (smtpmail-command-or-throw
! 	       process (format "MAIL FROM:<%s>%s%s"
! 			       envelope-from size-part body-part)))
  
  	    ;; RCPT TO:<recipient>
  	    (let ((n 0))
  	      (while (not (null (nth n recipient)))
! 		(smtpmail-send-command
! 		 process (format "RCPT TO:<%s>"
! 				 (smtpmail-maybe-append-domain
! 				  (nth n recipient))))
! 		(cond
! 		 ((smtpmail-ok-p (setq result (smtpmail-read-response process)))
! 		  ;; Success.
! 		  nil)
! 		 ((and auth-mechanisms
! 		       (not ask-for-password)
! 		       (= (car result) 550))
! 		  ;; We got a "550 relay not permitted", and the server
! 		  ;; accepts credentials, so we try again, but ask for a
! 		  ;; password first.
! 		  (smtpmail-send-command process "QUIT")
! 		  (smtpmail-read-response process)
! 		  (delete-process process)
! 		  (throw 'done
! 			 (smtpmail-via-smtp recipient smtpmail-text-buffer t)))
! 		 (t
! 		  ;; Return the error code.
! 		  (throw 'done
! 			 (smtpmail-response-text result))))
! 		(setq n (1+ n))))
  
! 	    ;; Send the contents.
! 	    (smtpmail-command-or-throw process "DATA")
  	    (smtpmail-send-data process smtpmail-text-buffer)
  	    ;; DATA end "."
! 	    (smtpmail-command-or-throw process ".")
! 	    ;; Return success.
! 	    nil))
!       (when (and process
! 		 (buffer-live-p process-buffer))
! 	(with-current-buffer (process-buffer process)
! 	  (smtpmail-send-command process "QUIT")
! 	  (smtpmail-read-response process)
! 	  (delete-process process)
! 	  (unless smtpmail-debug-info
! 	    (kill-buffer process-buffer)))))))
  
  
  (defun smtpmail-process-filter (process output)


[-- Attachment #3: Type: text/plain, Size: 103 bytes --]


-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: netrc field encryption in auth-source

[Ted Zlatanov]

On Wed, 15 Jun 2011 18:20:35 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Ted Zlatanov <tzz@lifelogs.com> writes:
>> If you want this done by Wednesday (including the creation code), please
>> let me know if my code looked OK and if you find the interface
>> acceptable.  I won't be able to do it in time otherwise.

LMI> So please apply and I'll get started with the smtpmail.el stuff.  :-)

Applied.  See `auth-source-save-secrets' for the only user-controllable
piece of the code.  It should really be using the EPA functions directly
instead of relying on the file handlers but I wasn't able to get that
working.  Maybe Daiki Ueno could give me a hint.  Or I will get to it
eventually...

I think I DTRT with `unwind-protect' and `delete-file' to ensure the
temporary stashfile is always cleaned out.

Test with this netrc entry:

machine supertest password gpg:jA0EAwMC2tUEaZgM7A5gyWM/owySdCOS/cjoFCuf8LI1d1kYX7z6cjsNkakM04u1geh/iesqyH3XQFI+SEVLb/oEC/EoQ0LIgRRoBiLyu9XZWN1ytY7MQxpPZniFz13oGV4/Dwl8yrP3Hba5LfQpHy2FZRM=

(funcall (plist-get (nth 0 (auth-source-search :host "supertest")) :secret))

which, with passphrase "123", should give you "hellfdhfhdkjfhskdjfhkdj fsdhkjfdshk fdshkj fdsfsdhkj fsdhkj fdshkj fd k fhdjk hjfsdkhjfd sk kfsd khfdskhjfskd  kfsdkhjfsdkhjfsdkhjfsdkhfdo"[1]

Thanks
Ted

[1] "Good morning" in Icelandic, according to Wikipedia.

Re: netrc field encryption in auth-source

[Robert Pluim]

Ted Zlatanov <tzz@lifelogs.com> writes:

> On Wed, 15 Jun 2011 18:20:35 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 
>
> LMI> Ted Zlatanov <tzz@lifelogs.com> writes:
>>> If you want this done by Wednesday (including the creation code), please
>>> let me know if my code looked OK and if you find the interface
>>> acceptable.  I won't be able to do it in time otherwise.
>
> LMI> So please apply and I'll get started with the smtpmail.el stuff.  :-)
>
> Applied.  See `auth-source-save-secrets' for the only user-controllable
> piece of the code.  It should really be using the EPA functions directly
> instead of relying on the file handlers but I wasn't able to get that
> working.  Maybe Daiki Ueno could give me a hint.  Or I will get to it
> eventually...

I just took a look at this, it contains

+(defcustom auth-source-save-secrets nil
+  "If set, auth-source will respect it for password tokens behavior."
+  :group 'auth-source
+  :version "23.2" ;; No Gnus
+  :type `(choice
+          :tag "auth-source new password token behavior"
+          (const :tag "Use GPG tokens" gpg)
+          (const :tag "Save unencrypted" nil)
+          (const :tag "Ask" ask)))

I'm glad auth-source will show respect, but that doc-string is almost
information free. How about something like

"This controls what auth-source will do with password tokens: save them,
ask, store as gpg tokens in .authinfo"

Also, does ask mean 'ask once', or does it mean 'ask every time'? I'm
personally looking for something that would give me "don't store
passwords and don't ask me about storing them except maybe the first
time".

Regards

Robert

Re: netrc field encryption in auth-source

[Ted Zlatanov]

On Thu, 16 Jun 2011 10:32:15 +0200 Robert Pluim <rpluim@gmail.com> wrote: 

RP> Ted Zlatanov <tzz@lifelogs.com> writes:
>> Applied.  See `auth-source-save-secrets' for the only user-controllable
>> piece of the code.  It should really be using the EPA functions directly
>> instead of relying on the file handlers but I wasn't able to get that
>> working.  Maybe Daiki Ueno could give me a hint.  Or I will get to it
>> eventually...

RP> I just took a look at this, it contains

RP> +(defcustom auth-source-save-secrets nil
RP> +  "If set, auth-source will respect it for password tokens behavior."
RP> +  :group 'auth-source
RP> +  :version "23.2" ;; No Gnus
RP> +  :type `(choice
RP> +          :tag "auth-source new password token behavior"
RP> +          (const :tag "Use GPG tokens" gpg)
RP> +          (const :tag "Save unencrypted" nil)
RP> +          (const :tag "Ask" ask)))

RP> I'm glad auth-source will show respect, but that doc-string is almost
RP> information free.

Damn it, someone noticed ;)

RP> How about something like

RP> "This controls what auth-source will do with password tokens: save them,
RP> ask, store as gpg tokens in .authinfo"

You're just listing the defcustom choices in the docstring itself.  What
if we add choices?  We have to edit the docstring again.

How about "Set this to tell auth-source how to handle password tokens in
unencrypted files."

RP> Also, does ask mean 'ask once', or does it mean 'ask every time'? I'm
RP> personally looking for something that would give me "don't store
RP> passwords and don't ask me about storing them except maybe the first
RP> time".

I'm not sure yet.  Right now it's once per Emacs session, if 'ask, but I
left the default nil since it's 100% experimental.  So users that update
blindly will not be affected at all, for now.

I can certainly use Customize to set it to 'gpg or nil forever after the
first time it's asked.  The problem is, I'm also not sure if it should
be a single global setting.  It feels like something that should be
decided for each individual netrc file.  And if that's the case, maybe
the defcustom should hold that choice in an alist with regex matching.

Ted

Re: netrc field encryption in auth-source

[Reiner Steib]

On Thu, Jun 16 2011, Ted Zlatanov wrote:

> On Thu, 16 Jun 2011 10:32:15 +0200 Robert Pluim <rpluim@gmail.com> wrote: 
> RP> "This controls what auth-source will do with password tokens: save them,
> RP> ask, store as gpg tokens in .authinfo"
>
> You're just listing the defcustom choices in the docstring itself.  

The user should be able to figure out the valid values without jumping
to the defcustom or using customize.

> What if we add choices?  We have to edit the docstring again.

Sure.  So what?  ;-)

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: netrc field encryption in auth-source

[Lars Magne Ingebrigtsen]

Reiner Steib <reinersteib+gmane@imap.cc> writes:

>> What if we add choices?  We have to edit the docstring again.
>
> Sure.  So what?  ;-)

Perhaps `describe-variable' should list the customisation values and
meanings?

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

should docstrings include all defcustom options? (was: netrc field encryption in auth-source)

[Ted Zlatanov]

On Thu, 16 Jun 2011 22:28:02 +0200 Reiner Steib <reinersteib+gmane@imap.cc> wrote: 

RS> On Thu, Jun 16 2011, Ted Zlatanov wrote:
>> You're just listing the defcustom choices in the docstring itself.  

RS> The user should be able to figure out the valid values without jumping
RS> to the defcustom or using customize.

Is this recommended anywhere in the ELisp manual?  First time I hear
about it; it's been years since I read the whole thing end to end so
apologies if I've missed or forgotten this recommendation.

>> What if we add choices?  We have to edit the docstring again.

RS> Sure.  So what?  ;-)

It's twice the work.  For me.  And it's prone to errors, and it doesn't
convey additional information to the user.

On Thu, 16 Jun 2011 23:05:38 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Perhaps `describe-variable' should list the customisation values and
LMI> meanings?

That would be smart.  I'd actually inline the customization interface
right in there, instead of forcing the user to follow a link in order to
customize.  Maybe as a tab or a collapsed dialog.

But all of that is besides the point of netrc field encryption...  So
I've changed the thread subject accordingly.

Ted

Re: netrc field encryption in auth-source

[Robert Pluim]

Ted Zlatanov <tzz@lifelogs.com> writes:

> On Thu, 16 Jun 2011 10:32:15 +0200 Robert Pluim <rpluim@gmail.com> wrote: 
> RP> How about something like
>
> RP> "This controls what auth-source will do with password tokens: save them,
> RP> ask, store as gpg tokens in .authinfo"
>
> You're just listing the defcustom choices in the docstring itself.  What
> if we add choices?  We have to edit the docstring again.
>
> How about "Set this to tell auth-source how to handle password tokens in
> unencrypted files."
>

Yes, that would be better (although I think an indication in the
defcustom choices that 'gpg will result in saving tokens to .authinfo
would be good as well)

> RP> Also, does ask mean 'ask once', or does it mean 'ask every time'? I'm
> RP> personally looking for something that would give me "don't store
> RP> passwords and don't ask me about storing them except maybe the first
> RP> time".
>
> I'm not sure yet.  Right now it's once per Emacs session, if 'ask, but I
> left the default nil since it's 100% experimental.  So users that update
> blindly will not be affected at all, for now.
>
> I can certainly use Customize to set it to 'gpg or nil forever after the
> first time it's asked.  The problem is, I'm also not sure if it should
> be a single global setting.  It feels like something that should be
> decided for each individual netrc file.  And if that's the case, maybe
> the defcustom should hold that choice in an alist with regex matching.

Hmm, a single global setting works for me, but I can envisage people
desiring to have different values for different servers, not just
different files. At that point we'd be stuffing server values in the
custom variables, and we're back at square one. Would a new keyword in
the file itself work?  save-token {cleartext,ask,gpg,no}? Tell me if I'm
over-engineering this :)

Robert

Re: netrc field encryption in auth-source

[Ted Zlatanov]

On Fri, 17 Jun 2011 09:17:13 +0200 Robert Pluim <rpluim@gmail.com> wrote: 

RP> Ted Zlatanov <tzz@lifelogs.com> writes:
RP> Also, does ask mean 'ask once', or does it mean 'ask every time'? I'm
RP> personally looking for something that would give me "don't store
RP> passwords and don't ask me about storing them except maybe the first
RP> time".
>> 
>> I'm not sure yet.  Right now it's once per Emacs session, if 'ask, but I
>> left the default nil since it's 100% experimental.  So users that update
>> blindly will not be affected at all, for now.
>> 
>> I can certainly use Customize to set it to 'gpg or nil forever after the
>> first time it's asked.  The problem is, I'm also not sure if it should
>> be a single global setting.  It feels like something that should be
>> decided for each individual netrc file.  And if that's the case, maybe
>> the defcustom should hold that choice in an alist with regex matching.

RP> Hmm, a single global setting works for me, but I can envisage people
RP> desiring to have different values for different servers, not just
RP> different files. At that point we'd be stuffing server values in the
RP> custom variables, and we're back at square one. 

Not quite square one, but yes, I know what you mean.  Per file regex is
the lowest granularity I would implement without bribery, because it's
what I would use.  But wait, we can do better if it's an alist...  Let's
use the EPA file pattern!  The default can then be:

`((,(car epa-file-auto-mode-alist-entry) nil)
  (t ask))

...and when the user says "yes, use GPG tokens for file xyz" we'd add
'("xyz" gpg) to the head of the alist and offer to save the defcustom.
We have to make the "never ask to add" choice 'never, because nil is now
a valid alist for the value.  So it could only be 'never or a valid
alist.  Yes, that would work.

RP> Would a new keyword in the file itself work?  save-token
RP> {cleartext,ask,gpg,no}? Tell me if I'm over-engineering this :)

Every line in a netrc file should be self-sufficient, so I'd rather not
add global keywords.  If you mean on each line, then yes, that's too
much.

Ted

Re: netrc field encryption in auth-source

[Robert Pluim]

Ted Zlatanov <tzz@lifelogs.com> writes:

> On Fri, 17 Jun 2011 09:17:13 +0200 Robert Pluim <rpluim@gmail.com> wrote: 
> RP> Hmm, a single global setting works for me, but I can envisage people
> RP> desiring to have different values for different servers, not just
> RP> different files. At that point we'd be stuffing server values in the
> RP> custom variables, and we're back at square one. 
>
> Not quite square one, but yes, I know what you mean.  Per file regex is
> the lowest granularity I would implement without bribery, because it's
> what I would use.  But wait, we can do better if it's an alist...  Let's
> use the EPA file pattern!  The default can then be:
>
> `((,(car epa-file-auto-mode-alist-entry) nil)
>   (t ask))
>
> ...and when the user says "yes, use GPG tokens for file xyz" we'd add
> '("xyz" gpg) to the head of the alist and offer to save the defcustom.
> We have to make the "never ask to add" choice 'never, because nil is now
> a valid alist for the value.  So it could only be 'never or a valid
> alist.  Yes, that would work.
>

That sounds perfect to me.

> RP> Would a new keyword in the file itself work?  save-token
> RP> {cleartext,ask,gpg,no}? Tell me if I'm over-engineering this :)
>
> Every line in a netrc file should be self-sufficient, so I'd rather not
> add global keywords.  If you mean on each line, then yes, that's too
> much.

I did mean on each line, but it does seem like overkill. We can always
tell people to split things up into separate files and set preferences
per file.

Robert

Re: netrc field encryption in auth-source

[Ted Zlatanov]

[-- Attachment #1: Type: text/plain, Size: 975 bytes --]

On Fri, 17 Jun 2011 04:32:42 -0500 Ted Zlatanov <tzz@lifelogs.com> wrote: 

TZ> But wait, we can do better if it's an alist...  Let's use the EPA
TZ> file pattern!  The default can then be:

TZ> `((,(car epa-file-auto-mode-alist-entry) nil)
TZ>   (t ask))

TZ> ...and when the user says "yes, use GPG tokens for file xyz" we'd add
TZ> '("xyz" gpg) to the head of the alist and offer to save the defcustom.
TZ> We have to make the "never ask to add" choice 'never, because nil is now
TZ> a valid alist for the value.  So it could only be 'never or a valid
TZ> alist.  Yes, that would work.

This made sense so I implemented a patch, replacing
`auth-source-save-secrets' with `auth-source-netrc-use-gpg-tokens' as
described above.  It uses `epa-file-auto-mode-alist-entry' if it's
bound.  I am not sure if I should just save the defcustom at the time
the user confirms or prompt instead.  Please take a look.  It makes
sense to me and the Customize interface looks nice.

Ted


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: auth-source-gpg-tokens2.patch --]
[-- Type: text/x-diff, Size: 6169 bytes --]

diff --git a/lisp/auth-source.el b/lisp/auth-source.el
index 83e12d6..17be7d5 100644
--- a/lisp/auth-source.el
+++ b/lisp/auth-source.el
@@ -164,15 +164,30 @@ let-binding."
           (const :tag "Never save" nil)
           (const :tag "Ask" ask)))
 
-(defcustom auth-source-save-secrets nil
-  "If set, auth-source will respect it for password tokens behavior."
+;; TODO: make the default (setq auth-source-netrc-use-gpg-tokens `((,(if (boundp 'epa-file-auto-mode-alist-entry) (car (symbol-value 'epa-file-auto-mode-alist-entry)) "\\.gpg\\'") never) (t gpg)))
+;; TODO: or maybe leave as (setq auth-source-netrc-use-gpg-tokens 'never)
+
+(defcustom auth-source-netrc-use-gpg-tokens 'never
+  "Set this to tell auth-source when to create GPG password
+tokens in netrc files.  It's either an alist or `never'."
   :group 'auth-source
   :version "23.2" ;; No Gnus
   :type `(choice
-          :tag "auth-source new password token behavior"
-          (const :tag "Use GPG tokens" gpg)
-          (const :tag "Save unencrypted" nil)
-          (const :tag "Ask" ask)))
+          (const :tag "Always use GPG password tokens" (t gpg))
+          (const :tag "Never use GPG password tokens" never)
+          (repeat :tag "Use a lookup list"
+                  (list
+                   (choice :tag "Matcher"
+                           (const :tag "Match anything" t)
+                           (const :tag "The EPA encrypted file extensions"
+                                  ,(if (boundp 'epa-file-auto-mode-alist-entry)
+                                       (car (symbol-value
+                                             'epa-file-auto-mode-alist-entry))
+                                     "\\.gpg\\'"))
+                           (regexp :tag "Regular expression"))
+                   (choice :tag "What to do"
+                           (const :tag "Save GPG-encrypted password tokens" gpg)
+                           (const :tag "Don't encrypt tokens" never))))))
 
 (defvar auth-source-magic "auth-source-magic ")
 
@@ -257,9 +272,11 @@ can get pretty complex."
                                           ,@auth-source-protocols-customize))
                                         (list :tag "User" :inline t
                                               (const :format "" :value :user)
-                                              (choice :tag "Personality/Username"
+                                              (choice
+                                               :tag "Personality/Username"
                                                       (const :tag "Any" t)
-                                                      (string :tag "Name")))))))))
+                                                      (string
+                                                       :tag "Name")))))))))
 
 (defcustom auth-source-gpg-encrypt-to t
   "List of recipient keys that `authinfo.gpg' encrypted to.
@@ -960,7 +977,7 @@ Note that the MAX parameter is used so we can exit the parse early."
                 (remove (symbol-value 'epa-file-handler)
                         file-name-handler-alist)
               file-name-handler-alist))
-         (find-file-hook
+         (,(if (boundp 'find-file-hook) 'find-file-hook 'find-file-hooks)
           ',(remove 'epa-file-find-file-hook find-file-hook))
          (auto-mode-alist
           ',(if (boundp 'epa-file-auto-mode-alist-entry)
@@ -1216,19 +1233,33 @@ See `auth-source-search' for details on SPEC."
               (cond
                ((and (null data) (eq r 'secret))
                 ;; Special case prompt for passwords.
-                ;; Respect `auth-source-save-secrets'
-                (let* ((ep (format "Do you want GPG password tokens? (%s)"
-                                   "see `auth-source-save-secrets'"))
+;; TODO: make the default (setq auth-source-netrc-use-gpg-tokens `((,(if (boundp 'epa-file-auto-mode-alist-entry) (car (symbol-value 'epa-file-auto-mode-alist-entry)) "\\.gpg\\'") nil) (t gpg)))
+;; TODO: or maybe leave as (setq auth-source-netrc-use-gpg-tokens 'never)
+                (let* ((ep (format "Use GPG password tokens in %s?" file))
                        (gpg-encrypt
-;;; FIXME: this relies on .gpg files being handled by EPA/EPG
-                        ;; don't put GPG tokens in GPG-encrypted files
-                        (and (not (equal "gpg" (file-name-extension file)))
-                             (or (eq auth-source-save-secrets 'gpg)
-                                 (and (eq auth-source-save-secrets 'ask)
-                                      (setq auth-source-save-secrets
-                                            (and (y-or-n-p ep) 'gpg))))))
+                        (cond
+                         ((eq auth-source-netrc-use-gpg-tokens 'never)
+                          'never)
+                         ((listp auth-source-netrc-use-gpg-tokens)
+                          (let ((check (copy-sequence
+                                        auth-source-netrc-use-gpg-tokens))
+                                item ret)
+                            (while check
+                              (setq item (pop check))
+                              (when (string-match (car item) file)
+                                (setq ret (cdr item))
+                                (setq check nil)))))
+                         (t 'never)))
                         (plain (read-passwd prompt)))
-                  (if (eq auth-source-save-secrets 'gpg)
+                  ;; ask if we don't know what to do (in which case
+                  ;; auth-source-netrc-use-gpg-tokens must be a list)
+                  (unless gpg-encrypt
+                    (setq gpg-encrypt (if (y-or-n-p ep) 'gpg 'never))
+                    ;; TODO: save the defcustom now? or ask?
+                    (setq auth-source-netrc-use-gpg-tokens
+                          (cons `(,file ,gpg-encrypt)
+                                auth-source-netrc-use-gpg-tokens)))
+                  (if (eq gpg-encrypt 'gpg)
                       (auth-source-epa-make-gpg-token plain file)
                     plain)))
                ((null data)

Re: netrc field encryption in auth-source

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> machine supertest password
> gpg:jA0EAwMC2tUEaZgM7A5gyWM/owySdCOS/cjoFCuf8LI1d1kYX7z6cjsNkakM04u1geh/iesqyH3XQFI+SEVLb/oEC/EoQ0LIgRRoBiLyu9XZWN1ytY7MQxpPZniFz13oGV4/Dwl8yrP3Hba5LfQpHy2FZRM=
>
> (funcall (plist-get (nth 0 (auth-source-search :host "supertest")) :secret))

This prompts me for the gpg password twice.  Once for ~/.authinfo, and
once for /tmp/gpg-token20359IIl.authinfo.gpg...

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: netrc field encryption in auth-source

[Ted Zlatanov]

On Tue, 21 Jun 2011 21:32:50 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Ted Zlatanov <tzz@lifelogs.com> writes:
>> machine supertest password
>> gpg:jA0EAwMC2tUEaZgM7A5gyWM/owySdCOS/cjoFCuf8LI1d1kYX7z6cjsNkakM04u1geh/iesqyH3XQFI+SEVLb/oEC/EoQ0LIgRRoBiLyu9XZWN1ytY7MQxpPZniFz13oGV4/Dwl8yrP3Hba5LfQpHy2FZRM=
>> 
>> (funcall (plist-get (nth 0 (auth-source-search :host "supertest")) :secret))

LMI> This prompts me for the gpg password twice.  Once for ~/.authinfo, and
LMI> once for /tmp/gpg-token20359IIl.authinfo.gpg...

It shouldn't do that (just the first passphrase should be used, and the
EPA passphrase alist is overridden so you don't see the second prompt).
I'll test it again tonight.

Ted

Committing new smtpmail.el later tonight (was: netrc field encryption in auth-source)

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> It shouldn't do that (just the first passphrase should be used, and the
> EPA passphrase alist is overridden so you don't see the second prompt).
> I'll test it again tonight.

I've now almost finished the smtpmail.el rewrite, and will probably be
committing later tonight.  The only missing pieces are the SMTP client
cert stuff, and saving the passwords.

When I call the :save-function from auth-source, it thinks that it's
going to save the credentials to ~/.authinfo.gpg, which it shouldn't.
What's the proper way to call the function so that the data is saved to
~/.authinfo?

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Committing new smtpmail.el later tonight

[Lars Magne Ingebrigtsen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> When I call the :save-function from auth-source, it thinks that it's
> going to save the credentials to ~/.authinfo.gpg, which it shouldn't.
> What's the proper way to call the function so that the data is saved to
> ~/.authinfo?

For now I've just altered the priority list in auth-source.el, which
makes this work well enough for now, and we can tinker with this more
later.

I am now going to commit the rewritten smtpmail.el.  I've tried testing
all the error conditions I could think of, but there's probably stuff I
haven't thought of.

Please give it a try and report back any errors you find.

It's mostly compatible with the previous version, but there are two
glaring incompatibilities:

`smtpmail-auth-credentials' no longer exists.  That variable could be
either ~/.authinfo (in which case you're fine -- you won't see any
difference), or it could be:

- ;;(setq smtpmail-auth-credentials
- ;;      '(("YOUR SMTP HOST" 25 "username" "password")))

If you had that, you will be prompted for a user name and a password,
which will then be saved in ~/.authinfo.

Similarly, if you had

- ;;(setq smtpmail-starttls-credentials
- ;;      '(("YOUR SMTP HOST" 25 "~/.my_smtp_tls.key" "~/.my_smtp_tls.cert")))

then you need to put

machine smtp.whatever.foo port 25 cert-key "~/.my_smtp_tls.key" cert-cert "~/.my_smtp_tls.cert"

in your ~/.authinfo file instead.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Committing new smtpmail.el later tonight

[Antoine Levitt]

21/06/11 23:01, Lars Magne Ingebrigtsen
> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
>
>> When I call the :save-function from auth-source, it thinks that it's
>> going to save the credentials to ~/.authinfo.gpg, which it shouldn't.
>> What's the proper way to call the function so that the data is saved to
>> ~/.authinfo?
>
> For now I've just altered the priority list in auth-source.el, which
> makes this work well enough for now, and we can tinker with this more
> later.
>
> I am now going to commit the rewritten smtpmail.el.  I've tried testing
> all the error conditions I could think of, but there's probably stuff I
> haven't thought of.
>
> Please give it a try and report back any errors you find.
>
> It's mostly compatible with the previous version, but there are two
> glaring incompatibilities:
>
> `smtpmail-auth-credentials' no longer exists.  That variable could be
> either ~/.authinfo (in which case you're fine -- you won't see any
> difference), or it could be:
>
> - ;;(setq smtpmail-auth-credentials
> - ;;      '(("YOUR SMTP HOST" 25 "username" "password")))
>
> If you had that, you will be prompted for a user name and a password,
> which will then be saved in ~/.authinfo.
>
> Similarly, if you had
>
> - ;;(setq smtpmail-starttls-credentials
> - ;;      '(("YOUR SMTP HOST" 25 "~/.my_smtp_tls.key" "~/.my_smtp_tls.cert")))
>
> then you need to put
>
> machine smtp.whatever.foo port 25 cert-key "~/.my_smtp_tls.key" cert-cert "~/.my_smtp_tls.cert"
>
> in your ~/.authinfo file instead.

Just pulled the changes, now I'm confused. Could you update the info on
http://www.emacswiki.org/emacs/GnusGmail#toc2 ? I get "Must issue a
STARTTLS command first" when I try to send a mail now. I just copied the
config over from the emacswiki page, and I have no idea how to tell the
new code to use STARTTLS.

Re: Committing new smtpmail.el later tonight

[Lars Magne Ingebrigtsen]

Antoine Levitt <antoine.levitt@gmail.com> writes:

> Just pulled the changes, now I'm confused. Could you update the info on
> http://www.emacswiki.org/emacs/GnusGmail#toc2 ? I get "Must issue a
> STARTTLS command first" when I try to send a mail now. I just copied the
> config over from the emacswiki page, and I have no idea how to tell the
> new code to use STARTTLS.

Is this when you try to send via smtp.gmail.com?

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Committing new smtpmail.el later tonight

[Antoine Levitt]

22/06/11 00:17, Lars Magne Ingebrigtsen
> Antoine Levitt <antoine.levitt@gmail.com> writes:
>
>> Just pulled the changes, now I'm confused. Could you update the info on
>> http://www.emacswiki.org/emacs/GnusGmail#toc2 ? I get "Must issue a
>> STARTTLS command first" when I try to send a mail now. I just copied the
>> config over from the emacswiki page, and I have no idea how to tell the
>> new code to use STARTTLS.
>
> Is this when you try to send via smtp.gmail.com?

Yes, with the config just like the emacswiki page. I tried adding

machine smtp.gmail.com login antoine.levitt@gmail.com password password
port 587 cert-key "" cert-cert ""

to my .authinfo, didn't change a thing (and same thing without the cert
stuff).

Re: Committing new smtpmail.el later tonight

[Lars Magne Ingebrigtsen]

Antoine Levitt <antoine.levitt@gmail.com> writes:

>> Is this when you try to send via smtp.gmail.com?
>
> Yes, with the config just like the emacswiki page.

I found a bug in the MAIL FROM auth handling, which made the auth not
work unless you already had a password in the ~/.authinfo file, but that
doesn't seem to be your problem...

Hm...

Ah, I think I know what the problem is.  You don't have gnutls built in
in your Emacs?  That is, does things start working if you say

(require 'gnutls)

and you've compiled your Emacs with gnutls support?  In that case, I
think I know what the problem is, and I'll fix it in a few jiffies...

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Committing new smtpmail.el later tonight

[Lars Magne Ingebrigtsen]

Yeah, I'm able to reproduce the bug if I don't use the built-in TLS
support.

`open-network-stream' only does opportunistic STARTTLS upgrades if
there's built-in support.  Mainly because the non-built-in support is
slow, and has historically been slightly buggy.

But the problem with smtp.gmail.com isn't that it just offers STARTTLS
support, but requires it.  So `open-network-stream' should, in this
case, do STARTTLS even if Emacs doesn't have built-in support.

So I'm wondering what's the best approach here.

1) `open-network-stream' can do opportunistic STARTTLS upgrades for all
protocols, using the external STARTTLS support.  This will be slower,
and may be more buggy.

2) I can add yet another parameter to `open-network-stream',
:always-use-starttls-if-possible, and have smtpmail.el set it.  This
will have least impact overall, but, like, adds yet another parameter,
so it's kinda tacky.

I think I'll do 2) for now to get things working, and we can reexamine
this later.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Committing new smtpmail.el later tonight

[Lars Magne Ingebrigtsen]

I've now applied the change.  Antoine, could you see whether you can
send email now?

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Committing new smtpmail.el later tonight

[Antoine Levitt]

22/06/11 00:57, Lars Magne Ingebrigtsen
> I've now applied the change.  Antoine, could you see whether you can
> send email now?

Yup, everything alright now. Thanks!

Re: Committing new smtpmail.el later tonight

[Robert Pluim]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Antoine Levitt <antoine.levitt@gmail.com> writes:
>
>>> Is this when you try to send via smtp.gmail.com?
>>
>> Yes, with the config just like the emacswiki page.
>
> I found a bug in the MAIL FROM auth handling, which made the auth not
> work unless you already had a password in the ~/.authinfo file, but that
> doesn't seem to be your problem...

I see you committed a fix for this, but I think things are still not
quite right. I had an existing username/port spec in .authinfo, but no
password. When sending mail, the connection comes up, but sending fails
because smtpmail has never offered to authenticate. Should it not be
prompting me for the password?

Robert

Re: Committing new smtpmail.el later tonight

[Lars Magne Ingebrigtsen]

Robert Pluim <rpluim@gmail.com> writes:

> I see you committed a fix for this, but I think things are still not
> quite right. I had an existing username/port spec in .authinfo, but no
> password. When sending mail, the connection comes up, but sending fails
> because smtpmail has never offered to authenticate. Should it not be
> prompting me for the password?

What is the error message you get?

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Committing new smtpmail.el later tonight

[Robert Pluim]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Robert Pluim <rpluim@gmail.com> writes:
>
>> I see you committed a fix for this, but I think things are still not
>> quite right. I had an existing username/port spec in .authinfo, but no
>> password. When sending mail, the connection comes up, but sending fails
>> because smtpmail has never offered to authenticate. Should it not be
>> prompting me for the password?
>
> What is the error message you get?


error: "Process smtpmail not running"

The smtp trace buffer has:

MAIL FROM:<rpluim@gmail.com> SIZE=314
530-5.5.1 Authentication Required. Learn more at                              
530 5.5.1 http://mail.google.com/support/bin/answer.py?answer=14257 h22sm137374wes.8
QUIT
221 2.0.0 closing connection h22sm137374wes.8
QUIT

Re: Committing new smtpmail.el later tonight

[Lars Magne Ingebrigtsen]

Robert Pluim <rpluim@gmail.com> writes:

> error: "Process smtpmail not running"

Do you get a backtrace here?

> The smtp trace buffer has:
>
> MAIL FROM:<rpluim@gmail.com> SIZE=314
> 530-5.5.1 Authentication Required. Learn more at                              
> 530 5.5.1 http://mail.google.com/support/bin/answer.py?answer=14257
> h22sm137374wes.8
> QUIT
> 221 2.0.0 closing connection h22sm137374wes.8
> QUIT

Ah, it's sending QUIT twice...  hm...

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Committing new smtpmail.el later tonight

[Lars Magne Ingebrigtsen]

I've now fixed the double-QUIT problem, so you should be getting a
better error message now, I think...

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Committing new smtpmail.el later tonight

[Robert Pluim]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> I've now fixed the double-QUIT problem, so you should be getting a
> better error message now, I think...

umm, yes. But still no prompt for a password.

My .authinfo has:

machine smtp.gmail.com login rpluim port 587

Does it need modification?

Robert

Re: Committing new smtpmail.el later tonight

[Lars Magne Ingebrigtsen]

Robert Pluim <rpluim@gmail.com> writes:

> My .authinfo has:
>
> machine smtp.gmail.com login rpluim port 587
>
> Does it need modification?

No, it's a bug.  It should query you for a password in this case, but
auth-source doesn't do that, since it finds an entry.  It's either a bug
in auth-source, or a bug in the way smtpmail.el uses auth-source.

I'll investigate later today, but I don't have time the next few hours,
I think...

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Committing new smtpmail.el later tonight

[Lars Magne Ingebrigtsen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> I'll investigate later today, but I don't have time the next few hours,
> I think...

Ok, I've now investigated, and it's both a bug in smtpmail.el, and a
misfeature in auth-source.el, I think.

What we want is to be able to say "give me the credentials for
server:port, and prompt and create the missing bits".  This doesn't seem
possible today.  If there is a "user" entry in ~/.authinfo, that should
obviously be used, and it should just prompt for the password.

From what I'm reading in the code, auth-source doesn't do that, so I'm
going to add that.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Committing new smtpmail.el later tonight

[Lars Magne Ingebrigtsen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> From what I'm reading in the code, auth-source doesn't do that, so I'm
> going to add that.

I've now done this, and it should (at least) make it possible for you to
send emails again.

However, if you save the passwords, it'll probably double up the lines,
which I need to fix somehow...

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Committing new smtpmail.el later tonight

[Robert Pluim]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
>
>> From what I'm reading in the code, auth-source doesn't do that, so I'm
>> going to add that.
>
> I've now done this, and it should (at least) make it possible for you to
> send emails again.
>

Indeed, although I'm getting the following message

gnutls.c: [0] (Emacs) fatal error: A TLS packet with unexpected length
was received.

and 

Process smtpmail connection broken by remote peer

in the SMTP trace buffer, which would seem to indicate that we're
disconnecting and reconnecting to the SMTP server, then doing AUTH? Or
is this a side effect of STARTTLS?

> However, if you save the passwords, it'll probably double up the lines,
> which I need to fix somehow...

I answered 'N' to the saving question, so this didn't happen to me :)

Robert

Re: Committing new smtpmail.el later tonight

[Lars Magne Ingebrigtsen]

Robert Pluim <rpluim@gmail.com> writes:

> Indeed, although I'm getting the following message
>
> gnutls.c: [0] (Emacs) fatal error: A TLS packet with unexpected length
> was received.
>
> and 
>
> Process smtpmail connection broken by remote peer
>
> in the SMTP trace buffer, which would seem to indicate that we're
> disconnecting and reconnecting to the SMTP server, then doing AUTH? Or
> is this a side effect of STARTTLS?

[...]

> I answered 'N' to the saving question, so this didn't happen to me :)

Oh, right.  The drop the connection/start again for AUTH is only
supposed to happen the first time, but since you're not saving, it's
happening every time, which is anti-social (and slower).

I'll fix it to not do that in the only-user-name-stored situation.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Committing new smtpmail.el later tonight

[Robert Pluim]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Robert Pluim <rpluim@gmail.com> writes:
>
>> Indeed, although I'm getting the following message
>>
>> gnutls.c: [0] (Emacs) fatal error: A TLS packet with unexpected length
>> was received.
>>
>> and 
>>
>> Process smtpmail connection broken by remote peer
>>
>> in the SMTP trace buffer, which would seem to indicate that we're
>> disconnecting and reconnecting to the SMTP server, then doing AUTH? Or
>> is this a side effect of STARTTLS?
>
> [...]
>
>> I answered 'N' to the saving question, so this didn't happen to me :)
>
> Oh, right.  The drop the connection/start again for AUTH is only
> supposed to happen the first time, but since you're not saving, it's
> happening every time, which is anti-social (and slower).
>
> I'll fix it to not do that in the only-user-name-stored situation.

Works fine now (and I'm not getting prompted to save anymore, which is
nice. Please don't change that.)

Robert

Re: Committing new smtpmail.el later tonight

[Ted Zlatanov]

On Wed, 22 Jun 2011 16:25:50 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
>> I'll investigate later today, but I don't have time the next few hours,
>> I think...

LMI> Ok, I've now investigated, and it's both a bug in smtpmail.el, and a
LMI> misfeature in auth-source.el, I think.

LMI> What we want is to be able to say "give me the credentials for
LMI> server:port, and prompt and create the missing bits".  This doesn't seem
LMI> possible today.  If there is a "user" entry in ~/.authinfo, that should
LMI> obviously be used, and it should just prompt for the password.

LMI> From what I'm reading in the code, auth-source doesn't do that, so I'm
LMI> going to add that.

Have you looked at :create and :require in `auth-source-search'?  Also
see the `auth-source-creation-{prompts,defaults}' dynamic overrides.

At least for netrc files, :create should DTRT as you describe.  It's
used that way in nnimap.el:

#+begin_src lisp
(defun nnimap-credentials (address ports user)
  (let* ((auth-source-creation-prompts
          '((user  . "IMAP user at %h: ")
            (secret . "IMAP password for %u@%h: ")))
         (found (nth 0 (auth-source-search :max 1
                                           :host address
                                           :port ports
                                           :user user
                                           :require '(:user :secret)
                                           :create t))))
    (if found
        (list (plist-get found :user)
	      (let ((secret (plist-get found :secret)))
		(if (functionp secret)
		    (funcall secret)
		  secret))
	      (plist-get found :save-function))
      nil)))

#+end_src


Ted

Re: Committing new smtpmail.el later tonight

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> At least for netrc files, :create should DTRT as you describe.  It's
> used that way in nnimap.el:

If I had

machine foo port bar login zot

and used

(auth-source-search :max 1
                     :host "foo"
                     :port "bar"
                     :require '(:user :secret))

it wouldn't return anything,

and if I added

                     :create t

then it would prompt for both the user name and the secret.

I think my latest changes to auth-source fixes this, but I'd appreciate
it if you'd go over the changes and check whether I was misunderstanding
something in the code.  :-)
                     
-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Committing new smtpmail.el later tonight

[Ted Zlatanov]

On Wed, 22 Jun 2011 21:24:00 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Ted Zlatanov <tzz@lifelogs.com> writes:
>> At least for netrc files, :create should DTRT as you describe.  It's
>> used that way in nnimap.el:

LMI> If I had

LMI> machine foo port bar login zot

LMI> and used

LMI> (auth-source-search :max 1
LMI>                      :host "foo"
LMI>                      :port "bar"
LMI>                      :require '(:user :secret))

LMI> it wouldn't return anything,

LMI> and if I added

LMI>                      :create t

LMI> then it would prompt for both the user name and the secret.

Didn't we do this for nnimap.el?  It saves the netrc line only if the
login is successful.  You can do a preliminary search to see if you find
the :host and :port without :max; if so you can fill the :user into the
`auth-source-creation-defaults' dynamic override.

So, something like the following untested code, partly from
`nnimap-credentials':

#+begin_src text
(let* ((users (delq nil
                    (loop for result
                          in (auth-source-search :host "foo":port "bar")
                          collect (plist-get result :user))))
       (auth-source-creation-prompts
        '((user  . "SMTP user at %h: ")
          (secret . "SMTP password for %u@%h: ")))
       (auth-source-creation-defaults
        '((user . (nth 0 users))))
       (found (nth 0 (auth-source-search :max 1
                                         :host "foo"
                                         :port "bar"
                                         :user users
                                         :require '(:user :secret)
                                         :create t))))
  (if found
      (list (plist-get found :user)
            (let ((secret (plist-get found :secret)))
              (if (functionp secret)
                  (funcall secret)
                secret))
            (plist-get found :save-function))
    nil))

#+end_src

This will create a new line iff you call the :save-function.

LMI> I think my latest changes to auth-source fixes this, but I'd appreciate
LMI> it if you'd go over the changes and check whether I was misunderstanding
LMI> something in the code.  :-)

I think they are reasonable.  But maybe the above will work?  It seems
to me it would work for `nnimap-credentials' as well, which perhaps
argues that it should be in the auth-source API.

Ted

Re: Committing new smtpmail.el later tonight

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> But maybe the above will work?  It seems to me it would work for
> `nnimap-credentials' as well, which perhaps argues that it should be
> in the auth-source API.

It's possible.  The SMTP situation is slightly special, though.  For
IMAP, we know that we want credentials.  For SMTP, we don't know, so we
first see if there are any there, and only if there are, we want to
:create the possibly missing `secret'.

I think doing that special-casing in smtpmail.el instead of trying to
shove it down into the auth-source functions is slightly cleaner.
Unless some new protocols appear that have the same requirements as
SMTP.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Committing new smtpmail.el later tonight

[Ted Zlatanov]

On Wed, 22 Jun 2011 22:43:48 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Ted Zlatanov <tzz@lifelogs.com> writes:
>> But maybe the above will work?  It seems to me it would work for
>> `nnimap-credentials' as well, which perhaps argues that it should be
>> in the auth-source API.

LMI> It's possible.  The SMTP situation is slightly special, though.  For
LMI> IMAP, we know that we want credentials.  For SMTP, we don't know, so we
LMI> first see if there are any there, and only if there are, we want to
LMI> :create the possibly missing `secret'.

LMI> I think doing that special-casing in smtpmail.el instead of trying to
LMI> shove it down into the auth-source functions is slightly cleaner.
LMI> Unless some new protocols appear that have the same requirements as
LMI> SMTP.

OK.  So really the case is "possibly anonymous" versus "definitely
authenticated" services.  NNTP is also like that and IMAP itself can
function unauthenticated if so configured.  IRC can be used anonymously
or not.  So maybe `auth-source-search' should be smart enough to
accomodate the two cases through a common API.

Ted

Re: Committing new smtpmail.el later tonight

[Eli Zaretskii]

> From: Lars Magne Ingebrigtsen <larsi@gnus.org>
> Date: Tue, 21 Jun 2011 23:01:33 +0200
> 
> `smtpmail-auth-credentials' no longer exists.  That variable could be
> either ~/.authinfo (in which case you're fine -- you won't see any
> difference), or it could be:
> 
> - ;;(setq smtpmail-auth-credentials
> - ;;      '(("YOUR SMTP HOST" 25 "username" "password")))
> 
> If you had that, you will be prompted for a user name and a password,
> which will then be saved in ~/.authinfo.

Why do we have to have this incompatibility?  Why cannot
smtpmail-auth-credentials continue be supported, in addition to the
new ~/.authinfo?  E.g., note the value of the variable and
automatically write ~/.authinfo?

Re: Committing new smtpmail.el later tonight

[Lars Magne Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Why do we have to have this incompatibility?  Why cannot
> smtpmail-auth-credentials continue be supported, in addition to the
> new ~/.authinfo?  E.g., note the value of the variable and
> automatically write ~/.authinfo?

I can (re-)implement that.

However, I thought it might make more sense to unconfuse the entire
credential situations.  In the past, all things that have required auth
of some kind have used their own methods for doing that, and by making a
"clean break", I think things might be slightly less confusing for the
user.

And it's not like the user has to know about this beforehand.
smtpmail.el will prompt for the user name and password, so things won't
mysteriously stop working totally.

If people feel that it would make more sense to still support
`smtpmail-auth-credentials', I'll put that back in there again.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Committing new smtpmail.el later tonight

[Robert Pluim]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>> Why do we have to have this incompatibility?  Why cannot
>> smtpmail-auth-credentials continue be supported, in addition to the
>> new ~/.authinfo?  E.g., note the value of the variable and
>> automatically write ~/.authinfo?
>
> I can (re-)implement that.
>
> However, I thought it might make more sense to unconfuse the entire
> credential situations.  In the past, all things that have required auth
> of some kind have used their own methods for doing that, and by making a
> "clean break", I think things might be slightly less confusing for the
> user.
>
> And it's not like the user has to know about this beforehand.
> smtpmail.el will prompt for the user name and password, so things won't
> mysteriously stop working totally.
>
> If people feel that it would make more sense to still support
> `smtpmail-auth-credentials', I'll put that back in there again.

No, what you've done is better than the old situation, let's not overdo
the backwards compatibility.

Robert

Re: Committing new smtpmail.el later tonight

[Eli Zaretskii]

> From: Lars Magne Ingebrigtsen <larsi@gnus.org>
> Date: Wed, 22 Jun 2011 16:53:40 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Why do we have to have this incompatibility?  Why cannot
> > smtpmail-auth-credentials continue be supported, in addition to the
> > new ~/.authinfo?  E.g., note the value of the variable and
> > automatically write ~/.authinfo?
> 
> I can (re-)implement that.
> 
> However, I thought it might make more sense to unconfuse the entire
> credential situations.  In the past, all things that have required auth
> of some kind have used their own methods for doing that, and by making a
> "clean break", I think things might be slightly less confusing for the
> user.

That is fine for newcomers to smtpmail.  But put myself in my shoes: I
configured my smtpmail years ago, and don't even remember the name of
my SMTP server, let alone the password.  People like me will not be
amused if their old setup suddenly stops working until they dust out
old instructions.

Unlike other software, Emacs didn't until now break customizations in
such a way, not without some sort of interim period when users were
told to migrate.

Re: Committing new smtpmail.el later tonight

[Ted Zlatanov]

On Wed, 22 Jun 2011 19:19:59 +0300 Eli Zaretskii <eliz@gnu.org> wrote: 

EZ> That is fine for newcomers to smtpmail.  But put myself in my shoes: I
EZ> configured my smtpmail years ago, and don't even remember the name of
EZ> my SMTP server, let alone the password.  People like me will not be
EZ> amused if their old setup suddenly stops working until they dust out
EZ> old instructions.

The variable is obsoleted.  Isn't that the normal way to do it, possibly
annoying people?  It's not particularly hard to migrate the info
yourself, as I mentioned.

EZ> Unlike other software, Emacs didn't until now break customizations in
EZ> such a way, not without some sort of interim period when users were
EZ> told to migrate.

Since 24.x is a major release, maybe it's OK.  How would you do it?

On Wed, 22 Jun 2011 19:51:29 +0300 Eli Zaretskii <eliz@gnu.org> wrote: 

EZ> Why do we have to have this incompatibility?  Why cannot
EZ> smtpmail-auth-credentials continue be supported, in addition to the
EZ> new ~/.authinfo?  E.g., note the value of the variable and
EZ> automatically write ~/.authinfo?
>> 
>> I think that leaves `smtpmail-auth-credentials' in place, where it will
>> confuse the user.

EZ> How can my own customization on my own .emacs confuse me?

There are now two places where the credentials live, if we follow your
scheme.  You can forget about ~/.authinfo and modify
`smtpmail-auth-credentials' which won't work.  I think that's a bad
situation.

>> So IMO the conversion should also unset `smtpmail-auth-credentials'.

EZ> I see no need for that.  Emacs doesn't modify .emacs except during
EZ> explicit use of Customize.

I'm sure in at least a few places I've seen a defcustom set explicitly
and saved.  But the point is, we want to *migrate* and that implies
dropping the old settings.  Otherwise it's *copying*.

Ted

Re: Committing new smtpmail.el later tonight

[Eli Zaretskii]

> From: Ted Zlatanov <tzz@lifelogs.com>
> Date: Wed, 22 Jun 2011 12:16:53 -0500
> 
> On Wed, 22 Jun 2011 19:19:59 +0300 Eli Zaretskii <eliz@gnu.org> wrote: 
> 
> EZ> That is fine for newcomers to smtpmail.  But put myself in my shoes: I
> EZ> configured my smtpmail years ago, and don't even remember the name of
> EZ> my SMTP server, let alone the password.  People like me will not be
> EZ> amused if their old setup suddenly stops working until they dust out
> EZ> old instructions.
> 
> The variable is obsoleted.  Isn't that the normal way to do it, possibly
> annoying people?

I understand that the variable is not just obsoleted, it no longer has
any effect whatsoever.  Isn't that so?

> It's not particularly hard to migrate the info yourself, as I
> mentioned.

I don't like to be forced to migrate.  As I explained, it's a nuisance
for me, because I need to dig up information I long forgot.

> EZ> Unlike other software, Emacs didn't until now break customizations in
> EZ> such a way, not without some sort of interim period when users were
> EZ> told to migrate.
> 
> Since 24.x is a major release, maybe it's OK.  How would you do it?

Declare it obsolete in NEWS and slate for removal in a couple of
releases.  Like we always did.

> EZ> How can my own customization on my own .emacs confuse me?
> 
> There are now two places where the credentials live, if we follow your
> scheme.  You can forget about ~/.authinfo and modify
> `smtpmail-auth-credentials' which won't work.

Well, I'm asking that it _does_ work, right?

Anyway, this conversation seems to go nowhere.  I made a request and
explained the reasons; if you want to ignore it, I guess I'll survive.

Re: Committing new smtpmail.el later tonight

[Lars Magne Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> I understand that the variable is not just obsoleted, it no longer has
> any effect whatsoever.  Isn't that so?

Yes.

> I don't like to be forced to migrate.  As I explained, it's a nuisance
> for me, because I need to dig up information I long forgot.

I agree, it's a nuisance.  I suspect, however, that the long-term
nuisance of having (possibly conflicting) passwords stored in two
different places will be bigger for users in general.

It isn't obvious to me which is the better way to do this, so if more
people could give their opinions on this, I'll implement whatever the
consensus is.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Committing new smtpmail.el later tonight

[Ted Zlatanov]

On Wed, 22 Jun 2011 22:50:57 +0300 Eli Zaretskii <eliz@gnu.org> wrote: 

>> From: Ted Zlatanov <tzz@lifelogs.com>
>> Date: Wed, 22 Jun 2011 12:16:53 -0500
>> 
>> On Wed, 22 Jun 2011 19:19:59 +0300 Eli Zaretskii <eliz@gnu.org> wrote: 
>> 
EZ> That is fine for newcomers to smtpmail.  But put myself in my shoes: I
EZ> configured my smtpmail years ago, and don't even remember the name of
EZ> my SMTP server, let alone the password.  People like me will not be
EZ> amused if their old setup suddenly stops working until they dust out
EZ> old instructions.
>> 
>> The variable is obsoleted.  Isn't that the normal way to do it, possibly
>> annoying people?

EZ> I understand that the variable is not just obsoleted, it no longer has
EZ> any effect whatsoever.  Isn't that so?

Yes.  I think Lars and Stefan explained why we should kill it better
than I could.  Sorry if it's inconvenient, but I don't see a way of
keeping backwards compatibility without ending up with duplicate
information.  We're really migrating from Lisp-based to file-based
authentication token storage, not just changing a variable's effect.
It's a big change.  Trying to plaster over it now will, I think, be even
more painful later.

Ted

Re: Committing new smtpmail.el later tonight

[Stefan Monnier]

> The variable is obsoleted.  Isn't that the normal way to do it, possibly
> annoying people?  It's not particularly hard to migrate the info
> yourself, as I mentioned.

Of course, but we can do better.

EZ> Unlike other software, Emacs didn't until now break customizations in
EZ> such a way, not without some sort of interim period when users were
EZ> told to migrate.
> Since 24.x is a major release, maybe it's OK.  How would you do it?

I'd at least give a hint to the user.  If the user suddenly gets
prompted for his username/password (or just his email fail to get sent
for lack of authentication), her first reaction is likely to be along
the lines of M-x report-emacs-bug or some other curse.

So I think that we have two situations to handle:
- smtpmail-auth-credentials is set and auth-source can't find
  credentials (i.e. the user hasn't migrated yet), so we should warn the
  user about the need to migrate and give some pointer of how to do it.
- smtpmail-auth-credentials is set and auth-source can also find credentials
  (i.e. the user has migrated but still has the old setting).  Here we
  can output a warning about the duplicate info, but it's not
  high-priority and should be discrete.  After all, the user may use
  such a config purposefully because she likes to share a single config
  for different machines where she uses different versions of Emacs.


        Stefan

Re: Committing new smtpmail.el later tonight

[Lars Magne Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> So I think that we have two situations to handle:
> - smtpmail-auth-credentials is set and auth-source can't find
>   credentials (i.e. the user hasn't migrated yet), so we should warn the
>   user about the need to migrate and give some pointer of how to do it.

It could just ask "Update ~/.authinfo with SMTP credentials?" and write
to the file, perhaps?

> - smtpmail-auth-credentials is set and auth-source can also find credentials
>   (i.e. the user has migrated but still has the old setting).

So we would prefer credentials from auth-source, which is OK.  My worry
is that the user would not necessarily be aware of this (unless we issue
warnings), and would tinker with `smtpmail-auth-credentials' in ~/.emacs
and not understand why the settings done there had any effect.

After all, most users just google stuff, and 95% of the web pages that
talk about this stuff will say "alter `smtpmail-auth-credentials'".  And
using that variable worked the first time Emacs 24 was used, but not
after altering the second time.

Seems like there would be plenty of possibilities to confuse the users
here. 

And at some point, we'll have to drop the variable.  I sort of feel that
we're prolonging the agony instead of making it swift and brutal.  :-)

Perhaps.

>   Here we can output a warning about the duplicate info, but it's not
>   high-priority and should be discrete.  After all, the user may use
>   such a config purposefully because she likes to share a single
>   config for different machines where she uses different versions of
>   Emacs.

Yes.  I'm not sure what would be the best way to do a discrete warning
that would still be visible enough that the user would actually see it.

If smtpmail.el just does

(message "Warning: Conflicting values from `smtpmail-auth-credentials'; using ~/.authinfo instead")

then that would, in virtually all cases, just be overwritten by the

Sending...done

message after sending it, so the user wouldn't see it...

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Committing new smtpmail.el later tonight

[Lars Magne Ingebrigtsen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> I sort of feel that we're prolonging the agony instead of making it
> swift and brutal.  :-)

I mean, the plan is still to default `send-mail-function' and
`message-send-mail-function' on all architectures in Emacs 24 to
`smtpmail-send-it', isn't it?  So there will be mail-related pain² for
old Emacs users, anyway, and we might as well get it over with in one
fell swoop.  :-)

--
²) The pain will be that the first time you send email on Emacs 24,
it'll prompt you for "Outgoing SMTP mail server: " instead of using the
local MTA, which is the current default on the majority of Linux
installations.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Committing new smtpmail.el later tonight

[Ted Zlatanov]

On Wed, 22 Jun 2011 05:52:19 +0300 Eli Zaretskii <eliz@gnu.org> wrote: 

>> From: Lars Magne Ingebrigtsen <larsi@gnus.org>
>> Date: Tue, 21 Jun 2011 23:01:33 +0200
>> 
>> `smtpmail-auth-credentials' no longer exists.  That variable could be
>> either ~/.authinfo (in which case you're fine -- you won't see any
>> difference), or it could be:
>> 
>> - ;;(setq smtpmail-auth-credentials
>> - ;;      '(("YOUR SMTP HOST" 25 "username" "password")))
>> 
>> If you had that, you will be prompted for a user name and a password,
>> which will then be saved in ~/.authinfo.

EZ> Why do we have to have this incompatibility?  Why cannot
EZ> smtpmail-auth-credentials continue be supported, in addition to the
EZ> new ~/.authinfo?  E.g., note the value of the variable and
EZ> automatically write ~/.authinfo?

I think that leaves `smtpmail-auth-credentials' in place, where it will
confuse the user.  So IMO the conversion should also unset
`smtpmail-auth-credentials'.

I don't know if the extra work is worthwhile, though.  The conversion is
trivial and maybe should simply be explained in the auth-source manual.

Ted

Re: Committing new smtpmail.el later tonight

[Eli Zaretskii]

> From: Ted Zlatanov <tzz@lifelogs.com>
> Date: Wed, 22 Jun 2011 10:55:26 -0500
> 
> EZ> Why do we have to have this incompatibility?  Why cannot
> EZ> smtpmail-auth-credentials continue be supported, in addition to the
> EZ> new ~/.authinfo?  E.g., note the value of the variable and
> EZ> automatically write ~/.authinfo?
> 
> I think that leaves `smtpmail-auth-credentials' in place, where it will
> confuse the user.

How can my own customization on my own .emacs confuse me?

> So IMO the conversion should also unset `smtpmail-auth-credentials'.

I see no need for that.  Emacs doesn't modify .emacs except during
explicit use of Customize.

Re: Committing new smtpmail.el later tonight

[Ted Zlatanov]

On Tue, 21 Jun 2011 22:19:01 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> When I call the :save-function from auth-source, it thinks that it's
LMI> going to save the credentials to ~/.authinfo.gpg, which it shouldn't.
LMI> What's the proper way to call the function so that the data is saved to
LMI> ~/.authinfo?

I see you switched the order, which is what I was going to propose.
Thanks.  Due to a power outage I couldn't help with any of your
questions sooner; sorry about that.

Ted

Re: netrc field encryption in auth-source

[Ted Zlatanov]

On Tue, 21 Jun 2011 14:51:38 -0500 Ted Zlatanov <tzz@lifelogs.com> wrote: 

TZ> On Tue, 21 Jun 2011 21:32:50 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 
LMI> Ted Zlatanov <tzz@lifelogs.com> writes:
>>> machine supertest password
>>> gpg:jA0EAwMC2tUEaZgM7A5gyWM/owySdCOS/cjoFCuf8LI1d1kYX7z6cjsNkakM04u1geh/iesqyH3XQFI+SEVLb/oEC/EoQ0LIgRRoBiLyu9XZWN1ytY7MQxpPZniFz13oGV4/Dwl8yrP3Hba5LfQpHy2FZRM=
>>> 
>>> (funcall (plist-get (nth 0 (auth-source-search :host "supertest")) :secret))

LMI> This prompts me for the gpg password twice.  Once for ~/.authinfo, and
LMI> once for /tmp/gpg-token20359IIl.authinfo.gpg...

TZ> It shouldn't do that (just the first passphrase should be used, and the
TZ> EPA passphrase alist is overridden so you don't see the second prompt).
TZ> I'll test it again tonight.

This should be resolved with my latest commit to Gnus (soon to be
synchronized with Emacs, when Katsumi Yamaoka runs the sync).

Thanks
Ted

Re: Opportunistic STARTTLS in smtpmail.el

[Stefan Monnier]

>> Oh, right, it should be smtpmail-use-auth.
> I don't think that's necessary.

I don't understand how you avoid the need for it:

> Here's my vision of how this would work for a person using Emacs 24 to
> send an email for the first time.

I'm more interested in the subsequent times:

> 1) smtpmail would see that there's no host setup, so it would prompt the
> user for the SMTP host and store it via customize.

For subsequent times, this is fine: no prompting.

> 2) smtpmail connects to the server, and possibly does STARTTLS

OK so far.

> III) It sees whether the server says that it accepts AUTH or not.  If it
> does, smtpmail then asks auth-source for credentials.  There will be
> none there, probably

If the user has a ~/.authinfo.gpg, the user will be prompted at this
point: *bad* since we don't know yet whether this prompt is needed.


        Stefan

Re: Opportunistic STARTTLS in smtpmail.el

[Lars Magne Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> III) It sees whether the server says that it accepts AUTH or not.  If it
>> does, smtpmail then asks auth-source for credentials.  There will be
>> none there, probably
>
> If the user has a ~/.authinfo.gpg, the user will be prompted at this
> point: *bad* since we don't know yet whether this prompt is needed.

That's true.  However, I don't think the user has a ~/.authinfo.gpg
file.  :-)

auth-source won't create that file by default (when we get the gpg:
tokens implemented), so the only way for that file to exist is if the
user has created it explicitly.  And if the user has created it
explicitly, I think it's likely that the user has already opened it to
connect to an IMAP server or the like, so it's most likely already
decoded.

If it exists, which it won't.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Opportunistic STARTTLS in smtpmail.el

[Stefan Monnier]

> auth-source won't create that file by default (when we get the gpg:
> tokens implemented), so the only way for that file to exist is if the
> user has created it explicitly.  And if the user has created it
> explicitly, I think it's likely that the user has already opened it to
> connect to an IMAP server or the like, so it's most likely already
> decoded.

> If it exists, which it won't.  :-)

That's what I said earlier: the new scheme can make sense, but at the
condition to deprecate/discourage authinfo.gpg.


        Stefan

client certs and CRL lists for GnuTLS (was: Opportunistic STARTTLS in smtpmail.el)

[Ted Zlatanov]

[-- Attachment #1: Type: text/plain, Size: 1535 bytes --]

On Mon, 02 May 2011 14:21:32 -0500 Ted Zlatanov <tzz@lifelogs.com> wrote: 

TZ> On Mon, 02 May 2011 20:59:18 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 
LMI> Ted Zlatanov <tzz@lifelogs.com> writes:
LMI> "--x509keyfile" "--x509certfile"
>>> 
LMI> to gnutlc-cli.  `open-network-stream' has no concept of these things,
LMI> and I'm not sure gnutls.c has, either.  Ted?
>>> 
>>> Yes, definitely, with the :keyfiles and :trustfiles parameters to
>>> `gnutls-boot'.

LMI> Right.  Would "--x509keyfile" correspond to :keyfiles and
LMI> "--x509certfile" to :trustfiles?

TZ> Oh wait, I think I'm wrong.  The key+cert files (client-side SSL certs)
TZ> are not the same as the trust files (which verify the server's SSL
TZ> cert).  Let me take a look, this may require another parameter or I'm
TZ> missing something.

OK, I found the problem.  See the attached patch.  Before I was
processing client key files as CRL lists (I meant to do client key+cert
pairs PLUS CRL lists and somehow merged the two mentally).

The attached patch adds a :keylist parameter to `gnutls-boot' which is a
list of (client key file, client cert file) pairs.  It also renames the
:keyfiles parameter to :crlfiles since it's for CRL lists.  So now you
can specify any number of client certs.  If the key files require a
passphrase, the decoding won't work because we don't set a callback.

`gnutls-negotiate' also gets the parameter changes (should I just make
it take a plist?)

Please look it over and I'll commit if you agree it's correct.

Thanks
Ted


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: gnutls-crl-and-client-certs.patch --]
[-- Type: text/x-diff, Size: 6400 bytes --]

=== modified file 'lisp/net/gnutls.el'
--- lisp/net/gnutls.el	2011-04-25 13:47:23 +0000
+++ lisp/net/gnutls.el	2011-05-03 15:14:03 +0000
@@ -86,7 +86,7 @@
 (declare-function gnutls-errorp "gnutls.c" (error))
 
 (defun gnutls-negotiate (proc type hostname &optional priority-string
-                              trustfiles keyfiles verify-flags
+                              trustfiles crlfiles keylist verify-flags
                               verify-error verify-hostname-error)
   "Negotiate a SSL/TLS connection.  Returns proc. Signals gnutls-error.
 TYPE is `gnutls-x509pki' (default) or `gnutls-anon'.  Use nil for the default.
@@ -94,7 +94,8 @@
 HOSTNAME is the remote hostname.  It must be a valid string.
 PRIORITY-STRING is as per the GnuTLS docs, default is \"NORMAL\".
 TRUSTFILES is a list of CA bundles.
-KEYFILES is a list of client keys.
+CRLFILES is a list of CRL files.
+KEYLIST is an alist of (client key file, client cert file) pairs.
 
 When VERIFY-HOSTNAME-ERROR is not nil, an error will be raised
 when the hostname does not match the presented certificate's host
@@ -141,7 +142,8 @@
                              :hostname ,hostname
                              :loglevel ,gnutls-log-level
                              :trustfiles ,trustfiles
-                             :keyfiles ,keyfiles
+                             :crlfiles ,crlfiles
+                             :keylist ,keylist
                              :verify-flags ,verify-flags
                              :verify-error ,verify-error
                              :verify-hostname-error ,verify-hostname-error

=== modified file 'src/gnutls.c'
--- src/gnutls.c	2011-05-02 02:49:06 +0000
+++ src/gnutls.c	2011-05-03 15:11:35 +0000
@@ -44,7 +44,8 @@
 /* The following are for the property list of `gnutls-boot'.  */
 static Lisp_Object Qgnutls_bootprop_priority;
 static Lisp_Object Qgnutls_bootprop_trustfiles;
-static Lisp_Object Qgnutls_bootprop_keyfiles;
+static Lisp_Object Qgnutls_bootprop_keylist;
+static Lisp_Object Qgnutls_bootprop_crlfiles;
 static Lisp_Object Qgnutls_bootprop_callbacks;
 static Lisp_Object Qgnutls_bootprop_loglevel;
 static Lisp_Object Qgnutls_bootprop_hostname;
@@ -412,7 +413,10 @@
 
 :trustfiles is a list of PEM-encoded trust files for `gnutls-x509pki'.
 
-:keyfiles is a list of PEM-encoded key files for `gnutls-x509pki'.
+:crlfiles is a list of PEM-encoded CRL lists for `gnutls-x509pki'.
+
+:keylist is an alist of PEM-encoded key files and PEM-encoded
+certificates for `gnutls-x509pki'.
 
 :callbacks is an alist of callback functions, see below.
 
@@ -471,7 +475,8 @@
   /* Placeholders for the property list elements.  */
   Lisp_Object priority_string;
   Lisp_Object trustfiles;
-  Lisp_Object keyfiles;
+  Lisp_Object crlfiles;
+  Lisp_Object keylist;
   /* Lisp_Object callbacks; */
   Lisp_Object loglevel;
   Lisp_Object hostname;
@@ -486,7 +491,8 @@
   hostname              = Fplist_get (proplist, Qgnutls_bootprop_hostname);
   priority_string       = Fplist_get (proplist, Qgnutls_bootprop_priority);
   trustfiles            = Fplist_get (proplist, Qgnutls_bootprop_trustfiles);
-  keyfiles              = Fplist_get (proplist, Qgnutls_bootprop_keyfiles);
+  keylist               = Fplist_get (proplist, Qgnutls_bootprop_keylist);
+  crlfiles              = Fplist_get (proplist, Qgnutls_bootprop_crlfiles);
   /* callbacks          = Fplist_get (proplist, Qgnutls_bootprop_callbacks); */
   loglevel              = Fplist_get (proplist, Qgnutls_bootprop_loglevel);
   verify_flags          = Fplist_get (proplist, Qgnutls_bootprop_verify_flags);
@@ -614,15 +620,41 @@
             }
         }
 
-      for (tail = keyfiles; !NILP (tail); tail = Fcdr (tail))
-	{
-	  Lisp_Object keyfile = Fcar (tail);
-          if (STRINGP (keyfile))
-            {
-              GNUTLS_LOG2 (1, max_log_level, "setting the keyfile: ",
+      for (tail = crlfiles; !NILP (tail); tail = Fcdr (tail))
+	{
+	  Lisp_Object crlfile = Fcar (tail);
+          if (STRINGP (crlfile))
+            {
+              GNUTLS_LOG2 (1, max_log_level, "setting the CRL file: ",
+                           SSDATA (crlfile));
+              ret = gnutls_certificate_set_x509_crl_file
+                (x509_cred,
+                 SSDATA (crlfile),
+                 file_format);
+
+              if (ret < GNUTLS_E_SUCCESS)
+                return gnutls_make_error (ret);
+            }
+          else
+            {
+              error ("Sorry, GnuTLS can't use non-string CRL file %s",
+                     SDATA (crlfile));
+            }
+        }
+
+      for (tail = keylist; !NILP (tail); tail = Fcdr (tail))
+	{
+	  Lisp_Object keyfile = Fcar (Fcar (tail));
+	  Lisp_Object certfile = Fcar (Fcdr (tail));
+          if (STRINGP (keyfile) && STRINGP (certfile))
+            {
+              GNUTLS_LOG2 (1, max_log_level, "setting the client key file: ",
                            SSDATA (keyfile));
-              ret = gnutls_certificate_set_x509_crl_file
+              GNUTLS_LOG2 (1, max_log_level, "setting the client cert file: ",
+                           SSDATA (certfile));
+              ret = gnutls_certificate_set_x509_key_file
                 (x509_cred,
+                 SSDATA (certfile),
                  SSDATA (keyfile),
                  file_format);
 
@@ -631,8 +663,12 @@
             }
           else
             {
-              error ("Sorry, GnuTLS can't use non-string keyfile %s",
-                     SDATA (keyfile));
+              if (STRINGP (keyfile))
+                error ("Sorry, GnuTLS can't use non-string client cert file %s",
+                       SDATA (certfile));
+              else
+                error ("Sorry, GnuTLS can't use non-string client key file %s",
+                       SDATA (keyfile));
             }
         }
     }
@@ -868,8 +904,11 @@
   Qgnutls_bootprop_trustfiles = intern_c_string (":trustfiles");
   staticpro (&Qgnutls_bootprop_trustfiles);
 
-  Qgnutls_bootprop_keyfiles = intern_c_string (":keyfiles");
-  staticpro (&Qgnutls_bootprop_keyfiles);
+  Qgnutls_bootprop_keylist = intern_c_string (":keylist");
+  staticpro (&Qgnutls_bootprop_keylist);
+
+  Qgnutls_bootprop_crlfiles = intern_c_string (":crlfiles");
+  staticpro (&Qgnutls_bootprop_crlfiles);
 
   Qgnutls_bootprop_callbacks = intern_c_string (":callbacks");
   staticpro (&Qgnutls_bootprop_callbacks);

Re: client certs and CRL lists for GnuTLS

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> The attached patch adds a :keylist parameter to `gnutls-boot' which is a
> list of (client key file, client cert file) pairs.  It also renames the
> :keyfiles parameter to :crlfiles since it's for CRL lists.  So now you
> can specify any number of client certs.  If the key files require a
> passphrase, the decoding won't work because we don't set a callback.

Right.  Hm...  if you specify a keyfile (that requires a password), does
starttls.el allow prompting for that password?  (I'm just wondering
whether the gnutls.c situation would be totally equivalent or not...)

> `gnutls-negotiate' also gets the parameter changes (should I just make
> it take a plist?)

[...]

>  (defun gnutls-negotiate (proc type hostname &optional priority-string
> -                              trustfiles keyfiles verify-flags
> +                              trustfiles crlfiles keylist verify-flags
>                                verify-error verify-hostname-error)

Heh.  Yes, I think it would be better to change this to a plist.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: client certs and CRL lists for GnuTLS

[Ted Zlatanov]

[-- Attachment #1: Type: text/plain, Size: 1191 bytes --]

On Tue, 03 May 2011 17:25:44 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Ted Zlatanov <tzz@lifelogs.com> writes:
>> The attached patch adds a :keylist parameter to `gnutls-boot' which is a
>> list of (client key file, client cert file) pairs.  It also renames the
>> :keyfiles parameter to :crlfiles since it's for CRL lists.  So now you
>> can specify any number of client certs.  If the key files require a
>> passphrase, the decoding won't work because we don't set a callback.

LMI> Right.  Hm...  if you specify a keyfile (that requires a password), does
LMI> starttls.el allow prompting for that password?  (I'm just wondering
LMI> whether the gnutls.c situation would be totally equivalent or not...)

I don't think so.  gnutls.c will eventually allow it if people need it.

>> (defun gnutls-negotiate (proc type hostname &optional priority-string
>> -                              trustfiles keyfiles verify-flags
>> +                              trustfiles crlfiles keylist verify-flags
>> verify-error verify-hostname-error)

LMI> Heh.  Yes, I think it would be better to change this to a plist.  :-)

Done, see attached (same patch with the plist change).

Ted


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: second try, with plist for `gnutls-negotiate' params --]
[-- Type: text/x-diff, Size: 8905 bytes --]

=== modified file 'lisp/net/gnutls.el'
--- lisp/net/gnutls.el	2011-04-25 13:47:23 +0000
+++ lisp/net/gnutls.el	2011-05-03 15:41:06 +0000
@@ -35,6 +35,8 @@
 
 ;;; Code:
 
+(eval-when-compile (require 'cl))
+
 (defgroup gnutls nil
   "Emacs interface to the GnuTLS library."
   :prefix "gnutls-"
@@ -72,9 +74,9 @@
 documentation for the specific parameters you can use to open a
 GnuTLS connection, including specifying the credential type,
 trust and key files, and priority string."
-  (gnutls-negotiate (open-network-stream name buffer host service)
-                    'gnutls-x509pki
-                    host))
+  (gnutls-negotiate :process (open-network-stream name buffer host service)
+                    :type 'gnutls-x509pki
+                    :hostname host))
 
 (put 'gnutls-error
      'error-conditions
@@ -85,16 +87,23 @@
 (declare-function gnutls-boot "gnutls.c" (proc type proplist))
 (declare-function gnutls-errorp "gnutls.c" (error))
 
-(defun gnutls-negotiate (proc type hostname &optional priority-string
-                              trustfiles keyfiles verify-flags
-                              verify-error verify-hostname-error)
+(defun* gnutls-negotiate
+    (&rest spec
+           &key process type hostname priority-string
+           trustfiles crlfiles keylist verify-flags
+           verify-error verify-hostname-error
+           &allow-other-keys)
   "Negotiate a SSL/TLS connection.  Returns proc. Signals gnutls-error.
+
+Note arguments are passed CL style, :type TYPE instead of just TYPE.
+
 TYPE is `gnutls-x509pki' (default) or `gnutls-anon'.  Use nil for the default.
-PROC is a process returned by `open-network-stream'.
+PROCESS is a process returned by `open-network-stream'.
 HOSTNAME is the remote hostname.  It must be a valid string.
 PRIORITY-STRING is as per the GnuTLS docs, default is \"NORMAL\".
 TRUSTFILES is a list of CA bundles.
-KEYFILES is a list of client keys.
+CRLFILES is a list of CRL files.
+KEYLIST is an alist of (client key file, client cert file) pairs.
 
 When VERIFY-HOSTNAME-ERROR is not nil, an error will be raised
 when the hostname does not match the presented certificate's host
@@ -141,7 +150,8 @@
                              :hostname ,hostname
                              :loglevel ,gnutls-log-level
                              :trustfiles ,trustfiles
-                             :keyfiles ,keyfiles
+                             :crlfiles ,crlfiles
+                             :keylist ,keylist
                              :verify-flags ,verify-flags
                              :verify-error ,verify-error
                              :verify-hostname-error ,verify-hostname-error
@@ -149,14 +159,14 @@
          ret)
 
     (gnutls-message-maybe
-     (setq ret (gnutls-boot proc type params))
+     (setq ret (gnutls-boot process type params))
      "boot: %s" params)
 
     (when (gnutls-errorp ret)
       ;; This is a error from the underlying C code.
-      (signal 'gnutls-error (list proc ret)))
+      (signal 'gnutls-error (list process ret)))
 
-    proc))
+    process))
 
 (declare-function gnutls-error-string "gnutls.c" (error))
 

=== modified file 'lisp/net/network-stream.el'
--- lisp/net/network-stream.el	2011-05-01 15:39:10 +0000
+++ lisp/net/network-stream.el	2011-05-03 15:41:58 +0000
@@ -45,9 +45,7 @@
 (require 'tls)
 (require 'starttls)
 
-(declare-function gnutls-negotiate "gnutls"
-                  (proc type host &optional priority-string trustfiles keyfiles
-                        verify-flags verify-error verify-hostname-error))
+(declare-function gnutls-negotiate "gnutls" (&rest spec))
 
 ;;;###autoload
 (defun open-network-stream (name buffer host service &rest parameters)
@@ -203,7 +201,7 @@
 			  (network-stream-command stream starttls-command eoc))
 	;; The server said it was OK to begin STARTTLS negotiations.
 	(if (fboundp 'open-gnutls-stream)
-	    (gnutls-negotiate stream nil host)
+	    (gnutls-negotiate :process stream :hostname host)
 	  (unless (starttls-negotiate stream)
 	    (delete-process stream)))
 	(if (memq (process-status stream) '(open run))

=== modified file 'src/gnutls.c'
--- src/gnutls.c	2011-05-02 02:49:06 +0000
+++ src/gnutls.c	2011-05-03 15:11:35 +0000
@@ -44,7 +44,8 @@
 /* The following are for the property list of `gnutls-boot'.  */
 static Lisp_Object Qgnutls_bootprop_priority;
 static Lisp_Object Qgnutls_bootprop_trustfiles;
-static Lisp_Object Qgnutls_bootprop_keyfiles;
+static Lisp_Object Qgnutls_bootprop_keylist;
+static Lisp_Object Qgnutls_bootprop_crlfiles;
 static Lisp_Object Qgnutls_bootprop_callbacks;
 static Lisp_Object Qgnutls_bootprop_loglevel;
 static Lisp_Object Qgnutls_bootprop_hostname;
@@ -412,7 +413,10 @@
 
 :trustfiles is a list of PEM-encoded trust files for `gnutls-x509pki'.
 
-:keyfiles is a list of PEM-encoded key files for `gnutls-x509pki'.
+:crlfiles is a list of PEM-encoded CRL lists for `gnutls-x509pki'.
+
+:keylist is an alist of PEM-encoded key files and PEM-encoded
+certificates for `gnutls-x509pki'.
 
 :callbacks is an alist of callback functions, see below.
 
@@ -471,7 +475,8 @@
   /* Placeholders for the property list elements.  */
   Lisp_Object priority_string;
   Lisp_Object trustfiles;
-  Lisp_Object keyfiles;
+  Lisp_Object crlfiles;
+  Lisp_Object keylist;
   /* Lisp_Object callbacks; */
   Lisp_Object loglevel;
   Lisp_Object hostname;
@@ -486,7 +491,8 @@
   hostname              = Fplist_get (proplist, Qgnutls_bootprop_hostname);
   priority_string       = Fplist_get (proplist, Qgnutls_bootprop_priority);
   trustfiles            = Fplist_get (proplist, Qgnutls_bootprop_trustfiles);
-  keyfiles              = Fplist_get (proplist, Qgnutls_bootprop_keyfiles);
+  keylist               = Fplist_get (proplist, Qgnutls_bootprop_keylist);
+  crlfiles              = Fplist_get (proplist, Qgnutls_bootprop_crlfiles);
   /* callbacks          = Fplist_get (proplist, Qgnutls_bootprop_callbacks); */
   loglevel              = Fplist_get (proplist, Qgnutls_bootprop_loglevel);
   verify_flags          = Fplist_get (proplist, Qgnutls_bootprop_verify_flags);
@@ -614,15 +620,41 @@
             }
         }
 
-      for (tail = keyfiles; !NILP (tail); tail = Fcdr (tail))
-	{
-	  Lisp_Object keyfile = Fcar (tail);
-          if (STRINGP (keyfile))
-            {
-              GNUTLS_LOG2 (1, max_log_level, "setting the keyfile: ",
+      for (tail = crlfiles; !NILP (tail); tail = Fcdr (tail))
+	{
+	  Lisp_Object crlfile = Fcar (tail);
+          if (STRINGP (crlfile))
+            {
+              GNUTLS_LOG2 (1, max_log_level, "setting the CRL file: ",
+                           SSDATA (crlfile));
+              ret = gnutls_certificate_set_x509_crl_file
+                (x509_cred,
+                 SSDATA (crlfile),
+                 file_format);
+
+              if (ret < GNUTLS_E_SUCCESS)
+                return gnutls_make_error (ret);
+            }
+          else
+            {
+              error ("Sorry, GnuTLS can't use non-string CRL file %s",
+                     SDATA (crlfile));
+            }
+        }
+
+      for (tail = keylist; !NILP (tail); tail = Fcdr (tail))
+	{
+	  Lisp_Object keyfile = Fcar (Fcar (tail));
+	  Lisp_Object certfile = Fcar (Fcdr (tail));
+          if (STRINGP (keyfile) && STRINGP (certfile))
+            {
+              GNUTLS_LOG2 (1, max_log_level, "setting the client key file: ",
                            SSDATA (keyfile));
-              ret = gnutls_certificate_set_x509_crl_file
+              GNUTLS_LOG2 (1, max_log_level, "setting the client cert file: ",
+                           SSDATA (certfile));
+              ret = gnutls_certificate_set_x509_key_file
                 (x509_cred,
+                 SSDATA (certfile),
                  SSDATA (keyfile),
                  file_format);
 
@@ -631,8 +663,12 @@
             }
           else
             {
-              error ("Sorry, GnuTLS can't use non-string keyfile %s",
-                     SDATA (keyfile));
+              if (STRINGP (keyfile))
+                error ("Sorry, GnuTLS can't use non-string client cert file %s",
+                       SDATA (certfile));
+              else
+                error ("Sorry, GnuTLS can't use non-string client key file %s",
+                       SDATA (keyfile));
             }
         }
     }
@@ -868,8 +904,11 @@
   Qgnutls_bootprop_trustfiles = intern_c_string (":trustfiles");
   staticpro (&Qgnutls_bootprop_trustfiles);
 
-  Qgnutls_bootprop_keyfiles = intern_c_string (":keyfiles");
-  staticpro (&Qgnutls_bootprop_keyfiles);
+  Qgnutls_bootprop_keylist = intern_c_string (":keylist");
+  staticpro (&Qgnutls_bootprop_keylist);
+
+  Qgnutls_bootprop_crlfiles = intern_c_string (":crlfiles");
+  staticpro (&Qgnutls_bootprop_crlfiles);
 
   Qgnutls_bootprop_callbacks = intern_c_string (":callbacks");
   staticpro (&Qgnutls_bootprop_callbacks);

Re: client certs and CRL lists for GnuTLS

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> Done, see attached (same patch with the plist change).

Looks good to me.

I'm wondering how to test this, though.  Although I could just convert it
blindly and hope that when it fails that someone will speak up.  :-)

The smtpmail.el conversion to opportunistic STARTTLS will require a bit
more violence to the smtpmail.el code than the pop3.el conversion, but I
hope to get to it this weekend.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: client certs and CRL lists for GnuTLS

[Ted Zlatanov]

On Tue, 03 May 2011 23:54:14 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Ted Zlatanov <tzz@lifelogs.com> writes:
>> Done, see attached (same patch with the plist change).

LMI> I'm wondering how to test this, though.  Although I could just convert it
LMI> blindly and hope that when it fails that someone will speak up.
LMI> :-)

Write some ERT tests.  They've been great for the *registry.el work.

LMI> The smtpmail.el conversion to opportunistic STARTTLS will require a bit
LMI> more violence to the smtpmail.el code than the pop3.el conversion, but I
LMI> hope to get to it this weekend.

No rush for me, though I appreciate it very much.

I'll push my patch since it's mostly trivial.

Ted

Re: client certs and CRL lists for GnuTLS

[Chong Yidong]

Ted Zlatanov <tzz@lifelogs.com> writes:

> LMI> I'm wondering how to test this, though.  Although I could just
> LMI> convert it blindly and hope that when it fails that someone will
> LMI> speak up.  :-)
>
> Write some ERT tests.  They've been great for the *registry.el work.

If you have ERT tests in the Gnus repository, could you arrage for these
files to be placed into (or mirrored into) the Emacs repository's
test/automated directory?  Then they'll be run by "make check", and will
show up in the Hydra coverage build.

Re: client certs and CRL lists for GnuTLS

[Ted Zlatanov]

On Sun, 08 May 2011 16:59:35 -0400 Chong Yidong <cyd@stupidchicken.com> wrote: 

CY> Ted Zlatanov <tzz@lifelogs.com> writes:
LMI> I'm wondering how to test this, though.  Although I could just
LMI> convert it blindly and hope that when it fails that someone will
LMI> speak up.  :-)
>> 
>> Write some ERT tests.  They've been great for the *registry.el work.

CY> If you have ERT tests in the Gnus repository, could you arrage for these
CY> files to be placed into (or mirrored into) the Emacs repository's
CY> test/automated directory?  Then they'll be run by "make check", and will
CY> show up in the Hydra coverage build.

They are embedded in the gnus/*registry.el files and I'd rather not
extract them into separate files.  Could that work with the Emacs ERT tests?

Ted

Re: client certs and CRL lists for GnuTLS

[Chong Yidong]

Ted Zlatanov <tzz@lifelogs.com> writes:

> CY> If you have ERT tests in the Gnus repository, could you arrage for
> CY> these files to be placed into (or mirrored into) the Emacs
> CY> repository's test/automated directory?  Then they'll be run by
> CY> "make check", and will show up in the Hydra coverage build.
>
> They are embedded in the gnus/*registry.el files and I'd rather not
> extract them into separate files.  Could that work with the Emacs ERT
> tests?

What's the rationale for not putting tests in separate files?  If it has
something to do with the way Gnus is developed, I can live with that,
though tests for other part of Emacs should go into test/, instead of
following this example.

As for Gnus, please add a file test/automated/gnus-tests.el, and have it
(require 'gnus-registry) and any other Gnus files that contain ERT
tests.  Then `make check' will be able to find those tests.

Gnus ERT tests inside Emacs (was: client certs and CRL lists for GnuTLS)

[Ted Zlatanov]

On Mon, 09 May 2011 11:00:12 -0400 Chong Yidong <cyd@stupidchicken.com> wrote: 

CY> Ted Zlatanov <tzz@lifelogs.com> writes:
CY> If you have ERT tests in the Gnus repository, could you arrage for
CY> these files to be placed into (or mirrored into) the Emacs
CY> repository's test/automated directory?  Then they'll be run by
CY> "make check", and will show up in the Hydra coverage build.
>> 
>> They are embedded in the gnus/*registry.el files and I'd rather not
>> extract them into separate files.  Could that work with the Emacs ERT
>> tests?

CY> What's the rationale for not putting tests in separate files?  If it has
CY> something to do with the way Gnus is developed, I can live with that,
CY> though tests for other part of Emacs should go into test/, instead of
CY> following this example.

It's more convenient IMO.  The ERT manual didn't specify tests had to be
separate, either.  For Gnus we have my tests internalized and some other
tests external...  I hope we're not forced to use just one style if
there's no technical reason for it.

CY> As for Gnus, please add a file test/automated/gnus-tests.el, and have it
CY> (require 'gnus-registry) and any other Gnus files that contain ERT
CY> tests.  Then `make check' will be able to find those tests.

I'll do mine but am not sure if the other Gnus tests (which David
Engster wrote IIRC) will work with the Emacs ERT run.  David, WDYT?  Do
you have anything that only works inside Gnus?

Ted

Re: Gnus ERT tests inside Emacs

[David Engster]

Ted Zlatanov writes:
> On Mon, 09 May 2011 11:00:12 -0400 Chong Yidong <cyd@stupidchicken.com> wrote: 
> CY> As for Gnus, please add a file test/automated/gnus-tests.el, and have it
> CY> (require 'gnus-registry) and any other Gnus files that contain ERT
> CY> tests.  Then `make check' will be able to find those tests.
>
> I'll do mine but am not sure if the other Gnus tests (which David
> Engster wrote IIRC) will work with the Emacs ERT run.  David, WDYT?  Do
> you have anything that only works inside Gnus?

No, but the NNTP test does things the Hydra guys probably aren't too
happy with (opening connection to Gmane, creation of files in /tmp), so
I'd suggest to leave it out.

-David

Re: Gnus ERT tests inside Emacs

[Ted Zlatanov]

On Mon, 09 May 2011 17:46:06 +0200 David Engster <deng@randomsample.de> wrote: 

DE> Ted Zlatanov writes:
>> On Mon, 09 May 2011 11:00:12 -0400 Chong Yidong <cyd@stupidchicken.com> wrote: 
CY> As for Gnus, please add a file test/automated/gnus-tests.el, and have it
CY> (require 'gnus-registry) and any other Gnus files that contain ERT
CY> tests.  Then `make check' will be able to find those tests.
>> 
>> I'll do mine but am not sure if the other Gnus tests (which David
>> Engster wrote IIRC) will work with the Emacs ERT run.  David, WDYT?  Do
>> you have anything that only works inside Gnus?

DE> No, but the NNTP test does things the Hydra guys probably aren't too
DE> happy with (opening connection to Gmane, creation of files in /tmp), so
DE> I'd suggest to leave it out.

OK, I'll just leave mine (*registry.el) in.

Ted

Re: Gnus ERT tests inside Emacs

[Ted Zlatanov]

On Mon, 09 May 2011 10:58:07 -0500 Ted Zlatanov <tzz@lifelogs.com> wrote: 

TZ> On Mon, 09 May 2011 17:46:06 +0200 David Engster <deng@randomsample.de> wrote: 
DE> Ted Zlatanov writes:
>>> On Mon, 09 May 2011 11:00:12 -0400 Chong Yidong <cyd@stupidchicken.com> wrote: 
CY> As for Gnus, please add a file test/automated/gnus-tests.el, and have it
CY> (require 'gnus-registry) and any other Gnus files that contain ERT
CY> tests.  Then `make check' will be able to find those tests.

TZ> OK, I'll just leave mine (*registry.el) in.

I think I did this correctly.  I removed a bad test in registry.el on
the Gnus side but until Katsumi Yamaoka synchronizes Gnus with Emacs,
that one test will fail.

Ted

Re: Emacs RPC security

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 370 bytes --]

On Mon, May 02 2011, Lars Magne Ingebrigtsen wrote:

> Hm...  perhaps I should convert smtpmail.el to use opportunistic
> STARTTLS while I'm at it.  Is that the only (major) network library that
> has escaped opportunist encryption so far?

In Gnus area, there's sieve, in Emacs, there might be erc and rcirc.

-- 
Julien Danjou
❱ http://julien.danjou.info

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: Emacs RPC security

[Ted Zlatanov]

On Mon, 02 May 2011 00:02:47 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
>> Having a GnuTLS server in Emacs would be nice.

LMI> I just had a horrible idea.

LMI> I converted pop3.el to use opportunistic STARTTLS upgrades now (one less
LMI> thing on my imaginary todo list -- only googleplex more to go), and it
LMI> occurred to me that the Emacs Server could use STARTTLS too.

LMI> Today you just send the shared secret and then the command, but we could
LMI> easily implement a CAPABILITY command, and offer STARTTLS and thereby
LMI> allow forward-and-backward compatibility between encrypted and
LMI> non-encrypted clients and servers.  :-)

Regardless of STARTTLS support (which would be good), a CAPABILITY
function would be good for this server-eval RPC you're building.

I already mentioned that given GnuTLS, we can associate client-side SSL
certificates with particular functions, so we authenticate on the
certificates and authorize based on the (certificate, function)
combination.  This seems to me much better, even if "orthogonal," than
the current "come visit my server and run anything you like" approach.

Ted

Re: Emacs RPC security

[Stefan Monnier]

> I already mentioned that given GnuTLS, we can associate client-side SSL
> certificates with particular functions, so we authenticate on the
> certificates and authorize based on the (certificate, function)
> combination.  This seems to me much better, even if "orthogonal," than
> the current "come visit my server and run anything you like" approach.

I think this is pushing server.el where it shouldn't go.  It's not meant
as "Emacs as a server for whichever network service you can think of",
but just "use your own Emacs from other processes".  If you want your
Emacs to offer services to various users (rather than just to yourself),
then you'll want to implement your own (probably based on GNUtls).


        Stefan

Re: Emacs RPC security

[Ted Zlatanov]

On Mon, 02 May 2011 16:48:17 -0300 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

>> I already mentioned that given GnuTLS, we can associate client-side SSL
>> certificates with particular functions, so we authenticate on the
>> certificates and authorize based on the (certificate, function)
>> combination.  This seems to me much better, even if "orthogonal," than
>> the current "come visit my server and run anything you like" approach.

SM> I think this is pushing server.el where it shouldn't go.  It's not meant
SM> as "Emacs as a server for whichever network service you can think of",
SM> but just "use your own Emacs from other processes".  If you want your
SM> Emacs to offer services to various users (rather than just to yourself),
SM> then you'll want to implement your own (probably based on GNUtls).

I'm saying the problem is that server.el doesn't know if you're offering
services just to yourself or to others as well, so you can't say it's OK
to be less secure for personal use.  Knowledge of the shared key is
sufficient.  Plus there is no authorization granularity so the shared
key grants full access.  Am I missing or misunderstanding something?

Ted

Re: Emacs RPC security

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> Knowledge of the shared key is sufficient.  Plus there is no
> authorization granularity so the shared key grants full access.  Am I
> missing or misunderstanding something?

Presumably you're not sharing the secret with anybody, so I think the
mechanism is safe enough for what it's used for.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Emacs RPC security

[Ted Zlatanov]

On Tue, 03 May 2011 00:56:42 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Ted Zlatanov <tzz@lifelogs.com> writes:
>> Knowledge of the shared key is sufficient.  Plus there is no
>> authorization granularity so the shared key grants full access.  Am I
>> missing or misunderstanding something?

LMI> Presumably you're not sharing the secret with anybody, so I think the
LMI> mechanism is safe enough for what it's used for.

Is the mechanism used for evaluating code remotely, or only locally?
In other words, can it be accessed over the network?

Ted

Re: Emacs RPC security

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> Is the mechanism used for evaluating code remotely, or only locally?
> In other words, can it be accessed over the network?

It's for evaluating code remotely over the network.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Emacs RPC security

[Ted Zlatanov]

On Tue, 03 May 2011 02:51:28 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Ted Zlatanov <tzz@lifelogs.com> writes:
>> Is the mechanism used for evaluating code remotely, or only locally?
>> In other words, can it be accessed over the network?

LMI> It's for evaluating code remotely over the network.

On Mon, 02 May 2011 21:35:46 -0300 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

>> I'm saying the problem is that server.el doesn't know if you're offering
>> services just to yourself or to others as well, so you can't say it's OK
>> to be less secure for personal use.

SM> server.el offers full service only.

Yes, I know!  I think it should at *least* have the option to limit the
access at the entry point when the code is eval-ed.  In Common Lisp you
can disable many of the Lisp reader's options that evaluate code, but I
don't know how the Emacs Lisp reader can do that.

SM> If you give access to it to someone else than yourself, it's your
SM> mistake, not server.el.

As I keep trying to explain, you don't know who is on the other end
because there is *no* authentication, or rather it's binary: you have
the shared secret or you don't.

At least let's associate a shared secret with an access level, so we do
not allow full access all the time.  The access level can be a list of
function symbols that can be called, for instance.

Ted

Re: Emacs RPC security

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> As I keep trying to explain, you don't know who is on the other end
> because there is *no* authentication, or rather it's binary: you have
> the shared secret or you don't.

Yes, it's like ssh + ssh-agent.  (Only without the encryption, of
course.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Emacs RPC security

[Ted Zlatanov]

On Tue, 03 May 2011 03:16:21 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Ted Zlatanov <tzz@lifelogs.com> writes:
>> As I keep trying to explain, you don't know who is on the other end
>> because there is *no* authentication, or rather it's binary: you have
>> the shared secret or you don't.

LMI> Yes, it's like ssh + ssh-agent.  (Only without the encryption, of
LMI> course.  :-)

ssh+ssh-agent has a user name and can authenticate the host keys.  It
has many other feature server.el doesn't, so it's like only root SSH
access was ever allowed.  Most importantly, it has PPK authentication so
there is no shared secret passed around unless the server allows
password authentication.

Ted

Re: Emacs RPC security

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> ssh+ssh-agent has a user name and can authenticate the host keys.

Emacs Server has a secret it authenticate, which is pretty much the same
thing. 

> It has many other feature server.el doesn't, so it's like only root
> SSH access was ever allowed.  Most importantly, it has PPK
> authentication so there is no shared secret passed around unless the
> server allows password authentication.

Well, if the Emacs Server connection was encrypted, it'd be rather
similar.  You need access to the agent to subvert ssh+ssh-agent, and you
need access to user-read-only files to access the Emacs Server.  Not
much difference (in principle), except for the transport layer security.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Emacs RPC security

[Stefan Monnier]

>> As I keep trying to explain, you don't know who is on the other end
>> because there is *no* authentication, or rather it's binary: you have
>> the shared secret or you don't.

> Yes, it's like ssh + ssh-agent.  (Only without the encryption, of
> course.  :-)

Exactly.


        Stefan

Re: Emacs RPC security

[Harald Hanche-Olsen]

[Ted Zlatanov <tzz@lifelogs.com> (2011-05-03 01:12:28 UTC)]

> In Common Lisp you can disable many of the Lisp reader's options
> that evaluate code, but I don't know how the Emacs Lisp reader can
> do that.

Perhaps you are thinking of CL's #. syntax, which causes the following
form to be evaluated at read time. There is a switch to turn off this
feature. AFAIK this is not available in elisp, so there is nothing to
disable.

The natural way (and only way, I believe) to provide a restricted
execution environment in Common Lisp is to leave the user in a package
that contains only "safe" symbols. As elisp does not have packages in
the CL sense, the only way I can think of would be to not use the
elisp reader at all, and provide you own – or else go through the read
form and excise forbidden symbols. This seems like a nontrivial (and
somewhat pointless?) endeavour.

- Harald

Re: Emacs RPC security

[Stefan Monnier]

> The natural way (and only way, I believe) to provide a restricted
> execution environment in Common Lisp is to leave the user in a package
> that contains only "safe" symbols. As elisp does not have packages in
> the CL sense, the only way I can think of would be to not use the
> elisp reader at all, and provide you own – or else go through the read
> form and excise forbidden symbols. This seems like a nontrivial (and
> somewhat pointless?) endeavour.

Indeed.  The way to do it in Elisp is to not use `eval' but instead
implement some other protocol/language.  Which is why it's outside the
scope of server.el.


        Stefan

Re: Emacs RPC security

[Stefan Monnier]

> I'm saying the problem is that server.el doesn't know if you're offering
> services just to yourself or to others as well, so you can't say it's OK
> to be less secure for personal use.

server.el offers full service only.  If you give access to it to someone
else than yourself, it's your mistake, not server.el.


        Stefan

Re: Emacs RPC

[Sebastian Rose]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
> I'm running a lot of Emacs-based servers here and there, and I
> communicate with them via "emacsclient --eval".  But it just occurred to
> me that it would be much more elegant to have an `emacs-client-rpc'
> function, so that I wouldn't have to parse the output from emacsclient
> in a shell.
>
> I've peeked through the info files and I couldn't find anything.  And it
> looks like it would be a simple enough thing to add to server.el.
> Anybody mind if I add a function like, er,
>
> (server-eval-at HOST COMMAND)
>
> to server.el?


There is org-protocol (still uses emacsclient though).  Don't know if
that's sufficient for your use case.

See:
http://orgmode.org/manual/Protocols.html#Protocols
http://orgmode.org/worg/org-contrib/org-protocol.html
http://orgmode.org/worg/org-tutorials/org-protocol-custom-handler.html
http://www.youtube.com/watch?v=h7Z2PiAcgh8
http://www.youtube.com/watch?v=G2xjwxEj-c8


A custom handler is a function you may call through emacsclient.  It is
triggered by an advice around `server-visit-files'.

It's used widely already, but I'd be glad to rewrite it to not use
emacsclient anymore.



    Sebastian


-- 

Nachdem ich festgestellt hatte, dass ich Fehler machen kann, ist mir
klar geworden, dass eine Ordnung ist in dem, was ich tue.
                                                     (Ornette Coleman)

Re: Emacs RPC

[Stefan Monnier]

> A custom handler is a function you may call through emacsclient.  It is
> triggered by an advice around `server-visit-files'.

Why not use the --eval arg instead of a defadvice?


        Stefan