Modifying Emacs to use the Mac OS X Keychain Services

Modifying Emacs to use the Mac OS X Keychain Services

[Ben Key]

[-- Attachment #1: Type: text/plain, Size: 705 bytes --]

Hello,

I have been looking into what it will take to add support for the Mac OS X
Keychain Services to Emacs.  It looks like what I need to do is to make
changes to lisp/net/secrets.el.  Specifically I need to modify the functions
secrets-create-item, secrets-delete-item, secrets-get-alias,
secrets-get-attributes, secrets-get-secret, secrets-list-collections, and
secrets-search-items so that on Mac OS X they call new built in functions
that use the Mac OS X Keychain Services instead of using dbus-call-method to
communicate with the "org.freedesktop.secrets" service.

Is that an accurate assessment of what needs to be done?  If it is, just let
me know and I will begin working on it this weekend.

[-- Attachment #2: Type: text/html, Size: 784 bytes --]

Re: Modifying Emacs to use the Mac OS X Keychain Services

[Michael Albinus]

Ben Key <bkey76@gmail.com> writes:

> Hello,

Hi,

> OS X Keychain Services to Emacs.  It looks like what I need to do is
> to make changes to lisp/net/secrets.el.  Specifically I need to modify
> the functions secrets-create-item, secrets-delete-item,
> secrets-get-alias, secrets-get-attributes, secrets-get-secret,
> secrets-list-collections, and secrets-search-items so that on Mac OS X
> they call new built in functions that use the Mac OS X Keychain
> Services instead of using dbus-call-method to communicate with the
> "org.freedesktop.secrets" service.

secrets.el was initiated as implementation of D-Bus'
org.freedesktop.secrets interface. Does there exist a a server under OS
X for that? I know the Gnome (keyring) and KDE (kwallet) implementations
so far, a short search does not return results for OS X.

The functions in secrets.el are designed to give simple access to that
interface.

> Is that an accurate assessment of what needs to be done?  If it is,
> just let me know and I will begin working on it this weekend.

Could be. But I believe a more clean solution would be to implement an
own package for OS X Keychain Services, which will be integrated into
auth-source then. Other Lisp packages will likely call auth-source
functions, instead those of secrets.el.

Best regards, Michael.

Re: Modifying Emacs to use the Mac OS X Keychain Services

[Ben Key]

[-- Attachment #1: Type: text/plain, Size: 2019 bytes --]

On Sat, May 28, 2011 at 6:09 AM, Michael Albinus <michael.albinus@gmx.de>wrote:

secrets.el was initiated as implementation of D-Bus'
> org.freedesktop.secrets interface. Does there exist a a server under OS
> X for that? I know the Gnome (keyring) and KDE (kwallet) implementations
> so far, a short search does not return results for OS X.
>
> The functions in secrets.el are designed to give simple access to that
> interface.
>
>
First, to give a little background.  Several months ago there was a request
to have auth-source.el use the Mac OS X keychain functions (see
http://lists.gnu.org/archive/html/emacs-devel/2011-03/msg00486.html).  These
functions are documented at
http://developer.apple.com/library/mac/#documentation/Security/Conceptual/keychainServConcepts/03tasks/tasks.html.
I agreed to do it but then got busy and later forgot about it.  I want to
finally keep my promise this weekend.

When I started looking into what would be needed to complete this task the
first thing I noticed is that auth-source.el makes use of the functions in
secrets.el I mentioned if the user customizes the auth-sources variable to
include the secrets API.  Since the Keychain Services API provides
functionality similar to that provided by the org.freedesktop.secrets
interface I figured that secrets.el would be a logical place to put the
change.  Putting the code there also makes it available to packages other
than Gnus.  Of course I could just define another authentication source in
auth-source.el as you suggest.

I thought that extending secrets.el was the more logical choice since much
like the org.freedesktop.secrets interface, the Keychain Services API
provides a Login key chain along with the possibility of using an
application defined keychain.

I am willing to be over ruled though.

Which approach should I take?  Whichever approach I take, I may need some
help writing the Lisp code necessary for my change.  While I know enough
Lisp to perform simple tasks, I am definitely NOT a Lisp programmer.

[-- Attachment #2: Type: text/html, Size: 2627 bytes --]

Re: Modifying Emacs to use the Mac OS X Keychain Services

[Michael Albinus]

Ben Key <bkey76@gmail.com> writes:

Hi Ben,

> When I started looking into what would be needed to complete this task
> the first thing I noticed is that auth-source.el makes use of the
> functions in secrets.el I mentioned if the user customizes the
> auth-sources variable to include the secrets API.  Since the Keychain
> Services API provides functionality similar to that provided by the
> org.freedesktop.secrets interface I figured that secrets.el would be a
> logical place to put the change.  Putting the code there also makes it
> available to packages other than Gnus.  Of course I could just define
> another authentication source in auth-source.el as you suggest.

auth-source.el is not used by Gnus only. For example, Tramp uses it as
well instead of secrets.el, although I'm the maintainer of Tramp and the
author of secrets.el :-)

And the intention is, that auth-source.el shall be *the* authentication
library to be used by other Emacs packages.

> I thought that extending secrets.el was the more logical choice since
> much like the org.freedesktop.secrets interface, the Keychain Services
> API provides a Login key chain along with the possibility of using an
> application defined keychain.

I fear a little bit about differences between Secrets Services API and
the Keychain Services API - secrets.el will follow the Secrets Service
API whenever there is a change. And I suspect there is not so much code
of secrets.el for reuse with Keychain Service API.

I'm not using OS X, could you, please, send me a pointer to the Keychain
Services API?

> Which approach should I take?  Whichever approach I take, I may need
> some help writing the Lisp code necessary for my change.  While I know
> enough Lisp to perform simple tasks, I am definitely NOT a Lisp
> programmer.

I'm pretty sure, people will help you. Including myself, although I'm
not as responsible as I would like due to other duties.

Best regards, Michael.

Re: Modifying Emacs to use the Mac OS X Keychain Services

[Ted Zlatanov]

On Sat, 28 May 2011 08:00:20 -0500 Ben Key <bkey76@gmail.com> wrote: 

BK> First, to give a little background.  Several months ago there was a request
BK> to have auth-source.el use the Mac OS X keychain functions (see
BK> http://lists.gnu.org/archive/html/emacs-devel/2011-03/msg00486.html).  These
BK> functions are documented at
BK> http://developer.apple.com/library/mac/#documentation/Security/Conceptual/keychainServConcepts/03tasks/tasks.html.
BK> I agreed to do it but then got busy and later forgot about it.  I want to
BK> finally keep my promise this weekend.

BK> When I started looking into what would be needed to complete this task the
BK> first thing I noticed is that auth-source.el makes use of the functions in
BK> secrets.el I mentioned if the user customizes the auth-sources variable to
BK> include the secrets API.  Since the Keychain Services API provides
BK> functionality similar to that provided by the org.freedesktop.secrets
BK> interface I figured that secrets.el would be a logical place to put the
BK> change.  Putting the code there also makes it available to packages other
BK> than Gnus.  Of course I could just define another authentication source in
BK> auth-source.el as you suggest.

BK> I thought that extending secrets.el was the more logical choice since much
BK> like the org.freedesktop.secrets interface, the Keychain Services API
BK> provides a Login key chain along with the possibility of using an
BK> application defined keychain.

They are superficially similar but internally very different (the
Keychain Services API uses a C API; the Secrets API uses D-BUS and an
entirely different protocol if I understand correctly).  So I think it's
better to supply a ELisp interface to the basic Keychain Services API,
which should be a simpler task than adapting secrets.el to use the
Keychain Services API.  You can try to model your code after the
secrets.el functions (list collections, search for items, etc.) but if
it becomes an impediment, just follow the C API you're adapting.

Ted

Re: Modifying Emacs to use the Mac OS X Keychain Services

[Ben Key]

[-- Attachment #1: Type: text/plain, Size: 2591 bytes --]

Michael Albinus <michael.albinus@gmx.de> wrote:

> I'm not using OS X, could you, please, send me a pointer to the Keychain
> Services API?
>
>
Some information on the Keychain Services API can be found at the Keychain
Services Tasks for Mac OS X page found at
http://developer.apple.com/library/mac/#documentation/Security/Conceptual/keychainServConcepts/03tasks/tasks.html.
More detailed information can be found at the Keychain Services Reference
page found at
http://developer.apple.com/library/mac/#documentation/Security/Reference/keychainservices/Reference/reference.html
.

While the Keychain Services API does not exactly mirror the
org.freedesktop.secrets interface it is possible to implement the same
concepts expressed in secrets.el via the Keychain Services API.  For
example, a ns-secrets-create-item function could be easily defined that has
the same parameters as secrets-create-item.  This function would use
SecKeychainOpen to open the keychain specified by the collection parameter,
and if the call to SecKeychainOpen failed, it could use SecKeychainCreate to
create the specified keychain.  It then could use
SecKeychainAddGenericPassword to add the item specified by the item
parameter with the password specified by the password parameter, to the
opened or newly created keychain.  The attributes parameter could be
processed by calling SecKeychainItemModifyAttributesAndData.

The ns-secrets-delete-item, ns-secrets-get-secret, and
ns-secrets-get-attributes functions could also be written similarly.  I am
not certain about the other functions, mainly because I have not yet
finished my research.

The point is, I can implement ns- equivalents in C for most, if not all of
the functions defined in secrets.el with the same parameters and nearly the
same behavior.  I may even be able to make them behave exactly the same as
their secrets.el counterparts.

Once the functions are defined in C, we can later decide how to call them
from Lisp.  We can decide to call them from secrets.el by having each of the
functions use the following psudeo code

(if (fboundp 'ns-{func})
  call ns-{func}
use dbus)

or we could add another auth source to auth-source.el.  Either way would
work for me.

Perhaps I should just focus on writing the ns- equivalents of each of the
secrets.el functions and we can decide later how to call them.  What do you
think?

Note: I am not exactly certain how to parse the attributes parameter of my
proposed ns-secrets-create-item function.  If someone can point me to an
example of how to process something like that in C it would be helpful.

[-- Attachment #2: Type: text/html, Size: 3272 bytes --]

Re: Modifying Emacs to use the Mac OS X Keychain Services

[Ted Zlatanov]

On Sat, 28 May 2011 12:16:53 -0500 Ben Key <bkey76@gmail.com> wrote: 

BK> The point is, I can implement ns- equivalents in C for most, if not all of
BK> the functions defined in secrets.el with the same parameters and nearly the
BK> same behavior.  I may even be able to make them behave exactly the same as
BK> their secrets.el counterparts.

BK> Once the functions are defined in C, we can later decide how to call them
BK> from Lisp.  We can decide to call them from secrets.el by having each of the
BK> functions use the following psudeo code

BK> (if (fboundp 'ns-{func})
BK>   call ns-{func}
BK> use dbus)

BK> or we could add another auth source to auth-source.el.  Either way would
BK> work for me.

It should definitely be another auth source.  auth-source.el could use
the similarities between the Secrets and Keychain APIs to simplify the
code as you suggest.

BK> Perhaps I should just focus on writing the ns- equivalents of each of the
BK> secrets.el functions and we can decide later how to call them.  What do you
BK> think?

Yes, please.

BK> Note: I am not exactly certain how to parse the attributes parameter of my
BK> proposed ns-secrets-create-item function.  If someone can point me to an
BK> example of how to process something like that in C it would be helpful.

Do you mean you want them to look like this (example from secrets.el):

((:user . "joe") (:host ."remote-host"))

You don't have to do it that way, but it's probably easiest to keep your
API similar to secrets.el.  So in C, just go through the list.  Start
with x = the list and do:

(x, cell, key, and val are Lisp_Objects)

while (! NILP (x))
{
 cell = Fcar(x)
 key = Fcar(cell)
 val = Fcdr(cell)
(... use key and val ...)
 x = Fcdr(x)
}

Expect the key to be a symbol and the val to be a string, so you may
want to convert them or throw an error if they are not.  I hope this is
helpful.

Ted

Re: Modifying Emacs to use the Mac OS X Keychain Services

[Ben Key]

[-- Attachment #1: Type: text/plain, Size: 547 bytes --]

Hello,

Thanks Ted for the example on how to implement support with for the
attributes parameter.  That is exactly what I needed.

I will begin working on writing ns- functions that mirror the signatures and
functionality of the secrets.el functions as closely as possible using the
Keychain Services API.  I hope to make some significant progress in this
task by Tuesday.

Once I have added the ns- functions to Emacs, I will most likely need
assistance modifying auth-source.el to use them since I do not know Lisp
nearly as well as I do C/C++.

[-- Attachment #2: Type: text/html, Size: 600 bytes --]

Re: Modifying Emacs to use the Mac OS X Keychain Services

[Michael Albinus]

Ben Key <bkey76@gmail.com> writes:

> Some information on the Keychain Services API can be found at the
> Keychain Services Tasks for Mac OS X page found at
> http://developer.apple.com/library/mac/#documentation/Security/Conceptual/keychainServConcepts/03tasks/tasks.
> html.  More detailed information can be found at the Keychain Services
> Reference page found at
> http://developer.apple.com/library/mac/#documentation/Security/Reference/keychainservices/Reference/reference.
> html.

Thanks, I'll have a look on it.

> Once the functions are defined in C, we can later decide how to call
> them from Lisp.  We can decide to call them from secrets.el by having
> each of the functions use the following psudeo code
>
> (if (fboundp 'ns-{func})
>   call ns-{func}
> use dbus)

What about when there will be a Secrets Services API implementation on
OS X?

> Perhaps I should just focus on writing the ns- equivalents of each of
> the secrets.el functions and we can decide later how to call them. 
> What do you think?

Yes, that might be a starting point.

Re: Modifying Emacs to use the Mac OS X Keychain Services

[Ben Key]

[-- Attachment #1: Type: text/plain, Size: 603 bytes --]

Hello,

I am working on this task.  I need some advice.

The Keychain Services allows you to store items in the Default Keychain, the
Login Keychain (which may be the Default Keychain), or in a separate
Keychain file.  Should I use the Default Keychain or use an Emacs specific
Keychain file?  The benefit of using the Default Keychain is that the user
should never be prompted to provide a password for the Keychain.  However,
keeping the passwords in a separate, Emacs specific, Keychain file would
have the benefit of keeping the passwords used by Emacs separate from other
passwords.

Any opinions?

[-- Attachment #2: Type: text/html, Size: 700 bytes --]

Re: Modifying Emacs to use the Mac OS X Keychain Services

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 817 bytes --]

On 5/29/11 6:08 PM, Ben Key wrote:
> Hello,
> 
> I am working on this task.  I need some advice.
> 
> The Keychain Services allows you to store items in the Default Keychain, the
> Login Keychain (which may be the Default Keychain), or in a separate
> Keychain file.  Should I use the Default Keychain or use an Emacs specific
> Keychain file?  The benefit of using the Default Keychain is that the user
> should never be prompted to provide a password for the Keychain.  However,
> keeping the passwords in a separate, Emacs specific, Keychain file would
> have the benefit of keeping the passwords used by Emacs separate from other
> passwords.

The natural option would be to make the choice a user preference,
setting the default to the default keychain (which is used by most other
programs).


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Modifying Emacs to use the Mac OS X Keychain Services

[Ted Zlatanov]

On Sun, 29 May 2011 20:08:26 -0500 Ben Key <bkey76@gmail.com> wrote: 

BK> The Keychain Services allows you to store items in the Default Keychain, the
BK> Login Keychain (which may be the Default Keychain), or in a separate
BK> Keychain file.  Should I use the Default Keychain or use an Emacs specific
BK> Keychain file?  The benefit of using the Default Keychain is that the user
BK> should never be prompted to provide a password for the Keychain.  However,
BK> keeping the passwords in a separate, Emacs specific, Keychain file would
BK> have the benefit of keeping the passwords used by Emacs separate from other
BK> passwords.

secrets.el understands session aliases, and the "default" collection is
typically an alias (I've seen it set to the "session" or "login"
keychains).  "session" is usually temporary for the login session.  Does
the Mac OS API not allow that?

In any case, I would not make a separate keychain for Emacs, only make
it possible to do so.  Your implementation should integrate in the
native OS as much as possible, so use the Default keychain by default
and give a function to change that.

auth-source specifies secrets.el session aliases like so in the
defcustom:

#+begin_src lisp
(choice :tag "Collection to use"
        (string :tag "Collection name")
        (const :tag "Default" 'default)
        (const :tag "Login" "Login")
        (const
         :tag "Temporary" "session"))
#+end_src

I hope that's helpful.

Ted

Modifying Emacs to use the Mac OS X Keychain Services

[Ben Key]

[-- Attachment #1: Type: text/plain, Size: 1700 bytes --]

Hello,

I am still working on this task.  I did not finish it by Today as I
expected.

So far I am working on the ns-keychain-create-item function and the
ns-keychain-get-secret function.  I will add more functions as I go on.

For those who are wondering, here are some details on how I am implementing
these functions.

The ns-keychain-create-item function has the same parameters as
secrets-create-item (collection, item, password, and &rest attributes).  It
essentially just calls the Keychain Services API functions
SecKeychainAddGenericPassword to create the password and
SecKeychainItemModifyAttributesAndData to process the attributes.  When I
call SecKeychainAddGenericPassword I map the collection parameter to the
serviceName parameter, the item parameter to the accountName, and the
password parameter to the passwordData parameter.

The ns-keychain-get-secret function has the same parameters as
secrets-get-secret (collection and item) and it essentially just calls
SecKeychainFindGenericPassword and returns the retrieved passwordData
parameter.  Again, when calling SecKeychainFindGenericPassword I map the
collection parameter to the serviceName parameter and the item parameter to
the accountName parameter.

Since I can only work on this project on nights and weekends, it may take me
several weeks to complete this task.  As Daniel Colascione and Ted Zlatanov
recommended I am making the Keychain file the passwords used by Emacs a
configurable option that defaults to the default Keychain file.

Once I am finished with this task, I will post a patch to this list so that
the community can review it before I check it in.  It is my hope that this
patch will be ready in 3 weeks.

[-- Attachment #2: Type: text/html, Size: 1971 bytes --]

Re: Modifying Emacs to use the Mac OS X Keychain Services

[Ted Zlatanov]

On Tue, 31 May 2011 21:04:00 -0500 Ben Key <bkey76@gmail.com> wrote: 

BK> Once I am finished with this task, I will post a patch to this list so that
BK> the community can review it before I check it in.  It is my hope that this
BK> patch will be ready in 3 weeks.

Thanks, Ben.  I hope we can get this functionality into the upcoming
Emacs pretest, which should happen if you keep that schedule.  But if
you can't, don't worry :)

You can post a partial patch if you have something that can be examined
by the emacs-devel readers.  I always find such feedback valuable.

Ted

Re: Modifying Emacs to use the Mac OS X Keychain Services

[Ben Key]

[-- Attachment #1: Type: text/plain, Size: 1378 bytes --]

Hello,

I am still working on this task.  I need some more advice though.

As I mentioned before, I am attempting to add ns-keychain functions that
match the secrets.el API as closely as possible.  I am adding a ns-keychain
function for each secrets- function that has the same parameters as the
secrets- function.  So far in my work, I have been mapping the collection
parameter to the serviceName parameter of the various Keychain Services API
functions.  By default, I am using the default Keychain to store the data
but adding a user configurable option to allow another keychain file to be
used.

But, after taking another look at auth-source.el, I am wondering if this was
the correct design decision.  I am wondering if I should instead be
interpreting the collection parameter as an indication of which keychain
file to store the passwords in.  I could interpret the value "default" for
the collection parameter as an indication that the default keychain should
be used.  I could interpret the value "Login" or "secrets:Login" as an
indication that the Login keychain should be used.  Other values for
collection parameter could be interpreted as an indication that the data
should be stored in a file {collection}.keychain in ~/Library/Keychains.

Which interpretation of the collection parameter is the correct one?  I ask
because I want to be sure to get this right.

[-- Attachment #2: Type: text/html, Size: 1491 bytes --]

Re: Modifying Emacs to use the Mac OS X Keychain Services

[Ted Zlatanov]

On Sun, 5 Jun 2011 13:54:20 -0500 Ben Key <bkey76@gmail.com> wrote: 

BK> As I mentioned before, I am attempting to add ns-keychain functions that
BK> match the secrets.el API as closely as possible.  I am adding a ns-keychain
BK> function for each secrets- function that has the same parameters as the
BK> secrets- function.  So far in my work, I have been mapping the collection
BK> parameter to the serviceName parameter of the various Keychain Services API
BK> functions.  By default, I am using the default Keychain to store the data
BK> but adding a user configurable option to allow another keychain file to be
BK> used.

Right, so the name of the collection is the serviceName as far as the OS
is concerned.  That means it could be stored anywhere, right?

BK> But, after taking another look at auth-source.el, I am wondering if this was
BK> the correct design decision.  I am wondering if I should instead be
BK> interpreting the collection parameter as an indication of which keychain
BK> file to store the passwords in.  I could interpret the value "default" for
BK> the collection parameter as an indication that the default keychain should
BK> be used.  I could interpret the value "Login" or "secrets:Login" as an
BK> indication that the Login keychain should be used.  Other values for
BK> collection parameter could be interpreted as an indication that the data
BK> should be stored in a file {collection}.keychain in ~/Library/Keychains.

BK> Which interpretation of the collection parameter is the correct one?  I ask
BK> because I want to be sure to get this right.

Does the user care what file it's stored in?  Can you have any
serviceName in any file, or is there some connection or dependency
between the two?

Ted

Re: Modifying Emacs to use the Mac OS X Keychain Services

[Ben Key]

[-- Attachment #1: Type: text/plain, Size: 2210 bytes --]

Ted Zlatanov wrote:

*> Right, so the name of the collection is the serviceName as far as the OS
> is concerned.  That means it could be stored anywhere, right?
*
Yes.  A keychain item for a given serviceName can be stored anywhere.

*> Does the user care what file it's stored in?  Can you have any
> serviceName in any file, or is there some connection or dependency
> between the two?
*
I do not know if the user cares what file it is stored in.  I just know what
I see in auth-source.el.  In the auth-sources defcustom I see several items
that appear to be related to using the secrets API as defined in
secrets.el.  They are as follows:
  (const :tag "Default Secrets API Collection" 'default)
  (const :tag "Login Secrets API Collection" "secrets:Login")
  (const :tag "Temp Secrets API Collection" "secrets:session")

The value assigned to the auth-sources defcustom appears to be associated
with the collection parameter that is used when calling secrets-get-secret
and secrets-get-attributes.  I do not know enough Lisp to be certain of
that.  It is the "secrets:Login" item that made me think I was
misinterpreting the collection parameter in my implementation.  This item
seems to exactly describe the Login keychain file in Mac OS X.  The default
item seems to be analogous to the default keychain in Mac OS X, which is by
default the Login keychain but can be changed to a user defined keychain
file using the Keychain Access application.

As for your question about whether you can have any serviceName in any file,
yes you can.  There is no dependency or connection between the serviceName
parameter and the keychain file.

I just want to do this right.  If the right approach is to use the default
keychain by default providing a way to customize Emacs to use another
keychain file and interpreting the collection parameter as the service name,
then that is the approach I will use.  But, if I should instead use the
collection parameter as an indication of which keychain file to use, as the
three "Secrets API Collection" items in the auth-sources defcustom imply I
should, then that is the approach I will use.

It seems that you favor the first approach.  Are there any other opinions?
*
*

[-- Attachment #2: Type: text/html, Size: 2455 bytes --]

Re: Modifying Emacs to use the Mac OS X Keychain Services

[Ted Zlatanov]

On Sun, 5 Jun 2011 18:23:59 -0500 Ben Key <bkey76@gmail.com> wrote: 

BK> This item seems to exactly describe the Login keychain file in Mac
BK> OS X.  The default item seems to be analogous to the default
BK> keychain in Mac OS X, which is by default the Login keychain but can
BK> be changed to a user defined keychain file using the Keychain Access
BK> application.

BK> As for your question about whether you can have any serviceName in any file,
BK> yes you can.  There is no dependency or connection between the serviceName
BK> parameter and the keychain file.

OK, I see.  The serviceName is just things like "SSH" and "FTP" right?

If that's the case, then yes, you want to treat the name parameter as
the name of the keychain file.  I guess it's called "collection" in the
Secrets API.

Ted

Re: Modifying Emacs to use the Mac OS X Keychain Services

[Michael Albinus]

Ben Key <bkey76@gmail.com> writes:

> Hello,

Hi Ben,

> I am still working on this task.  I need some more advice though.

Ted did answer already from the auth-source.el pov. Here are some more
details from secrets.el.

> But, after taking another look at auth-source.el, I am wondering if
> this was the correct design decision.  I am wondering if I should
> instead be interpreting the collection parameter as an indication of
> which keychain file to store the passwords in.  I could interpret the
> value "default" for the collection parameter as an indication that the
> default keychain should be used.  I could interpret the value "Login"
> or "secrets:Login" as an indication that the Login keychain should be
> used.  Other values for collection parameter could be interpreted as
> an indication that the data should be stored in a file
> {collection}.keychain in ~/Library/Keychains. 
>
> Which interpretation of the collection parameter is the correct one? 
> I ask because I want to be sure to get this right.

A collection in the Secrets Service API is just a bundle of passwords
(more precisely: secret items). It does not tell you how they are
stored. And there are even collections, which are not stored permanently.

With the GNOME Keyring (the implementation of the Secrets Service API I
prefer), there is usually just the collection "login". This is the only
physical collection, if you do not create other ones, and this
collection is stored in the file "~/.gnome2/keyrings/login.keyring".

However, the existence of this collection is not guaranteed by the
API. The only promise of the API is a collection called "default", which
is not a physical collection but an alias. No wonder, initially this
alias points to the "login" collection. You could change this link.

Another collection offered by the Secrets Service API is called
"session". This collection is not a persistent one; it has no physical
representation on your disk, and its lifetime is equal the Secret
Service session (per default, the running desktop session). This
collection is good for people who want a kind of password cache for
reuse, but who do not want it stored permanently.

Best regards, Michael.

Re: Modifying Emacs to use the Mac OS X Keychain Services

[Ben Key]

[-- Attachment #1: Type: text/plain, Size: 1512 bytes --]

Hello Michael,

I am not certain which interpretation of the collection parameter you are
advocating.  It sounds like you are suggesting that I use the collection
parameter to specify which keychain file to store the data in but I am not
certain.

In this implementation I could interpret the value "default" to mean the
default keychain (use nil for the keychain parameter for the various
Keychain Services API functions) and the value "login" or "secrets:Login" to
mean the login keychain (which is found at
~/Library/Keychains/login.keychain).

Note that the various Keychain Services API functions accept a keychain
parameter.  This parameter can be nil to indicate that the default keychain
should be used or a keychain reference opened by either SecKeychainOpen or
SecKeychainCreate.  Both SecKeychainOpen or SecKeychainCreate provide a
reference to a keychain file specified by a full path and file name.  As a
result, in this implementation if the collection parameter is "login" or
"secrets:Login" I would use SecKeychainOpen to open the keychain file
keychain file ~/Library/Keychains/login.keychain and use the returned
keychain reference.

There is no direct equivalent to the session keychain in Mac OS X.  I
suppose I could make an Emacs session keychain by using the file
~/Library/Keychains/emacs-session.keychain and using the function
SecKeychainDelete to clear the keychain file on first use.

Is this the interpretation of the collection parameter that I should use?
If so, just let me know.

[-- Attachment #2: Type: text/html, Size: 1640 bytes --]

Re: Modifying Emacs to use the Mac OS X Keychain Services

[Michael Albinus]

Ben Key <bkey76@gmail.com> writes:

> Hello Michael,

Hi Ben,

> I am not certain which interpretation of the collection parameter you
> are advocating.

I tried to avoid a recommendation, because I don't know the Mac OS X
Keychain Services.

> It sounds like you are suggesting that I use the collection parameter
> to specify which keychain file to store the data in but I am not
> certain.

Reading your explanation of keychain files, it sounds natural.

> In this implementation I could interpret the value "default" to mean
> the default keychain (use nil for the keychain parameter for the
> various Keychain Services API functions) and the value "login" or
> "secrets:Login" to mean the login keychain (which is found at
> ~/Library/Keychains/login.keychain).

OK.

> Note that the various Keychain Services API functions accept a
> keychain parameter.  This parameter can be nil to indicate that the
> default keychain should be used or a keychain reference opened by
> either SecKeychainOpen or SecKeychainCreate.  Both SecKeychainOpen or
> SecKeychainCreate provide a reference to a keychain file specified by
> a full path and file name.  As a result, in this implementation if the
> collection parameter is "login" or "secrets:Login" I would use
> SecKeychainOpen to open the keychain file keychain file
> ~/Library/Keychains/login.keychain and use the returned keychain
> reference.

OK.

> There is no direct equivalent to the session keychain in Mac OS X.  I
> suppose I could make an Emacs session keychain by using the file
> ~/Library/Keychains/emacs-session.keychain and using the function
> SecKeychainDelete to clear the keychain file on first use.

The major idea of the "session" collection is to NOT store secrets
permanently. Therefore, I would recommend that you raise an error "not
available" or something like this, if somebody uses the "session"
collection in your case.

> Is this the interpretation of the collection parameter that I should
> use?  If so, just let me know.

Yes, please go on.

Best regards, Michael.

Re: Modifying Emacs to use the Mac OS X Keychain Services

[Ben Key]

On Tue, Jun 7, 2011 at 2:58 AM, Michael Albinus <michael.albinus@gmx.de> wrote:

> Reading your explanation of keychain files, it sounds natural.

Since it seems that both you and Ted Zlatanov seem to prefer the
second interpretation of the collection parameter, this evening I
rewrote the ns-keychain functions I had already written so that they
interpret the collection parameter to be an indication of what
keychain file the data should be stored in.  The values "default" or
"secrets:default" are interpreted to mean that the default keychain
should be used.  The values "login" or "secrets:Login" are interpreted
to mean that the login keychain file should be used.  Other values for
collection such as "session" or "secrets:session" are not supported at
this time.  The values "session" or "secrets:session" cause an error
with an "At this time the 'session' collection is not supported on Mac
OS X" message.  Other non-supported values simply result in the
default keychain being used.

> The major idea of the "session" collection is to NOT store secrets
> permanently. Therefore, I would recommend that you raise an error "not
> available" or something like this, if somebody uses the "session"
> collection in your case.

I could mimic this capability by storing the data in a keychain file,
such as ~/Library/Keychains/emacs-session.keychain that gets deleted
using the function SecKeychainDelete when Emacs exits.  This would
make it an Emacs session keychain.  Of course if Emacs crashes, the
keychain file would not be deleted.  If we decide to do this later, I
can add that functionality at any time without too much difficulty.
For now, I just raise an error in this case as you suggested.

Since the collection parameter is now being interpreted as an
indication of which keychain file to store the data in, I had to pick
a value for the serviceName parameter that is used by functions like
SecKeychainAddGenericPassword and SecKeychainFindGenericPassword.  I
chose the value "GNU Emacs."  Are there any objections to this?  I
could have added a parameter to allow the user to select the value of
the serviceName parameter but this would have made it necessary to
abandon my plan of adding one ns-keychain function for each secrets
API function that has the same parameters, and as near to the same
behavior, as the corresponding secrets API function.

Re: Modifying Emacs to use the Mac OS X Keychain Services

[Ted Zlatanov]

On Wed, 8 Jun 2011 00:50:50 -0500 Ben Key <bkey76@gmail.com> wrote: 

BK> I rewrote the ns-keychain functions I had already written so that
BK> they interpret the collection parameter to be an indication of what
BK> keychain file the data should be stored in.  The values "default" or
BK> "secrets:default" are interpreted to mean that the default keychain
BK> should be used.  The values "login" or "secrets:Login" are
BK> interpreted to mean that the login keychain file should be
BK> used.  Other values for collection such as "session" or
BK> "secrets:session" are not supported at this time.  The values
BK> "session" or "secrets:session" cause an error with an "At this time
BK> the 'session' collection is not supported on Mac OS X" message.

This is good, but the error should be simply "The collection %s could
not be found" for any invalid collection, not just "session".

BK>  Other non-supported values simply result in the default keychain
BK> being used.

Please throw an error if the requested keychain doesn't exist, same as
above.

>> The major idea of the "session" collection is to NOT store secrets
>> permanently. Therefore, I would recommend that you raise an error "not
>> available" or something like this, if somebody uses the "session"
>> collection in your case.

BK> I could mimic this capability by storing the data in a keychain file,
BK> such as ~/Library/Keychains/emacs-session.keychain that gets deleted
BK> using the function SecKeychainDelete when Emacs exits.  This would
BK> make it an Emacs session keychain.  Of course if Emacs crashes, the
BK> keychain file would not be deleted.

I don't think this is useful and introduces unwanted security risks.
There's no need to mimic the Secrets API "session" transient storage if
it's not directly supported by the Keychain Services.

BK> Since the collection parameter is now being interpreted as an
BK> indication of which keychain file to store the data in, I had to pick
BK> a value for the serviceName parameter that is used by functions like
BK> SecKeychainAddGenericPassword and SecKeychainFindGenericPassword.  I
BK> chose the value "GNU Emacs."  Are there any objections to this?

Yes, please make it configurable.

BK>  I could have added a parameter to allow the user to select the
BK> value of the serviceName parameter but this would have made it
BK> necessary to abandon my plan of adding one ns-keychain function for
BK> each secrets API function that has the same parameters, and as near
BK> to the same behavior, as the corresponding secrets API function.

Don't be afraid of diverging from the Secrets API.  The Keychain
Services API is not the same thing, we know that.  Please add that
parameter to the calls.

Are you providing both SecKeychain*GenericPassword and
SecKeychain*InternetPassword?

Ted

Re: Modifying Emacs to use the Mac OS X Keychain Services

[Ben Key]

Hello,

Ted Zlatanov wrote:

> Please throw an error if the requested keychain doesn't exist, same as
> above.

It may be more useful to only throw the error for the "session" or
"secrets:session" values for the keychain parameter and to handle
values other than "login," "secrets:Login," "default,"
"secrets:default," "session", or "secrets:session" as an indication
that a keychain file named ~/Library/Keychains/{keychain}.keychain
should be used.  If this keychain file exists, I can open it using
SecKeychainOpen.  If the keychain file does not exist I can create it
with SecKeychainCreate.  What do you think of this idea?

> Don't be afraid of diverging from the Secrets API.  The Keychain
> Services API is not the same thing, we know that.  Please add that
> parameter to the calls.

OK.  I will abandon my attempt to conform to the Secrets API.  This
will actually make things a little easier.  I will provide a Keychain
and Service Name parameter for each of my new functions.

> Are you providing both SecKeychain*GenericPassword and
> SecKeychain*InternetPassword?

Yes, I will provide access to both the Generic Password and Internet
Password functions.

Re: Modifying Emacs to use the Mac OS X Keychain Services

[Ted Zlatanov]

On Fri, 10 Jun 2011 19:30:31 -0500 Ben Key <bkey76@gmail.com> wrote: 

BK> Ted Zlatanov wrote:

>> Please throw an error if the requested keychain doesn't exist, same as
>> above.

BK> It may be more useful to only throw the error for the "session" or
BK> "secrets:session" values for the keychain parameter and to handle
BK> values other than "login," "secrets:Login," "default,"
BK> "secrets:default," "session", or "secrets:session" as an indication
BK> that a keychain file named ~/Library/Keychains/{keychain}.keychain
BK> should be used.  If this keychain file exists, I can open it using
BK> SecKeychainOpen.  If the keychain file does not exist I can create it
BK> with SecKeychainCreate.  What do you think of this idea?

(we won't use the "secrets:" prefix, it will probably be "ns-keychain:"
or something like that)

Provide a `ns-keychain-open' and a `ns-keychain-create' (plus a
`ns-keychain-create-or-open' convenience function if you wish, and a
`ns-keychain-delete' if you think it's safe).

So the `ns-keychain-open' will throw an error if you try to open a
nonexistent keychain, including "session", and `ns-keychain-create' will
create whatever you ask (including "session"), only throwing an error if
the creation failed.

You may want to create access functions for keychain aliases
(create/delete/switch target) if the API supports that.

The rest we can handle at the auth-source level.

Thanks
Ted

Re: Modifying Emacs to use the Mac OS X Keychain Services

[Ben Key]

Ted Zlatanov wrote:

> Provide a `ns-keychain-open' and a `ns-keychain-create' (plus a
> `ns-keychain-create-or-open' convenience function if you wish, and a
> `ns-keychain-delete' if you think it's safe).

I had planed to allow each of the ns-keychain functions to take a
keychain parameter that was a string and open or create the keychain
automatically.  But the design you recommend will actually make things
easier.  This is the design I will use then.

Note that to avoid resource leaks, I will also have to provide a
ns-keychain-close function.

Re: Modifying Emacs to use the Mac OS X Keychain Services

[Ben Key]

Hello,

I need some advice.  Now that I am providing a `ns-keychain-open'
function and a `ns-keychain-create' function  I need to know how to
store a pointer to a SecKeychainRef object in a Lisp_Object variable.
Can I simply use a type cast?

Thanks.

Re: Modifying Emacs to use the Mac OS X Keychain Services

[Eli Zaretskii]

> From: Ben Key <bkey76@gmail.com>
> Date: Sat, 11 Jun 2011 23:18:22 -0500
> 
> I need some advice.  Now that I am providing a `ns-keychain-open'
> function and a `ns-keychain-create' function  I need to know how to
> store a pointer to a SecKeychainRef object in a Lisp_Object variable.
> Can I simply use a type cast?

It's tempting, but I don't think it's TRT.  I think we generally use
pseudovectors for this.  The advantage is that you can then use
standard macros to access it, and developers can use commands in
.gdbinit to debug the code that involves these objects.

Re: Modifying Emacs to use the Mac OS X Keychain Services

[Ted Zlatanov]

On Sat, 11 Jun 2011 21:28:19 -0500 Ben Key <bkey76@gmail.com> wrote: 

BK> Note that to avoid resource leaks, I will also have to provide a
BK> ns-keychain-close function.

Will it get cleaned up properly when Emacs exits if you don't close
explicitly?  If not, maybe it's nice to provide a helper macro,
something like this:

(defmacro with-ns-keychain (keychain &rest body)
  `(unwind-protect (progn
                    (ns-keychain-open ,keychain)
                    ,@body)
                   (ns-keychain-close ,keychain)))

...but that's easy to add.

Ted

Re: Modifying Emacs to use the Mac OS X Keychain Services

[Ted Zlatanov]

On Sun, 12 Jun 2011 19:40:16 +0300 Eli Zaretskii <eliz@gnu.org> wrote: 

>> From: Ben Key <bkey76@gmail.com>
>> Date: Sat, 11 Jun 2011 23:18:22 -0500
>> 
>> I need some advice.  Now that I am providing a `ns-keychain-open'
>> function and a `ns-keychain-create' function  I need to know how to
>> store a pointer to a SecKeychainRef object in a Lisp_Object variable.
>> Can I simply use a type cast?

EZ> It's tempting, but I don't think it's TRT.  I think we generally use
EZ> pseudovectors for this.  The advantage is that you can then use
EZ> standard macros to access it, and developers can use commands in
EZ> .gdbinit to debug the code that involves these objects.

Another approach is to do what frames do; see `frame-parameters' for
instance.

Ted

Re: Modifying Emacs to use the Mac OS X Keychain Services

[Ben Key]

Eli Zaretskii wrote:
> It's tempting, but I don't think it's TRT.  I think we generally use
> pseudovectors for this.  The advantage is that you can then use
> standard macros to access it, and developers can use commands in
> .gdbinit to debug the code that involves these objects.

Thanks.

Ted Zlatanov wrote:
> Another approach is to do what frames do; see `frame-parameters' for
> instance.

It looks like frames do use a pseudovector.  For example.

#define XSETFRAME(a, b) (XSETPSEUDOVECTOR (a, b, PVEC_FRAME))
#define FRAMEP(x) PSEUDOVECTORP (x, PVEC_FRAME)

Re: Modifying Emacs to use the Mac OS X Keychain Services

[Stefan Monnier]

> I need some advice.  Now that I am providing a `ns-keychain-open'
> function and a `ns-keychain-create' function  I need to know how to
> store a pointer to a SecKeychainRef object in a Lisp_Object variable.
> Can I simply use a type cast?

No, a simple type cast won't do.  If the Lisp_Object is fully under your
control (so you always know you're the one who built it and never need
to check what kind of Lisp_Object it is), you can use
a Lisp_Misc_Save_value.  "grep -i save_value src/*.[ch]" should get
you started.


        Stefan

Re: Modifying Emacs to use the Mac OS X Keychain Services

[Ben Key]

Stefan Monnier wrote:

> No, a simple type cast won't do.  If the Lisp_Object is fully under your
> control (so you always know you're the one who built it and never need
> to check what kind of Lisp_Object it is), you can use
> a Lisp_Misc_Save_value.  "grep -i save_value src/*.[ch]" should get
> you started.
>

The Lisp_Object is not always under my control.  The idea is to
provide three functions to open a Mac OS X keychain file.  The
functions are ns-keychain-open to open a keychain file,
ns-keychain-create to create a keychain file, and
ns-keychain-create-or-open to open a keychain file if it exists or
create it if it does not exist.  All three of these functions, which
will be callable from Lisp, need to return a SecKeychainRef object.
The SecKeychainRef object is just a pointer to a structure, allocated
by the operating system.  Additional functions such as
ns-keychain-add-generic-password and ns-keychain-get-generic-password
will expect to be called with the keychain parameter set to the value
returned by an earlier call to ns-keychain-open,  ns-keychain-create,
or ns-keychain-create-or-open.  Once the calling package is finished
with the keychain object it will be expected to call ns-keychain-close
to release the keychain reference.

Since the SecKeychainRef object will need to pass the boundary between
internal Emacs code and Lisp code it cannot be considered to be always
under my control.  As a result I am not certain that a
Lisp_Misc_Save_value is appropriate for my purpose.  I am not certain
that pseudovectors are appropriate either.  It seems that we use
pseudovectors to represent structures that are allocated by Emacs but
the SecKeychainRef object is allocated by the OS.

The problem is that I am not certain how to return a SecKeychainRef
object from ns-keychain-open,  ns-keychain-create, and
ns-keychain-create-or-open so that they are callable from Lisp.  Any
suggestions are welcome.

Re: Modifying Emacs to use the Mac OS X Keychain Services

[Ted Zlatanov]

On Tue, 14 Jun 2011 21:15:18 -0500 Ben Key <bkey76@gmail.com> wrote: 

BK> Stefan Monnier wrote:
>> No, a simple type cast won't do.  If the Lisp_Object is fully under your
>> control (so you always know you're the one who built it and never need
>> to check what kind of Lisp_Object it is), you can use
>> a Lisp_Misc_Save_value.  "grep -i save_value src/*.[ch]" should get
>> you started.

BK> The Lisp_Object is not always under my control.

Why not?  You're the only one providing ns-keychain-* functions.

I'm no expert on the Emacs internals, but I think you can do the
following:

The Lisp_Misc_Save_Value holds a void pointer and an integer.  Just
provide convenience functions for yourself that take a
Lisp_Misc_Save_Value and set the pointer to the SecKeychainRef or
extract it back out.  Use the `SAVE_VALUEP' and `XSAVE_VALUE' macros.

Then make a Lisp list of Lisp_Misc_Save_Value and that's your keychain
reference list.  Make the head of the Lisp list a static symbol and
GC-protect it (see gnutls.c, for instance, on how to intern a symbol and
use `staticpro' on it in the syms_of_gnutls function).  Emacs won't GC
the list then.

BK> Since the SecKeychainRef object will need to pass the boundary
BK> between internal Emacs code and Lisp code it cannot be considered to
BK> be always under my control.

Trust `staticpro' :)

BK> The problem is that I am not certain how to return a SecKeychainRef
BK> object from ns-keychain-open,  ns-keychain-create, and
BK> ns-keychain-create-or-open so that they are callable from Lisp.  Any
BK> suggestions are welcome.

I would return it as a Lisp_Misc_Save_Value and set a :ns-keychain-name
property on the value to get the keychain name (ditto for any other
keychain properties you want to preserve).

Ted

Re: Modifying Emacs to use the Mac OS X Keychain Services

[Andreas Schwab]

Ted Zlatanov <tzz@lifelogs.com> writes:

> I would return it as a Lisp_Misc_Save_Value

A Lisp_Misc_Save_Value is a purely internal type and must not escape to
the Lisp interpreter.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Modifying Emacs to use the Mac OS X Keychain Services

[Ted Zlatanov]

On Wed, 15 Jun 2011 18:30:36 +0200 Andreas Schwab <schwab@linux-m68k.org> wrote: 

AS> Ted Zlatanov <tzz@lifelogs.com> writes:
>> I would return it as a Lisp_Misc_Save_Value

AS> A Lisp_Misc_Save_Value is a purely internal type and must not escape to
AS> the Lisp interpreter.

I misunderstood it then.  Sorry.

Ted

Re: Modifying Emacs to use the Mac OS X Keychain Services

[Stefan Monnier]

> The Lisp_Misc_Save_Value holds a void pointer and an integer.  Just
> provide convenience functions for yourself that take a
> Lisp_Misc_Save_Value and set the pointer to the SecKeychainRef or
> extract it back out.  Use the `SAVE_VALUEP' and `XSAVE_VALUE' macros.

That only works if you know for sure that this save_value indeed holds
a SecKeychainRef, which is only the case if Elisp can never get its hands
on it.


        Stefan

Re: Modifying Emacs to use the Mac OS X Keychain Services

[Chong Yidong]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> The Lisp_Misc_Save_Value holds a void pointer and an integer.  Just
>> provide convenience functions for yourself that take a
>> Lisp_Misc_Save_Value and set the pointer to the SecKeychainRef or
>> extract it back out.  Use the `SAVE_VALUEP' and `XSAVE_VALUE' macros.
>
> That only works if you know for sure that this save_value indeed holds
> a SecKeychainRef, which is only the case if Elisp can never get its hands
> on it.

I don't like the idea of having a first class Emacs Lisp object that's a
proprietary Mac OS specific pointer object.

If there is really no other way (such as an id number) to identify the
SecKeychainRef, I think it's better to do what image.c does: make the
Lisp representation the list specification used to generate the
SecKeychainRef, keep an internal hash table that stores the
SecKeychainRef that's never exposed to Lisp.  Then the Lisp-visible
functions would take the list spec as its argument and look up the
SecKeychainRefs in the table.

Re: Modifying Emacs to use the Mac OS X Keychain Services

[Dave Abrahams]

Did anything come of this?  I am really tired of typing in my GPG key
every time I start Gnus.  I'd be more than happy to have a solution that
just used /usr/bin/security to look up the password; I don't need more
security than that.

I looked a bit at the "secrets" API but could understand it easily
enough to code something up.  I just want Emacs to run

   /usr/bin/security --find-internet-password -gs <hostname> <username>

to get the password for my mail server.

TIA,

Dave

on Wed Jun 08 2011, Ted Zlatanov <tzz-AT-lifelogs.com> wrote:

> On Wed, 8 Jun 2011 00:50:50 -0500 Ben Key <bkey76@gmail.com> wrote: 
>
> BK> I rewrote the ns-keychain functions I had already written so that
> BK> they interpret the collection parameter to be an indication of what
> BK> keychain file the data should be stored in.  The values "default" or
> BK> "secrets:default" are interpreted to mean that the default keychain
> BK> should be used.  The values "login" or "secrets:Login" are
> BK> interpreted to mean that the login keychain file should be
> BK> used.  Other values for collection such as "session" or
> BK> "secrets:session" are not supported at this time.  The values
> BK> "session" or "secrets:session" cause an error with an "At this time
> BK> the 'session' collection is not supported on Mac OS X" message.
>
> This is good, but the error should be simply "The collection %s could
> not be found" for any invalid collection, not just "session".
>
> BK>  Other non-supported values simply result in the default keychain
> BK> being used.
>
> Please throw an error if the requested keychain doesn't exist, same as
> above.
>
>>> The major idea of the "session" collection is to NOT store secrets
>>> permanently. Therefore, I would recommend that you raise an error "not
>>> available" or something like this, if somebody uses the "session"
>>> collection in your case.
>
> BK> I could mimic this capability by storing the data in a keychain file,
> BK> such as ~/Library/Keychains/emacs-session.keychain that gets deleted
> BK> using the function SecKeychainDelete when Emacs exits.  This would
> BK> make it an Emacs session keychain.  Of course if Emacs crashes, the
> BK> keychain file would not be deleted.
>
> I don't think this is useful and introduces unwanted security risks.
> There's no need to mimic the Secrets API "session" transient storage if
> it's not directly supported by the Keychain Services.
>
> BK> Since the collection parameter is now being interpreted as an
> BK> indication of which keychain file to store the data in, I had to pick
> BK> a value for the serviceName parameter that is used by functions like
> BK> SecKeychainAddGenericPassword and SecKeychainFindGenericPassword.  I
> BK> chose the value "GNU Emacs."  Are there any objections to this?
>
> Yes, please make it configurable.
>
> BK>  I could have added a parameter to allow the user to select the
> BK> value of the serviceName parameter but this would have made it
> BK> necessary to abandon my plan of adding one ns-keychain function for
> BK> each secrets API function that has the same parameters, and as near
> BK> to the same behavior, as the corresponding secrets API function.
>
> Don't be afraid of diverging from the Secrets API.  The Keychain
> Services API is not the same thing, we know that.  Please add that
> parameter to the calls.
>
> Are you providing both SecKeychain*GenericPassword and
> SecKeychain*InternetPassword?
>
> Ted

-- 
Dave Abrahams
BoostPro Computing                  Software Development        Training
http://www.boostpro.com             Clang/LLVM/EDG Compilers  C++  Boost

Re: Modifying Emacs to use the Mac OS X Keychain Services

[Harald Hanche-Olsen]

(This probably doesn't really belong on the emacs-devel mailing list,
but if we keep it short, maybe nobody will object. My apologies if
someone does object. I skip the copy to ding@gnus.org, since I am not
on that list myself, and don't even know what it's for.)

[Dave Abrahams <dave@boostpro.com> (2012-07-27 15:20:17 UTC)]

> I looked a bit at the "secrets" API but could understand it easily
> enough to code something up.

It's a bit messy, with output clearly meant for human consumption and
not for programs.

> I just want Emacs to run
> 
>    /usr/bin/security --find-internet-password -gs <hostname> <username>
> 
> to get the password for my mail server.

Since this scratches an itch of my own, I cooked up the following.
Feel free to adapt, bend and mutilate it to your heart's content.

(defun get-keychain-password (service)
  "Get a generic password from the OS X keychain.
The password is associated with the string SERVICE.
This corresponds to the Account field in the Keychain Access GUI.
BUG: If there is no matching password, or the output of /usr/bin/security
has an unexpected format, the function silently returns NIL."
  (with-temp-buffer
    ;; /usr/bin/security outputs all kinds of info on stdout
    ;; oddly, the password itself appears on stderr, in the form
    ;; password: "..." with a final newline
    ;; No escape mechanism appears to be used for quotes etc in the password
    (call-process "/bin/sh" nil (list (current-buffer) t) nil
		  "-c" "/usr/bin/security \"$@\" >/dev/null" "-"
		  "find-generic-password" "-gs" service)
    (let ((min (point-min))
	  (max (point-max)))
      (when (and (> (- max min) 13)
		 (string= (buffer-substring-no-properties min (+ min 11))
			  "password: \"")
		 (string= (buffer-substring-no-properties (- max 2) max)
			  "\"\n"))
	(buffer-substring-no-properties (+ min 11) (- max 2))))))

- Harald

Re: Modifying Emacs to use the Mac OS X Keychain Services

[Dave Abrahams]

on Sat Jul 28 2012, Harald Hanche-Olsen <hanche-AT-math.ntnu.no> wrote:

> Since this scratches an itch of my own, I cooked up the following.
> Feel free to adapt, bend and mutilate it to your heart's content.
>
> (defun get-keychain-password (service)

...

Haha, thanks, but writing a function to invoke /usr/bin/security and get
the password back is no problem for me; what baffles me is how to plug
this thing into Gnus so I don't need to supply a password when
.authinfo.gpg is decrypted.  If you get that part worked out, I'd love
to see it.

-- 
Dave Abrahams
BoostPro Computing                  Software Development        Training
http://www.boostpro.com             Clang/LLVM/EDG Compilers  C++  Boost

Re: Modifying Emacs to use the Mac OS X Keychain Services

[Harald Hanche-Olsen]

[Dave Abrahams <dave@boostpro.com> (2012-07-28 16:33:42 UTC)]

> Haha, thanks, but writing a function to invoke /usr/bin/security and get
> the password back is no problem for me;

Ah. Sorry, but it seemed to me that this was the problem you were
asking for help with. Sorry about the misunderstanding.

> what baffles me is how to plug this thing into Gnus so I don't need
> to supply a password when .authinfo.gpg is decrypted. If you get
> that part worked out, I'd love to see it.

Well, I can't help with that part.

- Harald

Re: Modifying Emacs to use the Mac OS X Keychain Services

[Ted Zlatanov]

On Fri, 27 Jul 2012 11:20:17 -0400 Dave Abrahams <dave@boostpro.com> wrote: 

DA> Did anything come of this?  I am really tired of typing in my GPG key
DA> every time I start Gnus.  I'd be more than happy to have a solution that
DA> just used /usr/bin/security to look up the password; I don't need more
DA> security than that.

DA> I looked a bit at the "secrets" API but could understand it easily
DA> enough to code something up.  I just want Emacs to run

DA>    /usr/bin/security --find-internet-password -gs <hostname> <username>

DA> to get the password for my mail server.

I don't think I knew about this utility :)  Thanks!

I haven't heard from Ben Key (CC-ed on this post) in a year so I figured
it's simpler to implement this myself.  I've pushed something into the
Gnus repo, which you can test.  It doesn't support creation or deletion,
but searching works.

The fundamental problem was that internet (I've spelled it with a
lowercase 'i' to be consistent with Apple) and generic keychains behave
very differently.  So I chose to make the user decide which one he
wants; the following are valid entries in `auth-sources':

#+begin_src lisp
(auth-source-backend-parse 'macos-keychain-internet)
(auth-source-backend-parse 'macos-keychain-generic)
(auth-source-backend-parse "macos-keychain-internet:/path/here.keychain")
(auth-source-backend-parse "macos-keychain-generic:/path/here.keychain")
(auth-source-backend-parse '(:source (:macos-keychain-internet default)))
(auth-source-backend-parse '(:source (:macos-keychain-generic "/path/here.keychain")))
#+end_src

...and here you can see the very first entry in each of your default
internet and generic keychains:

#+begin_src lisp
(let ((auth-sources '(macos-keychain-internet))) (auth-source-search :max 1))
(let ((auth-sources '(macos-keychain-generic))) (auth-source-search :max 1))
#+end_src

The hardest part was mapping internet and generic keychains into the
common auth-source format for searching and for providing results.  For
searching, I chose to map them as explained in the docstring of
`auth-source-macos-keychain-search', using the various /usr/bin/security
parameters.  For results, the logic is simple enough to show here:

#+begin_src lisp
(defun auth-source-macos-keychain-result-append (result generic k v)
  (push v result)
  (setq k (cond
           ((equal k "acct") "user")
           ;; for generic keychains, creator is host, service is port
           ((and generic (equal k "crtr")) "host")
           ((and generic (equal k "svce")) "port")
           ;; for internet keychains, protocol is port, server is host
           ((and (not generic) (equal k "ptcl")) "port")
           ((and (not generic) (equal k "srvr")) "host")
           (t k)))

  (push (intern (format ":%s" k)) result))
#+end_src

At most one result is returned, ever.  This is due to the way
/usr/bin/security works.  If I dump the whole keychain, the user would
get a thousand popup dialogs.

It should be pretty trivial to use the native keychain calls on Mac OS X
within this framework.  Ben, if you're still interested, please let us
know.

I am far from expert on Mac OS X; this worked for me and I hope it works
for you.  Patches welcome to improve it.

Ted

Re: Modifying Emacs to use the Mac OS X Keychain Services

[Michael Albinus]

Ted Zlatanov <tzz@lifelogs.com> writes:

> I haven't heard from Ben Key (CC-ed on this post) in a year so I figured
> it's simpler to implement this myself.  I've pushed something into the
> Gnus repo, which you can test.  It doesn't support creation or deletion,
> but searching works.

Wouldn't this be a good time to extend auth-source.el by a backend
concept? You have mentioned it as todo, and I belive it would pay. These
two new backend work only for Mac OS; the Secret Services backend is good
for GNU/Linux.

> Ted

Best regards, Michael.

Re: Modifying Emacs to use the Mac OS X Keychain Services

[Ted Zlatanov]

On Mon, 30 Jul 2012 15:34:45 +0200 Michael Albinus <michael.albinus@gmx.de> wrote: 

MA> Ted Zlatanov <tzz@lifelogs.com> writes:
>> I haven't heard from Ben Key (CC-ed on this post) in a year so I figured
>> it's simpler to implement this myself.  I've pushed something into the
>> Gnus repo, which you can test.  It doesn't support creation or deletion,
>> but searching works.

MA> Wouldn't this be a good time to extend auth-source.el by a backend
MA> concept? You have mentioned it as todo, and I belive it would pay. These
MA> two new backend work only for Mac OS; the Secret Services backend is good
MA> for GNU/Linux.

It already uses backends; see the following calls:

#+begin_src lisp
(auth-source-backend-parse "myfile.gpg")
(auth-source-backend-parse 'default)
(auth-source-backend-parse "secrets:Login")
(auth-source-backend-parse 'macos-keychain-internet)
(auth-source-backend-parse 'macos-keychain-generic)
(auth-source-backend-parse "macos-keychain-internet:/path/here.keychain")
(auth-source-backend-parse "macos-keychain-generic:/path/here.keychain")
#+end_src

The developer pain, at this point, seems to be that 

1) each backend has to implement its own treatment of the general search
parameters (:source :type :user :host :port :max), and interpret them in
its own domain; and

2) I use flat plists but most Emacs developers don't.  That's my
impression, at least, and perhaps the reason I haven't seen any
contributions to auth-source.el besides the `plstore' backend from Daiki
Ueno, which was mostly a copy of the `netrc' backend.

If you or anyone else has suggestions on redesigning auth-source.el and
perhaps changing its call interface, please speak up (with patches or
words).

Thanks
Ted

Re: Modifying Emacs to use the Mac OS X Keychain Services

[Dave Abrahams]

on Sun Jul 29 2012, Ted Zlatanov <tzz-AT-lifelogs.com> wrote:

> On Fri, 27 Jul 2012 11:20:17 -0400 Dave Abrahams <dave@boostpro.com> wrote: 
>
> DA> Did anything come of this?  I am really tired of typing in my GPG key
> DA> every time I start Gnus.  I'd be more than happy to have a solution that
> DA> just used /usr/bin/security to look up the password; I don't need more
> DA> security than that.
>
> DA> I looked a bit at the "secrets" API but could understand it easily
> DA> enough to code something up.  I just want Emacs to run
>
> DA>    /usr/bin/security --find-internet-password -gs <hostname> <username>
>
> DA> to get the password for my mail server.
>
> I don't think I knew about this utility :)  Thanks!
>
> I haven't heard from Ben Key (CC-ed on this post) in a year so I figured
> it's simpler to implement this myself.  I've pushed something into the
> Gnus repo, which you can test.  It doesn't support creation or deletion,
> but searching works.
>
> The fundamental problem was that internet (I've spelled it with a
> lowercase 'i' to be consistent with Apple) and generic keychains behave
> very differently.  So I chose to make the user decide which one he
> wants; the following are valid entries in `auth-sources':
>
> #+begin_src lisp
> (auth-source-backend-parse 'macos-keychain-internet)
> (auth-source-backend-parse 'macos-keychain-generic)
> (auth-source-backend-parse "macos-keychain-internet:/path/here.keychain")
> (auth-source-backend-parse "macos-keychain-generic:/path/here.keychain")
> (auth-source-backend-parse '(:source (:macos-keychain-internet default)))
> (auth-source-backend-parse '(:source (:macos-keychain-generic "/path/here.keychain")))
> #+end_src

And despite that, I am seeing 

auth-source-backend-parse: invalid backend spec: (quote macos-keychain-generic)
auth-source-backend-parse: invalid backend spec: (quote macos-keychain-internet)

>
> ...and here you can see the very first entry in each of your default
> internet and generic keychains:
>
> #+begin_src lisp
> (let ((auth-sources '(macos-keychain-internet))) (auth-source-search :max 1))
> (let ((auth-sources '(macos-keychain-generic))) (auth-source-search :max 1))
> #+end_src
>
> The hardest part was mapping internet and generic keychains into the
> common auth-source format for searching and for providing results.  For
> searching, I chose to map them as explained in the docstring of
> `auth-source-macos-keychain-search', using the various /usr/bin/security
> parameters.  For results, the logic is simple enough to show here:
>
> #+begin_src lisp
> (defun auth-source-macos-keychain-result-append (result generic k v)
>   (push v result)
>   (setq k (cond
>            ((equal k "acct") "user")
>            ;; for generic keychains, creator is host, service is port
>            ((and generic (equal k "crtr")) "host")
>            ((and generic (equal k "svce")) "port")
>            ;; for internet keychains, protocol is port, server is host
>            ((and (not generic) (equal k "ptcl")) "port")
>            ((and (not generic) (equal k "srvr")) "host")
>            (t k)))
>
>   (push (intern (format ":%s" k)) result))
> #+end_src
>
> At most one result is returned, ever.  This is due to the way
> /usr/bin/security works.  If I dump the whole keychain, the user would
> get a thousand popup dialogs.
>
> It should be pretty trivial to use the native keychain calls on Mac OS X
> within this framework.  Ben, if you're still interested, please let us
> know.
>
> I am far from expert on Mac OS X; this worked for me and I hope it works
> for you.  Patches welcome to improve it.
>
> Ted

-- 
Dave Abrahams
BoostPro Computing                  Software Development        Training
http://www.boostpro.com             Clang/LLVM/EDG Compilers  C++  Boost

Re: Modifying Emacs to use the Mac OS X Keychain Services

[Dave Abrahams]

[sorry about the last one; I misfired]

on Sun Jul 29 2012, Ted Zlatanov <tzz-AT-lifelogs.com> wrote:

> On Fri, 27 Jul 2012 11:20:17 -0400 Dave Abrahams <dave@boostpro.com> wrote: 
>
>
> #+begin_src lisp
> (auth-source-backend-parse 'macos-keychain-internet)
> (auth-source-backend-parse 'macos-keychain-generic)
> (auth-source-backend-parse "macos-keychain-internet:/path/here.keychain")
> (auth-source-backend-parse "macos-keychain-generic:/path/here.keychain")
> (auth-source-backend-parse '(:source (:macos-keychain-internet default)))
> (auth-source-backend-parse '(:source (:macos-keychain-generic "/path/here.keychain")))
> #+end_src

Hi Ted,

I finally got around to trying this.
Despite the fact that the above work, I'm seeing this message:

--8<---------------cut here---------------start------------->8---
auth-source-backend-parse: invalid backend spec: (quote macos-keychain-generic)
auth-source-backend-parse: invalid backend spec: (quote macos-keychain-internet)
--8<---------------cut here---------------end--------------->8---

I think maybe the customization interface has a couple extra single
quotes in it, because it left my auth-sources as

#+begin_src lisp
((quote macos-keychain-generic) (quote macos-keychain-internet) "~/.authinfo" "~/.authinfo.gpg" "~/.netrc")
#+end_src

-- 
Dave Abrahams
BoostPro Computing                  Software Development        Training
http://www.boostpro.com             Clang/LLVM/EDG Compilers  C++  Boost

Re: Modifying Emacs to use the Mac OS X Keychain Services

[Dave Abrahams]

on Mon Aug 20 2012, Dave Abrahams <dave-AT-boostpro.com> wrote:

> [sorry about the last one; I misfired]
>
> on Sun Jul 29 2012, Ted Zlatanov <tzz-AT-lifelogs.com> wrote:
>
>> On Fri, 27 Jul 2012 11:20:17 -0400 Dave Abrahams <dave@boostpro.com> wrote: 
>>
>>
>> #+begin_src lisp
>> (auth-source-backend-parse 'macos-keychain-internet)
>> (auth-source-backend-parse 'macos-keychain-generic)
>> (auth-source-backend-parse "macos-keychain-internet:/path/here.keychain")
>> (auth-source-backend-parse "macos-keychain-generic:/path/here.keychain")
>> (auth-source-backend-parse '(:source (:macos-keychain-internet default)))
>> (auth-source-backend-parse '(:source (:macos-keychain-generic "/path/here.keychain")))
>> #+end_src
>
> Hi Ted,
>
> I finally got around to trying this.
> Despite the fact that the above work, I'm seeing this message:
>
> auth-source-backend-parse: invalid backend spec: (quote macos-keychain-generic)
> auth-source-backend-parse: invalid backend spec: (quote macos-keychain-internet)
>
> I think maybe the customization interface has a couple extra single
> quotes in it, because it left my auth-sources as
>
> #+begin_src lisp
> ((quote macos-keychain-generic) (quote macos-keychain-internet) "~/.authinfo" "~/.authinfo.gpg" "~/.netrc")
> #+end_src

And once I fix that, I'm seeing:

--8<---------------cut here---------------start------------->8---
Opening connection to imap.gmail.com via tls...
Unable to open server nnimap+GMail due to: Wrong type argument: stringp, (993 "imaps" "imap" "993" "143")
Opening nnimap server on GMail...failed: 
nnimap read 0k from localhost
(No changes need to be saved)
Warning: Opening nnimap server on GMail...failed: ; Unable to open server nnimap+GMail due to: Wrong type argument: stringp, (993 "imaps" "imap" "993" "143")
--8<---------------cut here---------------end--------------->8---

-- 
Dave Abrahams
BoostPro Computing                  Software Development        Training
http://www.boostpro.com             Clang/LLVM/EDG Compilers  C++  Boost