Emacs Secret Service integration and KeePassXC issues

Emacs Secret Service integration and KeePassXC issues

[Liam Hupfer]

Hi Michael,

Thank you for integrating the Secret Servace API into Emacs; I've found
it very useful recently. KeePassXC, a popular FOSS cross-platform
password manager I use, recently merged the functionality to act as the
secrets provider as Linux (like Gnome Keyring or Kwallet). I'm trying to
use it with Emacs and having some issues. I asked about them in a
KeePassXC GitHub issue[1], and the implementer suggested Gnome
Keyring-isms might be an issue and that KeePassXC implements the full
spec.

Unfortunately, I know little about the Secret Service spec or Emacs'
implementation beyond basic usage, and I'm not sure how I'd begin to
debug this. I was wondering if you are capable of trying KeePassXC as
the secrets provider with Emacs? Specifically (as I mentioned in the
GitHub comment), I can't do things that write to the secrets collection,
like creating a secret with secrets-create-item. This means that when
TRAMP or Eshell prompts me to save a password to my secrets collection,
it silently fails to create the secret.

I emailed you directly because it seems you fully implemented the secret
service integration, but I CC'd help-gnu-emacs in case anyone else runs
into this issue or has information I'm missing. Thanks for the
implementation and any info you can provide!

Liam

PS: I'm new to mailing lists, so please let me know if I've committed
any faux pas here.

1: https://github.com/keepassxreboot/keepassxc/issues/3667#issuecomment-754380102

Re: Emacs Secret Service integration and KeePassXC issues

[Michael Albinus]

Liam Hupfer <liam@hpfr.net> writes:

> Hi Michael,

Hi Liam,

> Unfortunately, I know little about the Secret Service spec or Emacs'
> implementation beyond basic usage, and I'm not sure how I'd begin to
> debug this. I was wondering if you are capable of trying KeePassXC as
> the secrets provider with Emacs? Specifically (as I mentioned in the
> GitHub comment), I can't do things that write to the secrets collection,
> like creating a secret with secrets-create-item. This means that when
> TRAMP or Eshell prompts me to save a password to my secrets collection,
> it silently fails to create the secret.

Well, I've tried it. I'm running Fedora 33, and I've installed KeePassXC
2.6.3 via dnf.

Stopped the GNOME keyring daemon, started keepassxc. I've created a new
database, and two entries. Then I've enabled the Secret Service
Integration via Setings. And now I'm lost. I don't see any collection,
and so I don't know how to access.

> Liam
>
> PS: I'm new to mailing lists, so please let me know if I've committed
> any faux pas here.

No problem, everything is well as you have done it.

Best regards, Michael.

Re: Emacs Secret Service integration and KeePassXC issues

[Liam Hupfer via Users list for the GNU Emacs text editor]

Michael Albinus <michael.albinus@gmx.de> writes:

> Hi Liam,

Hello again Michael. Thank you very much for the prompt response; I must
apologize for the gargantuan delay. Hey, I got back to you before your
response’s first birthday, right?

Part of the reason for the wait was that I figured I’d have to create a
Fedora VM and reproduction instructions with vanilla Emacs (I use the
Doom distribution and have pretty much no idea how to navigate or
configure vanilla 😅) and KeePassXC.

I didn’t do that, but I did come back to this and actually read the
secrets.el and auth-sources.el source code as well as some of the
KeePassXC implementation comments.

> Well, I’ve tried it. I’m running Fedora 33, and I’ve installed KeePassXC
> 2.6.3 via dnf.
>
> Stopped the GNOME keyring daemon, started keepassxc. I’ve created a new
> database, and two entries. Then I’ve enabled the Secret Service
> Integration via Setings. And now I’m lost. I don’t see any collection,
> and so I don’t know how to access.

If you’re on Fedora 34, you should be able to get KeePassXC 2.6.6 now¹;
it’s what I’m using, so that’s probably our best bet for
reproducibility.

Firstly, I suspect the reason you don’t see a collection is because even
after enabling the Secret Service Integration in KeePassXC’s app-wide
settings, KeePassXC exposes no entries from the database to the service
by default. To expose some, you need to select `Database > Database
Settings > Secret Service Integration > Expose entries under this
group:', and then choose a group (likely only Root will exist for your
test database) to expose. Then in the app-wide secret service settings,
the group should show up under “Exposed database groups:”.

You might have to restart KeePassXC or Emacs at this point; I’m not 100%
sure how those interactions work, so I did it just to be safe. At this
point, invoking `(secrets-list-collections)' should return a list with
one string corresponding to your database name, and `M-x
secrets-show-secrets' should let you browse the entries and attributes.

Now for the main issue. Assuming my instructions are still working, you
have probably noticed that the “session” collection does not seem to be
present. I browsed the Emacs source and can’t find any
`CreateCollection' or `secrets-create-collection' references, but the
docs say the collection is “created automatically when Emacs uses the
Secret Service interface”. This does not seem to be happening. In any
case, KeePassXC maps collections to database files, so evaluating
`(secrets-create-collection)' results in KeePassXC prompting me for a
new database name and password and a place to save it. Since this
collection is intended to be ephemeral, it’s probably not great to go
through the whole database creation process every time you launch Emacs.
But that’s a KeePassXC issue to handle, and I’m fine with doing it for
the time being if it gets this working. Unfortunately, the session
collection does not seem to be created at all at the moment, despite
`(secrets-create-collection)' working fine. That part seems to be an
Emacs thing?

My only guess was that the `secrets-struct-secret-content-type'
`defconst' which creates a dummy item in the “session” collection
somehow creates the collection in the process.² Soon, I noticed another
thing: in KeePassXC’s app-wide secret service settings, there is an
“Authorization” tab that lists currently connected applications. For
some reason, Emacs was listed dozens of times when I opened this. I
watched the scroll-bar for the pane and it did shrink and grow when I
closed and reopened a session from Emacs, and I noticed that repeatedly
opening does in fact return the same session path, so I wasn’t sure how
that happened. But then I figured out how to try running the code in the
aforementioned `defconst', and I realized that it indeed opens a session
and attempts to create a dummy item to dynamically get the content-type.
However, because the “session” collection it uses doesn’t exist, it ends
up simply opening a session and then failing to create the item with
`dbus-call-method: D-Bus error: "No such object path
'/org/freedesktop/secrets/collection/session'"'. The removal call fails
because the result from the creation is nil. All of this is wrapped in
`ignore-errors', so it doesn’t get printed. But I think the result is
this somehow gets called repeatedly which creates the many open
sessions? Because once I got this far I just tried evaluating the
`defconst' repeatedly and it indeed opened many sessions, and the result
remained nil.

I supposed this is why `(secrets-create-item)' failed, but I noticed you
said the nil content type was backwards compatible so I gave it a try
and it gives /another/ error. I ran `(secrets-create-item "default"
"test emacs item" "test pw")' ³, and received:
┌────
│ No such method 'CreateItem' in interface 'org.freedesktop.Secret.Collection' at object path '/org/freedesktop/secrets/collection/main' (signature 'a{sv}(oayay)b')
└────

Interestingly, KeePassXC’s logs said:
┌────
│ Message signature does not match, expected "a{sv}(oayays)b" 3 got "a{sv}(oayay)b" 3
└────

In summary:
• Where is the “session” collection actually initialized?
• Do you have any idea what the issue is with the message signature
  issue for `CreateItem'?
• Can we get the “session” collection initialized (even if it involves
  waiting for me to interact with KeePassXC’s database creation UI)
  before the `defconst' evaluation in order to correctly grab the
  content-type?

Sorry this got long; writing about GUI navigation over email gets a
little verbose, I guess, and then I ended up doing a bunch more
debugging as I tried to write the email which resulted in me effectively
documenting my debugging process. I realize I’m asking a little much to
have the secrets.el maintainer debug interactions with an application he
doesn’t use, so let me be clear I’m not obliging you to try all this. If
you have suggestions or hunches I’m happy to try them myself. Also happy
to explore other debugging methods if you prefer!

Thanks for reading if you made it this far 😄.

—Liam

¹ <https://src.fedoraproject.org/rpms/keepassxc>
² <https://git.savannah.gnu.org/cgit/emacs.git/tree/lisp/net/secrets.el?id=194556425f140b8599467959b73d5954a59128e3#n317>
³ In KeePassXC’s implementation, the default alias refers to the currently focused open database, which is probably the only one you have.

Re: Emacs Secret Service integration and KeePassXC issues

[Michael Albinus]

Liam Hupfer <liam@hpfr.net> writes:

Hi Liam,

> Sorry this got long; writing about GUI navigation over email gets a
> little verbose, I guess, and then I ended up doing a bunch more
> debugging as I tried to write the email which resulted in me effectively
> documenting my debugging process. I realize I’m asking a little much to
> have the secrets.el maintainer debug interactions with an application he
> doesn’t use, so let me be clear I’m not obliging you to try all this. If
> you have suggestions or hunches I’m happy to try them myself. Also happy
> to explore other debugging methods if you prefer!

All of this sounds very interesting to me, thanks for the detailed
description. I'll dig into it next days when time permits, hopefully
still this year.

> Thanks for reading if you made it this far 😄.
>
> —Liam

Best regards, Michael.

Re: Emacs Secret Service integration and KeePassXC issues

[Michael Albinus]

Liam Hupfer <liam@hpfr.net> writes:

Hi Liam,

>> Well, I’ve tried it. I’m running Fedora 33, and I’ve installed KeePassXC
>> 2.6.3 via dnf.
>>
>> Stopped the GNOME keyring daemon, started keepassxc. I’ve created a new
>> database, and two entries. Then I’ve enabled the Secret Service
>> Integration via Setings. And now I’m lost. I don’t see any collection,
>> and so I don’t know how to access.
>
> If you’re on Fedora 34, you should be able to get KeePassXC 2.6.6 now¹;
> it’s what I’m using, so that’s probably our best bet for
> reproducibility.

I'm on Fedora 35 and KeePassXC 2.6.6-3.fc35 now.

> Firstly, I suspect the reason you don’t see a collection is because even
> after enabling the Secret Service Integration in KeePassXC’s app-wide
> settings, KeePassXC exposes no entries from the database to the service
> by default. To expose some, you need to select `Database > Database
> Settings > Secret Service Integration > Expose entries under this
> group:', and then choose a group (likely only Root will exist for your
> test database) to expose. Then in the app-wide secret service settings,
> the group should show up under “Exposed database groups:”.

Yep. This works for me, and I see the collection "Passwords", exposed
from KeePassXC. This maps to my database file name Passwords.kdbx.

> You might have to restart KeePassXC or Emacs at this point; I’m not 100%
> sure how those interactions work, so I did it just to be safe. At this
> point, invoking `(secrets-list-collections)' should return a list with
> one string corresponding to your database name, and `M-x
> secrets-show-secrets' should let you browse the entries and attributes.

Yes.

> Now for the main issue. Assuming my instructions are still working, you
> have probably noticed that the “session” collection does not seem to be
> present. I browsed the Emacs source and can’t find any
> `CreateCollection' or `secrets-create-collection' references, but the
> docs say the collection is “created automatically when Emacs uses the
> Secret Service interface”. This does not seem to be happening. In any
> case, KeePassXC maps collections to database files, so evaluating
> `(secrets-create-collection)' results in KeePassXC prompting me for a
> new database name and password and a place to save it. Since this
> collection is intended to be ephemeral, it’s probably not great to go
> through the whole database creation process every time you launch Emacs.
> But that’s a KeePassXC issue to handle, and I’m fine with doing it for
> the time being if it gets this working. Unfortunately, the session
> collection does not seem to be created at all at the moment, despite
> `(secrets-create-collection)' working fine. That part seems to be an
> Emacs thing?

The ephemeral "session" collection is created by the Secret Service
provider (GNOME keyring daemon in my tests). KeePassXC doesn't seem to
create it. I've scanned the docs whether this temporary collection is
requested (as I assume), but I couldn't find it. So maybe we shouldn't
trust its existence. I've added a respective note to the auth info
pages, where the Secret Service API is described.

> My only guess was that the `secrets-struct-secret-content-type'
> `defconst' which creates a dummy item in the “session” collection
> somehow creates the collection in the process.²

As the comment above the `secrets-struct-secret-content-type' definition
says, it was just a backward compatibility hack. 10 years have passed,
so we don't need it anymore. I've removed that hack.

> Soon, I noticed another
> thing: in KeePassXC’s app-wide secret service settings, there is an
> “Authorization” tab that lists currently connected applications. For
> some reason, Emacs was listed dozens of times when I opened this. I
> watched the scroll-bar for the pane and it did shrink and grow when I
> closed and reopened a session from Emacs, and I noticed that repeatedly
> opening does in fact return the same session path, so I wasn’t sure how
> that happened. But then I figured out how to try running the code in the
> aforementioned `defconst', and I realized that it indeed opens a session
> and attempts to create a dummy item to dynamically get the content-type.
> However, because the “session” collection it uses doesn’t exist, it ends
> up simply opening a session and then failing to create the item with
> `dbus-call-method: D-Bus error: "No such object path
> '/org/freedesktop/secrets/collection/session'"'. The removal call fails
> because the result from the creation is nil. All of this is wrapped in
> `ignore-errors', so it doesn’t get printed. But I think the result is
> this somehow gets called repeatedly which creates the many open
> sessions? Because once I got this far I just tried evaluating the
> `defconst' repeatedly and it indeed opened many sessions, and the result
> remained nil.

All of this shouldn't happen anymore with my recent change.

> I supposed this is why `(secrets-create-item)' failed, but I noticed you
> said the nil content type was backwards compatible so I gave it a try
> and it gives /another/ error. I ran `(secrets-create-item "default"
> "test emacs item" "test pw")' ³, and received:
> ┌────
> │ No such method 'CreateItem' in interface 'org.freedesktop.Secret.Collection' at object path '/org/freedesktop/secrets/collection/main' (signature 'a{sv}(oayay)b')
> └────
>
> Interestingly, KeePassXC’s logs said:
> ┌────
> │ Message signature does not match, expected "a{sv}(oayays)b" 3 got "a{sv}(oayay)b" 3
> └────

After my change, pushed to Emacs master, your example with
`secrets-create-item' in Emacs works as expected. And the item is
visible in KeePassXC. Wow!

> —Liam

Best regards, Michael.

Re: Emacs Secret Service integration and KeePassXC issues

[Liam Hupfer via Users list for the GNU Emacs text editor]

Hi Michael,

Glad to see you were able to reproduce the issues!

Michael Albinus <michael.albinus@gmx.de> writes:

> The ephemeral “session” collection is created by the Secret Service
> provider (GNOME keyring daemon in my tests). KeePassXC doesn’t seem to
> create it. I’ve scanned the docs whether this temporary collection is
> requested (as I assume), but I couldn’t find it. So maybe we shouldn’t
> trust its existence. I’ve added a respective note to the auth info
> pages, where the Secret Service API is described.

Makes sense, thanks. This is the conclusion I came to as well (that the
session collection is not part of the spec, but rather a
GNOME Keyring-specific feature).

>> Soon, I noticed another
>> thing: in KeePassXC’s app-wide secret service settings, there is an
>> “Authorization” tab that lists currently connected applications. For
>> some reason, Emacs was listed dozens of times when I opened this. I
>> watched the scroll-bar for the pane and it did shrink and grow when I
>> closed and reopened a session from Emacs, and I noticed that repeatedly
>> opening does in fact return the same session path, so I wasn’t sure how
>> that happened. But then I figured out how to try running the code in the
>> aforementioned `defconst’, and I realized that it indeed opens a session
>> and attempts to create a dummy item to dynamically get the content-type.
>> However, because the “session” collection it uses doesn’t exist, it ends
>> up simply opening a session and then failing to create the item with
>> `dbus-call-method: D-Bus error: “No such object path
>> ‘/org/freedesktop/secrets/collection/session’”’. The removal call fails
>> because the result from the creation is nil. All of this is wrapped in
>> `ignore-errors’, so it doesn’t get printed. But I think the result is
>> this somehow gets called repeatedly which creates the many open
>> sessions? Because once I got this far I just tried evaluating the
>> `defconst’ repeatedly and it indeed opened many sessions, and the result
>> remained nil.
>
> All of this shouldn’t happen anymore with my recent change.

I tried your change by downloading the updated `secrets.el' and
executing `(load "/path/to/new/secrets.el")' followed by `(require
'secrets)' in my config. This seems to use the new version, because
creating items works and `secrets-struct-secret-content-type' is set.
However, over a few hours running Emacs and KeePassXC with my
`auth-sources' set appropriately, it appears that many extra sessions
are still opened when I view the aforementioned `Tools > Settings >
Secret Service Integration > Authorization' in KeePassXC. I believe it
is specific to Emacs, because the Nextcloud client and the GNOME Online
Accounts daemon also show up and do not seem to ever create extra
sessions. I’m not really sure how to debug this. I think this would mean
something is calling `secrets-open-session' somewhere, but it only seems
to be called when `secrets.el' is loaded, so I have no idea how it’s
happening. If how I load your updated version has something to do with
it, is there a better way for me to override the `defconst' and `defun'
with your changes than what I’ve described?

> After my change, pushed to Emacs master, your example with
> `secrets-create-item’ in Emacs works as expected. And the item is
> visible in KeePassXC. Wow!

Indeed, excellent! Despite the remaining issue I have with extraneous
sessions, it seems to reliably work regardless of how many of those are
open. Deleting items also works as expected. This has significantly
improved dealing with secrets in Emacs for me. Thanks for taking the
time to go through my report!

—Liam

Re: Emacs Secret Service integration and KeePassXC issues

[Michael Albinus]

Liam Hupfer <liam@hpfr.net> writes:

> Hi Michael,

Hi Liam,

>> The ephemeral “session” collection is created by the Secret Service
>> provider (GNOME keyring daemon in my tests). KeePassXC doesn’t seem to
>> create it. I’ve scanned the docs whether this temporary collection is
>> requested (as I assume), but I couldn’t find it. So maybe we shouldn’t
>> trust its existence. I’ve added a respective note to the auth info
>> pages, where the Secret Service API is described.
>
> Makes sense, thanks. This is the conclusion I came to as well (that the
> session collection is not part of the spec, but rather a
> GNOME Keyring-specific feature).

I vaguely remember that I ran also tests with KDE wallet, and I've seen
the "session" collection there as well. But I cannot say it
bullet-proof, I'll dig further the docs to see where this collection
comes from.

> I tried your change by downloading the updated `secrets.el' and
> executing `(load "/path/to/new/secrets.el")' followed by `(require
> 'secrets)' in my config.

"load" shall do already what you need, no further "require" needed.

The canonical way to fix this is to add "/path/to/new" to your
load-path in your ~/.emacs, like

(add-to-list 'load-path "/path/to/new")

Then the modified version of secrets.el is loaded when needed.

> However, over a few hours running Emacs and KeePassXC with my
> `auth-sources' set appropriately, it appears that many extra sessions
> are still opened when I view the aforementioned `Tools > Settings >
> Secret Service Integration > Authorization' in KeePassXC. I believe it
> is specific to Emacs, because the Nextcloud client and the GNOME Online
> Accounts daemon also show up and do not seem to ever create extra
> sessions. I’m not really sure how to debug this. I think this would mean
> something is calling `secrets-open-session' somewhere, but it only seems
> to be called when `secrets.el' is loaded, so I have no idea how it’s
> happening. If how I load your updated version has something to do with
> it, is there a better way for me to override the `defconst' and `defun'
> with your changes than what I’ve described?

Did you restart a new Emacs session, after adapting the load-path to the
modified secrets.el?

If this still happens, please instrument `secrets-open-session' for
debugging. And when the debugger stops there, produce a backtrace by
typing "d". This might tell us what's up.

My crystal ball tells me that `auth-source-search' is involved.

> —Liam

Best regards, Michael.

Re: Emacs Secret Service integration and KeePassXC issues

[Michael Albinus]

Michael Albinus <michael.albinus@gmx.de> writes:

Hi Liam,

>> However, over a few hours running Emacs and KeePassXC with my
>> `auth-sources' set appropriately, it appears that many extra sessions
>> are still opened when I view the aforementioned `Tools > Settings >
>> Secret Service Integration > Authorization' in KeePassXC. I believe it
>> is specific to Emacs, because the Nextcloud client and the GNOME Online
>> Accounts daemon also show up and do not seem to ever create extra
>> sessions. I’m not really sure how to debug this. I think this would mean
>> something is calling `secrets-open-session' somewhere, but it only seems
>> to be called when `secrets.el' is loaded, so I have no idea how it’s
>> happening. If how I load your updated version has something to do with
>> it, is there a better way for me to override the `defconst' and `defun'
>> with your changes than what I’ve described?
>
> Did you restart a new Emacs session, after adapting the load-path to the
> modified secrets.el?
>
> If this still happens, please instrument `secrets-open-session' for
> debugging. And when the debugger stops there, produce a backtrace by
> typing "d". This might tell us what's up.
>
> My crystal ball tells me that `auth-source-search' is involved.

PS: Set `secrets-debug' to t. Whenever `secrets-open-session' is called,
you'll see a message in the minibuffer. This should give an
approximation, under which constellation it is called.

>> —Liam

Best regards, Michael.