oauth2 support for Emacs email clients

oauth2 support for Emacs email clients

[Roland Winkler]

A year ago, there was a long thread "Making GNUS continue to work
with Gmail".  Has there been any progress along these lines?

I am asking because my institution uses MS Outlook.  Recently, they
have disabled simple password-based authentication in favor of
oauth2.  Now, using oauth2.el from GNU Elpa, I have got the basics
(authentication and authorization) working.  This required some ugly
configuration within MS Azure.  But I am still some distance away
from a smooth workflow, say, using Gnus.  It is my understanding,
that support of oauth2 within the Emacs ecosystem is rather
incomplete.

Say, oauth2.el advises url-http-handle-authentication.  More
importantly, email clients need to regularly refresh the oauth2
access token.  I believe Emacs is not yet suited for this.  All this
brings me to the limits of my understanding of both the oauth2
protocol and the relevant parts of the emacs code base (I believe
this code resides largely in ./lisp/url.)

Has anybody else been looking into this?  Before I spent more time
on this, first I'd like to make sure that I am not reinventing the
wheel or missing something else.

Thanks,

Roland

Re: oauth2 support for Emacs email clients

[Uwe Brauer]

[-- Attachment #1: Type: text/plain, Size: 1188 bytes --]

>>> "RW" == Roland Winkler <winkler@gnu.org> writes:

> A year ago, there was a long thread "Making GNUS continue to work
> with Gmail".  Has there been any progress along these lines?

Not that I know off. Google announced that because of Covid they would
postpone their plans, but it seems like the  «Sword of Damocles»,
because somehow emacs should be then registered as a google approved
app, or something like this.

Since my institution uses Gmail (to save money)  I am very worried about
this issue and would be extremely interested in any progress.

> I am asking because my institution uses MS Outlook.  Recently, they
> have disabled simple password-based authentication in favor of
> oauth2.  Now, using oauth2.el from GNU Elpa, I have got the basics
> (authentication and authorization) working.  This required some ugly
> configuration within MS Azure.  
> Has anybody else been looking into this?  Before I spent more time
> on this, first I'd like to make sure that I am not reinventing the
> wheel or missing something else.

In my understanding no, unfortunately I don't know about about this
protocol and so can only offer my help for testing.

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 5673 bytes --]

Re: oauth2 support for Emacs email clients

[Andrew Cohen]

>>>>> "RW" == Roland Winkler <winkler@gnu.org> writes:

    RW> A year ago, there was a long thread "Making GNUS continue to
    RW> work with Gmail".  Has there been any progress along these
    RW> lines?

I have this working (my institution has required MS Azure
authentication for some time and I had to put this together to continue
using email :)) using oauth2.el 

It is not pretty, and not very user friendly (the setup on the MS and/or
gmail side is pretty horrible but you seem to have already survived this
part).

It requires two minor lisp changes (which I haven't yet pushed to
master): one for imap support of xoauth2, the other for smtpmail support
(see below).

But the truly hacky part---the information for refreshing the token is
stored in an auth-source file, where the "secret" is a function that
refreshes the token and returns the new credential.

Once set up it works pretty well (although it asks for the password for
the auth-source file frequently---I haven't had time to learn how
auth-source decides when a password is required so this should be easily
fixable---its not frequent enough to really bother me and my emacs time
is very limited these days).

If you want to try it let me know and I'll try to walk you through the setup.

diff --git a/lisp/gnus/nnimap.el b/lisp/gnus/nnimap.el
index 3e2a202a6c..3cf65453f3 100644
--- a/lisp/gnus/nnimap.el
+++ b/lisp/gnus/nnimap.el
@@ -599,6 +599,13 @@ nnimap-login
              (eq nnimap-authenticator 'anonymous)
             (eq nnimap-authenticator 'login)))
     (nnimap-command "LOGIN %S %S" user password))
+   ((and (nnimap-capability "AUTH=XOAUTH2")
+         (eq nnimap-authenticator 'xoauth2))
+    (nnimap-command  "AUTHENTICATE XOAUTH2 %s"
+                     (base64-encode-string
+                      (format "user=%s\001auth=Bearer %s\001\001"
+                              (nnimap-quote-specials user)
+                              (nnimap-quote-specials password)))))
    ((and (nnimap-capability "AUTH=CRAM-MD5")
         (or (null nnimap-authenticator)
             (eq nnimap-authenticator 'cram-md5)))
diff --git a/lisp/mail/smtpmail.el b/lisp/mail/smtpmail.el
index c1e2280033..a9c99aaf98 100644
--- a/lisp/mail/smtpmail.el
+++ b/lisp/mail/smtpmail.el
@@ -614,6 +614,14 @@ smtpmail-try-auth-method
           (base64-encode-string (concat "\0" user "\0" password) t))
    235))
 
+(cl-defmethod smtpmail-try-auth-method
+  (process (_mech (eql xoauth2)) user password)
+  (smtpmail-command-or-throw
+   process
+   (concat "AUTH XOAUTH2 "
+           (base64-encode-string
+            (concat "user=" user "\1auth=Bearer " password "\1\1") t))))
+
 (defun smtpmail-response-code (string)
   (when string
     (with-temp-buffer





-- 
Andrew Cohen

Re: oauth2 support for Emacs email clients

[Roland Winkler]

On Tue, Aug 03 2021, Andrew Cohen wrote:
>>>>>> "RW" == Roland Winkler <winkler@gnu.org> writes:
> But the truly hacky part---the information for refreshing the token is
> stored in an auth-source file, where the "secret" is a function that
> refreshes the token and returns the new credential.
>
> Once set up it works pretty well (although it asks for the password for
> the auth-source file frequently---I haven't had time to learn how
> auth-source decides when a password is required so this should be easily
> fixable---its not frequent enough to really bother me and my emacs time
> is very limited these days).

As I said in another reply to this thread, I believe that the problem
with frequent password requests is due to the fact that oauth2.el does
not support `access_type=offline' upon authentication.

> If you want to try it let me know and I'll try to walk you through the
> setup.

Thanks, I got this part working.  But I realized that davmail mentioned
elsewhere in this thread gives me, all in all, a smoother work flow.

> diff --git a/lisp/gnus/nnimap.el b/lisp/gnus/nnimap.el
>
> diff --git a/lisp/mail/smtpmail.el b/lisp/mail/smtpmail.el

I suggest to apply these steps to the emacs trunk, as one step towards
better oauth2 support.

Re: oauth2 support for Emacs email clients

[Andrew Cohen]

>>>>> "RW" == Roland Winkler <winkler@gnu.org> writes:

    RW> On Tue, Aug 03 2021, Andrew Cohen wrote:
    >>>>>>> "RW" == Roland Winkler <winkler@gnu.org> writes:
    >> But the truly hacky part---the information for refreshing the
    >> token is stored in an auth-source file, where the "secret" is a
    >> function that refreshes the token and returns the new credential.
    >> 
    >> Once set up it works pretty well (although it asks for the
    >> password for the auth-source file frequently---I haven't had time
    >> to learn how auth-source decides when a password is required so
    >> this should be easily fixable---its not frequent enough to really
    >> bother me and my emacs time is very limited these days).

    RW> As I said in another reply to this thread, I believe that the
    RW> problem with frequent password requests is due to the fact that
    RW> oauth2.el does not support `access_type=offline' upon
    RW> authentication.

No, this has nothing to do with oauth2---the automatic refreshing works
just fine. Its the auth-store that is asking for a password to read the
local plstore file (for some reason it caches the password for some
relatively short length of time, but then decides it needs the password
after a couple of hours).

Just to be really clear, all of the oauth2 stuff works automatically and
just fine (including for smtp for sending email).

    >> If you want to try it let me know and I'll try to walk you
    >> through the setup.

    RW> Thanks, I got this part working.  But I realized that davmail
    RW> mentioned elsewhere in this thread gives me, all in all, a
    RW> smoother work flow.

    >> diff --git a/lisp/gnus/nnimap.el b/lisp/gnus/nnimap.el
    >> 
    >> diff --git a/lisp/mail/smtpmail.el b/lisp/mail/smtpmail.el

    RW> I suggest to apply these steps to the emacs trunk, as one step
    RW> towards better oauth2 support.

Yes, I have been waiting for a good time to update my local emacs
repository so I can do a clean push. Not much time for emacs these
days. 

-- 

Re: oauth2 support for Emacs email clients

[Andrew Cohen]

>>>>> "AC" == Andrew Cohen <acohen@ust.hk> writes:

>>>>> "RW" == Roland Winkler <winkler@gnu.org> writes:
    RW> On Tue, Aug 03 2021, Andrew Cohen wrote:
    >>>>>>>> "RW" == Roland Winkler <winkler@gnu.org> writes:

[...]

    >>> Once set up it works pretty well (although it asks for the
    >>> password for the auth-source file frequently---I haven't had
    >>> time to learn how auth-source decides when a password is
    >>> required so this should be easily fixable---its not frequent
    >>> enough to really bother me and my emacs time is very limited
    >>> these days).

    RW> As I said in another reply to this thread, I believe that the
    RW> problem with frequent password requests is due to the fact that
    RW> oauth2.el does not support `access_type=offline' upon
    RW> authentication.

    AC> No, this has nothing to do with oauth2---the automatic
    AC> refreshing works just fine. Its the auth-store that is asking
    AC> for a password to read the local plstore file (for some reason
    AC> it caches the password for some relatively short length of time,
    AC> but then decides it needs the password after a couple of hours).

And finally taking the two minutes to read the documentation tells me
that 'auth-source-cache-expiry is, by default, 2 hours:) Set this to nil
and these annoying requests for my password go away.

If anyone is interested, it seems that MS requires re-authentication of
the client every 6 months (this has nothing to do with emacs). This hit
me for the second time just after I sent my message yesterday that
everything was working great. So now I know the exact date (one year ago)
when I started using oauth2 with gnus :)

-- 
Andrew Cohen

Re: oauth2 support for Emacs email clients

[Andreas Schwab]

On Aug 04 2021, Andrew Cohen wrote:

> No, this has nothing to do with oauth2---the automatic refreshing works
> just fine. Its the auth-store that is asking for a password to read the
> local plstore file (for some reason it caches the password for some
> relatively short length of time, but then decides it needs the password
> after a couple of hours).

plstore does no caching itself by default.  For asymmetric encryption it
depends on epg, and you can configure the cache time in gpg-agent.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: oauth2 support for Emacs email clients

[Andrew Cohen]

Ahh, thanks, that was it.

Too many timing variables-the access token has a lifetime of 60 minutes; 
auth-source was caching this for 2 hours; and gpg-agent has a default 
cache time of 2 hours as well.

But a few minutes of configuration and now I don't have to type a 
password too frequently:)

Aug 4, 2021 3:42:02 PM Andreas Schwab <schwab@linux-m68k.org>:

> On Aug 04 2021, Andrew Cohen wrote:
>
>> No, this has nothing to do with oauth2---the automatic refreshing 
>> works
>> just fine. Its the auth-store that is asking for a password to read 
>> the
>> local plstore file (for some reason it caches the password for some
>> relatively short length of time, but then decides it needs the 
>> password
>> after a couple of hours).
>
> plstore does no caching itself by default.  For asymmetric encryption 
> it
> depends on epg, and you can configure the cache time in gpg-agent.
>
> Andreas.
>

Re: oauth2 support for Emacs email clients

[Lars Ingebrigtsen]

Andrew Cohen <acohen@ust.hk> writes:

> It requires two minor lisp changes (which I haven't yet pushed to
> master): one for imap support of xoauth2, the other for smtpmail support
> (see below).

The patch looks good to me -- I can push it to Emacs 28 if you want?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: oauth2 support for Emacs email clients

[Andrew Cohen]

>>>>> "LI" == Lars Ingebrigtsen <larsi@gnus.org> writes:

    LI> Andrew Cohen <acohen@ust.hk> writes:
    >> It requires two minor lisp changes (which I haven't yet pushed to
    >> master): one for imap support of xoauth2, the other for smtpmail
    >> support (see below).

    LI> The patch looks good to me -- I can push it to Emacs 28 if you
    LI> want?

Yes please.

Ideally this should come with some documentation updates:) For imap
usage the relevant paragraph in "Customizing the IMAP Connection" should
read:

‘nnimap-authenticator’
     Some IMAP servers allow anonymous logins.  In that case, this
     should be set to ‘anonymous’.  If this variable isn’t set, the
     normal login methods will be used.  If you wish to specify a
     specific login method to be used, you can set this variable to
     either ‘login’ (the traditional IMAP login method), ‘plain’, 
     ‘cram-md5’, or ‘xoauth2’ (to use oauth2). 

For the smtp usage the last paragraph of the SMTP authentication section
needs updating. Actually, I think it is currently mistaken (it says that
it tries various authentication methods in order, but as I recall if one
method fails it doesn't fall through to others). So ideally this whole
paragraph would get rewritten. But a temporary fix is 


   The process by which the SMTP library authenticates you to the server
is known as “Simple Authentication and Security Layer” (SASL). There are
various SASL mechanisms, and this library supports four of them:
CRAM-MD5, PLAIN, LOGIN, and XOAUTH2 where the first uses a form of
encryption to obscure your password, the next two do not, and the fourth
uses the oauth2 protocol.  It tries each of them, in that order, until
one succeeds.  You can override this by assigning a specific
authentication mechanism to a server by including a key ‘smtp-auth’ with
the value of your preferred mechanism in the appropriate ‘~/.authinfo’
entry.

Thanks,
Andy

-- 

Re: oauth2 support for Emacs email clients

[Lars Ingebrigtsen]

Andrew Cohen <acohen@ust.hk> writes:

> Yes please.

Now done, including the doc updates suggested.

> For the smtp usage the last paragraph of the SMTP authentication section
> needs updating. Actually, I think it is currently mistaken (it says that
> it tries various authentication methods in order, but as I recall if one
> method fails it doesn't fall through to others). So ideally this whole
> paragraph would get rewritten.

I think that probably the smtpmail code should be fixed to match the
documentation instead, really.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: oauth2 support for Emacs email clients

[Gregory Heytings]

>
> A year ago, there was a long thread "Making GNUS continue to work with 
> Gmail".  Has there been any progress along these lines?
>

As far as I know, no.

>
> I am asking because my institution uses MS Outlook.  Recently, they have 
> disabled simple password-based authentication in favor of oauth2.  Now, 
> using oauth2.el from GNU Elpa, I have got the basics (authentication and 
> authorization) working.  This required some ugly configuration within MS 
> Azure.  But I am still some distance away from a smooth workflow, say, 
> using Gnus.  It is my understanding, that support of oauth2 within the 
> Emacs ecosystem is rather incomplete.
>

It is not incomplete, it works, but, as you said, it does not work 
"smoothly" because each users has to do some initial "ugly configuration" 
with Microsoft Azure or Google Cloud or...

A smooth workflow without that "ugly configuration" would require to 
register Gnus as an official / approved email client with each email 
provider, which as far as I can see is not going to happen.  For Google, 
even if Gnus were approved as an approved email client, it would not be 
possible to use the OAuth credentials obtained at the end of the approval 
process in Gnus, because doing this is explicitly forbidden by their TOS ( 
https://developers.google.com/terms ) 4.b.1: "You will keep your 
credentials confidential and make reasonable efforts to prevent and 
discourage other API Clients from using your credentials. Developer 
credentials may not be embedded in open source projects."  I did not check 
what the TOS of Microsoft are, tho.

Someone might agree to take the legal risk to violate these TOS.  IANAL, 
but I observe that no other "small" free software project (e.g. Mutt or 
Alpine) took that risk.  Two larger free software projects (Thunderbird 
and Kmail) took that risk, but their apps were registered by a legal 
person, not by a developer.

>
> Say, oauth2.el advises url-http-handle-authentication.  More 
> importantly, email clients need to regularly refresh the oauth2 access 
> token.
>

oauth2.el refreshes tokens automatically, so once the initial ugly 
configuration is done, everything should work smoothly.

Re: oauth2 support for Emacs email clients

[Roland Winkler]

On Tue Aug 3 2021 Gregory Heytings wrote:
> > It is my understanding, that support of oauth2 within the Emacs
> > ecosystem is rather incomplete.
> 
> It is not incomplete, 
>
[snip]
>
> oauth2.el refreshes tokens automatically, so once the initial ugly 
> configuration is done, everything should work smoothly.

Upon authentication, oauth2.el does not support `access_type=offline'.
It is my understanding that this is needed to get upon authorization
not only an access token, but also a refresh token.  The latter is
needed to grant an app continuous access beyond the short lifetime
of an access token.  However, the emacs url machinery also lacks
the code that is needed to use the referesh token in order to
request a new access token if the current access token has expired.
This goes beyond the advice of url-http-handle-authentication that
is currently used by oauth2.el.

Let me know if I misunderstand the oauth2 protocol.

Re: oauth2 support for Emacs email clients

[Gregory Heytings]

>
> Upon authentication, oauth2.el does not support `access_type=offline'. 
> It is my understanding that this is needed to get upon authorization not 
> only an access token, but also a refresh token.  The latter is needed to 
> grant an app continuous access beyond the short lifetime of an access 
> token.
>

Your understanding is correct, except that AFAIK it is not "needed", at 
least not in general.  But of course as you might guess each provider 
implemented their own variant / extensions of the OAuth protocol.  If you 
need these parameters in your case, you can add them in 
oauth2-request-authorization.

>
> However, the emacs url machinery also lacks the code that is needed to 
> use the referesh token in order to request a new access token if the 
> current access token has expired.
>

This should work AFAIU; I don't use oauth2.el myself, but a comment says 
"If the token needs to be refreshed, the code handles it automatically and 
store the new value of the access token", and that refreshing mechanism is 
an essential part of OAuth, if it is not present OAuth is not usable.

Re: oauth2 support for Emacs email clients

[Andrew Cohen]

>>>>> "RW" == Roland Winkler <winkler@gnu.org> writes:

    RW> On Tue Aug 3 2021 Gregory Heytings wrote:
    >> > It is my understanding, that support of oauth2 within the Emacs
    >> > ecosystem is rather incomplete.
    >> 
    >> It is not incomplete,
    >> 
    RW> [snip]
    >> 
    >> oauth2.el refreshes tokens automatically, so once the initial
    >> ugly configuration is done, everything should work smoothly.

    RW> Upon authentication, oauth2.el does not support
    RW> `access_type=offline'.  It is my understanding that this is
    RW> needed to get upon authorization not only an access token, but
    RW> also a refresh token.  The latter is needed to grant an app
    RW> continuous access beyond the short lifetime of an access token.
    RW> However, the emacs url machinery also lacks the code that is
    RW> needed to use the referesh token in order to request a new
    RW> access token if the current access token has expired.  This goes
    RW> beyond the advice of url-http-handle-authentication that is
    RW> currently used by oauth2.el.

I don't think this is quite right. Emacs can use the refresh token to
obtain the access token just fine (this is what I have been doing for
the past year).

As others have also mentioned, once the initial setup and authorization
has been done (i.e. registering and authorizing the client) emacs can
integrate the use of oauth2 in a way that is essentially
indistinguishable from any other password protected imap or stmp
server. I have the token information stored in auth-source storage, and
with the soon-to-be-pushed changes that I posted, the access token is
properly (and automatically) refreshed when needed (roughly every hour
given the MS expiration time).

Skip the following details if you aren't interested:

A plstore contains an entry for the oauth2-protected server (imap or
smtp), including the usual :host, :port, :user keys. The entry also
includes an oauth2 token structure with key :token. Finally for the
:secret (i.e. password) I use the name of a function that gets and
returns the access token (early on when I set this up I pushed a minor
modification to auth-source to allow functions to be called for the
secret rather than just holding the password).

Here is the function I use to return the refreshed access-token:

(defun gnus-refresh-access (plist)
    (let ((token (plist-get plist :token)))
      (oauth2-token-access-token
       (oauth2-refresh-access token))))

And then I configure the server (the nnimap-authenticator server
parameter or the smtp-auth parameter  'xoauth2) and that's all. As
mentioned in another post the plstore is encrypted and requires a
password to decrypt (as Andreas pointed out the gpg-agent controls
caching of this password).

Sorry for being long-winded---I make it sound complicated but it is very
straightforward and as I said it functions the same as any other
auth-source password protected server.

Best,
A.


-- 
Andrew Cohen

Re: oauth2 support for Emacs email clients

[Roland Winkler]

On Thu, Aug 05 2021, Andrew Cohen wrote:
> I don't think this is quite right. Emacs can use the refresh token to
> obtain the access token just fine (this is what I have been doing for
> the past year).

I just want to confirm I have read your posts with great interest, thank
you.  I am still in the process of digesting them.  These topics
(including emacs tools like plstore / auth-store / auth-source) have not
been my area of expertise.  So I am still trying to understand better
how to put these things together so that I get what I want / need.

In the meanwhile one question:

With your setup, what happens if you need to restart emacs?  Are you
storing the oauth2 tokens on disk so that you need not re-authenticate
(as long as the tokens themselves remain valid, which is 30 days for my
organizations)?

Thanks again!

Roland

Re: oauth2 support for Emacs email clients

[Andrew Cohen]

>>>>> "RW" == Roland Winkler <winkler@gnu.org> writes:


[...]


    RW> I just want to confirm I have read your posts with great
    RW> interest, thank you.  I am still in the process of digesting
    RW> them.  These topics (including emacs tools like plstore /
    RW> auth-store / auth-source) have not been my area of expertise.
    RW> So I am still trying to understand better how to put these
    RW> things together so that I get what I want / need.

I hope I can provide useful info. (I keep accidentally typing
"auth-store" instead of "auth-source" so I think that's one less topic to
worry about :))

    RW> In the meanwhile one question:

    RW> With your setup, what happens if you need to restart emacs?  Are
    RW> you storing the oauth2 tokens on disk so that you need not
    RW> re-authenticate (as long as the tokens themselves remain valid,
    RW> which is 30 days for my organizations)?

Yes, the oauth2 credentials are stored on disk. Everything works as
usual upon restarting emacs: gnus uses auth-source to retrieve the
appropriate credentials from the on-disk storage and the existing
refresh-token is used to automatically obtain a new access-token (a
password is requested to decrypt the on-disk storage if the cached value
has timed out, but otherwise no action by the user is needed).

auth-source is a great way to store and retrieve credentials for
accessing services (and other things too). It is used by gnus, smtpmail
(and I think rmail) and probably lots of other service-based things.

A typical entry includes host, user, port, secret (i.e. password) and
whatever other things you like. auth-source has several different
storage backends (netrc, plstore, json, secrets, and several more) and a
simple search mechanism that spans different backends to retrieve
appropriate authentication data.

For my oauth2 imap and smtp access I simply created appropriate entries
(each of which includes a token structure as defined in oauth2.el)
using the plstore backend (so the entry is stored on disk with the
sensitive parts encrypted).

Aside from the slightly funky structure of the auth-store entry
(including an ouath2 token) this works exactly as any other auth-store
entry to using imap or smtp---that was the reason for setting it up this
way, so it looked like any other SASL to the user and didn't require a
new package (just some minor tweaks to existing code).


Aside: I have another minor change to push to auth-source to make this
work. The plstore backend in auth-source only encrypts the :secret
entry, but I want to keep the client-secret and the oauth2 token
structure in the entry encrypted as well. This required some minor
tweaking to  auth-source to allow specifying which parts of the entry
are encrypted and which are unencrypted. Patch below (which I will push
myself in awhile). 

diff --git a/lisp/auth-source.el b/lisp/auth-source.el
index 6919738398..faddbdee1a 100644
--- a/lisp/auth-source.el
+++ b/lisp/auth-source.el
@@ -2120,11 +2120,17 @@ auth-source-plstore-create
          (base-secret '(secret))
          ;; we know (because of an assertion in auth-source-search) that the
          ;; :create parameter is either t or a list (which includes nil)
-         (create-extra (if (eq t create) nil create))
+        (create-secret-extra (plist-get create :encrypted))
+         (create-extra (if (eq t create) nil
+                        (or
+                         (append
+                          (plist-get create :unencrypted) create-secret-extra)
+                            create)))
          (current-data (car (auth-source-search :max 1
                                                 :host host
                                                 :port port)))
          (required (append base-required create-extra))
+        (required-secret (append base-secret create-secret-extra))
          ;; `valist' is an alist
          valist
          ;; `artificial' will be returned if no creation is needed


Best,
Andy


-- 
Andrew Cohen

Director, HKUST Jockey Club Institute for Advanced Study
Lam Woo Foundation Professor and Chair Professor of Physics

The Hong Kong University of Science and Technology

Re: oauth2 support for Emacs email clients

[Andrew Cohen]

Not that it matters much (I'll be pushing this at some point), but the
patch in my previous email was cutoff. Complete patch below:

diff --git a/lisp/auth-source.el b/lisp/auth-source.el
index 6919738398..faddbdee1a 100644
--- a/lisp/auth-source.el
+++ b/lisp/auth-source.el
@@ -2120,11 +2120,17 @@ auth-source-plstore-create
          (base-secret '(secret))
          ;; we know (because of an assertion in auth-source-search) that the
          ;; :create parameter is either t or a list (which includes nil)
-         (create-extra (if (eq t create) nil create))
+        (create-secret-extra (plist-get create :encrypted))
+         (create-extra (if (eq t create) nil
+                        (or
+                         (append
+                          (plist-get create :unencrypted) create-secret-extra)
+                            create)))
          (current-data (car (auth-source-search :max 1
                                                 :host host
                                                 :port port)))
          (required (append base-required create-extra))
+        (required-secret (append base-secret create-secret-extra))
          ;; `valist' is an alist
          valist
          ;; `artificial' will be returned if no creation is needed
@@ -2213,7 +2219,7 @@ auth-source-plstore-create
                            (eval default)))))
 
         (when data
-          (if (member r base-secret)
+          (if (member r required-secret)
               (setq secret-artificial
                     (plist-put secret-artificial
                                (auth-source--symbol-keyword r)


-- 

Re: oauth2 support for Emacs email clients

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Could you please post a summary of what part of the overall problem
is solved by your changes, with auth-source?

Which services can users use, in which circumstances -- and what
conditions does this depend on?


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: oauth2 support for Emacs email clients

[Andrew Cohen]

>>>>> "RS" == Richard Stallman <rms@gnu.org> writes:

    RS> [[[ To any NSA and FBI agents reading my email: please consider
    RS> ]]] [[[ whether defending the US Constitution against all
    RS> enemies, ]]] [[[ foreign or domestic, requires you to follow
    RS> Snowden's example. ]]]

    RS> Could you please post a summary of what part of the overall
    RS> problem is solved by your changes, with auth-source?

    RS> Which services can users use, in which circumstances -- and what
    RS> conditions does this depend on?

The changes to auth-source have nothing to do with oauth2 nor do they
involve any particular service. Plstore is code that allows storing
plists on disk with arbitrary parts of the list stored in plaintext and
other parts stored in encrypted form. The current plstore backend of the
auth-source code however only allows the :secret to be encrypted but
nothing else. The code change I am suggesting enhances the auth-source
plstore backend to allow other parts of the entry to be encrypted as
well, according to a user specification. The default is unchanged (the
:secret entry is encrypted but nothing else), while passing a :create
key to auth-source-search allows selecting exactly which parts should be
stored encrypted and which parts should be stored unencrypted in the
on-disk storage.

If/when I push this there will be a simple documentation update to make
this clear.

Why did this come up in the context of oauth2? I am using the plstore
backend of auth-source to store my oauth2 credentials (this is a user
choice: other backends, like the netrc backend, work fine as well) and
wanted the oauth2 security tokens that are included in my plstore
authentication entry to be stored on disk in an encrypted form. This
change makes that slightly more convenient to do (it can be already be
done by invoking plstore directly, but I find it convenient to use
auth-source to manage my service authentication entries).

-- 

Re: oauth2 support for Emacs email clients

[Arthur Miller]

Gregory Heytings <gregory@heytings.org> writes:

> Someone might agree to take the legal risk to violate these TOS.  IANAL, but I
> observe that no other "small" free software project (e.g. Mutt or Alpine) took
> that risk.  Two larger free software projects (Thunderbird and Kmail) took that
> risk, but their apps were registered by a legal person, not by a developer.

There you have something, isn't FSF a legal person? Is GNU an organisation right?

Re: oauth2 support for Emacs email clients

[Gregory Heytings]

>> Someone might agree to take the legal risk to violate these TOS. 
>> IANAL, but I observe that no other "small" free software project (e.g. 
>> Mutt or Alpine) took that risk.  Two larger free software projects 
>> (Thunderbird and Kmail) took that risk, but their apps were registered 
>> by a legal person, not by a developer.
>
> There you have something, isn't FSF a legal person? Is GNU an 
> organisation right?
>

It is indeed.  If the lawyers of the FSF agree to do that, I would (of 
course) have no objection.

Re: oauth2 support for Emacs email clients

[Eric Abrahamsen]

Gregory Heytings <gregory@heytings.org> writes:

>>> Someone might agree to take the legal risk to violate these TOS.
>>> IANAL, but I observe that no other "small" free software project
>>> (e.g. Mutt or Alpine) took that risk.  Two larger free software
>>> projects (Thunderbird and Kmail) took that risk, but their apps
>>> were registered by a legal person, not by a developer.
>>
>> There you have something, isn't FSF a legal person? Is GNU an
>> organisation right?
>>
>
> It is indeed.  If the lawyers of the FSF agree to do that, I would (of
> course) have no objection.

But even so, that would only get us to the same very tenuous legal
position that Thunderbird and Kmail are in now, right?

Re: oauth2 support for Emacs email clients

[Gregory Heytings]

>>>> Someone might agree to take the legal risk to violate these TOS. 
>>>> IANAL, but I observe that no other "small" free software project 
>>>> (e.g. Mutt or Alpine) took that risk.  Two larger free software 
>>>> projects (Thunderbird and Kmail) took that risk, but their apps were 
>>>> registered by a legal person, not by a developer.
>>>
>>> There you have something, isn't FSF a legal person? Is GNU an 
>>> organisation right?
>>
>> It is indeed.  If the lawyers of the FSF agree to do that, I would (of 
>> course) have no objection.
>
> But even so, that would only get us to the same very tenuous legal 
> position that Thunderbird and Kmail are in now, right?
>

AFAIU, yes, but again IANAL, and the lawyers of the FSF might have another 
opinion.

Re: oauth2 support for Emacs email clients

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > >> Someone might agree to take the legal risk to violate these TOS. 
  > >> IANAL, but I observe that no other "small" free software project (e.g. 
  > >> Mutt or Alpine) took that risk.  Two larger free software projects 
  > >> (Thunderbird and Kmail) took that risk, but their apps were registered 
  > >> by a legal person, not by a developer.
  > >
  > > There you have something, isn't FSF a legal person? Is GNU an 
  > > organisation right?

Can someoe point me at the _precise_ agreement text for this?
Then the FSF could actually study the question.

Part of what we would have to think about is whether agreeing would
enable us to fix the problem by including these tokens in the Emacs release.
If we didn't include them, the agreement would (as far as I can see)
do no good.

(BTW, I think every human being is a "legal person" too.)

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: oauth2 support for Emacs email clients

[Gregory Heytings]

[-- Attachment #1: Type: text/plain, Size: 1626 bytes --]


>
> Can someoe point me at the _precise_ agreement text for this? Then the 
> FSF could actually study the question.
>
> Part of what we would have to think about is whether agreeing would 
> enable us to fix the problem by including these tokens in the Emacs 
> release. If we didn't include them, the agreement would (as far as I can 
> see) do no good.
>

Sure.  Google's TOS are available here : 
https://developers.google.com/terms (accessible without JavaScript). 
Just in case, I attach a PDF version.

The two points that are relevant to this discussion are :

4.a.1: "When using the APIs, you may not (or allow those acting on your 
behalf to) [s]ublicense an API for use by a third party. Consequently, you 
will not create an API Client that functions substantially the same as the 
APIs and offer it for use by third parties."

4.b.1: "You will keep your credentials confidential and make reasonable 
efforts to prevent and discourage other API Clients from using your 
credentials. Developer credentials may not be embedded in open source 
projects."

Point 4.a.1 prohibits your idea of "tak[ing] a part of Thunderbird and 
modify it into a mail-fetcher program that produces an inbox file, for use 
in place of GNU movemail".

Point 4.b.1 prohibits the inclusion of application credentials in free 
software projects.  Because of this each user has to create their own 
application credentials, which makes the authentication process much more 
complicated than with already registered applications, in which the 
authentication process only requires one to enter credentials (login and 
password) in a browser.

[-- Attachment #2: Type: application/pdf, Size: 56612 bytes --]

Re: oauth2 support for Emacs email clients

[Ulrich Mueller]

>>>>> On Sat, 14 Aug 2021, Gregory Heytings wrote:

> 4.b.1: "You will keep your credentials confidential and make
> reasonable efforts to prevent and discourage other API Clients from
> using your credentials. Developer credentials may not be embedded in
> open source projects."

Good that Emacs isn't open source but free software. :)

Re: Re: oauth2 support for Emacs email clients

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > Can someoe point me at the _precise_ agreement text for this? Then the 
  > > FSF could actually study the question.

Thanks for sending them.

  > 4.a.1: "When using the APIs, you may not (or allow those acting on your 
  > behalf to) [s]ublicense an API for use by a third party. Consequently, you 
  > will not create an API Client that functions substantially the same as the 
  > APIs and offer it for use by third parties."

I am not sure concretely what this is talking about.  Does the FSF
ever do anything remotely like "sublicense an API"?  I don't think so,
but I can't be really sure until I know what it would mean.

Also, what is "an API Client that functions substantially the same as
the APIs"?  You seem to believe that movemail fits those words, but I
doubt it.  I don't know what that API can do, but I would expect it
includes a lot more funtionality than just "download all the incoming
mail as an inbox file and delete it on the server."

However, supposing that extending a program to "download all the
incoming mail as an inbox file and delete it on the server" did fall
under that description, I see nothing to stop us from doing it.  Any
free license permits adding features, and some of us have never agreed
to these Google terms.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Making your own application credentials as a user

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]


  >   Because of this each user has to create their own 
  > application credentials,

Has anyone here actually done this?
If so, could you please post a practical description of what you had to do?
How long did it take you to do that?

Did it impose any additional onerous conditions (beyond the conditions for
using Gmail, since by assumption you've accepted those).


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Making your own application credentials as a user

[Tim Cross]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>
>   >   Because of this each user has to create their own 
>   > application credentials,
>
> Has anyone here actually done this?
> If so, could you please post a practical description of what you had to do?
> How long did it take you to do that?
>
> Did it impose any additional onerous conditions (beyond the conditions for
> using Gmail, since by assumption you've accepted those).

Yes, I did this a while back. It is a pain, but not terribly onerous.
Basically, you have to register as a developer and agree to Google's
T&C. However, in my case, as I have no plans to actually develop any
application to be released and am only registering to get the developer
ID in order to use it as part of my personal workflow to get the
necessary oauth2 tokens to allow access to IMAP and SMTP, those T&C are
largely irrelevant. I'm not using it now - mainly because when I did it,
there were other issues - main one being I use mbsync to access the IMAP
server and I had problems getting that working correctly. So for now,
I'm still using the old application password facility. 

I think the key point here is that requiring all users who want to
access  an oauth based email provider to register as a developer in
order to use Emacs to access the mail service is too much of a barrier,
especially for less technical users. People are more likely to change
their email client rather than jump through those hoops.

It should be noted that users do jump through very similar hoops for
other oauth based Emacs packages. For example the spotify, slack,
stakcOverflow, forge and paradox packages all require the user visit a
web site to create a token or get an application ID. In some of those cases, you also have to
register as a developer (spotify and possibly slack - I can't remember).

Of course we also need to note that if Emacs was able to get registered
as an approved app with Google, then uses would not need to register as
a developer. However, this only addresses one mail provider. It does
nothing to improve the workflow for other oauth based service providers.
How far is Emacs prepared to bend over in order to enable users to use
Gmail?

Re: oauth2 support for Emacs email clients

[Eric S Fraga]

On Tuesday,  3 Aug 2021 at 00:00, Roland Winkler wrote:
> I am asking because my institution uses MS Outlook.  Recently, they
> have disabled simple password-based authentication in favor of
> oauth2.  Now, using oauth2.el from GNU Elpa, I have got the basics
> (authentication and authorization) working.  This required some ugly
> configuration within MS Azure.  But I am still some distance away
> from a smooth workflow, say, using Gnus.  

My institution did the same.  I use gnus.  The easiest solution, in the
end, for me was to install and run davmail to get my emails from the
institution.  I then use pop (could have used imap but I prefer pop in
any case) to get email from "localhost" running davmail.  Davmail takes
care of multi-factor authentication for me.

-- 
Eric S Fraga via Emacs 28.0.50 & org 9.4.6 on Debian 11.0

Re: oauth2 support for Emacs email clients

[Tim Cross]

Eric S Fraga <e.fraga@ucl.ac.uk> writes:

> On Tuesday,  3 Aug 2021 at 00:00, Roland Winkler wrote:
>> I am asking because my institution uses MS Outlook.  Recently, they
>> have disabled simple password-based authentication in favor of
>> oauth2.  Now, using oauth2.el from GNU Elpa, I have got the basics
>> (authentication and authorization) working.  This required some ugly
>> configuration within MS Azure.  But I am still some distance away
>> from a smooth workflow, say, using Gnus.  
>
> My institution did the same.  I use gnus.  The easiest solution, in the
> end, for me was to install and run davmail to get my emails from the
> institution.  I then use pop (could have used imap but I prefer pop in
> any case) to get email from "localhost" running davmail.  Davmail takes
> care of multi-factor authentication for me.

Eric, what do you do for sending mail? As I understand it, oauth2 will
be required for authentication for smtp as well. While I know davmail
will get the messages, does it also send/relay them to the gmail smtp
server?

I also wonder if the 'ban' on putting credentials into the source
(public) is that 'clear cut'. From what I've read, the 'applicaiton
key', was never supposed to be secret - this was apparently an oversight
in the initial oauth specs - obviously other parts of the credentials
do need to be secret. (I do wonder if you can actually get the
application key from registered apps by just running 'strings' over the
binaries!). Of course, the chance of getting a decision from the right
person at either google or MS is next to zero, so I guess we are stuck. 

I guess in the end, all we can really do is try to find a way of
streamlining the process to get a developer key for each user as this
seems to be the main barrier to a more straight-forward setup. I have
had to jump through those hoops with other oauth2 systems which have an
emacs client. The good news is that once you have that key, the
oauth2.el library seems to take care of renewal of session tokens, so
once setup, things should just work.

Re: oauth2 support for Emacs email clients

[Gregory Heytings]

>
> I also wonder if the 'ban' on putting credentials into the source 
> (public) is that 'clear cut'.
>

Again, IANAL, but I at least would never take the risk to deliberately and 
publicly violate the terms of a contract I signed with Google or Microsoft 
(or, for that matter, with anyone else).

>
> From what I've read, the 'applicaiton key', was never supposed to be 
> secret - this was apparently an oversight in the initial oauth specs
>

It is indeed "security through obscurity".  But it is (a kind of) security 
nonetheless.  The application key is used by the provider to identify the 
application that requests access to the resources (in this case emails). 
If Mr. Black Hat copies the application key of (say) Gnus in his malware 
(which he obviously did not submit for approval to Google), its users will 
see an approval screen "Do you allow Gnus to access your emails?".  If Mr. 
Black Hat's names its application "Gnus", its users will believe its 
application is a legitimate and approved one, and will click "OK".

>
> Of course, the chance of getting a decision from the right person at 
> either google or MS is next to zero, so I guess we are stuck.
>

Indeed.

>
> I guess in the end, all we can really do is try to find a way of 
> streamlining the process to get a developer key for each user as this 
> seems to be the main barrier to a more straight-forward setup.
>

I fear that's not possible either, each email provider has their own 
process to create an application key, which they adapt from time to time 
(at least from a user experience viewpoint).

>
> The good news is that once you have that key, the oauth2.el library 
> seems to take care of renewal of session tokens, so once setup, things 
> should just work.
>

Indeed.

Re: oauth2 support for Emacs email clients

[tomas]

[-- Attachment #1: Type: text/plain, Size: 454 bytes --]

On Tue, Aug 03, 2021 at 12:55:45PM +0000, Gregory Heytings wrote:

> >From what I've read, the 'applicaiton key', was never supposed to
> >be secret - this was apparently an oversight in the initial oauth
> >specs
> >
> 
> It is indeed "security through obscurity".  But it is (a kind of)
> security nonetheless [...]

I can't help it. To me it smells of some semi-hidden variant of
"market control".

But I might be wrong.

Cheers
 - t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: oauth2 support for Emacs email clients

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > It is indeed "security through obscurity".  But it is (a kind of)
  > > security nonetheless [...]

  > I can't help it. To me it smells of some semi-hidden variant of
  > "market control".

We can't tell what "their" intentions were, and indeed, those that
approved this spec may have had different intentions.  The companies
that implement the spec may have different intentions now.  And each
company may change its intentions in 2022 or 2023.

Thus. I think speculating about intentions is unprofitable.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: oauth2 support for Emacs email clients

[Eric S Fraga]

On Tuesday,  3 Aug 2021 at 21:17, Tim Cross wrote:
> Eric, what do you do for sending mail? As I understand it, oauth2 will
> be required for authentication for smtp as well. While I know davmail
> will get the messages, does it also send/relay them to the gmail smtp
> server?

I have avoided worrying about this aspect as, at least in our
institution, SMTP does not yet require oauth2.  When this changes, I
will start panicking...

-- 
Eric S Fraga via Emacs 28.0.50 & org 9.4.6 on Debian 11.0

Re: oauth2 support for Emacs email clients

[Roland Winkler]

On Tue, Aug 03 2021, Eric S Fraga wrote:
> On Tuesday,  3 Aug 2021 at 21:17, Tim Cross wrote:
>> Eric, what do you do for sending mail? As I understand it, oauth2 will
>> be required for authentication for smtp as well. While I know davmail
>> will get the messages, does it also send/relay them to the gmail smtp
>> server?
>
> I have avoided worrying about this aspect as, at least in our
> institution, SMTP does not yet require oauth2.  When this changes, I
> will start panicking...

Am I missing something about davmail?  My debian 10 came with davmail
5.1.  For sending mail this seems to take care of oauth2 which, I believe,
is required by my institution.  (But I am still learning things.  Maybe
I am overlooking something...)

Re: oauth2 support for Emacs email clients

[Eric S Fraga]

On Tuesday,  3 Aug 2021 at 14:45, Roland Winkler wrote:
> Am I missing something about davmail?  My debian 10 came with davmail
> 5.1.  For sending mail this seems to take care of oauth2 which, I believe,
> is required by my institution.  (But I am still learning things.  Maybe
> I am overlooking something...)

No, sorry, I am not suggesting that davmail doesn't handle SMTP.  I
don't know one way or another.  The point I was making is that I haven't
needed to solve the problem of oauth2 and SMTP to date.

If davmail supports SMTP & oauth2, excellent!  Makes me less worried
about my setup breaking in the future.

-- 
Eric S Fraga via Emacs 28.0.50 & org 9.4.6 on Debian 11.0

Re: oauth2 support for Emacs email clients

[Roland Winkler]

On Tue, Aug 03 2021, Eric S Fraga wrote:
> The easiest solution, in the end, for me was to install and run
> davmail to get my emails from the institution.  I then use pop (could
> have used imap but I prefer pop in any case) to get email from
> "localhost" running davmail.  Davmail takes care of multi-factor
> authentication for me.

Thanks.  I tried this.  Indeed, this seems to work quite smoothly.

Re: oauth2 support for Emacs email clients

[Eric S Fraga]

On Tuesday,  3 Aug 2021 at 14:41, Roland Winkler wrote:
> On Tue, Aug 03 2021, Eric S Fraga wrote:
>> The easiest solution, in the end, for me was to install and run
>> davmail to get my emails from the institution.  I then use pop (could
>
> Thanks.  I tried this.  Indeed, this seems to work quite smoothly.

Excellent!  And thanks for the other post re: SMTP.

-- 
Eric S Fraga via Emacs 28.0.50 & org 9.4.6 on Debian 11.0

Re: oauth2 support for Emacs email clients

[Thomas Fitzsimmons]

Hi Roland,

Roland Winkler <winkler@gnu.org> writes:

> On Tue, Aug 03 2021, Eric S Fraga wrote:
>> The easiest solution, in the end, for me was to install and run
>> davmail to get my emails from the institution.  I then use pop (could
>> have used imap but I prefer pop in any case) to get email from
>> "localhost" running davmail.  Davmail takes care of multi-factor
>> authentication for me.
>
> Thanks.  I tried this.  Indeed, this seems to work quite smoothly.

DavMail is a good fallback, but from a Free Software perspective it's
also helpful to use/test/mention the native IMAP/SMTP support in Emacs.
DavMail speaks a proprietary protocol to the email server.  IT
departments should configure the email server to also provide support
for IETF-standard email protocols.

IMAP and SMTP can use OAuth2 for authentication.  But to properly
support OAuth2 in Emacs, the FSF needs to talk to big email providers
like Microsoft and Google and get Emacs registered as an OAuth2
application (like Thunderbird has done), or advocate for some better
solution than embedding keys in the application binaries.  This was
discussed in bug 41386 [1]; I hope the FSF is working on it.

Thomas

1. https://debbugs.gnu.org/cgi/bugreport.cgi?bug=41386#104

Re: oauth2 support for Emacs email clients

[Tim Cross]

Thomas Fitzsimmons <fitzsim@fitzsim.org> writes:

> IMAP and SMTP can use OAuth2 for authentication.  But to properly
> support OAuth2 in Emacs, the FSF needs to talk to big email providers
> like Microsoft and Google and get Emacs registered as an OAuth2
> application (like Thunderbird has done), or advocate for some better
> solution than embedding keys in the application binaries.  This was
> discussed in bug 41386 [1]; I hope the FSF is working on it.
>

but isn't this the issue - a 'registered' application is just one which
has a registered application id key, but the T&C for the major mail
providers does not allow that registered ID token to be public, which
means it cannot be put into Emacs source code. This means there is no
practical advantage in Emacs being registered.  

Re: oauth2 support for Emacs email clients

[Thomas Fitzsimmons]

Tim Cross <theophilusx@gmail.com> writes:

> Thomas Fitzsimmons <fitzsim@fitzsim.org> writes:
>
>> IMAP and SMTP can use OAuth2 for authentication.  But to properly
>> support OAuth2 in Emacs, the FSF needs to talk to big email providers
>> like Microsoft and Google and get Emacs registered as an OAuth2
>> application (like Thunderbird has done), or advocate for some better
>> solution than embedding keys in the application binaries.  This was
>> discussed in bug 41386 [1]; I hope the FSF is working on it.
>
> but isn't this the issue - a 'registered' application is just one which
> has a registered application id key, but the T&C for the major mail
> providers does not allow that registered ID token to be public, which
> means it cannot be put into Emacs source code.

Only the FSF, as the copyright holder of Emacs, and with its legal
expertise, is in a position to try to research and solve these issues
and do what may be necessary (e.g., advocating for changes to terms and
conditions that in practice prevent the use of Free Software email
clients for these services, publishing safe usage guidelines if such
advocacy fails, etc.).

Thomas

Re: oauth2 support for Emacs email clients

[Ulrich Mueller]

>>>>> On Thu, 05 Aug 2021, Tim Cross wrote:

> Thomas Fitzsimmons <fitzsim@fitzsim.org> writes:

>> IMAP and SMTP can use OAuth2 for authentication.  But to properly
>> support OAuth2 in Emacs, the FSF needs to talk to big email providers
>> like Microsoft and Google and get Emacs registered as an OAuth2
>> application (like Thunderbird has done), or advocate for some better
>> solution than embedding keys in the application binaries.  This was
>> discussed in bug 41386 [1]; I hope the FSF is working on it.

> but isn't this the issue - a 'registered' application is just one which
> has a registered application id key, but the T&C for the major mail
> providers does not allow that registered ID token to be public, which
> means it cannot be put into Emacs source code. This means there is no
> practical advantage in Emacs being registered.  

Where do proprietary clients store their ID token, if not in the
application itself?

Re: oauth2 support for Emacs email clients

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > DavMail is a good fallback, but from a Free Software perspective it's
  > also helpful to use/test/mention the native IMAP/SMTP support in Emacs.
  > DavMail speaks a proprietary protocol to the email server.

I'd like to make sure there is no miscommunication here.
Could you explain what you mean when you say it is "proprietary"?

Is DevMail free software?  Does it implement that protocol with free
software?


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: oauth2 support for Emacs email clients

[Eric S Fraga]

On Saturday,  7 Aug 2021 at 23:58, Richard Stallman wrote:
> Is DevMail free software?  Does it implement that protocol with free
> software?

GPL (http://davmail.sourceforge.net/licenses.html).
-- 
Eric S Fraga via Emacs 28.0.50 & org 9.4.6 on Debian 11.0

Re: oauth2 support for Emacs email clients

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > Is DevMail free software?  Does it implement that protocol with free
  > > software?

  > GPL (http://davmail.sourceforge.net/licenses.html).

That's good.  (Which GPL versions does it allow?)

So, what part of the problem does DevMail solve?
Which cases is it usable in?

What part of the problem remains to be solved?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: oauth2 support for Emacs email clients

[Roland Winkler]

On Wed, Aug 11 2021, Richard Stallman wrote:
> > > Is DevMail free software?  Does it implement that protocol with
> > > free software?
>
> > GPL (http://davmail.sourceforge.net/licenses.html).
>
> That's good.  (Which GPL versions does it allow?)

davmail is GPL version 2 (no later version).

> So, what part of the problem does DevMail solve?
> Which cases is it usable in?

It is a local gateway for Microsoft Exchange / Office365 so that a local
IMAP / SMTP client such as gnus can communicate with the server.  For
this, davmail performs the OAuth2 authentication and authorization with
the Microsoft server.

davmail supports only Microsoft Exchange / Office365.  Other systems
such as gmail are not supported.

> What part of the problem remains to be solved?

I just confirmed: the Microsoft server requires that javascript is
enabled in the browser.  When I disable javascript in my firefox
browser, it gives me the error message:

  Microsoft
  We can't sign you in
  JavaScript is required to sign in. Your browser either
  does not support JavaScript or it is being blocked.
  Enable JavaScript in your browser or use one which supports it.

  See https://login.microsoftonline.com/jsdisabled

In oauth2.el, this problem is hidden in oauth2-request-authorization,
where browse-url delegates the actual OAuth2 authentication to an
(external) browser.  If oauth2.el is used with Microsoft Exchange /
Office365, it only works with a browser that supports javascript.

(I not know about the implementation of OAuth2 for other systems such as
Gmail.)

Re: oauth2 support for Emacs email clients

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  >   Now, using oauth2.el from GNU Elpa, I have got the basics
  > (authentication and authorization) working.  This required some ugly
  > configuration within MS Azure.  But I am still some distance away
  > from a smooth workflow, say, using Gnus.

Can you tell us more details of
(1) what you did to make this work, and
(2) what problems remain?

(1) will help people now, and (2) will show us what to work on.

  > Say, oauth2.el advises url-http-handle-authentication.

That is one thing we should change.  url-http-handle-authentication
should provide whatever hooks are needed so oauth2.el can do its job
without needing to do advice.

That change will make our code cleaner, but won't directly affect
what it can do and what it can't do.

  >   More
  > importantly, email clients need to regularly refresh the oauth2
  > access token.  I believe Emacs is not yet suited for this.

Can you tell us any more about what "regularly" concretely means?
For instance, is this based on elapsed time?  Actions carried out?
It doesn't sound like something that will be difficult.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: oauth2 support for Emacs email clients

[Roland Winkler]

> > Now, using oauth2.el from GNU Elpa, I have got the basics
> > (authentication and authorization) working.  This required some ugly
> > configuration within MS Azure.  But I am still some distance away
> > from a smooth workflow, say, using Gnus.
> 
> Can you tell us more details of
> (1) what you did to make this work, and

My starting point was oauth2.el in GNU elpa.  Its main entry point is
oauth2-auth that requires several arguments

auth-url:   URL for authentication
token-url:  URL for requesting authorization

client-id
client-secret
redirect-uri:  I got these from MS Azure by registering my "personnal app"
               for communicating with Office365

> (2) what problems remain?

- The first problem has been documentation: both the Office365
  machinery and oauth2.el assume that users know what they need to
  do.  I spent a good amount of time trying to figure out

  - how the OAuth2 protocol works in general

  - how the OAuth2 protocal relates to the particalur problem I
    needed to address

  - what functionality is provided by oauth2.el (it "implements the
    OAuth 2.0 draft")

  - how the general functionality provided by oauth2.el relates to
    the particalur problem I needed to address

  - what remaining steps were not (yet) implemented in relevant
    emacs packages (imap and smtpmail)

- Others in this thread have mentioned davmail.  This is a project
  (published under GPL 2) that works as a local proxy server that
  allows IMAP and SMTP clients to connect to Office365.  So its main
  purpose of existence are essentially the issues we are discussing
  here.  I have tried davmail and it seems to work.

  I am mentioning it here because some discussions on its mailing list
  gave me the impression that the entire topic is a hairy one which does
  not permit a "one size fits all" solutions because, say, institutions
  that use Office365 (i.e., institutions like the one I am working for
  and that's forcing me to go through this pain), they have some freedom
  to configure what OAuth2 means for their users (say: multi-factor
  authentication), and they can also configure to what extent their
  users can configure MS Azure as mentioned above.

  So whatever I am writing here may be more or less irrelvant for
  others, even if they likewise have to use Office365 (not to
  mention Gmail).

  Others have mentioned "officially" registering Emacs as IMAP/SMTP
  clients for Office365 (and possibly Gmail), similar to what seems
  to be the case for Thunderbird.  I am wondering how davmail is
  doing this.

- It is my understanding of the internals of its code that oauth2.el
  effectively delegates authentication to a browser that receives
  the authentication token as a redirect uri.  The last line of code
  in oauth2-request-authorization is:

    (read-string "Enter the code your browser displayed: "))

  This step is awkward, even if it is only rarely needed.

> > More importantly, email clients need to regularly refresh the oauth2
> > access token.  I believe Emacs is not yet suited for this.
> 
> Can you tell us any more about what "regularly" concretely means?
> For instance, is this based on elapsed time?  Actions carried out?
> It doesn't sound like something that will be difficult.

The OAuth2 protocol uses an access token and a refresh token.  The
access token is used to access the server, but it has a shorter
lifetime.  When the access token has expired, it can be refreshed
during a running session.  Others reported in this thread that
oauth2.el supports this (for them), so I stand corrected.

From the code base of oauth2.el I haven't been able to figure out
what happens when the refresh tokens expires.  With my institution,
this happens after 30 days.  So this may be a less pressing issue.
Still it is probably good (and not too difficult) to handle this
gracefully.

Also, I have not yet been able to figure out what happens when a new
emacs session is started.  oauth2.el uses plstore.  Does this work
across multiple emacs sessions, or is it then necessary to go again
through the (more tedious) multi-factor authentication?  Depending
on a user's work flow with emacs and also depending on what
authentication means for this user, that may be an issue or not.
Maybe others can comment here.

And yes, I believe that earlier this week both imap.el and
smtpmail.el have been updated in trunk to support oauth2.  It may
make sense to promote oauth2.el from GNU Elpa to emacs core.

Re: oauth2 support for Emacs email clients

[Andreas Schwab]

On Aug 08 2021, Roland Winkler wrote:

> Also, I have not yet been able to figure out what happens when a new
> emacs session is started.  oauth2.el uses plstore.  Does this work
> across multiple emacs sessions, or is it then necessary to go again
> through the (more tedious) multi-factor authentication?

As long as the refresh token remains valid, oauth2.el can refresh the
auth token automatically with the refresh token.  The refresh token is
all you need to be able to access the IMAP account.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: oauth2 support for Emacs email clients

[Roland Winkler]

On Sun, Aug 08 2021, Andreas Schwab wrote:
> On Aug 08 2021, Roland Winkler wrote:
>
>> Also, I have not yet been able to figure out what happens when a new
>> emacs session is started.  oauth2.el uses plstore.  Does this work
>> across multiple emacs sessions, or is it then necessary to go again
>> through the (more tedious) multi-factor authentication?
>
> As long as the refresh token remains valid, oauth2.el can refresh the
> auth token automatically with the refresh token.  The refresh token is
> all you need to be able to access the IMAP account.

Thanks.  My reading of the code of oauth2.el is that oauth2-compute-id
derives the id for plstore from auth-url, token-url, and scope.  It
seems to me that this will get confused with, say, two different
Office365 accounts that have the same auth-url, token-url, and scope.
Should the code use also additional information such as the user name?
(I do not know much about plstore.el used by oauth2.el.)

Re: oauth2 support for Emacs email clients

[Andrew Cohen]

>>>>> "RW" == Roland Winkler <winkler@gnu.org> writes:

[...]

    RW> Thanks.  My reading of the code of oauth2.el is that
    RW> oauth2-compute-id derives the id for plstore from auth-url,
    RW> token-url, and scope.  It seems to me that this will get
    RW> confused with, say, two different Office365 accounts that have
    RW> the same auth-url, token-url, and scope.  Should the code use
    RW> also additional information such as the user name?  (I do not
    RW> know much about plstore.el used by oauth2.el.)

oauth2 is primarily focused on accessing web-based resources, not
service based ones such as imap or smtp. For imap the most common way to
control authentication is through auth-source which does indeed use the
user name (and host name and port) to distinguish entries. As I
described in an earlier email this all works quite well (for me)
although the plstore is handled separately from oauth2. And this method
isn't tied to plstore, but works fine with the other auth-source
backends (like an .authinfo file). 

Having said that, it may not hurt to extend/modify oauth2 to interact
better with auth-store. My preference would be to focus ouath2 on the
token management and leave the actual resource interaction (including
storing tokens) to the tools that are already doing that.

Best,
Andy

-- 

Re: oauth2 support for Emacs email clients

[David Engster]

>   Others have mentioned "officially" registering Emacs as IMAP/SMTP
>   clients for Office365 (and possibly Gmail), similar to what seems
>   to be the case for Thunderbird.  I am wondering how davmail is
>   doing this.

Microsoft has actually recognized that it does not make sense for
desktop applications to embed secrets into their code, so they
distinguish between "public" and "confidential" client applications:

https://docs.microsoft.com/en-us/azure/active-directory/develop/msal-client-applications

Public client applications do not have a client secret but only an ID
which can simply be embedded into the application, which is how DavMail
does it. Public client applications are only allowed to access web APIs
on behalf of the user, but this is usually enough.

-David

Re: oauth2 support for Emacs email clients

[Thomas Fitzsimmons]

David Engster <deng@randomsample.de> writes:

>>   Others have mentioned "officially" registering Emacs as IMAP/SMTP
>>   clients for Office365 (and possibly Gmail), similar to what seems
>>   to be the case for Thunderbird.  I am wondering how davmail is
>>   doing this.
>
> Microsoft has actually recognized that it does not make sense for
> desktop applications to embed secrets into their code, so they
> distinguish between "public" and "confidential" client applications:
>
> https://docs.microsoft.com/en-us/azure/active-directory/develop/msal-client-applications
>
> Public client applications do not have a client secret but only an ID
> which can simply be embedded into the application, which is how DavMail
> does it. Public client applications are only allowed to access web APIs
> on behalf of the user, but this is usually enough.

Interesting, but are public client applications allowed to use
IMAP/SMTP?  Or must public client applications use WebDAV to communicate
with Microsoft servers, like DavMail does?

It seems like Thunderbird could act as a public client application,
however I believe it is currently acting as a confidential client
application.  I wonder why.

Thomas

Re: oauth2 support for Emacs email clients

[David Engster]

> David Engster <deng@randomsample.de> writes:
>
>>>   Others have mentioned "officially" registering Emacs as IMAP/SMTP
>>>   clients for Office365 (and possibly Gmail), similar to what seems
>>>   to be the case for Thunderbird.  I am wondering how davmail is
>>>   doing this.
>>
>> Microsoft has actually recognized that it does not make sense for
>> desktop applications to embed secrets into their code, so they
>> distinguish between "public" and "confidential" client applications:
>>
>> https://docs.microsoft.com/en-us/azure/active-directory/develop/msal-client-applications
>>
>> Public client applications do not have a client secret but only an ID
>> which can simply be embedded into the application, which is how DavMail
>> does it. Public client applications are only allowed to access web APIs
>> on behalf of the user, but this is usually enough.
>
> Interesting, but are public client applications allowed to use
> IMAP/SMTP?  Or must public client applications use WebDAV to communicate
> with Microsoft servers, like DavMail does?

As I've written: Public client applications are only allowed to access
web APIs, so no IMAP/SMTP. I usually use DavMail to get my mail
downloaded to a locally running IMAP server.

So yes, simply registering Gnus as a public client is not enough, one
would also need a new backend specifically for Exchange.

> It seems like Thunderbird could act as a public client application,
> however I believe it is currently acting as a confidential client
> application.  I wonder why.

Because they want to use IMAP/SMTP.

-David

Re: oauth2 support for Emacs email clients

[Thomas Fitzsimmons]

David Engster <deng@randomsample.de> writes:

>> David Engster <deng@randomsample.de> writes:
>>
>>>>   Others have mentioned "officially" registering Emacs as IMAP/SMTP
>>>>   clients for Office365 (and possibly Gmail), similar to what seems
>>>>   to be the case for Thunderbird.  I am wondering how davmail is
>>>>   doing this.
>>>
>>> Microsoft has actually recognized that it does not make sense for
>>> desktop applications to embed secrets into their code, so they
>>> distinguish between "public" and "confidential" client applications:
>>>
>>> https://docs.microsoft.com/en-us/azure/active-directory/develop/msal-client-applications
>>>
>>> Public client applications do not have a client secret but only an ID
>>> which can simply be embedded into the application, which is how DavMail
>>> does it. Public client applications are only allowed to access web APIs
>>> on behalf of the user, but this is usually enough.
>>
>> Interesting, but are public client applications allowed to use
>> IMAP/SMTP?  Or must public client applications use WebDAV to communicate
>> with Microsoft servers, like DavMail does?
>
> As I've written: Public client applications are only allowed to access
> web APIs, so no IMAP/SMTP.

OK; I wasn't sure if by "web APIs" you meant only "OAuth-related web
APIs".  Thanks for confirming.

I wonder why Microsoft does not allow public client applications to use
IMAP/SMTP.

> I usually use DavMail to get my mail downloaded to a locally running
> IMAP server.
>
> So yes, simply registering Gnus as a public client is not enough, one
> would also need a new backend specifically for Exchange.

Hmm, yeah.  I'd prefer to keep using IMAP/SMTP, standards designed for
email.  Excorporate does some email operations via EWS, but it seems
strange to extend Excorporate (and make a Gnus backend for it) to handle
all of email just to avoid application registration issues with a new
IMAP/SMTP authentication method.

IMAP/SMTP are already implemented and work fine for other email
services, and they can authenticate via OAuth (assuming registration is
sorted out).

>> It seems like Thunderbird could act as a public client application,
>> however I believe it is currently acting as a confidential client
>> application.  I wonder why.
>
> Because they want to use IMAP/SMTP.

Maybe the FSF could request that Emacs be registered as a public client
application and also be allowed to use IMAP/SMTP.  That would solve the
"embedding a secret in Free Software" part of the OAuth registration
issue, at least for Microsoft servers.

Thomas

Re: oauth2 support for Emacs email clients

[David Engster]

>> As I've written: Public client applications are only allowed to access
>> web APIs, so no IMAP/SMTP.
>
> OK; I wasn't sure if by "web APIs" you meant only "OAuth-related web
> APIs".  Thanks for confirming.
>
> I wonder why Microsoft does not allow public client applications to use
> IMAP/SMTP.

Actually, after I've looked a bit into this, I think I'm mistaken and it
is possible to register a public client with IMAP access.

> Maybe the FSF could request that Emacs be registered as a public client
> application and also be allowed to use IMAP/SMTP.  That would solve the
> "embedding a secret in Free Software" part of the OAuth registration
> issue, at least for Microsoft servers.

In a company setting based on Microsoft services, the problem is not
really so much to get your client registered, but to get your IT
department to enable IMAP and to explicitly allow your application
access to your mailboxes.

-David

Re: oauth2 support for Emacs email clients

[Roland Winkler]

On Sun, Aug 08 2021, David Engster wrote:
> In a company setting based on Microsoft services, the problem is not
> really so much to get your client registered, but to get your IT
> department to enable IMAP and to explicitly allow your application
> access to your mailboxes.

I believe it's an important issue that these things can be handled quite
differently in different companies and institutions.  I happen to have
two Office365 accounts through two different institutions.  One required
OAuth2 with multi-factor authentication already a few years ago, whereas
the other one disabled password-based access to IMAP and SMTP only two
weeks ago.  I have not yet explored in more detail how (dis)similar
these two institution have setup Office365.

Re: oauth2 support for Emacs email clients

[Thomas Fitzsimmons]

David Engster <deng@randomsample.de> writes:

>>> As I've written: Public client applications are only allowed to access
>>> web APIs, so no IMAP/SMTP.
>>
>> OK; I wasn't sure if by "web APIs" you meant only "OAuth-related web
>> APIs".  Thanks for confirming.
>>
>> I wonder why Microsoft does not allow public client applications to use
>> IMAP/SMTP.
>
> Actually, after I've looked a bit into this, I think I'm mistaken and it
> is possible to register a public client with IMAP access.

OK, thanks for checking.  That's encouraging.

>> Maybe the FSF could request that Emacs be registered as a public client
>> application and also be allowed to use IMAP/SMTP.  That would solve the
>> "embedding a secret in Free Software" part of the OAuth registration
>> issue, at least for Microsoft servers.
>
> In a company setting based on Microsoft services, the problem is not
> really so much to get your client registered, but to get your IT
> department to enable IMAP and to explicitly allow your application
> access to your mailboxes.

It depends on the organization, I guess.  My organization has IMAP
enabled.  To use Thunderbird, I attempted to authenticate IMAP using
OAuth.  The administrator received a request for access from
"Thunderbird" (identified by Thunderbird's Microsoft-assigned globally
unique identifier (GUID)), which they granted, after which I could use
Thunderbird with IMAP.

I haven't tried yet, but I suspect using Emacs in my organization would
be harder because, what GUID would I use?

I think it's worth the FSF getting such a globally unique identifier
assigned for GNU Emacs, so that the permission-granting workflow would
be as easy for Emacs as it is for Thunderbird.

Thomas

Re: oauth2 support for Emacs email clients

[David Engster]

> It depends on the organization, I guess.  My organization has IMAP
> enabled.  To use Thunderbird, I attempted to authenticate IMAP using
> OAuth.  The administrator received a request for access from
> "Thunderbird" (identified by Thunderbird's Microsoft-assigned globally
> unique identifier (GUID)), which they granted, after which I could use
> Thunderbird with IMAP.

You're lucky. :-)

> I haven't tried yet, but I suspect using Emacs in my organization would
> be harder because, what GUID would I use?

Well, you could just use Thunderbird's. It's all right here:

https://github.com/mozilla/releases-comm-central/blob/master/mailnews/base/src/OAuth2Providers.jsm

You shouldn't though, because that could get you in trouble. While all
this stuff is essentially security theatre, good luck explaining that to
your IT security department...

-David

Re: oauth2 support for Emacs email clients

[Thomas Fitzsimmons]

David Engster <deng@randomsample.de> writes:

>> It depends on the organization, I guess.  My organization has IMAP
>> enabled.  To use Thunderbird, I attempted to authenticate IMAP using
>> OAuth.  The administrator received a request for access from
>> "Thunderbird" (identified by Thunderbird's Microsoft-assigned globally
>> unique identifier (GUID)), which they granted, after which I could use
>> Thunderbird with IMAP.
>
> You're lucky. :-)

Yes, I'm lucky in that my organization listened to me when I requested
that they leave IMAP and SMTP enabled during the transition to OAuth
(there was a proposal to disable IMAP and SMTP, and an apparent
perception that those protocols would only work with basic
authentication, which I helped correct).

I would recommend that others in organizations with Microsoft email
servers make similar requests to their administrators.

I suspect the request process would be easier for Emacs users if the
administrators saw "Emacs (<Emacs's official GUID>)" in the OAuth
message.

>> I haven't tried yet, but I suspect using Emacs in my organization would
>> be harder because, what GUID would I use?
>
> Well, you could just use Thunderbird's. It's all right here:
>
> https://github.com/mozilla/releases-comm-central/blob/master/mailnews/base/src/OAuth2Providers.jsm
>
> You shouldn't though, because that could get you in trouble.
>
> While all this stuff is essentially security theatre, good luck
> explaining that to your IT security department...

I explained to my IT administrator that I'd like to use Emacs for email.
Others should too.  There's no shame in it. :-)

The FSF should do the legal legwork to figure out a best practice for
using Emacs and OAuth with Microsoft email servers, like Thunderbird
apparently has.  Otherwise, how can we officially document how to
configure Emacs for such servers?

From what you've found, it sounds like a viable approach would be for
the FSF to register Emacs with Microsoft as a public client application,
then we'd default a defcustom in oauth2.el to Emacs's Microsoft-assigned
GUID.

Thomas

Re: oauth2 support for Emacs email clients

[David Engster]

>> Well, you could just use Thunderbird's. It's all right here:
>>
>> https://github.com/mozilla/releases-comm-central/blob/master/mailnews/base/src/OAuth2Providers.jsm
>>
>> You shouldn't though, because that could get you in trouble.
>>
>> While all this stuff is essentially security theatre, good luck
>> explaining that to your IT security department...
>
> I explained to my IT administrator that I'd like to use Emacs for email.
> Others should too.  There's no shame in it. :-)

That's not what I meant. Your administrators might think that they have
only allowed Thunderbird to access IMAP, while in reality they have
whitelisted everything, because you can trivially copy the client-ID and
-secret from Thunderbird's source. That's what I mean with "security
theatre" - everyone's just sticking to the script.

-David

Re: oauth2 support for Emacs email clients

[Alexandre Garreau]

Le mardi 10 août 2021, 17:56:58 CEST David Engster a écrit :
> >> Well, you could just use Thunderbird's. It's all right here:
> >> 
> >> https://github.com/mozilla/releases-comm-central/blob/master/mailnews
> >> /base/src/OAuth2Providers.jsm
> >> 
> >> You shouldn't though, because that could get you in trouble.
> >> 
> >> While all this stuff is essentially security theatre, good luck
> >> explaining that to your IT security department...
> > 
> > I explained to my IT administrator that I'd like to use Emacs for
> > email. Others should too.  There's no shame in it. :-)
> 
> That's not what I meant. Your administrators might think that they have
> only allowed Thunderbird to access IMAP, while in reality they have
> whitelisted everything, because you can trivially copy the client-ID and
> -secret from Thunderbird's source. That's what I mean with "security
> theatre" - everyone's just sticking to the script.

If there is a trivial, secure and documented way of “whitelisting” some 
app from network, it means they would have a way to verify the secret, 
hence it would be somewhat trivial to get in not from sources but from the 
binaries of many programs as well, I believe u.u

Re: oauth2 support for Emacs email clients

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > I haven't tried yet, but I suspect using Emacs in my organization would
  > > be harder because, what GUID would I use?

  > Well, you could just use Thunderbird's. It's all right here:

  > https://github.com/mozilla/releases-comm-central/blob/master/mailnews/base/src/OAuth2Providers.jsm

  > You shouldn't though, because that could get you in trouble. While all
  > this stuff is essentially security theatre, good luck explaining that to
  > your IT security department...

It's possible that you're right.  But, because your statement is takes
a leap and doesn't describe the logical steps, it is hard to verify
that.

Would you like to help determine whether that approach is practical?
And, if not, why not?

For instance, why would the organization's IT security team care
whether he runs Thunderbird, or runs Emacs and tells some company it is
Thunderbird?

How would it even notice that he's using Emacs rather than Thunderbird?

Emacs can be thought of as a modified version of Thunderbird, right?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: oauth2 support for Emacs email clients

[David Engster]

>   > > I haven't tried yet, but I suspect using Emacs in my organization would
>   > > be harder because, what GUID would I use?
>
>   > Well, you could just use Thunderbird's. It's all right here:
>
>   > https://github.com/mozilla/releases-comm-central/blob/master/mailnews/base/src/OAuth2Providers.jsm
>
>   > You shouldn't though, because that could get you in trouble. While all
>   > this stuff is essentially security theatre, good luck explaining that to
>   > your IT security department...
>
> It's possible that you're right.  But, because your statement is takes
> a leap and doesn't describe the logical steps, it is hard to verify
> that.
>
> Would you like to help determine whether that approach is practical?
> And, if not, why not?
>
> For instance, why would the organization's IT security team care
> whether he runs Thunderbird, or runs Emacs and tells some company it is
> Thunderbird?

This is not a technical issue, this is just about company policies what
you are and are not allowed to do on your company laptop. Whether these
policies are only written down, or are also enforced through some
technical measure, is really not that important.

What I consider to be more important is that the Thunderbird developers
kindly ask to not copy their client ID/secret for other applications,
and we should respect that.

> How would it even notice that he's using Emacs rather than Thunderbird?

By looking over his shoulder, through a security audit of his laptop, or
he might simply tell...

-David

Re: oauth2 support for Emacs email clients

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > What I consider to be more important is that the Thunderbird developers
  > kindly ask to not copy their client ID/secret for other applications,
  > and we should respect that.

We were talking about using Thunderbird to convince a server to
authenticate you, then using the authenticated connection with
movemail to read the mail in Emacs.  I think you've changed to a
different question about a different scenario.

  > > How would it even notice that he's using Emacs rather than Thunderbird?

  > By looking over his shoulder, through a security audit of his laptop, or
  > he might simply tell...

That might perhaps happen.  Just as an employer might perhaps tolerate
use of Thunderbird but not Emacs.  This is not impossible.

But is that situation likely enough that we need to worry about it?
We could say, "Unfortunately, we don't know a way to cope with that
situation.  We hope you find a less rigid employer."

If an employer objects to installing Emacs on the company's
laptop, I think the Emacs-using employee will face bigger problems
in using Emacs than how to read mail with it.

My message ended with this:

  Emacs can be thought of as a modified version of Thunderbird, right?

That's a serious point.  Emacs and Thunderbird can, in principle, be
merged in many different proportions.  There is no clear, correct
boundary in that almost-continuum with "Emacs" on one side and
"Thunderbird" on the other.  You could modify Thunderbird step by step
and eventually get Emacs.

I wonder, is it possible to take a part of Thunderbird and modify it
into a mail-fetcher program that produces an inbox file, for use in
place of GNU movemail?

Does Thunderbird as released have that capability?  Maybe people could
simply run Thunderbird to fetch the mail using oauth2.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: oauth2 support for Emacs email clients

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]


  > I haven't tried yet, but I suspect using Emacs in my organization would
  > be harder because, what GUID would I use?

  > I think it's worth the FSF getting such a globally unique identifier
  > assigned for GNU Emacs, so that the permission-granting workflow would
  > be as easy for Emacs as it is for Thunderbird.

Can someone please find out _precisely_ what we would have to do?
And _precisely_ what we would have to agree to?

In discussion months ago, previous reports said we would have
to apply to get a certain kind of code, and promise to keep it secret.
We would not be able to keep it secret.

Does "GUID" refer to that same code?  Is this the same option
or is it a different option?

To keep the number of messages down, please don't reply with speculations.
Someone else will reply with verified facts.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: oauth2 support for Emacs email clients

[Thomas Fitzsimmons]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>
>   > I haven't tried yet, but I suspect using Emacs in my organization would
>   > be harder because, what GUID would I use?
>
>   > I think it's worth the FSF getting such a globally unique identifier
>   > assigned for GNU Emacs, so that the permission-granting workflow would
>   > be as easy for Emacs as it is for Thunderbird.
>
> Can someone please find out _precisely_ what we would have to do?

The official overview is on this web page:

https://docs.microsoft.com/en-us/azure/active-directory/develop/quickstart-register-app

> And _precisely_ what we would have to agree to?

Someone attempting this procedure will be presented with the required
legal agreements as they go through the steps.  I looked but I couldn't
find a simple web page with all the terms of service.

> In discussion months ago, previous reports said we would have
> to apply to get a certain kind of code, and promise to keep it secret.
> We would not be able to keep it secret.

That discussion, in bug #41386, was about some of Google's email
services.

This discussion is about Microsoft's email services.

> Does "GUID" refer to that same code?

No.  GUID here refers to a Microsoft-assigned public identifier.

> Is this the same option or is it a different option?

Microsoft's implementation of OAuth2 provides an application
identification option that is different from the one that Google
provides.  It is the "public client application" option that was
identified by David and discussed in this thread.

With this option, there is no secret code, only the Microsoft-assigned
public identifier that identifies Emacs.  In principle it would be
possible for Emacs source code to include that public identifier since
it is not meant to be kept secret.

Thomas

Re: oauth2 support for Emacs email clients

[Gregory Heytings]

[-- Attachment #1: Type: text/plain, Size: 1172 bytes --]


>
> That discussion, in bug #41386, was about some of Google's email 
> services. This discussion is about Microsoft's email services.
>

No, both discussions are about OAuth2.  For Google, Microsoft, Yahoo, AOL, 
an others.

>
> Microsoft's implementation of OAuth2 provides an application 
> identification option that is different from the one that Google 
> provides.  It is the "public client application" option that was 
> identified by David and discussed in this thread.
>

As David also said, that option will not provide the same function from a 
user's viewpoint, as you cannot use IMAP and SMTP with a public client 
application.  So registering Emacs or Gnus as a public client application 
would be useless.

Thunderbird uses an application id / secret, see [1].  And "All activities 
that occur using your Access Credentials are your responsibility. Access 
Credentials are non-transferable and non-assignable. Keep them secret." 
(see 2.b in [2], and in attached file).

[1] https://raw.githubusercontent.com/mozilla/releases-comm-central/master/mailnews/base/src/OAuth2Providers.jsm
[2] https://docs.microsoft.com/en-us/legal/microsoft-apis/terms-of-use

[-- Attachment #2: Type: application/pdf, Size: 162484 bytes --]

Re: oauth2 support for Emacs email clients

[Tim Cross]

Thomas Fitzsimmons <fitzsim@fitzsim.org> writes:

> David Engster <deng@randomsample.de> writes:
>
>>> David Engster <deng@randomsample.de> writes:
>>>
>>>>>   Others have mentioned "officially" registering Emacs as IMAP/SMTP
>>>>>   clients for Office365 (and possibly Gmail), similar to what seems
>>>>>   to be the case for Thunderbird.  I am wondering how davmail is
>>>>>   doing this.
>>>>
>>>> Microsoft has actually recognized that it does not make sense for
>>>> desktop applications to embed secrets into their code, so they
>>>> distinguish between "public" and "confidential" client applications:
>>>>
>>>> https://docs.microsoft.com/en-us/azure/active-directory/develop/msal-client-applications
>>>>
>>>> Public client applications do not have a client secret but only an ID
>>>> which can simply be embedded into the application, which is how DavMail
>>>> does it. Public client applications are only allowed to access web APIs
>>>> on behalf of the user, but this is usually enough.
>>>
>>> Interesting, but are public client applications allowed to use
>>> IMAP/SMTP?  Or must public client applications use WebDAV to communicate
>>> with Microsoft servers, like DavMail does?
>>
>> As I've written: Public client applications are only allowed to access
>> web APIs, so no IMAP/SMTP.
>
> OK; I wasn't sure if by "web APIs" you meant only "OAuth-related web
> APIs".  Thanks for confirming.
>
> I wonder why Microsoft does not allow public client applications to use
> IMAP/SMTP.
>

MS doesn't like people using IMAP/SMTP mainly because email is really
only just a part of their 'environment'. Office 365 and Exchange are not
mail servers - they are a 'unified communication stack', which includes
email, calendaring, chat, document sharing, etc.

The other reason they don't want direct access to IMAP and even SMTP is
because they are also adding lots of other 'enterprise' and security
features - for example, not allowing attachments which have not got the
right policy classification, preventing 'sensitive' data being sent to
external parties and adding features like read receipts, ability to
recall messages and even have emails which 'auto destruct' or timeout.

They cannot add these features to IMAP or SMTP and if they allow these
protocols, then all these other 'features' can be bypassed. MS will
advise organisations not to enable IMAP or external direct SMTP access
and the C level execs will follow that advice because if they don't and
something goes wrong (even if unrelated), they will be blamed. Things
will still go wrong, but at least they can say they were following the
'experts' advice and 'best practice'. 

I wouldn't be at all surprised if MS didn't remove IMAP support
altogether at some point in the future. There is even a growing
resistance to Email in the corporate sector and try talking to young
people about Email - most of them only deal with it under sufferance. My
plumber actually told me last week that they no longer send invoices via
email - instead, they send an SMS with a link to the invoice on a
server. If they weren't such good and reliable plumbers, I would
consider changing companies.

>> I usually use DavMail to get my mail downloaded to a locally running
>> IMAP server.
>>
>> So yes, simply registering Gnus as a public client is not enough, one
>> would also need a new backend specifically for Exchange.
>
> Hmm, yeah.  I'd prefer to keep using IMAP/SMTP, standards designed for
> email.  Excorporate does some email operations via EWS, but it seems
> strange to extend Excorporate (and make a Gnus backend for it) to handle
> all of email just to avoid application registration issues with a new
> IMAP/SMTP authentication method.
>
> IMAP/SMTP are already implemented and work fine for other email
> services, and they can authenticate via OAuth (assuming registration is
> sorted out).
>
>>> It seems like Thunderbird could act as a public client application,
>>> however I believe it is currently acting as a confidential client
>>> application.  I wonder why.
>>
>> Because they want to use IMAP/SMTP.
>
> Maybe the FSF could request that Emacs be registered as a public client
> application and also be allowed to use IMAP/SMTP.  That would solve the
> "embedding a secret in Free Software" part of the OAuth registration
> issue, at least for Microsoft servers.
>

I think this is unlikely due to the reason outlined above. MS isn't
really that interested in either the 'individual' or simply providing
email. They are selling a much bigger picture with a focus on the
'enterprise' and selling snake oil to those C level executives who are
worried about security and PR who think the solution is to manage and
restrict what users can do.

It is likely that if you have to use Office 365/Outlook, then davmail
may be the best solution. At least it is GPL'd software. 

Re: oauth2 support for Emacs email clients

[Eric S Fraga]

On Monday,  9 Aug 2021 at 02:05, Tim Cross wrote:
> and the C level execs will follow that advice because if they don't and
> something goes wrong (even if unrelated), they will be blamed. Things
> will still go wrong, but at least they can say they were following the
> 'experts' advice and 'best practice'. 

Ah, yes, the equivalent of the 70s mainframe world: "Nobody got fired
for buying IBM."

(I used to work for an organization that did buy an IBM mainframe clone
system [called PCM in those days] and took that risk, mind you.)

:-(

-- 
Eric S Fraga via Emacs 28.0.50 & org 9.4.6 on Debian 11.0

Re: oauth2 support for Emacs email clients

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Public client applications do not have a client secret but only an ID
  > which can simply be embedded into the application, which is how DavMail
  > does it. Public client applications are only allowed to access web APIs
  > on behalf of the user, but this is usually enough.

Is this something that movemail could in principle do?
If so, what are the obstacles?  Is it just a matter of writing some code?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: oauth2 support for Emacs email clients

[Tim Cross]

[-- Attachment #1: Type: text/plain, Size: 1888 bytes --]

I guess it would be possible - at least for the GNU Mailutils version (not
the Emacs version as it does not support imap or encrypted POP3). However,
I guess it would also involve adding a whole HTTP request library and an
Oauth2 library to obtain authentication/refresh tokens. This is of course
assuming that MS does allow an application ID to be used with IMAP (to be
confirmed).

This also only helps with MS Office365/Outlook access - it doesn't help
with Gmail or any other provider who transitions to nly support Oauth2. As
mentioned by others, the big stumbling block here is that each provider is
able to implement Oauth2 with their own custom workflow, which makes a
general generic solution difficult to define. The solution will probably
require some sort of 'pluggable' Oauth2 layer, which might include Oauth2
authenticators for popular mail providers and a facility to add a custom
one for others.

On Tue, 10 Aug 2021 at 13:30, Richard Stallman <rms@gnu.org> wrote:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > Public client applications do not have a client secret but only an ID
>   > which can simply be embedded into the application, which is how DavMail
>   > does it. Public client applications are only allowed to access web APIs
>   > on behalf of the user, but this is usually enough.
>
> Is this something that movemail could in principle do?
> If so, what are the obstacles?  Is it just a matter of writing some code?
>
> --
> Dr Richard Stallman (https://stallman.org)
> Chief GNUisance of the GNU Project (https://gnu.org)
> Founder, Free Software Foundation (https://fsf.org)
> Internet Hall-of-Famer (https://internethalloffame.org)
>
>
>
>

-- 
regards,

Tim

--
Tim Cross

[-- Attachment #2: Type: text/html, Size: 2737 bytes --]

Re: oauth2 support for Emacs email clients

[Roland Winkler]

On Tue, Aug 10 2021, Tim Cross wrote:
> I guess it would be possible - at least for the GNU Mailutils version
> (not the Emacs version as it does not support imap or encrypted
> POP3). However, I guess it would also involve adding a whole HTTP
> request library and an Oauth2 library to obtain authentication/refresh
> tokens. This is of course assuming that MS does allow an application
> ID to be used with IMAP (to be confirmed). 
>
> This also only helps with MS Office365/Outlook access - it doesn't
> help with Gmail or any other provider who transitions to nly support
> Oauth2.

I am not sure I understand the last statement.  My organizations only
allow its users to access their Office365 mail accounts via OAuth2 (and
multi-factor authentication).  However, the way this requirement is
implemented, a tool like davmail is able to perform the necessary steps
so that locally you can use any IMAP client you like.  What is different
here regarding what Gmail is aiming for?