bug#41386: 28.0.50; Gnus nnimap OAuth 2.0 support

bug#41386: 28.0.50; Gnus nnimap OAuth 2.0 support

[Thomas Fitzsimmons]

Hi,

Some email services seem to be moving toward requiring MUAs to support
OAuth 2.0, one reason being that they would like MUAs to do two factor
authentication.  It would be nice to have this capability in Gnus's
nnimap.

The Gnus manual makes no mention of OAuth 2.0, I can't find any
suggested configurations online, and there's nothing mentioned in
debbugs about this.

Is support for OAuth 2.0 -- perhaps via oauth2.el available in GNU ELPA
-- something that Gnus could eventually implement?  If so, I guess this
bug report could be where the idea is discussed.

Thanks,
Thomas

bug#41386: 28.0.50; Gnus nnimap OAuth 2.0 support

[Lars Ingebrigtsen]

Thomas Fitzsimmons <fitzsim@fitzsim.org> writes:

> Is support for OAuth 2.0 -- perhaps via oauth2.el available in GNU ELPA
> -- something that Gnus could eventually implement?  If so, I guess this
> bug report could be where the idea is discussed.

I don't think there's any way to ship Emacs with built-in oauth2 support
for doing auth with Gmail -- it requires distributions with API secrets
and stuff, and there's no secrets in the Emacs distribution.

Or is there a way to do that now?  I haven't been paying attention the
last few months.  I remember Thunderbird including some credentials in
the source code and saying, jokily, "remember, these are secret".
Somebody would have to register the Emacs "app" with Google, and for
Emacs, that would have to be the FSF, right?  And I don't see that
happening ever, ideologically.

But somebody could definitely write a package and put that on MELPA, and
do the registration, I think?  (With the same joke, of course.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#41386: 28.0.50; Gnus nnimap OAuth 2.0 support

[Noam Postavsky]

On Tue, 19 May 2020 at 08:47, Lars Ingebrigtsen <larsi@gnus.org> wrote:

> > Is support for OAuth 2.0 -- perhaps via oauth2.el available in GNU ELPA
> > -- something that Gnus could eventually implement?  If so, I guess this
> > bug report could be where the idea is discussed.
>
> I don't think there's any way to ship Emacs with built-in oauth2 support
> for doing auth with Gmail -- it requires distributions with API secrets

There is something called xoauth2, is that related? Someone posted a
package for it, but the docs are a bit thin...

https://github.com/ccrusius/auth-source-xoauth2

bug#41386: 28.0.50; Gnus nnimap OAuth 2.0 support

[Lars Ingebrigtsen]

Noam Postavsky <npostavs@gmail.com> writes:

> There is something called xoauth2, is that related? Someone posted a
> package for it, but the docs are a bit thin...
>
> https://github.com/ccrusius/auth-source-xoauth2

Below are the instructions for end-user usage.  I think you can sum that
up as "er, no".  :-/

   1. Go to the developer console,  
   https://console.developers.google.com/project  
   2. Create a new project (if necessary), and select it once created.  
   3. Select \"APIs & Services\" from the navigation menu.  
   4. Select \"Credentials\".  
   5. Create new credentials of type \"OAuth Client ID\".  
   6. Choose application type \"Other\".  
   7. Choose a name for the client.  
     
   This should get you all the values but for the refresh token. For that one:  
     
   1. Clone the https://github.com/google/gmail-oauth2-tools repository  
   2. Execute the following command in the cloned repository:  
     
   python2.7 python/oauth2.py  
   --generate_oauth2_token \  
   --client_id=<client id from previous steps> \  
   --client_secret=<client secret from previous steps>  
     
   You should now have all the required values (the :token-url value should  
   be \"https://accounts.google.com/o/oauth2/token\").")  


-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#41386: 28.0.50; Gnus nnimap OAuth 2.0 support

[Thomas Fitzsimmons]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Thomas Fitzsimmons <fitzsim@fitzsim.org> writes:
>
>> Is support for OAuth 2.0 -- perhaps via oauth2.el available in GNU ELPA
>> -- something that Gnus could eventually implement?  If so, I guess this
>> bug report could be where the idea is discussed.
>
> I don't think there's any way to ship Emacs with built-in oauth2 support
> for doing auth with Gmail -- it requires distributions with API secrets
> and stuff, and there's no secrets in the Emacs distribution.
>
> Or is there a way to do that now?  I haven't been paying attention the
> last few months.  I remember Thunderbird including some credentials in
> the source code and saying, jokily, "remember, these are secret".
> Somebody would have to register the Emacs "app" with Google, and for
> Emacs, that would have to be the FSF, right?  And I don't see that
> happening ever, ideologically.

I suppose it depends on what Google wants during the registration
process; I've never tried this registration process before so I don't
know what's involved.  Maybe someone from the FSF could research this?
Maybe a solution could be found for Free Software like Emacs.
Thunderbird is mentioned as a not-less-secure-app, so they seem to have
solved this problem to Thunderbird/Google's satisfaction.

According to communications I've received for my Google "G Suite" email
service, Google plans to change these "less secure app" policies such
that next year they won't allow Gnus to connect using username/password
authentication like it does today.

> But somebody could definitely write a package and put that on MELPA, and
> do the registration, I think?  (With the same joke, of course.)

OK, maybe Google could relax the secrecy requirement for Emacs though,
since I'd hope they'd be sufficiently Free-Software-friendly to work
something out.  I assume, given what Thunderbird is doing, that the
secrecy requirement isn't something fundamental to OAuth 2.0's security.

Thomas

bug#41386: 28.0.50; Gnus nnimap OAuth 2.0 support

[David Engster]

> Maybe a solution could be found for Free Software like Emacs.
> Thunderbird is mentioned as a not-less-secure-app, so they seem to have
> solved this problem to Thunderbird/Google's satisfaction.

No, they haven't. It's just that at the moment, no one cares.

It is pretty obvious that OAuth2 client id/secrets do not make sense in
desktop applications (whether (F)OSS or not), making their whole point
moot. Google admits this much in their documentation, where they say

  The process results in a client ID and, in some cases, a client secret,
  which you embed in the source code of your application. (In this
  context, the client secret is obviously not treated as a secret.)

(see https://developers.google.com/identity/protocols/oauth2)

People took this paragraph as permission to simply put client id and
secret directly into the source code. However, Google *explicitly*
forbids this in their developer TOS:

  Developer credentials (such as passwords, keys, and client IDs) are
  intended to be used by you and identify your API Client. You will keep
  your credentials confidential and make reasonable efforts to prevent and
  discourage other API Clients from using your credentials. Developer
  credentials may not be embedded in open source projects.

(see https://developers.google.com/terms)

The Thunderbird people simply ignore this and do it anyway, but it's not
like they have much choice.

> OK, maybe Google could relax the secrecy requirement for Emacs though,
> since I'd hope they'd be sufficiently Free-Software-friendly to work
> something out.

Google does not care one bit.

The solution to this problem is to choose another mail provider.

-David

bug#41386: 28.0.50; Gnus nnimap OAuth 2.0 support

[Lars Ingebrigtsen]

Thomas Fitzsimmons <fitzsim@fitzsim.org> writes:

> I suppose it depends on what Google wants during the registration
> process; I've never tried this registration process before so I don't
> know what's involved.

You register a developer account (with your name and address and stuff),
and you then register an application.  Everybody connecting to Gmail
will use this application ID, so you are "responsible" in some degree
for the users of your application.  Rate-limiting, for instance, are
based on the application ID.

> OK, maybe Google could relax the secrecy requirement for Emacs though,
> since I'd hope they'd be sufficiently Free-Software-friendly to work
> something out.  I assume, given what Thunderbird is doing, that the
> secrecy requirement isn't something fundamental to OAuth 2.0's security.

It is.  OAuth login without secrets isn't any more secure than normal
user name/password logins, so making apps run through these hoops is
just obfuscation.  It's obvious what Google's end game here is: They
will stop IMAP access altogether to Gmail as soon as they are able to
without losing too many of the users.

This OAuth 2.0 stuff is just a sop they can point people towards while
they're closing off access to their walled garden: "See!  We're still
open!"  And people bite.  Some hackernews commented something like "I
don't see why people are complaining...  they just have to run a
script..."

The only solution here is to leave Gmail.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#41386: 28.0.50; Gnus nnimap OAuth 2.0 support

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Some email services seem to be moving toward requiring MUAs to support
  > OAuth 2.0, one reason being that they would like MUAs to do two factor
  > authentication.  It would be nice to have this capability in Gnus's
  > nnimap.

This may raise some moral issues -- depending on what practical
requirements there are.  Would you like to explain to me how the
two-factor authentication works in practice, and what the user needs
to have?

For instance, can the user do this using nothing but Emacs running on
some operating system (GNU/Linux, we hope)?  If not, what else does
the user need to have?

Does it require the user to have a portable phone?

Do those email services plan to require two-factor authentication
or offer it as an option for the user?

Are there free implementations of OAuth 2.0?  What are their licenses?

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

bug#41386: 28.0.50; Gnus nnimap OAuth 2.0 support

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

This is alarming news.  If these reports are correct, they mean that
Google will entirely cut off access to Gmail from the Free World.

Can you find a web page that describes the danger in a clear way,
with references to substantiate the warning?  That is the first thing
we need in order to campaign to convince Google not to do it.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

bug#41386: 28.0.50; Gnus nnimap OAuth 2.0 support

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > There is something called xoauth2, is that related? Someone posted a
  > package for it, but the docs are a bit thin...

  > https://github.com/ccrusius/auth-source-xoauth2

I read the response to your message, but I could not tell whether it
involves some nonfree software.  Does it?

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

bug#41386: 28.0.50; Gnus nnimap OAuth 2.0 support

[Stefan Kangas]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
> This is alarming news.  If these reports are correct, they mean that
> Google will entirely cut off access to Gmail from the Free World.
>
> Can you find a web page that describes the danger in a clear way,
> with references to substantiate the warning?  That is the first thing
> we need in order to campaign to convince Google not to do it.

Here are two relevant announcements, I think:

Turning off less secure app access to G Suite accounts
https://gsuiteupdates.googleblog.com/2019/12/less-secure-apps-oauth-google-username-password-incorrect.html

Less secure app turn-off suspended until further notice
https://gsuiteupdates.googleblog.com/2020/03/less-secure-app-turn-off-suspended.html

In the second of them, we read:

    Last December, we announced that we’d be turning off less secure app
    (LSA) access to G Suite accounts, and that you should migrate to OAuth
    authentication instead. The first phase of the LSA turn-down was
    scheduled for June 15, 2020. As many organizations deal with the impact
    of COVID-19 and are now focused on supporting a remote workforce, we
    want to minimize potential disruptions for customers unable to complete
    migrations in this timeframe.

    As a result, we are suspending the LSA turn-off until further
    notice. All previously announced timeframes no longer apply.

So I guess this means that in the mid-term, people should seriously
consider leaving Google.

Hope that helps.

Best regards,
Stefan Kangas

bug#41386: 28.0.50; Gnus nnimap OAuth 2.0 support

[Thomas Fitzsimmons]

Hi Richard,

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > Some email services seem to be moving toward requiring MUAs to support
>   > OAuth 2.0, one reason being that they would like MUAs to do two factor
>   > authentication.  It would be nice to have this capability in Gnus's
>   > nnimap.
>
> This may raise some moral issues -- depending on what practical
> requirements there are.  Would you like to explain to me how the
> two-factor authentication works in practice, and what the user needs
> to have?

Thanks for having a look at this bug report.

I could explain in a general way how this might work for Gnus's nnimap
backend.  However, given what I've learned from Lars's and David's
responses, I would prefer to keep this bug report solely about OAuth 2.0
support, with a focus on Google's email services.

Two factor authentication can be implemented by a service without
requiring OAuth 2.0, and vice versa.  (Given what I've learned from Lars
and David I probably should not have even mentioned it in the initial
report.)  Therefore I think lumping the two concepts together in this
bug report will be too confusing.

We could open a new bug report, or open a discussion on emacs-devel,
about two factor authentication support in Emacs generally.  However, I
don't have a specific example to discuss or a bug to report about a
specific service, so I don't think the resulting discussion would be
focused enough to be productive.

Thomas

bug#41386: 28.0.50; Gnus nnimap OAuth 2.0 support

[Thomas Fitzsimmons]

Hi Stefan,

Stefan Kangas <stefan@marxist.se> writes:

> Richard Stallman <rms@gnu.org> writes:
>
>> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
>> [[[ whether defending the US Constitution against all enemies,     ]]]
>> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>>
>> This is alarming news.  If these reports are correct, they mean that
>> Google will entirely cut off access to Gmail from the Free World.
>>
>> Can you find a web page that describes the danger in a clear way,
>> with references to substantiate the warning?  That is the first thing
>> we need in order to campaign to convince Google not to do it.
>
> Here are two relevant announcements, I think:
>
> Turning off less secure app access to G Suite accounts
> https://gsuiteupdates.googleblog.com/2019/12/less-secure-apps-oauth-google-username-password-incorrect.html
> 
> Less secure app turn-off suspended until further notice
> https://gsuiteupdates.googleblog.com/2020/03/less-secure-app-turn-off-suspended.html
>
> [...]

Yes, those are the relevant announcements, thanks.

Quoting from the first announcement:

"If you are using Thunderbird or another email client, re-add your
Google Account and configure it to use IMAP with OAuth."

It seems like Google is acknowledging Thunderbird as a valid option
post-"less-secure-app" turn-off there.

Richard, I think Lars's and David's summary of Thunderbird's situation
highlights the non-technical issues blocking GNU Emacs from supporting
OAuth 2.0 access to the G Suite IMAP service.  I'm wondering if the FSF
could help with those issues.

> So I guess this means that in the mid-term, people should seriously
> consider leaving Google.

Agreed, and I am [1], but maybe a better outcome could be reached.  It
sounds like from Thunderbird's situation, changes to Google's developer
terms of service, giving some consideration to Free Software projects
using these APIs, could solve the problem.

It might take non-technical discussions between the FSF and Google, but
that's one of the FSF's functions, AIUI.  Maybe the result could be a
model for how to handle other similar situations, since Emacs needing
"API keys" to access network services is not unique to G Suite or
Google.

Thomas

1. I'm tempted to try Lars's self-hosted email script. ;-)

bug#41386: 28.0.50; Gnus nnimap OAuth 2.0 support

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

It seems to me that the requirement for a secret key in a user program
is fundamentally incompatible with free software.  A free system
cannot keep secrets from the user.  However, if Google is willing to leave a space to use free clients, something could be worked out.

First I have to finish studying all this.  I will send mail now to fetch
those articles.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

bug#41386: 28.0.50; Gnus nnimap OAuth 2.0 support

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Two factor authentication can be implemented by a service without
  > requiring OAuth 2.0, and vice versa.

Ok.  But my concern is not about OAuth in particular.  The crucial
question is, will Gmail in the future permit access from the Free World
in any manner whatsoever?

In other words, if we look at all the ways of logging in that Gmail
permits in the future, will _any_ of them be usable in the Free World?

Can you help me find the answer?  If it is no, we had better put all our
strength into changing that answer.



-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

bug#41386: 28.0.50; Gnus nnimap OAuth 2.0 support

[Thomas Fitzsimmons]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > Two factor authentication can be implemented by a service without
>   > requiring OAuth 2.0, and vice versa.
>
> Ok.  But my concern is not about OAuth in particular.  The crucial
> question is, will Gmail in the future permit access from the Free World
> in any manner whatsoever?
>
> In other words, if we look at all the ways of logging in that Gmail
> permits in the future, will _any_ of them be usable in the Free World?
>
> Can you help me find the answer?  If it is no, we had better put all our
> strength into changing that answer.

OK, I'll try to help, but I don't know what Google or Gmail will do in
the future, in general, obviously.  I wanted this bug report to be
specifically about Gnus, IMAP and OAuth 2.0 (which the responding
experts correctly inferred was more precisely about Gnus, IMAP and
Gmail's proposed OAuth 2.0 policy change).

Your question reminded me that Gmail provides "basic HTML view" which
does not require any Javascript whatsoever for sign-in or for the main
interface, and thus is fully useable with Emacs Web Wowser (eww).  For
me that isn't a viable alternative to Gnus, so it's tangential to this
bug report, but that access method is feasible using entirely Free
Software.  Hopefully that answers your question.

Will "basic HTML view" still be available after Gmail starts mandating
OAuth 2.0 for IMAP access (assuming they do, sometime in the future)?  I
suspect it will, but I don't know.

Thomas

bug#41386: 28.0.50; Gnus nnimap OAuth 2.0 support

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]


  > I don't think there's any way to ship Emacs with built-in oauth2 support
  > for doing auth with Gmail -- it requires distributions with API secrets
  > and stuff, and there's no secrets in the Emacs distribution.

There are no real secrets in any free operating system.  The only way
I know of to have a key and effectively keep it secret with software
is to bury it in a nonfree excutable, and that is not a solution; it
only replaces one impassable obstacle with another impassable
obstacle.

  > Or is there a way to do that now?  I haven't been paying attention the
  > last few months.  I remember Thunderbird including some credentials in
  > the source code and saying, jokily, "remember, these are secret".
  > Somebody would have to register the Emacs "app" with Google, and for
  > Emacs, that would have to be the FSF, right?  And I don't see that
  > happening ever, ideologically.

If what Thunderbird is doing does not involve nonfree software,
I see no reason we couldn't do the same.

But I doubt that Google will continue accepting this forever.  Google
might eventually decide to kick off Thunderbird users, and Gnus users
along with them.

  > But somebody could definitely write a package and put that on MELPA, and
  > do the registration, I think?  (With the same joke, of course.)

To the extent that this approach is usable, we woultn't need to
relegate it to MELPA.  We could put it straight into Gnus.

The two Google announcements clearly describe how Google plans to
block access with anything other than OAuth 2.  They don't go into
much detail about what OAuth 2 requires, and don't describe how this
conflicts with free software.

Can someone find a page describing this issue in a careful
and thorough way, written by someone who knows the subject?

Can someone get in touch with a Thunderbird developer or expert
who would like to discuss the issue?

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

bug#41386: 28.0.50; Gnus nnimap OAuth 2.0 support

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > OK, I'll try to help, but I don't know what Google or Gmail will do in
  > the future, in general, obviously.

I don't expect you to practice precognition.  But it may be possible
to deduce something by putting their announcements together with other
known facts.  Maybe we could figure out questions to ask them.

  > Your question reminded me that Gmail provides "basic HTML view" which
  > does not require any Javascript whatsoever for sign-in or for the main
  > interface, and thus is fully useable with Emacs Web Wowser (eww).

That is a significant fact.  It means that using Gmail in the free
world won't become entirely impossible.  Thanks.
 
So I modify the question:

   Will Gmail in the future permit access from the Free World
   to read mail with a free local MUA in any manner whatsoever?
   
And there is another followup question:

Is it possible to log in on basic HTML view
passing via a proxy that will hide your actual location?
It is possible that basic HTML view won't allow this.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

bug#41386: 28.0.50; Gnus nnimap OAuth 2.0 support

[David Engster]

> The two Google announcements clearly describe how Google plans to
> block access with anything other than OAuth 2.  They don't go into
> much detail about what OAuth 2 requires, and don't describe how this
> conflicts with free software.
>
> Can someone find a page describing this issue in a careful
> and thorough way, written by someone who knows the subject?

I've described the main issue in another mail in this bug thread. The
problem is that Google's terms of service explicitly forbid to put
client IDs into "open source projects". You can read their terms of
service here:

  https://developers.google.com/terms

Section 4b, paragraph 1:

  Developer credentials (such as passwords, keys, and client IDs) are
  intended to be used by you and identify your API Client. You will keep
  your credentials confidential and make reasonable efforts to prevent and
  discourage other API Clients from using your credentials. Developer
  credentials may not be embedded in open source projects.

So for authors of (F)OSS non-web applications, this in effect makes it
impossible to use OAuth2 with Google services if a client id/secret is
required. The client id/secret is used to identify the application that
is making the request.

-David

bug#41386: 28.0.50; Gnus nnimap OAuth 2.0 support

[Thomas Fitzsimmons]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > OK, I'll try to help, but I don't know what Google or Gmail will do in
>   > the future, in general, obviously.
>
> I don't expect you to practice precognition.  But it may be possible
> to deduce something by putting their announcements together with other
> known facts.  Maybe we could figure out questions to ask them.
>
>   > Your question reminded me that Gmail provides "basic HTML view" which
>   > does not require any Javascript whatsoever for sign-in or for the main
>   > interface, and thus is fully useable with Emacs Web Wowser (eww).
>
> That is a significant fact.  It means that using Gmail in the free
> world won't become entirely impossible.  Thanks.
>  
> So I modify the question:
>
>    Will Gmail in the future permit access from the Free World
>    to read mail with a free local MUA in any manner whatsoever?
>    
> And there is another followup question:
>
> Is it possible to log in on basic HTML view
> passing via a proxy that will hide your actual location?

Yes.  Here is the recipe I tested:

   sudo apt install proxychains
   sudo sed 's/#quiet_mode/quiet_mode/' -i /etc/proxychains.conf
   ssh -ND 9050 <host-from-which-to-access-mail> &
   HOME=$(mktemp -d) proxychains emacs -Q -nw
   M-x eww RET https://gmail.com RET
   [Use the usual web-based log-in procedure via Emacs Web Wowser.]

The log-in procedure was successful and I was able to use the basic HTML
view interface.

To confirm the results of the test, I also logged into the same service
via Firefox and a direct connection.  After I did the procedure above,
the Firefox session showed:

"This account is currently being used in 1 other location (<address>)."

Where <address> was the IP address of <host-from-which-to-access-mail>.

The above is just an example, other methods may be superior in various
ways, but I think this proves that Gmail is useable in Emacs Web Wowser
via a proxy.

Thomas

P.S. Prior to installing proxychains, I tried using Emacs's built-in
proxy support in the same way.  There are references to SOCKS in the URL
manual, but I couldn't find an easy way to make EWW use localhost:9050
as a SOCKS proxy.  I debugged until I got to the comment "Should check
for socks" in url-default-find-proxy-for-url.

bug#41386: 28.0.50; Gnus nnimap OAuth 2.0 support

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > Is it possible to log in on basic HTML view
  > > passing via a proxy that will hide your actual location?

  > Yes.  Here is the recipe I tested:

That is a little less bad.

Is there any positive indication that Google will let people continue
using this basic HTML mode?

I wonder -- is it possible for a program talking to that proxy to
convert the messages in the Gmail server into a mailbox file which you
could then pass to an MUA?


-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

bug#41386: 28.0.50; Gnus nnimap OAuth 2.0 support

[Thomas Fitzsimmons]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > > Is it possible to log in on basic HTML view
>   > > passing via a proxy that will hide your actual location?
>
>   > Yes.  Here is the recipe I tested:
>
> That is a little less bad.
>
> Is there any positive indication that Google will let people continue
> using this basic HTML mode?

It seems like basic HTML view is the old mode that is still being
maintained.  Just after logging in, an HTML page mentions that "standard
view" requires JavaScript and that to use standard view one must enable
JavaScript.  But the same page then provides the option, "To use [...]
basic HTML view, which does not require JavaScript, click here.".  After
logging in, one can set basic HTML as the default view.  So it seems
like it is still fully supported.

(As an aside: basic HTML view uses old HTML-table-based, rather than new
CSS-based, layout, so it lays out nicely in EWW; in other words, it
provides a user interface that is friendly to simple browsers.  It's by
no means an efficient alternative to Emacs's native MUAs though.)

> I wonder -- is it possible for a program talking to that proxy to
> convert the messages in the Gmail server into a mailbox file which you
> could then pass to an MUA?

I don't see a way of downloading mailbox files via simple links.

It is technically possible to implement the conversion you're
describing, since one can click through several links for a given
message and eventually get the original text.  However, it would be an
awkward, fragile HTML-scraping-type implementation that I don't expect
would be viable in practice.

Thomas

bug#41386: 28.0.50; Gnus nnimap OAuth 2.0 support

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > I wonder -- is it possible for a program talking to that proxy to
  > > convert the messages in the Gmail server into a mailbox file which you
  > > could then pass to an MUA?

  > I don't see a way of downloading mailbox files via simple links.

I don't see how "simple links" relates to what I had in mind, so I have
a feeling this is a misunderstanding.

  > It is technically possible to implement the conversion you're
  > describing, since one can click through several links for a given
  > message and eventually get the original text.  However, it would be an
  > awkward, fragile HTML-scraping-type implementation that I don't expect
  > would be viable in practice.

That is what I had in mind.  Is it "fragile"?  If Google changes the
UI of the basic HTML mode, it would break.  Does Google change that
very often?

In any case, if Google breaks all advertized other ways of
transferring a Gmail account's incoming mail into a free MUA, scraping
would be MUCH better than no way at all.


-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

bug#41386: 28.0.50; Gnus nnimap OAuth 2.0 support

[Lars Ingebrigtsen]

Richard Stallman <rms@gnu.org> writes:

> It seems to me that the requirement for a secret key in a user program
> is fundamentally incompatible with free software.  A free system
> cannot keep secrets from the user.  However, if Google is willing to
> leave a space to use free clients, something could be worked out.

At present, there doesn't seem to be any way to follow Google's explicit
instructions as to how OAuth 2.0 support should be done in free
software -- even if they tacitly are saying through unofficial channels
"but we don't really mean it" by saying that Thunderbird is still
working.

So I'm closing this bug report, as this isn't something that can be
fixed here, but it something the FSF will have to negotiate with Google.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#41386: 28.0.50; Gnus nnimap OAuth 2.0 support

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > So I'm closing this bug report, as this isn't something that can be
  > fixed here, but it something the FSF will have to negotiate with Google.

I guess so -- if we can find someone there to talk with.  Can someone
find out whether Google still has a representative concerned with
"open source"?

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

bug#41386: 28.0.50; Gnus nnimap OAuth 2.0 support

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

I got this information from someone who knows
who says it is all from published information.

Does this enable us to cope with Google's planned changes?

======================================================================
GNUS could follow the same process of the the maintainers of KMail to 
get approved:

 
https://www.reddit.com/r/kde/comments/gi5bol/kmailkontact_oauth_signin_with_gmail_enabled_again/

This had nothing to do with shutting off access to free MUAs. It's a 
matter of hardening the security of GMail accounts and having all 
3rd-party clients (free and non-free alike) publish a privacy policy 
where they describe their usage of the GMail APIs, as described in this 
thread:

https://www.reddit.com/r/kde/comments/cj7t7c/kmail_doesnt_let_me_sign_in_with_gmail/evchyih/

In particular:

   Google is not concerned with the legal "significance", they just care
   about the part about what features of their API we use and how, which
   seems to be incomplete, judging from the last email I got from them
   just yesterday, which is my responsibility to update, as I'm more or
   less the Google integration maintainer in PIM, since I wrote most of
   the stuff (silly me!) :D

The implementation details for XOAUTH2 are here, along with sample code 
and links to the relevant RFCs:

   https://developers.google.com/gmail/imap/xoauth2-protocol
-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

bug#41386: 28.0.50; Gnus nnimap OAuth 2.0 support

[Lars Ingebrigtsen]

Richard Stallman <rms@gnu.org> writes:

> I got this information from someone who knows
> who says it is all from published information.
>
> Does this enable us to cope with Google's planned changes?
>
> ======================================================================
> GNUS could follow the same process of the the maintainers of KMail to 
> get approved:
>
> https://www.reddit.com/r/kde/comments/gi5bol/kmailkontact_oauth_signin_with_gmail_enabled_again/

The link doesn't same what the process is, so I don't know whether it
enables us or not.  In particular, it doesn't say anything about how the
free software application is supposed to keep the secret keys secret,
which is (of course) totally impossible for an application like Emacs.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#41386: 28.0.50; Gnus nnimap OAuth 2.0 support

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > The link doesn't same what the process is, so I don't know whether it
  > enables us or not.  In particular, it doesn't say anything about how the
  > free software application is supposed to keep the secret keys secret,
  > which is (of course) totally impossible for an application like Emacs.

Can you try contacting and asking the developers of Kmail about any points
that are not clearly stated in what I sent you?

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

bug#41386: 28.0.50; Gnus nnimap OAuth 2.0 support

[Lars Ingebrigtsen]

Richard Stallman <rms@gnu.org> writes:

>   > The link doesn't same what the process is, so I don't know whether it
>   > enables us or not.  In particular, it doesn't say anything about how the
>   > free software application is supposed to keep the secret keys secret,
>   > which is (of course) totally impossible for an application like Emacs.
>
> Can you try contacting and asking the developers of Kmail about any points
> that are not clearly stated in what I sent you?

If the Emacs maintainers wants to get involved with Google on this,
that's fine by me.  My stance is that Google is obviously not doing
these changes on a good-faith basis, but as a long-term way to shut off
all avenues of using Gmail without going through the official interface.

So I'm not interested in spending any time on this.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#41386: 28.0.50; Gnus nnimap OAuth 2.0 support

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > If the Emacs maintainers wants to get involved with Google on this,
  > that's fine by me.

Do you mean, that Eli or Stefan should address this problem in Gnus?
I don't think we can possibly do anything on this.

  >   My stance is that Google is obviously not doing
  > these changes on a good-faith basis, but as a long-term way to shut off
  > all avenues of using Gmail without going through the official interface.

That doesn't seem to be what the Kmail developers think.

Many Gnus users will be unhappy if it stops working with Gmail.


-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

bug#41386: 28.0.50; Gnus nnimap OAuth 2.0 support

[Lars Ingebrigtsen]

Richard Stallman <rms@gnu.org> writes:

>   > If the Emacs maintainers wants to get involved with Google on this,
>   > that's fine by me.
>
> Do you mean, that Eli or Stefan should address this problem in Gnus?
> I don't think we can possibly do anything on this.

No, I don't expect them to implement this.  But somebody has to open a
developer account at Google and register the Emacs "application", and
that has to be the FSF.  They will then be able to generate the API
secret key that will then be included, non-secretly, specifically
countering the contract you enter into with Google, when you then
publish it openly in the Emacs sources.

>   >   My stance is that Google is obviously not doing
>   > these changes on a good-faith basis, but as a long-term way to shut off
>   > all avenues of using Gmail without going through the official interface.
>
> That doesn't seem to be what the Kmail developers think.
>
> Many Gnus users will be unhappy if it stops working with Gmail.

Definitely.  But it's not something I can help with.  Blame Google.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#41386: 28.0.50; Gnus nnimap OAuth 2.0 support

[황병희]

>> Many Gnus users will be unhappy if it stops working with Gmail.
>
> Definitely.  But it's not something I can help with.  Blame Google.

I just myself mark as IMPORTANT this PR because i use Gmail + Google Apps.

Sincerely,

-- 
^고맙습니다 _救濟蒼生_ 감사합니다_^))//

bug#41386: 28.0.50; Gnus nnimap OAuth 2.0 support

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Can someone please make contact with the Kmail developers so I
can ask them for advice?

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

bug#41386: 28.0.50; Gnus nnimap OAuth 2.0 support

[Simon Leinen]

> I guess so -- if we can find someone there to talk with.  Can
> someone find out whether Google still has a representative concerned
> with "open source"?

I'd try Chris DiBona <cdibona@google.com>
-- 
Simon.

bug#41386: 28.0.50; Gnus nnimap OAuth 2.0 support

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Chris DiBona <cdibona@google.com>

He was Google's "open source" person.  Do you know whether he still is?

I got an answer from a KDE developer, and with luck that should
be enough.  If it isn't, we can try DiBona.

Thanks.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

bug#41386: 28.0.50; Gnus nnimap OAuth 2.0 support

[Simon Leinen]

>> Chris DiBona <cdibona@google.com>

> He was Google's "open source" person.  Do you know whether he still is?

His LinkedIn profile and Wikipedia page still list him as Director of
Open Source at Google.

Several years ago I talked with him about license options (specifically
GPL variants) on Google's now-defunct source-code hosting service, and
found him knowledgeable and willing to help.

> I got an answer from a KDE developer, and with luck that should
> be enough.  If it isn't, we can try DiBona.

Sounds good!

All the best,
-- 
Simon.

bug#41386: 28.0.50; Gnus nnimap OAuth 2.0 support

[Björn Bidar via Bug reports for GNU Emacs, the Swiss army knife of text editors]

I know this bug is marked as wontfix however more and more providers are
moving to Oauth2, even those that use plain standards like
imap/{cal,card}dav/smpt, thous increasing the importance of Oauth2
support in Gnus.

The main advantage I see is that oauth allows for two factor
authentication and the invalidation of the "password" that the app
stores. The password or token that the app has usually only lasts for a
duration of time and can be invalidated if needed. Like if the person no
loner works for the employer or the device has been stolen.

Some providers like Microsoft require it next year and the employer
can already enforce the use of Oauth2 [1].
The argument "just use another email provider" doesn't really work in
such cases.

SailfishOS recently addeded oauth2 support for Microsoft Oauth and
KDE also does support it[2].
In the case of Microsoft there are no "secrets" that can be stored publicly just the
application id[3].

Without proper OAuth2 support there is no use for Gnus for such users,
except to try third party solutions that can help.

On Elpa there's oauth2.el which provides Oauth2 support for Emacs. There
are externals who implemented oauth for Gmail[4] and Microsoft 365[5]
through the use of oauth2.el.

However these don't handle the oauth workflow of acquiring the token.
It is possible to try to do that inside emacs or use an external browser
and then catch the response or make the user copy the response address
into Emacs.

The main issue to implement this I think is to have an official "appid"
for Emacs and add the Oauth2 workflow.
I don't know about google right now but for Microsoft 365 this seams
feasible as there's just an appid that can be stored publicly. 


Br,

Björn Bidar

---

[1] https://techcommunity.microsoft.com/t5/exchange-team-blog/improving-security-together/ba-p/805892
[2] https://invent.kde.org/pim/kdepim-runtime/-/tree/master/resources/ews/ewsclient/auth
[3] https://learn.microsoft.com/en-us/azure/active-directory/develop/v2-oauth2-auth-code-flow
[4] https://github.com/ggervasio/gnus-gmail-oauth/
[5] https://gitlab.com/Binary-Eater/gnus-o365-oauth2/-/tree/master

bug#41386: 28.0.50; Gnus nnimap OAuth 2.0 support

[Deus Max]

On Sat, Oct 29 2022, Björn Bidar via "Bug reports for GNU Emacs, the Swiss army knife of text editors" wrote:

> I know this bug is marked as wontfix however more and more providers are
> moving to Oauth2, even those that use plain standards like
> imap/{cal,card}dav/smpt, thous increasing the importance of Oauth2
> support in Gnus.
>
Ditto. OAuth2 authentication is now needed, not just nice to have.
Please re-open ticket.
   
> Some providers like Microsoft require it next year and the employer
> can already enforce the use of Oauth2 [1].
> The argument "just use another email provider" doesn't really work in
> such cases.

Microsoft disabled Basic Authentication for POP, IMAP and other
protocols on Oct. 01, 2022. As you may know, a lot of work places use
office365.
https://answers.microsoft.com/en-us/outlook_com/forum/all/imap-password-not-working/d4432df0-f7ea-42b7-90e7-2cf1e0e02e4a

My understanding from reading M$ documentation is that now OAuth2 is the
only method for a client to connect to office365 IMAP. Correct me please
if I'm wrong.

>
> Without proper OAuth2 support there is no use for Gnus for such users,
> except to try third party solutions that can help.
>

Sadly true.