[GNU ELPA] New package: url-http-oauth

[GNU ELPA] New package: url-http-oauth

[Thomas Fitzsimmons]

Hi,

I would like to add one or two new packages to GNU ELPA.

The main one is url-http-oauth, which adds OAuth 2.0 support to the URL
library, via "url-auth" hooks, like url-http-basic, url-http-digest and
url-http-ntlm.  It provides auth-source integration for secrets, using
the netrc backend.

   https://git.sr.ht/~fitzsim/url-http-oauth

This package is unrelated to oauth2.el in GNU ELPA, which provides new
oauth2-url-retrieve and oauth2-url-retrieve-synchronously functions, and
has plstore instead of auth-source integration.  For Excorporate, I
needed something that would work with the built-in url-retrieve
functions and I couldn't see how to do that with oauth2.el.  I haven't
tested, but I see no reason that the two packages would interfere with
one another.

For the next release of Excorporate I want to depend on url-http-oauth
to fix the longstanding bug#50113, "Excorporate: Communicating with
domain that requires SSO?".  I have the changes ready; I am using them
daily.

The OAuth 2.0 standard encodes the use of a user-agent (i.e., web
browser) for authorization steps which differ per OAuth 2.0 provider,
and are not defined by the specification.  I have provided support for
package authors and users to write custom functions to automate these
web browser interactions in arbitrary ways:

   AUTHORIZATION-CODE-FUNCTION is an elisp function that takes an
   authorization URL as a string argument, and returns, as a string, a
   full URL containing a code value in its query string.

By default though, url-http-oauth will prompt the user to copy-n-paste
URLs to and from the web browser.  This is the most general default I
could think of; for example, this allows performing authorization in a
local web browser then pasting the result to an Emacs session running
three SSH hops away, where `browse-url' may not do the right thing.

Users and package authors can design automatic user-agent interactions,
but those ways are so varied that I wanted to see how they would evolve.
For example, I would like to see someone write an
authorization-code-function for Sourcehut that would use EWW inline.
For other OAuth 2.0 providers whose authorization steps require
JavaScript, EWW would not work.

I have published another, tiny package:

   https://git.sr.ht/~fitzsim/url-http-oauth-demo

It demonstrates the use of url-http-oauth against the Emacs-friendliest
OAuth 2.0 implementation I've found: Sourcehut.  Sourcehut's
implementation is entirely Free Software, and it does not require
JavaScript in the authorization steps.  Its client registration process
does not have onerous terms of use.  Any Sourcehut user should be able
to get url-http-oauth-demo working.  Maybe this package makes sense in
GNU ELPA, or perhaps it could be part of url-http-oauth's documentation.

I wrote these packages myself [1] and I have copyright assignment
paperwork on file.

Thomas

1. Except url-http-oauth--netrc-delete, which borrows lots of code from
   auth-source.el.

Re: [GNU ELPA] New package: url-http-oauth

[Philip Kaludercic]

Thomas Fitzsimmons <fitzsim@fitzsim.org> writes:

> Hi,
>
> I would like to add one or two new packages to GNU ELPA.

I have no objections to this, but I don't have any comments either.

> The main one is url-http-oauth, which adds OAuth 2.0 support to the URL
> library, via "url-auth" hooks, like url-http-basic, url-http-digest and
> url-http-ntlm.  It provides auth-source integration for secrets, using
> the netrc backend.
>
>    https://git.sr.ht/~fitzsim/url-http-oauth
>
> This package is unrelated to oauth2.el in GNU ELPA, which provides new
> oauth2-url-retrieve and oauth2-url-retrieve-synchronously functions, and
> has plstore instead of auth-source integration.  For Excorporate, I
> needed something that would work with the built-in url-retrieve
> functions and I couldn't see how to do that with oauth2.el.  I haven't
> tested, but I see no reason that the two packages would interfere with
> one another.
>
> For the next release of Excorporate I want to depend on url-http-oauth
> to fix the longstanding bug#50113, "Excorporate: Communicating with
> domain that requires SSO?".  I have the changes ready; I am using them
> daily.
>
> The OAuth 2.0 standard encodes the use of a user-agent (i.e., web
> browser) for authorization steps which differ per OAuth 2.0 provider,
> and are not defined by the specification.  I have provided support for
> package authors and users to write custom functions to automate these
> web browser interactions in arbitrary ways:
>
>    AUTHORIZATION-CODE-FUNCTION is an elisp function that takes an
>    authorization URL as a string argument, and returns, as a string, a
>    full URL containing a code value in its query string.
>
> By default though, url-http-oauth will prompt the user to copy-n-paste
> URLs to and from the web browser.  This is the most general default I
> could think of; for example, this allows performing authorization in a
> local web browser then pasting the result to an Emacs session running
> three SSH hops away, where `browse-url' may not do the right thing.
>
> Users and package authors can design automatic user-agent interactions,
> but those ways are so varied that I wanted to see how they would evolve.
> For example, I would like to see someone write an
> authorization-code-function for Sourcehut that would use EWW inline.
> For other OAuth 2.0 providers whose authorization steps require
> JavaScript, EWW would not work.
>
> I have published another, tiny package:
>
>    https://git.sr.ht/~fitzsim/url-http-oauth-demo
>
> It demonstrates the use of url-http-oauth against the Emacs-friendliest
> OAuth 2.0 implementation I've found: Sourcehut.  Sourcehut's
> implementation is entirely Free Software, and it does not require
> JavaScript in the authorization steps.  Its client registration process
> does not have onerous terms of use.  Any Sourcehut user should be able
> to get url-http-oauth-demo working.  Maybe this package makes sense in
> GNU ELPA, or perhaps it could be part of url-http-oauth's documentation.
>
> I wrote these packages myself [1] and I have copyright assignment
> paperwork on file.
>
> Thomas
>
> 1. Except url-http-oauth--netrc-delete, which borrows lots of code from
>    auth-source.el.
>
>

-- 
Philip Kaludercic

Re: [GNU ELPA] New package: url-http-oauth

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 3560 bytes --]

I think packages such as this one are very welcome, and I appreciate the
UI-agnostic architecture you describe.

Url.el itself, last i looked, seemed a bit outdated and hard to work with.
Are there other http client libraries for Emacs? Could they also take
advantage of your new library?

João

On Sat, May 6, 2023, 06:07 Thomas Fitzsimmons <fitzsim@fitzsim.org> wrote:

> Hi,
>
> I would like to add one or two new packages to GNU ELPA.
>
> The main one is url-http-oauth, which adds OAuth 2.0 support to the URL
> library, via "url-auth" hooks, like url-http-basic, url-http-digest and
> url-http-ntlm.  It provides auth-source integration for secrets, using
> the netrc backend.
>
>    https://git.sr.ht/~fitzsim/url-http-oauth
>
> This package is unrelated to oauth2.el in GNU ELPA, which provides new
> oauth2-url-retrieve and oauth2-url-retrieve-synchronously functions, and
> has plstore instead of auth-source integration.  For Excorporate, I
> needed something that would work with the built-in url-retrieve
> functions and I couldn't see how to do that with oauth2.el.  I haven't
> tested, but I see no reason that the two packages would interfere with
> one another.
>
> For the next release of Excorporate I want to depend on url-http-oauth
> to fix the longstanding bug#50113, "Excorporate: Communicating with
> domain that requires SSO?".  I have the changes ready; I am using them
> daily.
>
> The OAuth 2.0 standard encodes the use of a user-agent (i.e., web
> browser) for authorization steps which differ per OAuth 2.0 provider,
> and are not defined by the specification.  I have provided support for
> package authors and users to write custom functions to automate these
> web browser interactions in arbitrary ways:
>
>    AUTHORIZATION-CODE-FUNCTION is an elisp function that takes an
>    authorization URL as a string argument, and returns, as a string, a
>    full URL containing a code value in its query string.
>
> By default though, url-http-oauth will prompt the user to copy-n-paste
> URLs to and from the web browser.  This is the most general default I
> could think of; for example, this allows performing authorization in a
> local web browser then pasting the result to an Emacs session running
> three SSH hops away, where `browse-url' may not do the right thing.
>
> Users and package authors can design automatic user-agent interactions,
> but those ways are so varied that I wanted to see how they would evolve.
> For example, I would like to see someone write an
> authorization-code-function for Sourcehut that would use EWW inline.
> For other OAuth 2.0 providers whose authorization steps require
> JavaScript, EWW would not work.
>
> I have published another, tiny package:
>
>    https://git.sr.ht/~fitzsim/url-http-oauth-demo
>
> It demonstrates the use of url-http-oauth against the Emacs-friendliest
> OAuth 2.0 implementation I've found: Sourcehut.  Sourcehut's
> implementation is entirely Free Software, and it does not require
> JavaScript in the authorization steps.  Its client registration process
> does not have onerous terms of use.  Any Sourcehut user should be able
> to get url-http-oauth-demo working.  Maybe this package makes sense in
> GNU ELPA, or perhaps it could be part of url-http-oauth's documentation.
>
> I wrote these packages myself [1] and I have copyright assignment
> paperwork on file.
>
> Thomas
>
> 1. Except url-http-oauth--netrc-delete, which borrows lots of code from
>    auth-source.el.
>
>

[-- Attachment #2: Type: text/html, Size: 4320 bytes --]

Re: [GNU ELPA] New package: url-http-oauth

[Philip Kaludercic]

João Távora <joaotavora@gmail.com> writes:

> I think packages such as this one are very welcome, and I appreciate the
> UI-agnostic architecture you describe.
>
> Url.el itself, last i looked, seemed a bit outdated and hard to work with.
> Are there other http client libraries for Emacs? Could they also take
> advantage of your new library?

Adam Porter recently wrote a library called "plz" which was added to GNU
ELPA: http://elpa.gnu.org/packages/plz.html

> João
>
> On Sat, May 6, 2023, 06:07 Thomas Fitzsimmons <fitzsim@fitzsim.org> wrote:
>
>> Hi,
>>
>> I would like to add one or two new packages to GNU ELPA.
>>
>> The main one is url-http-oauth, which adds OAuth 2.0 support to the URL
>> library, via "url-auth" hooks, like url-http-basic, url-http-digest and
>> url-http-ntlm.  It provides auth-source integration for secrets, using
>> the netrc backend.
>>
>>    https://git.sr.ht/~fitzsim/url-http-oauth
>>
>> This package is unrelated to oauth2.el in GNU ELPA, which provides new
>> oauth2-url-retrieve and oauth2-url-retrieve-synchronously functions, and
>> has plstore instead of auth-source integration.  For Excorporate, I
>> needed something that would work with the built-in url-retrieve
>> functions and I couldn't see how to do that with oauth2.el.  I haven't
>> tested, but I see no reason that the two packages would interfere with
>> one another.
>>
>> For the next release of Excorporate I want to depend on url-http-oauth
>> to fix the longstanding bug#50113, "Excorporate: Communicating with
>> domain that requires SSO?".  I have the changes ready; I am using them
>> daily.
>>
>> The OAuth 2.0 standard encodes the use of a user-agent (i.e., web
>> browser) for authorization steps which differ per OAuth 2.0 provider,
>> and are not defined by the specification.  I have provided support for
>> package authors and users to write custom functions to automate these
>> web browser interactions in arbitrary ways:
>>
>>    AUTHORIZATION-CODE-FUNCTION is an elisp function that takes an
>>    authorization URL as a string argument, and returns, as a string, a
>>    full URL containing a code value in its query string.
>>
>> By default though, url-http-oauth will prompt the user to copy-n-paste
>> URLs to and from the web browser.  This is the most general default I
>> could think of; for example, this allows performing authorization in a
>> local web browser then pasting the result to an Emacs session running
>> three SSH hops away, where `browse-url' may not do the right thing.
>>
>> Users and package authors can design automatic user-agent interactions,
>> but those ways are so varied that I wanted to see how they would evolve.
>> For example, I would like to see someone write an
>> authorization-code-function for Sourcehut that would use EWW inline.
>> For other OAuth 2.0 providers whose authorization steps require
>> JavaScript, EWW would not work.
>>
>> I have published another, tiny package:
>>
>>    https://git.sr.ht/~fitzsim/url-http-oauth-demo
>>
>> It demonstrates the use of url-http-oauth against the Emacs-friendliest
>> OAuth 2.0 implementation I've found: Sourcehut.  Sourcehut's
>> implementation is entirely Free Software, and it does not require
>> JavaScript in the authorization steps.  Its client registration process
>> does not have onerous terms of use.  Any Sourcehut user should be able
>> to get url-http-oauth-demo working.  Maybe this package makes sense in
>> GNU ELPA, or perhaps it could be part of url-http-oauth's documentation.
>>
>> I wrote these packages myself [1] and I have copyright assignment
>> paperwork on file.
>>
>> Thomas
>>
>> 1. Except url-http-oauth--netrc-delete, which borrows lots of code from
>>    auth-source.el.
>>
>>

Re: [GNU ELPA] New package: url-http-oauth

[T.V Raman]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=gb18030, Size: 681 bytes --]


url.el is definitely outdated and  likely one of the bottlenecks that
holds back EWW; the other is of course JS.

I looked at plz.el when it came out and it is nice; by  depending on
Curl, it immediately brings Emacs up-to-date with various URL bits that
have languished in url.el. That said, for Emacs to be able to move on
from url.el, I suspect we'll need an 80/20 plz wrapper that if loaded by
the user causes calls to the url library to be overridden by plz so that
one can then use EWW to validate the effectiveness and completeness of
plz.

Any Takers?


-- 

Thanks,

--Raman(I Search, I Find, I Misplace, I Research)
7©4 Id: kg:/m/0285kf1  •0Ü8

Re: [GNU ELPA] New package: url-http-oauth

[João Távora]

On Mon, May 8, 2023 at 3:22 PM T.V Raman <raman@google.com> wrote:
>
>
> url.el is definitely outdated and  likely one of the bottlenecks that
> holds back EWW; the other is of course JS.
>
> I looked at plz.el when it came out and it is nice; by  depending on
> Curl, it immediately brings Emacs up-to-date with various URL bits that
> have languished in url.el. That said, for Emacs to be able to move on
> from url.el, I suspect we'll need an 80/20 plz wrapper that if loaded by
> the user causes calls to the url library to be overridden by plz so that
> one can then use EWW to validate the effectiveness and completeness of
> plz.

Hmm, OK, thanks for mentioning this.  Does it depend on curl or libcurl?
Starting a new process for every request seems not ideal, and
defeats the purpose of HTTP keep-alive.  But I'm just speculating
here, maybe that's not how it works.

An elisp HTTP client based on process-filters shouldn't be _that_
hard to write though.  AFAIR the ugly bits of url.el weren't about
that, but just run-of-the-mill API quirks about response codes and
such.

João

Re: [GNU ELPA] New package: url-http-oauth

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Mon, 8 May 2023 17:19:30 +0100
> Cc: Thomas Fitzsimmons <fitzsim@fitzsim.org>, emacs-devel <emacs-devel@gnu.org>
> Does it depend on curl or libcurl?

The former.

Re: [GNU ELPA] New package: url-http-oauth

[T.V Raman]

I agree that using libcurl would be better, but that proposal has gone
nowhere. plz uses curl s a separate process but it might be a good
stepping stone to  eventually moving to a libcurl based solution.


João Távora writes:
 > On Mon, May 8, 2023 at 3:22 PM T.V Raman <raman@google.com> wrote:
 > >
 > >
 > > url.el is definitely outdated and  likely one of the bottlenecks that
 > > holds back EWW; the other is of course JS.
 > >
 > > I looked at plz.el when it came out and it is nice; by  depending on
 > > Curl, it immediately brings Emacs up-to-date with various URL bits that
 > > have languished in url.el. That said, for Emacs to be able to move on
 > > from url.el, I suspect we'll need an 80/20 plz wrapper that if loaded by
 > > the user causes calls to the url library to be overridden by plz so that
 > > one can then use EWW to validate the effectiveness and completeness of
 > > plz.
 > 
 > Hmm, OK, thanks for mentioning this.  Does it depend on curl or libcurl?
 > Starting a new process for every request seems not ideal, and
 > defeats the purpose of HTTP keep-alive.  But I'm just speculating
 > here, maybe that's not how it works.
 > 
 > An elisp HTTP client based on process-filters shouldn't be _that_
 > hard to write though.  AFAIR the ugly bits of url.el weren't about
 > that, but just run-of-the-mill API quirks about response codes and
 > such.
 > 
 > João

-- 

Thanks,

--Raman(I Search, I Find, I Misplace, I Research)
♉ Id: kg:/m/0285kf1  🦮

--

Thanks,

--Raman(I Search, I Find, I Misplace, I Research)
♉ Id: kg:/m/0285kf1  🦮

Re: [GNU ELPA] New package: url-http-oauth

[Thomas Fitzsimmons]

Hi João,

João Távora <joaotavora@gmail.com> writes:

> I think packages such as this one are very welcome, and I appreciate the
> UI-agnostic architecture you describe.

Thanks.

> Url.el itself, last i looked, seemed a bit outdated and hard to work
> with.

I find it OK to work with.  It's primarily elisp, vs some other
proposals I've seen, so I can step through most of it with edebug.
There's one FIXME in url-http-oauth currently that may represent a
url.el oddity (or may represent me doing something wrong):

;; FIXME: if anything goes wrong during the authentication steps,
;; `url-http-end-of-document-sentinel' calls back into
;; `url-oauth-auth' somehow.  Maybe `url-http-no-retry' can help here?

I'm sure other client libraries would have oddities too though.

Are there bugs filed for the specific ways in which url.el may be
outdated or hard to work with?  Maybe someone could maintain a
collection of links to those bug reports, to see how contenders measure
up?  Of course, that list wouldn't represent the upsides of url.el, such
as the elisp purity I mentioned.

> Are there other http client libraries for Emacs?

Today url.el works for me, so I don't have plans to change; I'll
probably just follow what makes it into Emacs core.

> Could they also take advantage of your new library?

If the new library implemented url.el's APIs, then sure.  I would accept
well-defined/minimal compatibility patches for other client libraries
too, of course.

Thomas

Re: [GNU ELPA] New package: url-http-oauth

[João Távora]

On Mon, May 8, 2023 at 4:27 PM Thomas Fitzsimmons <fitzsim@fitzsim.org> wrote:

> > Url.el itself, last i looked, seemed a bit outdated and hard to work
> > with.
>
> I find it OK to work with.  It's primarily elisp, vs some other
> proposals I've seen, so I can step through most of it with edebug.

Yes, being elisp is indeed a nice property.

> There's one FIXME in url-http-oauth currently that may represent a
> url.el oddity (or may represent me doing something wrong):
>
> ;; FIXME: if anything goes wrong during the authentication steps,
> ;; `url-http-end-of-document-sentinel' calls back into
> ;; `url-oauth-auth' somehow.  Maybe `url-http-no-retry' can help here?
>
> I'm sure other client libraries would have oddities too though.

Well, the point was to pick one that didn't have as many
oddities, i.e. a good one.

> Are there bugs filed for the specific ways in which url.el may be
> outdated or hard to work with?  Maybe someone could maintain a
> collection of links to those bug reports, to see how contenders measure
> up?  Of course, that list wouldn't represent the upsides of url.el, such
> as the elisp purity I mentioned.
>
> > Are there other http client libraries for Emacs?
>
> Today url.el works for me, so I don't have plans to change; I'll
> probably just follow what makes it into Emacs core.

I remember url.el and finding it satisfactory enough
for many things.  Can't really remember any specific
criticism, just the overall impression that it was
a bit overcomplicated and didn't make use of lexical
binding, for example.

> > Could they also take advantage of your new library?
>
> If the new library implemented url.el's APIs, then sure.  I would accept
> well-defined/minimal compatibility patches for other client libraries
> too, of course.

Yes, makes sense.  Well, anyway don't let this comment stop
url-http-oauth from entering into ELPA.

João