Emacs HTTP libraries [was: Re: How to contribute new package to GNU ELPA?]

[Adam Porter <adam@alphapapa.net>]

stardiviner <numbchild@gmail.com> writes:

>  More importantly: what do you want to do about `request`?
>
> About this, I really suggest ELPA can include it. Because I hate to
> write more complex url requests with `url.el`.  If someone want to
> adopt kiwix.el request to use `url.el`. I also can't accept
> it. Because I still need to write code on it.  Also as you said, a
> wrapper on `url.el` is good for developer. Emacs will need it. Really
> helpful.

Emacs definitely needs an alternative to url.el.  It has some issues
that are not easily solved nor worked around, such as potentially
calling callbacks multiple times[0], requests not timing out, and some
of its API being in the form of undocumented variables that must be
let-bound around calls to its functions.

I wrote a macro that wraps `url-retrieve' to help alleviate some of the
issues.[1]  Using it in matrix-client.el seems to make its use of url.el
much more reliable (requests time out properly, etc).  But I don't know
if it would be suitable for general use, and it basically "papers over"
some of these issues.

request.el is a widely used library, and it works well for many things.
However, it has its own not-easily-resolved issues that, IME, forced me
to use url.el in the end--except for the things which url.el couldn't
handle properly, such as some file uploads, which forced me to use
request.el for those.  Having to use two different HTTP libraries in
matrix-client.el is...not ideal.

Besides that, I doubt that request.el could be easily merged into ELPA
since, apparently, not all of its contributors have done FSF copyright
assignment.  An issue that was raised about merging it into ELPA was
closed without discussion or consideration.[2]  And that leads to another
concern, one I raised about its current governance, which has been
ignored for a year now.[3]

As Chris Wellons noted in some discussions I've had with him[4], it's
relatively simple to write a Curl wrapper in Elisp that precludes having
to use url.el.  He did this for Elfeed, and it works very well.
However, quite understandably, he isn't interested in factoring it out
and maintaining it as a library.

So in Ement.el[5], I've been using a simple Curl wrapper library that
I've started.[6]  It's rather primitive, but it seems to work well.
It's relatively easy to use, IMHO.  I haven't written documentation yet
(other than docstrings), but you can see how it's used in the tests.[7]
If it were useful, I'd be glad to contribute it to ELPA.

Of course, it would be ideal if Emacs had a built-in HTTP library that
didn't require an external dependency like Curl, but Curl does work
well.  The only issue I have with it is that certain data can only be
passed via either the command line (leaving it visible to the whole
system) or temporary files (which have to be carefully handled), which
is a concern for, e.g. cookies, authentication tokens, passwords, etc.
Unfortunately, when that issue was raised and a solution proposed in
2003, it was quickly rejected.[8]

It's primarily that issue that has left me hesitant to advertise plz.el,
because I wouldn't want to recommend that anyone use it for sensitive
applications, even if the risk is probably minimal on one's personal
system.  Nevertheless, that issue remains in request.el when it uses its
Curl backend, so it's not worse in that respect.

Well, food for thought.  :)

Adam

0: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=20159
1: https://github.com/alphapapa/elexandria/blob/83a1b08d0711fdce07a5b33525535cc3a457c6ee/elexandria.el#L429
2: https://github.com/tkf/emacs-request/issues/112
3: https://github.com/tkf/emacs-request/issues/172
4: https://github.com/skeeto/elisp-latch/issues/1#issuecomment-397910988
5: https://github.com/alphapapa/ement.el
6: https://github.com/alphapapa/plz.el
7: https://github.com/alphapapa/plz.el/blob/master/tests/test-plz.el
8: https://curl.se/mail/archive-2003-08/0099.html