Re: ELPA submission: plz-see

[Adam Porter <adam@alphapapa.net>]

>> On Sat, Nov 04, 2023 at 08:49:24AM +0200, Eli Zaretskii wrote:
>> > > Date: Sat, 4 Nov 2023 00:08:52 -0500
>> > > Cc: brickviking@gmail.com, eliz@gnu.org, emacs-devel@gnu.org,
>> > >  philipk@posteo.net
>> > > From: Adam Porter <adam@alphapapa.net>
>> > > 
>> > > As has been said on emacs-devel several times over the years, the next 
>> > > significant step in this field would most likely be for Emacs to 
>> > > integrate libcurl directly.  Maybe someday that will happen...
>> > 
>> > Before that happens, I'd like to see a good analysis of why we cannot
>> > make our own implementation of network communications performant
>> > enough to avoid yet another external library.
>> 
>> Any pointers as to where the problems with url lie?
> 
> I don't know.  If anyone else does, please speak up.  the claim is
> that fetching a URL with our built-in primitives is too slow.

Forgive me, but that is not the claim.  There are numerous problems 
experienced with using url.el "in anger," as well as with other HTTP 
client libraries for Emacs.  Stefan K. was kind enough to link one of my 
posts about this issue a few years ago, when plz.el was earlier in its 
development (the issues mentioned having been solved now)[0].  Briefly, 
the problems include unreliability (e.g. callbacks being called multiple 
times or not at all, timeouts that don't happen), very awkward API (e.g. 
having to let-bind often-undocumented variables to do basic HTTP 
operations), and, perhaps also, inferior performance compared to a 
curl-based implementation (curl being implemented in a separate process, 
in C).

To reiterate what was said over a year ago when plz.el was added to GNU 
ELPA: I spent years trying to *not* develop plz.el, trying to work with 
and around the built-in url.el and the then-MELPA-hosted request.el, in 
packages of mine that used HTTP extensively.  I wrote wrapper macros and 
workarounds and hacks that used both libraries as necessary to work 
around one problem or another.

Finally, with the advice and consent of Chris Wellons, a highly 
respected hacker both inside and outside the Emacs arena, who had spent 
even more time dealing with those issues than I had (and he filed bug 
reports about them, as well), I developed plz.el based on his 
elfeed-curl.el library, which he had developed for his Elfeed RSS 
client, having been battle-tested for years.

Over the past 3 years I hammered on it further, eventually solving 
issues related to Emacs' process-related idiosyncrasies (e.g. [1], [2]). 
  It is now, as far as I know, completely reliable: never dropping 
requests, never failing to call callbacks or calling them multiple 
times, never failing to timeout, etc, even when hundreds or thousands of 
requests are made in parallel from a single Emacs process.

What began as a last resort has, with the help of several other 
developers and users to whom I am grateful, become what is, IMHO, the 
best HTTP client library for Emacs (modulo any yet-to-be-implemented 
features which may be present in other clients).

I confess to feeling a bit annoyed at having to re-justify the inclusion 
of plz.el in GNU ELPA (or, at least, I feel like I'm having to do so). 
As I said, there are multiple downstream clients that depend on it, 
including my own Ement.el in GNU ELPA, which is a widely used client for 
the Matrix protocol.

As well, plz.el is now distributed directly in Debian, Ubuntu, Devuan, 
Kali Linux, PureOS, Raspbian, Gentoo, GNU Guix, and NixOS.

Having said all that, of course it would be best if Emacs's built-in 
library were the best one.  But as the author of url.el himself has 
admitted, it has some fundamental issues which are not easy to solve, 
and his own branch related to improving it hasn't been touched in 6 
years[3].  And so far, no one has stepped up to integrate libcurl or 
similar.

So this is where we are with respect to Emacs HTTP libraries.  I 
understand that some people don't like the name, and it wasn't my first 
choice, either.  But the name is of very little importance, and surely 
no more difficult to remember than an API, of which there are 
innumerable ones in Emacs.

--Adam

0: https://lists.gnu.org/r/emacs-devel/2020-12/msg01217.html
1: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=50166
2: https://github.com/alphapapa/plz.el/issues/3
3: 
https://git.savannah.gnu.org/cgit/emacs.git/log/?h=scratch/with-fetched-url