ELPA submission: plz-see

ELPA submission: plz-see

[Augusto Stoffel]

Hi,

I would like to submit the following package to ELPA:

  https://github.com/astoff/plz-see.el

It is an interactive HTTP client in the sense that request responses are
pretty-printed in a pop-up buffer. It is useful e.g. to test REST APIs.

There exist similar packages, such as restclient or verb.  Unlike those
alternatives, plz-see is based on Elisp functions rather than reading
instructions from some sort of notebook-like plain text file.

Re: ELPA submission: plz-see

[Eli Zaretskii]

> From: Augusto Stoffel <arstoffel@gmail.com>
> Date: Mon, 30 Oct 2023 08:17:30 +0100
> 
> I would like to submit the following package to ELPA:
> 
>   https://github.com/astoff/plz-see.el
> 
> It is an interactive HTTP client in the sense that request responses are
> pretty-printed in a pop-up buffer. It is useful e.g. to test REST APIs.

I wonder if you could come up with a name that tells something about
the package's purpose.  "plz-see" tells me "please see", and I don't
see any relation to HTTP, REST, pretty-printing, or popup buffers.

Thanks.

Re: ELPA submission: plz-see

[Stefan Kangas]

Augusto Stoffel <arstoffel@gmail.com> writes:

> I would like to submit the following package to ELPA:
>
>   https://github.com/astoff/plz-see.el

To clarify, do you mean GNU ELPA here?

Re: ELPA submission: plz-see

[Augusto Stoffel]

On Mon, 30 Oct 2023 at 13:56, Eli Zaretskii wrote:

>> From: Augusto Stoffel <arstoffel@gmail.com>
>> Date: Mon, 30 Oct 2023 08:17:30 +0100
>> 
>> I would like to submit the following package to ELPA:
>> 
>>   https://github.com/astoff/plz-see.el
>> 
>> It is an interactive HTTP client in the sense that request responses are
>> pretty-printed in a pop-up buffer. It is useful e.g. to test REST APIs.
>
> I wonder if you could come up with a name that tells something about
> the package's purpose.  "plz-see" tells me "please see", and I don't
> see any relation to HTTP, REST, pretty-printing, or popup buffers.

It is based on the plz package, which users will know.  As to the
relation with seeing, when a buffer pops up, you see its content.

Re: ELPA submission: plz-see

[Augusto Stoffel]

On Mon, 30 Oct 2023 at 15:33, Stefan Kangas wrote:

> Augusto Stoffel <arstoffel@gmail.com> writes:
>
>> I would like to submit the following package to ELPA:
>>
>>   https://github.com/astoff/plz-see.el
>
> To clarify, do you mean GNU ELPA here?

Yes!

Re: ELPA submission: plz-see

[Philip Kaludercic]

[-- Attachment #1: Type: text/plain, Size: 274 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Augusto Stoffel <arstoffel@gmail.com>
>> Date: Mon, 30 Oct 2023 08:17:30 +0100
>> 
>> I would like to submit the following package to ELPA:
>> 
>>   https://github.com/astoff/plz-see.el

Loks good, I only have a few comments:


[-- Attachment #2: Type: text/plain, Size: 1492 bytes --]

diff --git a/plz-see.el b/plz-see.el
index 2e5451f..74d62b1 100644
--- a/plz-see.el
+++ b/plz-see.el
@@ -4,6 +4,8 @@
 
 ;; Author: Augusto Stoffel <arstoffel@gmail.com>
 ;; Keywords: comm, network, http
+;; Version: 1.0.0
+;; Package-Requires: ((emacs "29.1"))
 
 ;; This program is free software; you can redistribute it and/or modify
 ;; it under the terms of the GNU General Public License as published by
@@ -20,10 +22,12 @@
 
 ;;; Commentary:
 
-;; 
+;; Please write a brief commentary section, that is useful when
+;; querying the package from a `describe-package' buffer.
 
 ;;; Code:
 
+(eval-when-compile (require 'cl-lib))
 (require 'json)
 (require 'plz)
 
@@ -57,7 +61,7 @@ If nil, never delete old response buffers."
 
 (defcustom plz-see-display-action nil
   "The ACTION argument `plz-see' passes to `display-buffer'."
-  :type 'sexp)
+  :type display-buffer--action-custom-type)
 
 (defcustom plz-see-header-line-format
   (let ((headers '(plz-see-header-line-status
@@ -181,7 +185,7 @@ call and AS specifies the argument type they expect."
                      ('response response)
                      ('buffer buffer)
                      ((or 'binary 'string 'file `(file ,_))
-                      (user-error "plz-see does not accept :as %s" as))
+                      (user-error "`plz-see' does not accept :as %s" as))
                      ((pred functionp)
                       (with-temp-buffer
                         (insert (plz-response-body response))

[-- Attachment #3: Type: text/plain, Size: 617 bytes --]


>> It is an interactive HTTP client in the sense that request responses are
>> pretty-printed in a pop-up buffer. It is useful e.g. to test REST APIs.
>
> I wonder if you could come up with a name that tells something about
> the package's purpose.  "plz-see" tells me "please see", and I don't
> see any relation to HTTP, REST, pretty-printing, or popup buffers.

I agree, from what I see this is just an implementation detail.  Plz is
a peculiar enough name as it is, it shouldn't be inherited by packages
that depend on it.  Or is there any reason why the package couldn't also
use url.el?

-- 
Philip Kaludercic

Re: ELPA submission: plz-see

[Mauro Aranda]

On 30/10/23 04:17, Augusto Stoffel wrote:
 >
 > I would like to submit the following package to ELPA:
 >
 >    https://github.com/astoff/plz-see.el
 >
 >
 > (defcustom plz-see-header-line-format
 >   (let ((headers '(plz-see-header-line-status
 >                    plz-see-header-line-content-type
 >                    plz-see-header-line-content-length
 >                    plz-see-header-line-show-headers)))
 >     (dolist (sym headers)
 >       (put sym 'risky-local-variable t))
 >     (cons "" headers))
 >   "Header line format for `plz-see' result buffers."
 >   :type 'sexp)

Do you really need all that in the standard expression?

Re: ELPA submission: plz-see

[Augusto Stoffel]

Thanks for the comments!

On Tue, 31 Oct 2023 at 08:43, Philip Kaludercic wrote:

> +(eval-when-compile (require 'cl-lib))
>  (require 'json)

By the way, the latter is kind of a non-essential require, would you
rather write this near line 88?

(defcustom plz-see-content-type-alist
  `(...
    ("\\`application/json" . ,(lambda ()
                                (require 'json)
                                (declare-function 'json-pretty-print-buffer "json.el")
                                (json-pretty-print-buffer)
                                (js-json-mode)))
  ...))


> I agree, from what I see this is just an implementation detail.  Plz is
> a peculiar enough name as it is, it shouldn't be inherited by packages
> that depend on it.  Or is there any reason why the package couldn't also
> use url.el?

The packages is really closely tied to plz's API, which by the way is a
very nice one.

Re: ELPA submission: plz-see

[Augusto Stoffel]

On Tue, 31 Oct 2023 at 06:33, Mauro Aranda wrote:

> On 30/10/23 04:17, Augusto Stoffel wrote:
>>
>> I would like to submit the following package to ELPA:
>>
>>    https://github.com/astoff/plz-see.el
>>
>>
>> (defcustom plz-see-header-line-format
>>   (let ((headers '(plz-see-header-line-status
>>                    plz-see-header-line-content-type
>>                    plz-see-header-line-content-length
>>                    plz-see-header-line-show-headers)))
>>     (dolist (sym headers)
>>       (put sym 'risky-local-variable t))
>>     (cons "" headers))
>>   "Header line format for `plz-see' result buffers."
>>   :type 'sexp)
>
> Do you really need all that in the standard expression?

You're right, this doesn't actually work since the default value is not
evaluated if you set that variable before loading the package.  Thanks.

Re: ELPA submission: plz-see

[Stefan Kangas]

Augusto Stoffel <arstoffel@gmail.com> writes:

>> To clarify, do you mean GNU ELPA here?
>
> Yes!

OK.  Let us know when you feel like you've addressed the feedback you've
gotten, and let's add it to the archive.

Re: ELPA submission: plz-see

[Augusto Stoffel]

On Tue, 31 Oct 2023 at 17:37, Stefan Kangas wrote:

> Augusto Stoffel <arstoffel@gmail.com> writes:
>
>>> To clarify, do you mean GNU ELPA here?
>>
>> Yes!
>
> OK.  Let us know when you feel like you've addressed the feedback you've
> gotten, and let's add it to the archive.

It's ready for publishing.  Thanks again!

Re: ELPA submission: plz-see

[Philip Kaludercic]

Augusto Stoffel <arstoffel@gmail.com> writes:

> Thanks for the comments!
>
> On Tue, 31 Oct 2023 at 08:43, Philip Kaludercic wrote:
>
>> +(eval-when-compile (require 'cl-lib))
>>  (require 'json)
>
> By the way, the latter is kind of a non-essential require, would you
> rather write this near line 88?
>
> (defcustom plz-see-content-type-alist
>   `(...
>     ("\\`application/json" . ,(lambda ()
>                                 (require 'json)
>                                 (declare-function 'json-pretty-print-buffer "json.el")
>                                 (json-pretty-print-buffer)
>                                 (js-json-mode)))
>   ...))

I don't have a strong opinion either way.

>> I agree, from what I see this is just an implementation detail.  Plz is
>> a peculiar enough name as it is, it shouldn't be inherited by packages
>> that depend on it.  Or is there any reason why the package couldn't also
>> use url.el?
>
> The packages is really closely tied to plz's API, which by the way is a
> very nice one.

But is this fact exposed to the user?

-- 
Philip Kaludercic

Re: ELPA submission: plz-see

[Augusto Stoffel]

On Wed,  1 Nov 2023 at 09:14, Philip Kaludercic wrote:

>>> I agree, from what I see this is just an implementation detail.  Plz is
>>> a peculiar enough name as it is, it shouldn't be inherited by packages
>>> that depend on it.  Or is there any reason why the package couldn't also
>>> use url.el?
>>
>> The packages is really closely tied to plz's API, which by the way is a
>> very nice one.
>
> But is this fact exposed to the user?

Totally.  Perhaps if you had a look into plz you would appreciate this
fact :-).

Re: ELPA submission: plz-see

[brickviking]

[-- Attachment #1: Type: text/plain, Size: 721 bytes --]

Before this package reached my radar, I'd never even heard of plz. :-)

Regards, brickviking

On Thu, 2 Nov 2023 at 07:36, Augusto Stoffel <arstoffel@gmail.com> wrote:

> On Wed,  1 Nov 2023 at 09:14, Philip Kaludercic wrote:
>
> >>> I agree, from what I see this is just an implementation detail.  Plz is
> >>> a peculiar enough name as it is, it shouldn't be inherited by packages
> >>> that depend on it.  Or is there any reason why the package couldn't
> also
> >>> use url.el?
> >>
> >> The packages is really closely tied to plz's API, which by the way is a
> >> very nice one.
> >
> > But is this fact exposed to the user?
>
> Totally.  Perhaps if you had a look into plz you would appreciate this
> fact :-).
>
>

[-- Attachment #2: Type: text/html, Size: 1148 bytes --]

Re: ELPA submission: plz-see

[Stefan Kangas]

Augusto Stoffel <arstoffel@gmail.com> writes:

> It's ready for publishing.  Thanks again!

Thanks, I've now added it to GNU ELPA.  It should be available for
install within 24-48 hours.

Re: ELPA submission: plz-see

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > Totally.  Perhaps if you had a look into plz you would appreciate this
  > > fact :-).

I looked at plz (disambiguation) in Wikipedia, and found nothing
that seems to be what you mean here.  What is the plz in question?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: ELPA submission: plz-see

[brickviking]

[-- Attachment #1: Type: text/plain, Size: 1026 bytes --]

I have it listed here in my elpa packages listing. A very short summary
from
the package itself is as follows:

`plz' is an HTTP library for Emacs.  It uses `curl' as a backend, which
avoids some of the issues with using Emacs's built-in `url' library.

HTH, brickviking


On Fri, 3 Nov 2023 at 15:31, Richard Stallman <rms@gnu.org> wrote:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > > Totally.  Perhaps if you had a look into plz you would appreciate
> this
>   > > fact :-).
>
> I looked at plz (disambiguation) in Wikipedia, and found nothing
> that seems to be what you mean here.  What is the plz in question?
>
> --
> Dr Richard Stallman (https://stallman.org)
> Chief GNUisance of the GNU Project (https://gnu.org)
> Founder, Free Software Foundation (https://fsf.org)
> Internet Hall-of-Famer (https://internethalloffame.org)
>
>
>
>

[-- Attachment #2: Type: text/html, Size: 1770 bytes --]

Re: ELPA submission: plz-see

[Philip Kaludercic]

brickviking <brickviking@gmail.com> writes:

> I have it listed here in my elpa packages listing. A very short summary
> from
> the package itself is as follows:
>
> `plz' is an HTTP library for Emacs.  It uses `curl' as a backend, which
> avoids some of the issues with using Emacs's built-in `url' library.

And yes, the name is not indicative of what the packages does, but the
author had a strong preference of using a "fun" name and has advocated
this on multiple occasions, so it is extremely unlikely that it will
ever change -- not worth attempting to discuss again.  My only hope was
that this wouldn't proliferate further to other package names :/

> HTH, brickviking
>
>
> On Fri, 3 Nov 2023 at 15:31, Richard Stallman <rms@gnu.org> wrote:
>
>> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
>> [[[ whether defending the US Constitution against all enemies,     ]]]
>> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>>
>>   > > Totally.  Perhaps if you had a look into plz you would appreciate
>> this
>>   > > fact :-).
>>
>> I looked at plz (disambiguation) in Wikipedia, and found nothing
>> that seems to be what you mean here.  What is the plz in question?
>>
>> --
>> Dr Richard Stallman (https://stallman.org)
>> Chief GNUisance of the GNU Project (https://gnu.org)
>> Founder, Free Software Foundation (https://fsf.org)
>> Internet Hall-of-Famer (https://internethalloffame.org)
>>
>>
>>
>>

Re: ELPA submission: plz-see

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: rms@gnu.org,  emacs-devel@gnu.org
> Date: Fri, 03 Nov 2023 07:50:45 +0000
> 
> brickviking <brickviking@gmail.com> writes:
> 
> > I have it listed here in my elpa packages listing. A very short summary
> > from
> > the package itself is as follows:
> >
> > `plz' is an HTTP library for Emacs.  It uses `curl' as a backend, which
> > avoids some of the issues with using Emacs's built-in `url' library.
> 
> And yes, the name is not indicative of what the packages does, but the
> author had a strong preference of using a "fun" name and has advocated
> this on multiple occasions, so it is extremely unlikely that it will
> ever change

Too bad, FIWIW.  The name is not helpful, and I personally don't even
find it "funny" in any sense of the word.

By proliferating packages with unhelpful names, we make ELPA and Emacs
in general harder to use.

Re: ELPA submission: plz-see

[Adam Porter]

>> From: Philip Kaludercic <philipk@posteo.net>
>> Cc: rms@gnu.org,  emacs-devel@gnu.org
>> Date: Fri, 03 Nov 2023 07:50:45 +0000
>> 
>> brickviking <brickviking@gmail.com> writes:
>> 
>> > I have it listed here in my elpa packages listing. A very short summary
>> > from
>> > the package itself is as follows:
>> >
>> > `plz' is an HTTP library for Emacs.  It uses `curl' as a backend, which
>> > avoids some of the issues with using Emacs's built-in `url' library.
>> 
>> And yes, the name is not indicative of what the packages does, but the
>> author had a strong preference of using a "fun" name and has advocated
>> this on multiple occasions, so it is extremely unlikely that it will
>> ever change
> 
> Too bad, FIWIW.  The name is not helpful, and I personally don't even
> find it "funny" in any sense of the word.
> 
> By proliferating packages with unhelpful names, we make ELPA and Emacs
> in general harder to use.

Please (ha) don't misunderstand me: I don't advocate for "fun" or 
"funny" names to be used exclusively for their own sake, nor at the 
expense of usefulness.  There are a number of criteria used in choosing 
a library's name, and it is ultimately a judgment call.

In the case of plz.el, all the obvious names for Elisp HTTP libraries 
were already taken (url, http, request, web).  I spent a good deal of 
time thinking about and choosing the name.  Then, when I submitted the 
package to ELPA (after having developed it for a few years), the 
alternatives suggested were long and awkward.  Then ensued a long 
discussion about how a name should be chosen, names vs. descriptions, 
concise names vs. long ones, memorable ones vs. descriptive ones, etc.

Were one of the obvious names available, I would have used it at the 
time; since none were, I chose what was the best in my judgment.  Those 
who weight criteria differently may arrive at different conclusions. 
And, yes, "fun" is one of the criteria, as is memorability.

In the end, once one finds and begins to use a tool, one tends to 
remember its name.  I would not search for "emacs" to find a text 
editor--but having found it and used it, I know what it is.

--Adam

Re: ELPA submission: plz-see

[Stefan Kangas]

Adam Porter <adam@alphapapa.net> writes:

> In the case of plz.el, all the obvious names for Elisp HTTP libraries
> were already taken (url, http, request, web).  I spent a good deal of
> time thinking about and choosing the name.  Then, when I submitted the
> package to ELPA (after having developed it for a few years), the
> alternatives suggested were long and awkward. [...]
>
> Were one of the obvious names available, I would have used it at the
> time; since none were, I chose what was the best in my judgment.

FWIW, I'm not always a huge fan of packages taking the "obvious" names.
I'd prefer it if we could mostly reserve those for features in core.

For example, we were unable to rename `eglot' to `lsp' when we moved it
to core, as the name was already "taken".  So now users that want LSP
enabled in Emacs can't say `M-x lsp'.

Re: ELPA submission: plz-see

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > > I have it listed here in my elpa packages listing. A very short summary
  > > > from
  > > > the package itself is as follows:
  > > >
  > > > `plz' is an HTTP library for Emacs.

Does having this in ELPA add something significant to Emacs?

         It uses `curl' as a backend, which
  > > > avoids some of the issues with using Emacs's built-in `url' library.

Would it be feasible to improve the `url' library to make `plz' unnecessary?
Would it be feasible to change `url' compatibly to do its work using `plz'?
Either way, we would have just one package for this job, and that would
be simpler than having two.


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: ELPA submission: plz-see

[Adam Porter]

>   > > > I have it listed here in my elpa packages listing. A very short summary
>   > > > from
>   > > > the package itself is as follows:
>   > > >
>   > > > `plz' is an HTTP library for Emacs.
> 
> Does having this in ELPA add something significant to Emacs?
> 
>          It uses `curl' as a backend, which
>   > > > avoids some of the issues with using Emacs's built-in `url' library.
> 
> Would it be feasible to improve the `url' library to make `plz' unnecessary?
> Would it be feasible to change `url' compatibly to do its work using `plz'?
> Either way, we would have just one package for this job, and that would
> be simpler than having two.

As you may remember, we had this discussion at the time plz.el was added 
to GNU ELPA.  plz.el is not even the only HTTP client library on ELPA. 
It was developed after years of consideration of the shortcomings of the 
existing alternatives, and submitted to GNU ELPA after years of careful 
development.  Since then, it has matured and become a reliable, 
performant, easy-to-use (if I do say so myself) library that solves 
several problems in the alternatives, and is used by other ELPA and 
non-ELPA packages, some of which are widely used.

As has been said on emacs-devel several times over the years, the next 
significant step in this field would most likely be for Emacs to 
integrate libcurl directly.  Maybe someday that will happen...

Re: ELPA submission: plz-see

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: philipk@posteo.net, brickviking@gmail.com, emacs-devel@gnu.org
> Date: Fri, 03 Nov 2023 22:10:53 -0400
> 
> Would it be feasible to improve the `url' library to make `plz' unnecessary?

It should be, but no one is working on that, or expressed the intent
of doing so, AFAIK.

> Would it be feasible to change `url' compatibly to do its work using `plz'?

FWIW, I don't think it would be a good idea, since (AFAIU) plz uses
curl under the hood, and that program is not guaranteed to be
installed on all the supported platforms.  Nor do I like the idea of
having Emacs depend on external software where we have the same
capabilities built in, i.e. when the only reason for using external
programs is that our implementation is in some use cases not
performant enough.  We should instead improve our implementation.

Re: ELPA submission: plz-see

[Eli Zaretskii]

> Date: Sat, 4 Nov 2023 00:08:52 -0500
> Cc: brickviking@gmail.com, eliz@gnu.org, emacs-devel@gnu.org,
>  philipk@posteo.net
> From: Adam Porter <adam@alphapapa.net>
> 
> As has been said on emacs-devel several times over the years, the next 
> significant step in this field would most likely be for Emacs to 
> integrate libcurl directly.  Maybe someday that will happen...

Before that happens, I'd like to see a good analysis of why we cannot
make our own implementation of network communications performant
enough to avoid yet another external library.

Re: ELPA submission: plz-see

[tomas]

[-- Attachment #1: Type: text/plain, Size: 731 bytes --]

On Sat, Nov 04, 2023 at 08:49:24AM +0200, Eli Zaretskii wrote:
> > Date: Sat, 4 Nov 2023 00:08:52 -0500
> > Cc: brickviking@gmail.com, eliz@gnu.org, emacs-devel@gnu.org,
> >  philipk@posteo.net
> > From: Adam Porter <adam@alphapapa.net>
> > 
> > As has been said on emacs-devel several times over the years, the next 
> > significant step in this field would most likely be for Emacs to 
> > integrate libcurl directly.  Maybe someday that will happen...
> 
> Before that happens, I'd like to see a good analysis of why we cannot
> make our own implementation of network communications performant
> enough to avoid yet another external library.

Any pointers as to where the problems with url lie?

Cheers
-- 
t

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: ELPA submission: plz-see

[Eli Zaretskii]

> Date: Sat, 4 Nov 2023 09:51:46 +0100
> From: <tomas@tuxteam.de>
> 
> On Sat, Nov 04, 2023 at 08:49:24AM +0200, Eli Zaretskii wrote:
> > > Date: Sat, 4 Nov 2023 00:08:52 -0500
> > > Cc: brickviking@gmail.com, eliz@gnu.org, emacs-devel@gnu.org,
> > >  philipk@posteo.net
> > > From: Adam Porter <adam@alphapapa.net>
> > > 
> > > As has been said on emacs-devel several times over the years, the next 
> > > significant step in this field would most likely be for Emacs to 
> > > integrate libcurl directly.  Maybe someday that will happen...
> > 
> > Before that happens, I'd like to see a good analysis of why we cannot
> > make our own implementation of network communications performant
> > enough to avoid yet another external library.
> 
> Any pointers as to where the problems with url lie?

I don't know.  If anyone else does, please speak up.  the claim is
that fetching a URL with our built-in primitives is too slow.

Re: ELPA submission: plz-see

[tomas]

[-- Attachment #1: Type: text/plain, Size: 661 bytes --]

On Sat, Nov 04, 2023 at 11:07:27AM +0200, Eli Zaretskii wrote:
> > Date: Sat, 4 Nov 2023 09:51:46 +0100
> > From: <tomas@tuxteam.de>
> > 
> > On Sat, Nov 04, 2023 at 08:49:24AM +0200, Eli Zaretskii wrote:

[...]

> > > Before that happens, I'd like to see a good analysis of why we cannot
> > > make our own implementation of network communications performant
> > > enough to avoid yet another external library.
> > 
> > Any pointers as to where the problems with url lie?
> 
> I don't know.  If anyone else does, please speak up.  the claim is
> that fetching a URL with our built-in primitives is too slow.

I see, thanks.

Cheers
-- 
t

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: ELPA submission: plz-see

[Stefan Kangas]

<tomas@tuxteam.de> writes:

> Any pointers as to where the problems with url lie?

We've discussed this before, for example:

    https://lists.gnu.org/r/emacs-devel/2020-12/msg01217.html

Perhaps this will also help:

    https://nullprogram.com/blog/2016/06/16/

Re: ELPA submission: plz-see

[Visuwesh]

[சனி நவம்பர் 04, 2023] Stefan Kangas wrote:

> <tomas@tuxteam.de> writes:
>
>> Any pointers as to where the problems with url lie?
>
> We've discussed this before, for example:
>
>     https://lists.gnu.org/r/emacs-devel/2020-12/msg01217.html
>
> Perhaps this will also help:
>
>     https://nullprogram.com/blog/2016/06/16/

This discussion might also be of interest:

    https://debbugs.gnu.org/cgi/bugreport.cgi?bug=37577

and the bug that was merged with it:

    https://debbugs.gnu.org/cgi/bugreport.cgi?bug=22921

I get bit by this regularly.

Re: ELPA submission: plz-see

[tomas]

[-- Attachment #1: Type: text/plain, Size: 416 bytes --]

On Sat, Nov 04, 2023 at 03:38:11AM -0700, Stefan Kangas wrote:
> <tomas@tuxteam.de> writes:
> 
> > Any pointers as to where the problems with url lie?
> 
> We've discussed this before, for example:
> 
>     https://lists.gnu.org/r/emacs-devel/2020-12/msg01217.html
> 
> Perhaps this will also help:
> 
>     https://nullprogram.com/blog/2016/06/16/

Thanks, both are good references.

Cheers
-- 
t

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: ELPA submission: plz-see

[tomas]

[-- Attachment #1: Type: text/plain, Size: 366 bytes --]

On Sat, Nov 04, 2023 at 05:58:19PM +0530, Visuwesh wrote:

[...]

> This discussion might also be of interest:
> 
>     https://debbugs.gnu.org/cgi/bugreport.cgi?bug=37577
> 
> and the bug that was merged with it:
> 
>     https://debbugs.gnu.org/cgi/bugreport.cgi?bug=22921
> 
> I get bit by this regularly.

Thanks for those, too.

Cheers
-- 
t

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: ELPA submission: plz-see

[Adam Porter]

>> On Sat, Nov 04, 2023 at 08:49:24AM +0200, Eli Zaretskii wrote:
>> > > Date: Sat, 4 Nov 2023 00:08:52 -0500
>> > > Cc: brickviking@gmail.com, eliz@gnu.org, emacs-devel@gnu.org,
>> > >  philipk@posteo.net
>> > > From: Adam Porter <adam@alphapapa.net>
>> > > 
>> > > As has been said on emacs-devel several times over the years, the next 
>> > > significant step in this field would most likely be for Emacs to 
>> > > integrate libcurl directly.  Maybe someday that will happen...
>> > 
>> > Before that happens, I'd like to see a good analysis of why we cannot
>> > make our own implementation of network communications performant
>> > enough to avoid yet another external library.
>> 
>> Any pointers as to where the problems with url lie?
> 
> I don't know.  If anyone else does, please speak up.  the claim is
> that fetching a URL with our built-in primitives is too slow.

Forgive me, but that is not the claim.  There are numerous problems 
experienced with using url.el "in anger," as well as with other HTTP 
client libraries for Emacs.  Stefan K. was kind enough to link one of my 
posts about this issue a few years ago, when plz.el was earlier in its 
development (the issues mentioned having been solved now)[0].  Briefly, 
the problems include unreliability (e.g. callbacks being called multiple 
times or not at all, timeouts that don't happen), very awkward API (e.g. 
having to let-bind often-undocumented variables to do basic HTTP 
operations), and, perhaps also, inferior performance compared to a 
curl-based implementation (curl being implemented in a separate process, 
in C).

To reiterate what was said over a year ago when plz.el was added to GNU 
ELPA: I spent years trying to *not* develop plz.el, trying to work with 
and around the built-in url.el and the then-MELPA-hosted request.el, in 
packages of mine that used HTTP extensively.  I wrote wrapper macros and 
workarounds and hacks that used both libraries as necessary to work 
around one problem or another.

Finally, with the advice and consent of Chris Wellons, a highly 
respected hacker both inside and outside the Emacs arena, who had spent 
even more time dealing with those issues than I had (and he filed bug 
reports about them, as well), I developed plz.el based on his 
elfeed-curl.el library, which he had developed for his Elfeed RSS 
client, having been battle-tested for years.

Over the past 3 years I hammered on it further, eventually solving 
issues related to Emacs' process-related idiosyncrasies (e.g. [1], [2]). 
  It is now, as far as I know, completely reliable: never dropping 
requests, never failing to call callbacks or calling them multiple 
times, never failing to timeout, etc, even when hundreds or thousands of 
requests are made in parallel from a single Emacs process.

What began as a last resort has, with the help of several other 
developers and users to whom I am grateful, become what is, IMHO, the 
best HTTP client library for Emacs (modulo any yet-to-be-implemented 
features which may be present in other clients).

I confess to feeling a bit annoyed at having to re-justify the inclusion 
of plz.el in GNU ELPA (or, at least, I feel like I'm having to do so). 
As I said, there are multiple downstream clients that depend on it, 
including my own Ement.el in GNU ELPA, which is a widely used client for 
the Matrix protocol.

As well, plz.el is now distributed directly in Debian, Ubuntu, Devuan, 
Kali Linux, PureOS, Raspbian, Gentoo, GNU Guix, and NixOS.

Having said all that, of course it would be best if Emacs's built-in 
library were the best one.  But as the author of url.el himself has 
admitted, it has some fundamental issues which are not easy to solve, 
and his own branch related to improving it hasn't been touched in 6 
years[3].  And so far, no one has stepped up to integrate libcurl or 
similar.

So this is where we are with respect to Emacs HTTP libraries.  I 
understand that some people don't like the name, and it wasn't my first 
choice, either.  But the name is of very little importance, and surely 
no more difficult to remember than an API, of which there are 
innumerable ones in Emacs.

--Adam

0: https://lists.gnu.org/r/emacs-devel/2020-12/msg01217.html
1: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=50166
2: https://github.com/alphapapa/plz.el/issues/3
3: 
https://git.savannah.gnu.org/cgit/emacs.git/log/?h=scratch/with-fetched-url

Re: ELPA submission: plz-see

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1317 bytes --]

On Sat, Nov 04, 2023 at 06:57:54AM -0400, dick wrote:
> EZ> ... fetching a URL with our built-in primitives is too slow.
> 
> Quite the contrary.  Last I checked `url-retrieve` was several times
> faster than ELPA request.el (and presumably any other package that
> crudely spawns a curl instance).

The problem is that "slow" means many things for different situations.

Most of the time it just means "blocking", i.e. Emacs becomes unresponsive
while some network thing is twiddling its thumbs.

Whereas non-blocking read and write are more or less (but see below)
straightforward, a non-blocking network open (connect(2) and friends)
is surprisingly hard to get right portably [1]. Same with the resolver
(is gethostbyname going out to query DNS? Is local info sufficient?
The only non-blocking way I know of is cheaping out and punting it
to another process/thread which blocks for us (if anyone knows better
I'd like to learn :)

And, of course, if you are doing it all in one process, you've to
get used to that continuation passing style "do this request and,
on success, call that function, on failure that other". I don't
mind that, and Javascript programmers would feel right at home,
but not everyone likes that :-)

Cheers

[1] https://cr.yp.to/docs/connect.html
-- 
tomás

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: ELPA submission: plz-see

[Eli Zaretskii]

> Date: Sat, 4 Nov 2023 08:30:47 -0500
> Cc: emacs-devel@gnu.org, tomas@tuxteam.de
> From: Adam Porter <adam@alphapapa.net>
> 
> > The claim is that fetching a URL with our built-in primitives is
> > too slow.
> 
> Forgive me, but that is not the claim.  There are numerous problems 
> experienced with using url.el "in anger," as well as with other HTTP 
> client libraries for Emacs.  Stefan K. was kind enough to link one of my 
> posts about this issue a few years ago, when plz.el was earlier in its 
> development (the issues mentioned having been solved now)[0].  Briefly, 
> the problems include unreliability (e.g. callbacks being called multiple 
> times or not at all, timeouts that don't happen), very awkward API (e.g. 
> having to let-bind often-undocumented variables to do basic HTTP 
> operations)

These all sound like bugs, and I'd be surprised if bugs were cited as
a reason to go to a completely different implementation, and one that
is based on an external program on top of that.  Bugs should be fixed;
they should not cause us to throw away our code just because it has
bugs.  Especially since those bugs cannot be so bad, as our
implementation of network communications does work: I'm sending this
mail using it, for example.

> and, perhaps also, inferior performance compared to a 
> curl-based implementation (curl being implemented in a separate process, 
> in C).

So is performance an issue or isn't it?

> I confess to feeling a bit annoyed at having to re-justify the inclusion 
> of plz.el in GNU ELPA (or, at least, I feel like I'm having to do so). 

This is a misunderstanding.  I never said anything about inclusion of
the package in ELPA or the need to justify it.  I responded to the
suggestion to integrate curl with the Emacs core:

>> > > As has been said on emacs-devel several times over the years, the next 
>> > > significant step in this field would most likely be for Emacs to 
>> > > integrate libcurl directly.  Maybe someday that will happen...

Please don't misrepresent what I said and why, because the above
sounds like you are saying I questioned your reasons for developing
the package or asked you to explain why it was needed.  I did not.

> Having said all that, of course it would be best if Emacs's built-in 
> library were the best one.  But as the author of url.el himself has 
> admitted, it has some fundamental issues which are not easy to solve, 
> and his own branch related to improving it hasn't been touched in 6 
> years[3].  And so far, no one has stepped up to integrate libcurl or 
> similar.

If url.el has fundamental problems, they can be fixed by thorough
modifications or even with a completely new line of APIs which don't
have those problems.  That is not the issue that bothers me here; the
issue that bothers me is the idea that instead of improving our own
implementation, we should use an external program.  It should be clear
to anyone who understands how this stuff is integrated into the core
that basing this on external programs has serious issues of its own,
because Emacs is not a garden-variety program, it needs to support
many low-level hooks into the communications level which are hard or
even impossible with external programs and libraries.  I very much
doubt that going this way will ever be able to support everything we
support now with network "processes" and applications based on them.

It is indeed unfortunate that we don't have anyone working on these
issues.  I still hope that someone will step forward soon enough.

> But the name is of very little importance, and surely no more
> difficult to remember than an API, of which there are innumerable
> ones in Emacs.

My original request to find a better name was for plz-see, not for
plz.  So once again, please don't misrepresent what I said and why.

Re: ELPA submission: plz-see

[tomas]

[-- Attachment #1: Type: text/plain, Size: 389 bytes --]

On Sat, Nov 04, 2023 at 10:13:13AM -0400, dick wrote:
> t> a non-blocking network open is surprisingly hard to get right
> 
> I point out url-retrieve is theoretically and empirically faster than
> having the OS load another process into memory [...]

I think it's not about "fast", but about "blocking". I tried
to explain that, but it seems I failed miserably.

Cheers
-- 
t

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: ELPA submission: plz-see

[Adam Porter]

Hello Eli,
>> Date: Sat, 4 Nov 2023 08:30:47 -0500
>> Cc: emacs-devel@gnu.org, tomas@tuxteam.de
>> From: Adam Porter <adam@alphapapa.net>
>> 
>> > The claim is that fetching a URL with our built-in primitives is
>> > too slow.
>> 
>> Forgive me, but that is not the claim.  There are numerous problems 
>> experienced with using url.el "in anger," as well as with other HTTP 
>> client libraries for Emacs.  Stefan K. was kind enough to link one of my 
>> posts about this issue a few years ago, when plz.el was earlier in its 
>> development (the issues mentioned having been solved now)[0].  Briefly, 
>> the problems include unreliability (e.g. callbacks being called multiple 
>> times or not at all, timeouts that don't happen), very awkward API (e.g. 
>> having to let-bind often-undocumented variables to do basic HTTP 
>> operations)
> 
> These all sound like bugs, and I'd be surprised if bugs were cited as
> a reason to go to a completely different implementation, and one that
> is based on an external program on top of that.  Bugs should be fixed;
> they should not cause us to throw away our code just because it has
> bugs.  Especially since those bugs cannot be so bad, as our
> implementation of network communications does work: I'm sending this
> mail using it, for example.

Ideally, of course, I would agree with you.  In practice, experience has 
shown that fixing these bugs is not so simple.  As even Lars said years 
ago[0]:

     "And the more I looked at this, the more apparently it became 
[clear] that Emacs' network stuff should just be totally reimplemented, 
because it's so full of...  oddnesses geared toward processes.  And 
processes and network connections are very different things.

     "So it's a Big Project.  :-/"

The file process.c is known for being hard to debug, especially with 
regard to these kinds of problems.  I feel fortunate that I was able to, 
with the help of a few other people, work around them in plz.el to 
resolve issues with reliability.

>> and, perhaps also, inferior performance compared to a 
>> curl-based implementation (curl being implemented in a separate process, 
>> in C).
> 
> So is performance an issue or isn't it?

As others have noted here and on the bug tracker, Emacs's built-in 
network processes seem to have hangs with regard to DNS, TLS, etc, and 
having the external curl process handle those issues avoids those 
problems.  Some of this discussion happened three years ago, where even 
Richard suggested that using an external process like curl to solve it 
seemed reasonable.[1]

>> I confess to feeling a bit annoyed at having to re-justify the inclusion 
>> of plz.el in GNU ELPA (or, at least, I feel like I'm having to do so). 
> 
> This is a misunderstanding.  I never said anything about inclusion of
> the package in ELPA or the need to justify it.  I responded to the
> suggestion to integrate curl with the Emacs core:

>>> > > As has been said on emacs-devel several times over the years, the next 
>>> > > significant step in this field would most likely be for Emacs to 
>>> > > integrate libcurl directly.  Maybe someday that will happen...
> 
> Please don't misrepresent what I said and why, because the above
> sounds like you are saying I questioned your reasons for developing
> the package or asked you to explain why it was needed.  I did not.

Forgive me, then, for misunderstanding you.  It's never my intention to 
misrepresent.

>> Having said all that, of course it would be best if Emacs's built-in 
>> library were the best one.  But as the author of url.el himself has 
>> admitted, it has some fundamental issues which are not easy to solve, 
>> and his own branch related to improving it hasn't been touched in 6 
>> years[3].  And so far, no one has stepped up to integrate libcurl or 
>> similar.
> 
> If url.el has fundamental problems, they can be fixed by thorough
> modifications or even with a completely new line of APIs which don't
> have those problems.  That is not the issue that bothers me here; the
> issue that bothers me is the idea that instead of improving our own
> implementation, we should use an external program.  It should be clear
> to anyone who understands how this stuff is integrated into the core
> that basing this on external programs has serious issues of its own,
> because Emacs is not a garden-variety program, it needs to support
> many low-level hooks into the communications level which are hard or
> even impossible with external programs and libraries.  I very much
> doubt that going this way will ever be able to support everything we
> support now with network "processes" and applications based on them.

Of course, I don't disagree with you.  The fallback to an external 
program is a pragmatic decision, one which plz.el is in a long line of 
similar packages to make.

> It is indeed unfortunate that we don't have anyone working on these
> issues.  I still hope that someone will step forward soon enough.
> 
>> But the name is of very little importance, and surely no more
>> difficult to remember than an API, of which there are innumerable
>> ones in Emacs.
> 
> My original request to find a better name was for plz-see, not for
> plz.  So once again, please don't misrepresent what I said and why.

Indeed, I misunderstood.  Thanks for patiently clarifying what you meant.

--Adam

0: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=22921#19
1: https://lists.gnu.org/r/emacs-devel/2020-12/msg01456.html