bug#34763: 27.0.50; url-retrieve-synchronously misbehaves inside eldoc-documentation-function

bug#34763: 27.0.50; url-retrieve-synchronously misbehaves inside eldoc-documentation-function

[Dmitry Gutov]

[-- Attachment #1: Type: text/plain, Size: 1076 bytes --]

1. Evaluate the attached .el file.

2. Start typing, randomly and with certain pauses, trying to align the
interval to the "Contacting host" messages.

At first, Emacs is responsive, and the requests are duly aborted, like
'while-no-input' obliges them to. A little bit later (maybe 10-30
seconds later), I see errors like "Transfer interrupted", and Emacs
stutters when that happens. Later on, any HTTP request to the same host
freezes Emacs until I press C-g. The attached screenshot shows the
messages log and the list of processes at the end.

I guessing this became much easier to trigger with
12e922156c86a26fa4bb2cb9e7d2b3fd639e4707 when eldoc started using
when-no-input. But whatever underlying problem is there in url-http,
it's probably been there longer.

In GNU Emacs 27.0.50 (build 20, x86_64-pc-linux-gnu, GTK+ Version 3.22.30)
  of 2019-03-05 built on zappa
Repository revision: 9b93e3b0759d562989283eaecf32e075f984c18c
Repository branch: master
Windowing system distributor 'The X.Org Foundation', version 11.0.12001000
System Description: Ubuntu 18.04.2 LTS

[-- Attachment #2: Screenshot from 2019-03-05 16-36-55.png --]
[-- Type: image/png, Size: 439329 bytes --]

[-- Attachment #3: url-retrieve-synchronously-bug.el --]
[-- Type: text/x-emacs-lisp, Size: 262 bytes --]

(defun silly-eldoc ()
  (dotimes (_ 10)
    (url-retrieve-synchronously "http://www.google.com"))
  "done")

(setq-local eldoc-documentation-function #'silly-eldoc)

(setq-local eldoc-idle-delay 0.05)

; asdadasdasdasdas

; aadasdsddsdasddddasds
; asssddasaaaaa

bug#34763: 27.0.50; url-retrieve-synchronously misbehaves inside eldoc-documentation-function

[Dmitry Gutov]

Can anybody reproduce this?

On 05.03.2019 23:34, Dmitry Gutov wrote:
> 1. Evaluate the attached .el file.
> 
> 2. Start typing, randomly and with certain pauses, trying to align the
> interval to the "Contacting host" messages.
> 
> At first, Emacs is responsive, and the requests are duly aborted, like
> 'while-no-input' obliges them to. A little bit later (maybe 10-30
> seconds later), I see errors like "Transfer interrupted", and Emacs
> stutters when that happens. Later on, any HTTP request to the same host
> freezes Emacs until I press C-g. The attached screenshot shows the
> messages log and the list of processes at the end.
> 
> I guessing this became much easier to trigger with
> 12e922156c86a26fa4bb2cb9e7d2b3fd639e4707 when eldoc started using
> when-no-input. But whatever underlying problem is there in url-http,
> it's probably been there longer.
> 
> In GNU Emacs 27.0.50 (build 20, x86_64-pc-linux-gnu, GTK+ Version 3.22.30)
>   of 2019-03-05 built on zappa
> Repository revision: 9b93e3b0759d562989283eaecf32e075f984c18c
> Repository branch: master
> Windowing system distributor 'The X.Org Foundation', version 11.0.12001000
> System Description: Ubuntu 18.04.2 LTS

bug#34763: 27.0.50; url-retrieve-synchronously misbehaves inside eldoc-documentation-function

[Eli Zaretskii]

> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Mon, 11 Mar 2019 02:18:54 +0200
> 
> Can anybody reproduce this?

Does it help to replace this:

  (defsubst url-http-debug (&rest args)
    (if quit-flag

with this:

  (defsubst url-http-debug (&rest args)
    (if (eq quit-flag t)

in url-http.el?

bug#34763: 27.0.50; url-retrieve-synchronously misbehaves inside eldoc-documentation-function

[Dmitry Gutov]

On 11.03.2019 16:30, Eli Zaretskii wrote:

> Does it help to replace this:
> 
>    (defsubst url-http-debug (&rest args)
>      (if quit-flag
> 
> with this:
> 
>    (defsubst url-http-debug (&rest args)
>      (if (eq quit-flag t)
> 
> in url-http.el?

No change, unfortunately. I double-checked.

bug#34763: 27.0.50; url-retrieve-synchronously misbehaves inside eldoc-documentation-function

[Dmitry Gutov]

Some further investigation follows.

I've checked out the revision before the one where eldoc started to use 
while-no-input (6d898918980be4cb29a182ecde7f8e1f95a08462) and repeated 
the scenario.

Predictably, Emacs was considerably less responsive, lots of pauses 
waiting for the replies from the remote server.

Also predictably, no "Transfer interrupted" messages until I press C-g.

But I still reach the state where Emacs seemingly just freezes with 
"Contacting host: www.google.com:80" in the echo area, and even faster 
than with the current Emacs master (within the first 10-20 seconds).

I can abort that with tapping C-g, see that "Transfer interrupted" 
message (Emacs still frozen), and then tapping C-g again to see "eldoc 
error: (error Interrupted!)", wait a second, and see "Quit".

The message log is only slightly different: it consistently says:

"error in process sentinel: url-http-end-of-document-sentinel: Transfer 
interrupted!"

Whereas previously, I mostly seen other sentinels mentioned here.

After Emacs freezes this way once and I C-g out of it, similarly to the 
original problem description, any subsequent call to silly-eldoc freezes 
right away. Calling url-retrieve-synchronously from eval-expression, 
meanwhile, works (although kind of slowly).

This reminds me strongly of a problem with my third-party package 
reported by several people, which I never managed to cleanly reproduce: 
https://github.com/dgutov/robe/issues/105

Also tried this in Emacs 25.2, the scenario reproduces almost exactly.

bug#34763: 27.0.50; url-retrieve-synchronously misbehaves inside eldoc-documentation-function

[Eli Zaretskii]

> Cc: 34763@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Tue, 12 Mar 2019 11:08:06 +0200
> 
> On 11.03.2019 16:30, Eli Zaretskii wrote:
> 
> > Does it help to replace this:
> > 
> >    (defsubst url-http-debug (&rest args)
> >      (if quit-flag
> > 
> > with this:
> > 
> >    (defsubst url-http-debug (&rest args)
> >      (if (eq quit-flag t)
> > 
> > in url-http.el?
> 
> No change, unfortunately. I double-checked.

Crystal ball says there _was_ in fact a change: the error message text
has changed slightly.  Which means there's one more place with a
similar problem.

The below should fix both; please see if it does, and also whether it
fixes your original real-life use case.

Thanks.

diff --git a/lisp/url/url-http.el b/lisp/url/url-http.el
index 76faac1..651a2cc 100644
--- a/lisp/url/url-http.el
+++ b/lisp/url/url-http.el
@@ -150,7 +150,7 @@ url-https-default-port
 ;; These routines will allow us to implement persistent HTTP
 ;; connections.
 (defsubst url-http-debug (&rest args)
-  (if quit-flag
+  (if (eq quit-flag t)
       (let ((proc (get-buffer-process (current-buffer))))
 	;; The user hit C-g, honor it!  Some things can get in an
 	;; incredibly tight loop (chunked encoding)
diff --git a/lisp/url/url-util.el b/lisp/url/url-util.el
index cb80ec6..72ff4f1 100644
--- a/lisp/url/url-util.el
+++ b/lisp/url/url-util.el
@@ -61,7 +61,7 @@ url-debug
 
 ;;;###autoload
 (defun url-debug (tag &rest args)
-  (if quit-flag
+  (if (eq quit-flag t)
       (error "Interrupted!"))
   (if (or (eq url-debug t)
 	  (numberp url-debug)

bug#34763: 27.0.50; url-retrieve-synchronously misbehaves inside eldoc-documentation-function

[Dmitry Gutov]

On 12.03.2019 19:18, Eli Zaretskii wrote:
>> Cc: 34763@debbugs.gnu.org
>> From: Dmitry Gutov <dgutov@yandex.ru>
>> Date: Tue, 12 Mar 2019 11:08:06 +0200
>>
>> On 11.03.2019 16:30, Eli Zaretskii wrote:
>>
>>> Does it help to replace this:
>>>
>>>     (defsubst url-http-debug (&rest args)
>>>       (if quit-flag
>>>
>>> with this:
>>>
>>>     (defsubst url-http-debug (&rest args)
>>>       (if (eq quit-flag t)
>>>
>>> in url-http.el?
>>
>> No change, unfortunately. I double-checked.
> 
> Crystal ball says there _was_ in fact a change: the error message text
> has changed slightly.

I'm sorry, you are right. The text did change.

> The below should fix both; please see if it does, and also whether it
> fixes your original real-life use case.

The patch does away with the first part of the complaint: no "Transfer 
interrupted" or "Interrupted" messages anymore. Thanks!

The HTTP requests do take longer and longer times to complete, though. 
After enough typing, the 10th request never seems to complete. I can 
monitor that with changing the function's definition to

(defun silly-eldoc ()
   (dotimes (i 10)
     (url-retrieve-synchronously "http://www.google.com")
     (message "try %i" i))
   "done")

and watching the Messages buffer.

The counter fairly quickly reaches 0, but rarely goes past 3, and almost 
never past 5 (now that I've been calling that eldoc function for several 
minutes, albeit with big pauses).

M-x list-processes shows 12 lines with very similar contents.

I don't know whether having 10 requests to the same URL counts as a 
real-life case, though.

bug#34763: 27.0.50; url-retrieve-synchronously misbehaves inside eldoc-documentation-function

[Eli Zaretskii]

> Cc: 34763@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Tue, 12 Mar 2019 19:44:04 +0200
> 
> > Crystal ball says there _was_ in fact a change: the error message text
> > has changed slightly.
> 
> I'm sorry, you are right. The text did change.

It's similar enough, so I can understand the confusion and
disappointment ;-).

> > The below should fix both; please see if it does, and also whether it
> > fixes your original real-life use case.
> 
> The patch does away with the first part of the complaint: no "Transfer 
> interrupted" or "Interrupted" messages anymore. Thanks!
> 
> The HTTP requests do take longer and longer times to complete, though. 
> After enough typing, the 10th request never seems to complete. I can 
> monitor that with changing the function's definition to
> 
> (defun silly-eldoc ()
>    (dotimes (i 10)
>      (url-retrieve-synchronously "http://www.google.com")
>      (message "try %i" i))
>    "done")
> 
> and watching the Messages buffer.
> 
> The counter fairly quickly reaches 0, but rarely goes past 3, and almost 
> never past 5 (now that I've been calling that eldoc function for several 
> minutes, albeit with big pauses).
> 
> M-x list-processes shows 12 lines with very similar contents.
> 
> I don't know whether having 10 requests to the same URL counts as a 
> real-life case, though.

Isn't the above expected, given the recipe?  You are initiating an
HTTP retrieval inside while-no-input, and then provide input which
aborts the retrieval.  What else can we expect from this code?

The point of my changes was to avoid signaling errors from the
retrieval routines when some other unrelated code, such as the "real"
eldoc function, is running inside while-no-input.  But if you run
url-retrieve-synchronously itself inside while-no-input, you should
expect it to be aborted by arriving input, right?  Or am I missing
something?

bug#34763: 27.0.50; url-retrieve-synchronously misbehaves inside eldoc-documentation-function

[Dmitry Gutov]

On 12.03.2019 20:00, Eli Zaretskii wrote:
> Isn't the above expected, given the recipe?  You are initiating an
> HTTP retrieval inside while-no-input, and then provide input which
> aborts the retrieval.  What else can we expect from this code?

I'm looking at two distinct things at once:

- The requests to be aborted in time without extra stutters or alarming 
messages.
- The requests to complete promptly enough if I don't abort them. E.g. 
type 'a' and just wait for "done" to show up in the echo area.

The scenario I described triggers, in my testing, problems in both of 
them. The first item seems fixed with your patch.

The second is more or less unchanged. It's fairly likely an orthogonal 
problem, and this is just the way I've found to trigger it. I can file a 
new report, if you like.

To reiterate, the way to reproduce the problem:

- Evaluate the file attached to the report and do some typing, as though 
to reproduce the first problem.

- Continue typing, pause from time to time and wait for "done" to show 
up in the echo area. After a little while, it will stop arriving, 
because the requests will stall for some reason.

bug#34763: 27.0.50; url-retrieve-synchronously misbehaves inside eldoc-documentation-function

[Eli Zaretskii]

> Cc: 34763@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Tue, 12 Mar 2019 22:05:59 +0200
> 
> - Evaluate the file attached to the report and do some typing, as though 
> to reproduce the first problem.
> 
> - Continue typing, pause from time to time and wait for "done" to show 
> up in the echo area. After a little while, it will stop arriving, 
> because the requests will stall for some reason.

They don't stall here, they just take more time to complete.

If you do the same from an interactive function, not from eldoc, does
the problem happen there as well?

bug#34763: 27.0.50; url-retrieve-synchronously misbehaves inside eldoc-documentation-function

[Dmitry Gutov]

On 13.03.2019 18:49, Eli Zaretskii wrote:

>> - Continue typing, pause from time to time and wait for "done" to show
>> up in the echo area. After a little while, it will stop arriving,
>> because the requests will stall for some reason.
> 
> They don't stall here, they just take more time to complete.

How much more time? In my current testing, it can go up 20x. And maybe more.

So normally all 10 requests to GOOG complete in 1.1s (meaning 0.11s per 
request), but after some typing, it slows down and takes ~20s. That's 
~1.8s per request.

At the same time, if I open a second instance of Emacs and try the 
scenario there, the requests start out slow already (same as in the 
already open instance).

But at the same time, opening www.google.com in Firefox is still fast. 
As well as doing it from a console with CURL ('time curl 
http://www.google.com' shows 0.3s).

Sounds like an external library problem, maybe?

> If you do the same from an interactive function, not from eldoc, does
> the problem happen there as well?

More or less. I managed to make it slower by ~20x as well.

Try this:

(defun silly-eldoc ()
   (let ((start (time-to-seconds)))
     (dotimes (i 10)
       (url-retrieve-synchronously "http://www.google.com")
       (message "try %i" i))
     (format "done in %f" (- (time-to-seconds) start))))

(defun silly-command ()
   (interactive)
   (message "%s" (silly-eldoc)))

(global-set-key (kbd "C-o") #'silly-command)

Call the command a few time with C-o, it should run the timer and finish 
with calculation how much time it took to run the requests. Now:

1. Alternate between pressing C-o and C-g, with varying speeds.

The goal is to see

  error in process filter: Transfer interrupted!
  error in process filter: url-http-generic-filter: Transfer interrupted!

a few times.

2. Then see if C-o still runs to completion. If it does, and does not 
look slow enough, repeat 1.

BUT! I have tried to reproduce this slowdown behavior with a local 
server, and so far no luck. Ping to Google (or almost any other website) 
is 100ms from my current location, so maybe it makes a difference.

bug#34763: 27.0.50; url-retrieve-synchronously misbehaves inside eldoc-documentation-function

[Dmitry Gutov]

And regardless of whether the slowdown is caused by external factors in 
my examples, here are two aspects that worry me either way:

1. When I get to "Transfer interrupted", I need to press 'C-g' a second 
time to actually abort right away.

2. When that happens, 'M-x list-processes' shows an extra new line with 
status 'open'. Interrupt enough requests this way - and there are many 
such lines there. That doesn't look good.

bug#34763: 27.0.50; url-retrieve-synchronously misbehaves inside eldoc-documentation-function

[Eli Zaretskii]

> Cc: 34763@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Fri, 15 Mar 2019 16:24:44 +0200
> 
> Sounds like an external library problem, maybe?

GnuTLS?

> > If you do the same from an interactive function, not from eldoc, does
> > the problem happen there as well?
> 
> More or less. I managed to make it slower by ~20x as well.
> 
> Try this:
> 
> (defun silly-eldoc ()
>    (let ((start (time-to-seconds)))
>      (dotimes (i 10)
>        (url-retrieve-synchronously "http://www.google.com")
>        (message "try %i" i))
>      (format "done in %f" (- (time-to-seconds) start))))
> 
> (defun silly-command ()
>    (interactive)
>    (message "%s" (silly-eldoc)))
> 
> (global-set-key (kbd "C-o") #'silly-command)
> 
> Call the command a few time with C-o, it should run the timer and finish 
> with calculation how much time it took to run the requests. Now:
> 
> 1. Alternate between pressing C-o and C-g, with varying speeds.
> 
> The goal is to see
> 
>   error in process filter: Transfer interrupted!
>   error in process filter: url-http-generic-filter: Transfer interrupted!
> 
> a few times.
> 
> 2. Then see if C-o still runs to completion. If it does, and does not 
> look slow enough, repeat 1.

So it sounds like quitting in the middle of a URL retrieval leaves
some leftovers that get in the way of additional retrieval attempts.

> BUT! I have tried to reproduce this slowdown behavior with a local 
> server, and so far no luck. Ping to Google (or almost any other website) 
> is 100ms from my current location, so maybe it makes a difference.

Maybe with a local server you aren't fast enough to interrupt the
transfer before it completes?

bug#34763: 27.0.50; url-retrieve-synchronously misbehaves inside eldoc-documentation-function

[Eli Zaretskii]

> Cc: 34763@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Fri, 15 Mar 2019 16:29:10 +0200
> 
> And regardless of whether the slowdown is caused by external factors in 
> my examples, here are two aspects that worry me either way:
> 
> 1. When I get to "Transfer interrupted", I need to press 'C-g' a second 
> time to actually abort right away.
> 
> 2. When that happens, 'M-x list-processes' shows an extra new line with 
> status 'open'. Interrupt enough requests this way - and there are many 
> such lines there. That doesn't look good.

The code in the functions that react to quit-flag being non-nil should
be audited.  They don't look clean to me, FWIW, and we don't generally
do such things from Lisp, we normally quit on the C level.  Maybe this
whole sub-feature, whereby we show a message for interrupted
transfers, should be rethought.

bug#34763: 27.0.50; url-retrieve-synchronously misbehaves inside eldoc-documentation-function

[Dmitry Gutov]

On 15.03.2019 17:47, Eli Zaretskii wrote:
>> Cc: 34763@debbugs.gnu.org
>> From: Dmitry Gutov <dgutov@yandex.ru>
>> Date: Fri, 15 Mar 2019 16:24:44 +0200
>>
>> Sounds like an external library problem, maybe?
> 
> GnuTLS?

Except the requests only use HTTP, not HTTPS. So... IDK?

> So it sounds like quitting in the middle of a URL retrieval leaves
> some leftovers that get in the way of additional retrieval attempts.

Maybe so.

>> BUT! I have tried to reproduce this slowdown behavior with a local
>> server, and so far no luck. Ping to Google (or almost any other website)
>> is 100ms from my current location, so maybe it makes a difference.
> 
> Maybe with a local server you aren't fast enough to interrupt the
> transfer before it completes?

I do manage to trigger the "Transfer interrupted" errors, though. So 
maybe not. Or not exactly.

bug#34763: 27.0.50; url-retrieve-synchronously misbehaves inside eldoc-documentation-function

[Dmitry Gutov]

On 15.03.2019 17:49, Eli Zaretskii wrote:

> The code in the functions that react to quit-flag being non-nil should
> be audited.  They don't look clean to me, FWIW, and we don't generally
> do such things from Lisp, we normally quit on the C level.  Maybe this
> whole sub-feature, whereby we show a message for interrupted
> transfers, should be rethought.

Apparently the idea is to handle quits, including contexts where 
inhibit-quit is t (947612be2c71d2478179694e8dfa538b9e8e07c1).

So url-retrieve-synchronously uses with-local-quit around 
accept-process-output, and I guess the idea was to handle quits inside 
debugging output calls.

So url-http-debug also does the job of clearing the sentinel and the 
filter of the connection's process.

But maybe that's not enough: it doesn't kill the process and/or remove 
it from url-http-open-connections. (I'm guessing 
url-http-mark-connection-as-free would normally be called later if the 
sentinel wasn't removed).

I'm not quite clear on how error handling should work here anyway. 
Asynchronous code is tricky that way.

bug#34763: 27.0.50; url-retrieve-synchronously misbehaves inside eldoc-documentation-function

[Eli Zaretskii]

> Cc: 34763@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Thu, 4 Apr 2019 02:37:49 +0300
> 
> On 15.03.2019 17:49, Eli Zaretskii wrote:
> 
> > The code in the functions that react to quit-flag being non-nil should
> > be audited.  They don't look clean to me, FWIW, and we don't generally
> > do such things from Lisp, we normally quit on the C level.  Maybe this
> > whole sub-feature, whereby we show a message for interrupted
> > transfers, should be rethought.
> 
> Apparently the idea is to handle quits, including contexts where 
> inhibit-quit is t (947612be2c71d2478179694e8dfa538b9e8e07c1).
> 
> So url-retrieve-synchronously uses with-local-quit around 
> accept-process-output, and I guess the idea was to handle quits inside 
> debugging output calls.

If that's the intent, why does it need to explicitly quit in Lisp?
Just letting the code leave the form where inhibit-quit is non-nil
will let Emacs quit at the first opportunity, with no need for any
Lisp code for that.

bug#34763: 27.0.50; url-retrieve-synchronously misbehaves inside eldoc-documentation-function

[Dmitry Gutov]

On 04.04.2019 16:04, Eli Zaretskii wrote:
> If that's the intent, why does it need to explicitly quit in Lisp?
> Just letting the code leave the form where inhibit-quit is non-nil
> will let Emacs quit at the first opportunity, with no need for any
> Lisp code for that.

Because accept-process-output doesn't abort on user input? And 
url-retrieve-synchronously calls it with 1 second timeout (which is a 
fairly long wait).

I wonder if the original author remembers the idea, now 15 years after 
the code was submitted.

We can try removing the 'error' calls and see if things improve, but:

1. I guess we'll also need to wrap the accept-process-output call in 
while-no-input. And concerns?

2. I wonder if there are cases where some part of the asynchronous code 
takes too long, where it should be allowed to be aborted by the user 
right away. Meaning when url-retrieve is used, not 
url-retrieve-synchronously.

bug#34763: 27.0.50; url-retrieve-synchronously misbehaves inside eldoc-documentation-function

[Eli Zaretskii]

> Cc: 34763@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Thu, 4 Apr 2019 16:29:24 +0300
> 
> On 04.04.2019 16:04, Eli Zaretskii wrote:
> > If that's the intent, why does it need to explicitly quit in Lisp?
> > Just letting the code leave the form where inhibit-quit is non-nil
> > will let Emacs quit at the first opportunity, with no need for any
> > Lisp code for that.
> 
> Because accept-process-output doesn't abort on user input?

It doesn't?  Are you sure?  Or are you talking about calling
accept-process-output with inhibit-quit non-nil?

> And url-retrieve-synchronously calls it with 1 second timeout (which
> is a fairly long wait).

That could be dealt with by making the timeout shorter.

> 2. I wonder if there are cases where some part of the asynchronous code 
> takes too long, where it should be allowed to be aborted by the user 
> right away. Meaning when url-retrieve is used, not 
> url-retrieve-synchronously.

I always thought that C-g aborts accept-process-output.

bug#34763: 27.0.50; url-retrieve-synchronously misbehaves inside eldoc-documentation-function

[Dmitry Gutov]

On 04.04.2019 17:36, Eli Zaretskii wrote:

>> Because accept-process-output doesn't abort on user input?
> 
> It doesn't?  Are you sure?  Or are you talking about calling
> accept-process-output with inhibit-quit non-nil?

Sorry, you're right. That's the only thing that's necessary.

>> And url-retrieve-synchronously calls it with 1 second timeout (which
>> is a fairly long wait).
> 
> That could be dealt with by making the timeout shorter.

No need.

So, I tried the patch below (did you have that change in mind exactly?), 
and I see no adverse effects so far.

Unfortunately, I don't see all the improvement I was hoping for either.

Good (probably): No "interrupted" messages now, or weird 1-second 
pauses. Or full freezes (so far).

Bad:

The requests still get slower after I've been typing a while, and the 
original speed is never recovered. Even after I wait 10 minutes or so.

And this scenario still spawns lots of processes for the same host and port.

>> 2. I wonder if there are cases where some part of the asynchronous code
>> takes too long, where it should be allowed to be aborted by the user
>> right away. Meaning when url-retrieve is used, not
>> url-retrieve-synchronously.
> 
> I always thought that C-g aborts accept-process-output.

url-retrieve doesn't use accept-process-output.


diff --git a/lisp/url/url-http.el b/lisp/url/url-http.el
index cf1952066a..d76ad63eef 100644
--- a/lisp/url/url-http.el
+++ b/lisp/url/url-http.el
@@ -158,7 +158,8 @@ url-http-debug
  	    (progn
  	      (set-process-sentinel proc nil)
  	      (set-process-filter proc nil)))
-	(error "Transfer interrupted!")))
+	;; (error "Transfer interrupted!")
+        ))
    (apply 'url-debug 'http args))

  (defun url-http-mark-connection-as-busy (host port proc)
diff --git a/lisp/url/url-util.el b/lisp/url/url-util.el
index 72ff4f171c..5b8350642f 100644
--- a/lisp/url/url-util.el
+++ b/lisp/url/url-util.el
@@ -61,8 +61,6 @@ url-debug

  ;;;###autoload
  (defun url-debug (tag &rest args)
-  (if (eq quit-flag t)
-      (error "Interrupted!"))
    (if (or (eq url-debug t)
  	  (numberp url-debug)
  	  (and (listp url-debug) (memq tag url-debug)))

bug#34763: 27.0.50; url-retrieve-synchronously misbehaves inside eldoc-documentation-function

[Eli Zaretskii]

> Cc: 34763@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Fri, 5 Apr 2019 03:29:55 +0300
> 
> So, I tried the patch below (did you have that change in mind exactly?), 
> and I see no adverse effects so far.

I had something like that in mind, yes.  I think this should be
installed.

> The requests still get slower after I've been typing a while, and the 
> original speed is never recovered. Even after I wait 10 minutes or so.
> 
> And this scenario still spawns lots of processes for the same host and port.

If you manually kill all processes but one, say, does the problem of
slower transfer go away?  IOW, do we have two separate problems here
or just one?

> >> 2. I wonder if there are cases where some part of the asynchronous code
> >> takes too long, where it should be allowed to be aborted by the user
> >> right away. Meaning when url-retrieve is used, not
> >> url-retrieve-synchronously.
> > 
> > I always thought that C-g aborts accept-process-output.
> 
> url-retrieve doesn't use accept-process-output.

I don't think it matters, because any function that reads from a
process will eventually call the same low-level code as
accept-process-output does, and that low-level code does abort on C-g,
AFAIR.

bug#34763: 27.0.50; url-retrieve-synchronously misbehaves inside eldoc-documentation-function

[Dmitry Gutov]

On 05.04.2019 09:14, Eli Zaretskii wrote:
>> Cc: 34763@debbugs.gnu.org
>> From: Dmitry Gutov <dgutov@yandex.ru>
>> Date: Fri, 5 Apr 2019 03:29:55 +0300
>>
>> So, I tried the patch below (did you have that change in mind exactly?),
>> and I see no adverse effects so far.
> 
> I had something like that in mind, yes.  I think this should be
> installed.

I think this patch makes clear at least one problem: the cleanup in case 
of a quit is spread apart different functions, which isn't good for 
reliability.

E.g. why doesn't url-http-debug also kill the process (but 
url-retrieve-synchronously does)? And what happens if the function is 
interrupted before url-http-debug has had a chance to be called? Or what 
if it's interrupted by a different signal than 'quit'? Or what if it's 
interrupted by a symbol being thrown, set up by while-no-input?

> If you manually kill all processes but one, say, does the problem of
> slower transfer go away?  IOW, do we have two separate problems here
> or just one?

It kind of does. Killing all processes, by itself, doesn't change things.

But if I also wait the aforementioned 10 minutes, transfers are fast 
once again (until I repeat the reproduction scenario).

> I don't think it matters, because any function that reads from a
> process will eventually call the same low-level code as
> accept-process-output does, and that low-level code does abort on C-g,
> AFAIR.

Err, what "function that reads from a process"? If the call is 
asynchronous, chance is, the caller will use the continuation-passing style.

Anyway, I was referring to something else: the comment in url-http-debug 
mentions (IIUC) that the url-http package might use some CPU-intensive 
handling of the process output, url-http-debug being the sole quick 
escape hatch (by the virtue of it signaling an error).

And apparently it can be called called in contexts where inhibit-quit is 
non-nil, or else (eq quit-flag t) would never manage to evaluate to 
true, right?

bug#34763: 27.0.50; url-retrieve-synchronously misbehaves inside eldoc-documentation-function

[Eli Zaretskii]

> Cc: 34763@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Mon, 8 Apr 2019 13:25:36 +0300
> 
> On 05.04.2019 09:14, Eli Zaretskii wrote:
> >> Cc: 34763@debbugs.gnu.org
> >> From: Dmitry Gutov <dgutov@yandex.ru>
> >> Date: Fri, 5 Apr 2019 03:29:55 +0300
> >>
> >> So, I tried the patch below (did you have that change in mind exactly?),
> >> and I see no adverse effects so far.
> > 
> > I had something like that in mind, yes.  I think this should be
> > installed.
> 
> I think this patch makes clear at least one problem: the cleanup in case 
> of a quit is spread apart different functions, which isn't good for 
> reliability.
> 
> E.g. why doesn't url-http-debug also kill the process (but 
> url-retrieve-synchronously does)?

After installing the patch, I think we should indeed try adding code
to kill the process.  That's why I asked you to try that manually
first: to see if doing that will have some positive effect.

> And what happens if the function is interrupted before
> url-http-debug has had a chance to be called?

Not sure why you are bothered by that.  Why would we need to handle
this situation differently from the others?

> Or what if it's interrupted by a different signal than 'quit'?

That's a different issue: in general Emacs retries the calls
interrupted by signals internally.

> Or what if it's interrupted by a symbol being thrown, set up by
> while-no-input?

It shouldn't, that's what the change I proposed does, doesn't it?

> > If you manually kill all processes but one, say, does the problem of
> > slower transfer go away?  IOW, do we have two separate problems here
> > or just one?
> 
> It kind of does. Killing all processes, by itself, doesn't change things.
> 
> But if I also wait the aforementioned 10 minutes, transfers are fast 
> once again (until I repeat the reproduction scenario).

Hmmm... now I'm confused: originally you said something different
about those 10 minutes:

> Bad:
> 
> The requests still get slower after I've been typing a while, and the 
> original speed is never recovered. Even after I wait 10 minutes or so.

Now you seem to say that after 10-min wait the problems do go away?

> > I don't think it matters, because any function that reads from a
> > process will eventually call the same low-level code as
> > accept-process-output does, and that low-level code does abort on C-g,
> > AFAIR.
> 
> Err, what "function that reads from a process"? If the call is 
> asynchronous, chance is, the caller will use the continuation-passing style.

In Emacs, "continuation-passing" means setting up a process filter,
right?  And the process filter does read from the process, right?  My
point was that being interruptible by C-g is implemented in the low
level code used by both accept-process-output and reading process
output that is fed to a filter.

> Anyway, I was referring to something else: the comment in url-http-debug 
> mentions (IIUC) that the url-http package might use some CPU-intensive 
> handling of the process output, url-http-debug being the sole quick 
> escape hatch (by the virtue of it signaling an error).

Wouldn't that be a bug of the code which does that processing?
Background processing shouldn't hog the CPU, because that makes Emacs
unresponsive.

> And apparently it can be called called in contexts where inhibit-quit is 
> non-nil, or else (eq quit-flag t) would never manage to evaluate to 
> true, right?

No, quit-flag could be non-nil because we don't magically test for it
everywhere, only in some strategic places.  If your Lisp takes a long
time to get to one of those places, it might see quit-flag non-nil at
some point.

bug#34763: 27.0.50; url-retrieve-synchronously misbehaves inside eldoc-documentation-function

[Dmitry Gutov]

On 08.04.2019 18:20, Eli Zaretskii wrote:

>> E.g. why doesn't url-http-debug also kill the process (but
>> url-retrieve-synchronously does)?
> 
> After installing the patch, I think we should indeed try adding code
> to kill the process.  That's why I asked you to try that manually
> first: to see if doing that will have some positive effect.

OK then. But if we're killing the process, should we worry about the 
sentinel and the filter?

url-retrieve-synchronously doesn't bother to reset them, just kills the 
process. Speaking of, shouldn't that be enough for our scenario?

>> And what happens if the function is interrupted before
>> url-http-debug has had a chance to be called?
> 
> Not sure why you are bothered by that.  Why would we need to handle
> this situation differently from the others?

Well, if url-http-debug is never entered, its cleanup logic will never 
be executed. Shouldn't we consider that a problem as well?

>> Or what if it's interrupted by a different signal than 'quit'?
> 
> That's a different issue: in general Emacs retries the calls
> interrupted by signals internally.

I mean, like, interrupted by a different kind of error. Not a signal 
that's cleanly handled in Emacs's internals.

>> Or what if it's interrupted by a symbol being thrown, set up by
>> while-no-input?
> 
> It shouldn't, that's what the change I proposed does, doesn't it?

I mean the running code is interrupted, in general, by a symbol being 
thrown. That would also be the case of url-http-debug's cleanup never 
being run.

>>> If you manually kill all processes but one, say, does the problem of
>>> slower transfer go away?  IOW, do we have two separate problems here
>>> or just one?
>>
>> It kind of does. Killing all processes, by itself, doesn't change things.
>>
>> But if I also wait the aforementioned 10 minutes, transfers are fast
>> once again (until I repeat the reproduction scenario).
> 
> Hmmm... now I'm confused: originally you said something different
> about those 10 minutes:
> 
>> Bad:
>>
>> The requests still get slower after I've been typing a while, and the
>> original speed is never recovered. Even after I wait 10 minutes or so.
> 
> Now you seem to say that after 10-min wait the problems do go away?

*If* I kill the running processes before doing that 10-minute wait, yes.

At least that's what I meant. But, sorry to report, repeating the same 
couple of experiments again doesn't yield the same result (killing the 
processes didn't give any measurable impact compared to not killing them 
and simply waiting).

So we seem to have two problems, yes.

Simply waiting for a some amount of time tends to get the problem 
"unstuck", though the improvement is gradual and fairly unpredictable.

> In Emacs, "continuation-passing" means setting up a process filter,
> right?

Sure, as one example.

> And the process filter does read from the process, right?  My
> point was that being interruptible by C-g is implemented in the low
> level code used by both accept-process-output and reading process
> output that is fed to a filter.

Okay. But that is referring to the code that reads the output, not 
whatever CPU-intensive loops can be inside the filter function, right?

>> Anyway, I was referring to something else: the comment in url-http-debug
>> mentions (IIUC) that the url-http package might use some CPU-intensive
>> handling of the process output, url-http-debug being the sole quick
>> escape hatch (by the virtue of it signaling an error).
> 
> Wouldn't that be a bug of the code which does that processing?
> Background processing shouldn't hog the CPU, because that makes Emacs
> unresponsive.

Well, URL is a general purpose package, not always used in the 
background. To make it suitable for background processing is the goal of 
the current bug report, isn't it?

And as for "bug of the code", I'm saying that there must be some code 
that can hog the CPU (the comment refers to it), and we might want to 
handle that carefully. I wish somebody who knows URL's code could 
comment on that.

>> And apparently it can be called called in contexts where inhibit-quit is
>> non-nil, or else (eq quit-flag t) would never manage to evaluate to
>> true, right?
> 
> No, quit-flag could be non-nil because we don't magically test for it
> everywhere, only in some strategic places.  If your Lisp takes a long
> time to get to one of those places, it might see quit-flag non-nil at
> some point.

I see, thank you.

It probably means that seeing non-nil quit-flag is unreliable anyway, 
though, so doing cleanup or not depending on the value of that variable 
seems unwise.

bug#34763: 27.0.50; url-retrieve-synchronously misbehaves inside eldoc-documentation-function

[Eli Zaretskii]

> Cc: 34763@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Mon, 8 Apr 2019 19:42:04 +0300
> 
> > After installing the patch, I think we should indeed try adding code
> > to kill the process.  That's why I asked you to try that manually
> > first: to see if doing that will have some positive effect.
> 
> OK then. But if we're killing the process, should we worry about the 
> sentinel and the filter?

Using delete-process would take care of that.  The sentinel will run,
of course, but I see no reason to worry about that.

> url-retrieve-synchronously doesn't bother to reset them, just kills the 
> process. Speaking of, shouldn't that be enough for our scenario?

I think so, yes.

> >> And what happens if the function is interrupted before
> >> url-http-debug has had a chance to be called?
> > 
> > Not sure why you are bothered by that.  Why would we need to handle
> > this situation differently from the others?
> 
> Well, if url-http-debug is never entered, its cleanup logic will never 
> be executed. Shouldn't we consider that a problem as well?

Depends on how the cleanup code will be written.  It can be written
such that it works regardless where the interruption happens.

> >> Or what if it's interrupted by a different signal than 'quit'?
> > 
> > That's a different issue: in general Emacs retries the calls
> > interrupted by signals internally.
> 
> I mean, like, interrupted by a different kind of error. Not a signal 
> that's cleanly handled in Emacs's internals.

A bug, then?

> >> Or what if it's interrupted by a symbol being thrown, set up by
> >> while-no-input?
> > 
> > It shouldn't, that's what the change I proposed does, doesn't it?
> 
> I mean the running code is interrupted, in general, by a symbol being 
> thrown.

That'd also be a bug, IMO.  We can expect bugs to behave abnormally.

> *If* I kill the running processes before doing that 10-minute wait, yes.
> 
> At least that's what I meant. But, sorry to report, repeating the same 
> couple of experiments again doesn't yield the same result (killing the 
> processes didn't give any measurable impact compared to not killing them 
> and simply waiting).
> 
> So we seem to have two problems, yes.
> 
> Simply waiting for a some amount of time tends to get the problem 
> "unstuck", though the improvement is gradual and fairly unpredictable.

Is it related in any way with the outstanding connections being
completed/closed?  What does netstat show?

> > And the process filter does read from the process, right?  My
> > point was that being interruptible by C-g is implemented in the low
> > level code used by both accept-process-output and reading process
> > output that is fed to a filter.
> 
> Okay. But that is referring to the code that reads the output, not 
> whatever CPU-intensive loops can be inside the filter function, right?

Yes.

> And as for "bug of the code", I'm saying that there must be some code 
> that can hog the CPU (the comment refers to it), and we might want to 
> handle that carefully.

A simple C-g should theoretically take handle that carefully.

> I wish somebody who knows URL's code could comment on that.

Seconded.

> It probably means that seeing non-nil quit-flag is unreliable anyway, 
> though, so doing cleanup or not depending on the value of that variable 
> seems unwise.

Yes, this part is better left to the cleanup we do in C.

bug#34763: 27.0.50; url-retrieve-synchronously misbehaves inside eldoc-documentation-function

[Dmitry Gutov]

On 08.04.2019 20:13, Eli Zaretskii wrote:

>> url-retrieve-synchronously doesn't bother to reset them, just kills the
>> process. Speaking of, shouldn't that be enough for our scenario?
> 
> I think so, yes.

So I guess the first step would be to figure out why 
url-retrieve-synchronously does that isn't enough.

>> Well, if url-http-debug is never entered, its cleanup logic will never
>> be executed. Shouldn't we consider that a problem as well?
> 
> Depends on how the cleanup code will be written.  It can be written
> such that it works regardless where the interruption happens.

Do you have a "good" approach in mind that would still use url-http-debug?

>> I mean, like, interrupted by a different kind of error. Not a signal
>> that's cleanly handled in Emacs's internals.
> 
> A bug, then?

Possibly. Or I imagine something like a response timeout (in an 
already-established connection) could also signal some kind of error.

> That'd also be a bug, IMO.  We can expect bugs to behave abnormally.

Sometimes it's useful to perform cleanup properly anyway. If feasible.

>> And as for "bug of the code", I'm saying that there must be some code
>> that can hog the CPU (the comment refers to it), and we might want to
>> handle that carefully.
> 
> A simple C-g should theoretically take handle that carefully.

That also speaks for removing the cleanup code from url-http-debug, I think.

 >> Simply waiting for a some amount of time tends to get the problem
 >> "unstuck", though the improvement is gradual and fairly unpredictable.
 >
 > Is it related in any way with the outstanding connections being
 > completed/closed?  What does netstat show?

Feel free to suggest a particular invocation, but here's some results. 
I'm calling it while filtering out some irrelevant programs:

netstat -tp --wide | grep -v qbittorrent | grep -v ...

Output at the beginning, when everything seems okay:

Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address           Foreign Address 
State       PID/Program name
tcp        0    300 zappa:53781 
cust-138-33-109-94.dyn.as47377.net:23048 FIN_WAIT1   -
tcp        0      0 zappa:54612 
mrs08s03-in-f10.1e100.net:https TIME_WAIT   -
tcp        0      1 zappa:33575 
ec2-3-83-213-10.compute-1.amazonaws.com:49839 FIN_WAIT1   - 

tcp        0      1 zappa:55515 
svaio-12.dyn.pool1.garodo.net:6881 SYN_SENT    -
tcp        0     69 zappa:48829 
cust-138-33-109-94.dyn.as47377.net:20142 FIN_WAIT1   -

Here's the output right after I repro'd the problem:
(Not all processes could be identified, non-owned process info
  will not be shown, you would have to be root to see it all.)
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address           Foreign Address 
State       PID/Program name
tcp        0     69 zappa:44551 
cust-138-33-109-94.dyn.as47377.net:20142 FIN_WAIT1   -
tcp        0      0 zappa:50064 
mrs08s02-in-f4.1e100.net:http ESTABLISHED 4594/src/emacs
tcp        0      0 zappa:50066 
mrs08s02-in-f4.1e100.net:http ESTABLISHED 4594/src/emacs
tcp        0      0 zappa:55858 
mrs08s04-in-f4.1e100.net:http ESTABLISHED 4594/src/emacs
tcp        0      0 zappa:55930 
mrs08s04-in-f4.1e100.net:http ESTABLISHED 4594/src/emacs
tcp        0      0 zappa:50076 
mrs08s02-in-f4.1e100.net:http ESTABLISHED 4594/src/emacs
tcp        0      0 zappa:55928 
mrs08s04-in-f4.1e100.net:http ESTABLISHED 4594/src/emacs
tcp        0      1 zappa:48910 
ns345059.ip-176-31-225.eu:http LAST_ACK    -
tcp        0      0 zappa:55926 
mrs08s04-in-f4.1e100.net:http ESTABLISHED 4594/src/emacs
tcp        0      0 zappa:55856 
mrs08s04-in-f4.1e100.net:http ESTABLISHED 4594/src/emacs
tcp        0      0 zappa:38389 
nz184l178.bb122100.ctm.net:32766 TIME_WAIT   -
tcp        0    516 zappa:46189 
S0106f0f249850973.wp.shawcable.net:62318 LAST_ACK    -
tcp        0      0 zappa:50062 
mrs08s02-in-f4.1e100.net:http ESTABLISHED 4594/src/emacs
tcp        0      0 zappa:55920 
mrs08s04-in-f4.1e100.net:http ESTABLISHED 4594/src/emacs
tcp        0      0 zappa:50068 
mrs08s02-in-f4.1e100.net:http ESTABLISHED 4594/src/emacs
tcp        0      0 zappa:50070 
mrs08s02-in-f4.1e100.net:http ESTABLISHED 4594/src/emacs
tcp        0      0 zappa:55922 
mrs08s04-in-f4.1e100.net:http ESTABLISHED 4594/src/emacs
tcp        0      0 zappa:55832 
mrs08s04-in-f4.1e100.net:http ESTABLISHED 4594/src/emacs
tcp        0      0 zappa:55924 
mrs08s04-in-f4.1e100.net:http ESTABLISHED 4594/src/emacs
tcp        0      0 zappa:34997 
ec2-3-83-213-10.compute-1.amazonaws.com:49839 FIN_WAIT2   - 

tcp        0      0 zappa:50074 
mrs08s02-in-f4.1e100.net:http ESTABLISHED 4594/src/emacs
tcp        0      0 zappa:50072 
mrs08s02-in-f4.1e100.net:http ESTABLISHED 4594/src/emacs

Here it is after a few minutes has passed:

(Not all processes could be identified, non-owned process info
  will not be shown, you would have to be root to see it all.)
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address           Foreign Address 
State       PID/Program name
tcp        0     69 zappa:44551 
cust-138-33-109-94.dyn.as47377.net:20142 FIN_WAIT1   -
tcp        0      0 zappa:50064 
mrs08s02-in-f4.1e100.net:http ESTABLISHED 4594/src/emacs
tcp        0      0 zappa:50066 
mrs08s02-in-f4.1e100.net:http ESTABLISHED 4594/src/emacs
tcp        0      0 zappa:55858 
mrs08s04-in-f4.1e100.net:http ESTABLISHED 4594/src/emacs
tcp        0      0 zappa:55930 
mrs08s04-in-f4.1e100.net:http ESTABLISHED 4594/src/emacs
tcp        0      0 zappa:50076 
mrs08s02-in-f4.1e100.net:http ESTABLISHED 4594/src/emacs
tcp        0      0 zappa:55928 
mrs08s04-in-f4.1e100.net:http ESTABLISHED 4594/src/emacs
tcp        0      0 zappa:54792 
mrs08s03-in-f10.1e100.net:https TIME_WAIT   -
tcp        0      0 zappa:55926 
mrs08s04-in-f4.1e100.net:http ESTABLISHED 4594/src/emacs
tcp        0      0 zappa:55856 
mrs08s04-in-f4.1e100.net:http ESTABLISHED 4594/src/emacs
tcp        0      0 zappa:50062 
mrs08s02-in-f4.1e100.net:http ESTABLISHED 4594/src/emacs
tcp        0      0 zappa:55920 
mrs08s04-in-f4.1e100.net:http ESTABLISHED 4594/src/emacs
tcp        0      0 zappa:50068 
mrs08s02-in-f4.1e100.net:http ESTABLISHED 4594/src/emacs
tcp        0      0 zappa:50070 
mrs08s02-in-f4.1e100.net:http ESTABLISHED 4594/src/emacs
tcp        0      0 zappa:55922 
mrs08s04-in-f4.1e100.net:http ESTABLISHED 4594/src/emacs
tcp        0      0 zappa:55832 
mrs08s04-in-f4.1e100.net:http ESTABLISHED 4594/src/emacs
tcp        0      0 zappa:55924 
mrs08s04-in-f4.1e100.net:http ESTABLISHED 4594/src/emacs
tcp        0      0 zappa:50074 
mrs08s02-in-f4.1e100.net:http ESTABLISHED 4594/src/emacs
tcp        0      0 zappa:50072 
mrs08s02-in-f4.1e100.net:http ESTABLISHED 4594/src/emacs

The list gets shorter over time.

bug#34763: 27.0.50; url-retrieve-synchronously misbehaves inside eldoc-documentation-function

[Eli Zaretskii]

> Cc: 34763@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Tue, 9 Apr 2019 03:48:17 +0300
> 
> The list gets shorter over time.

That's why I proposed to kill the processes manually, but you say it
didn't help, which is strange.  Does killing processes also make the
list shorter?

(I didn't ignore the rest of what you wrote, I just don't have good
answers for that, and not enough time to study the code and come up
with such answers.  Sorry.)

bug#34763: 27.0.50; url-retrieve-synchronously misbehaves inside eldoc-documentation-function

[Dmitry Gutov]

On 09.04.2019 9:34, Eli Zaretskii wrote:

> (I didn't ignore the rest of what you wrote, I just don't have good
> answers for that, and not enough time to study the code and come up
> with such answers.  Sorry.)

That's totally fine, I have my own print-debugging to do anyway.

>> The list gets shorter over time.
> 
> That's why I proposed to kill the processes manually, but you say it
> didn't help, which is strange.  Does killing processes also make the
> list shorter?

It does, yes. If I kill the processes inside Emacs, the connections 
first become "orphaned", but in a few minutes, they disappear, if I 
don't create any new ones. That doesn't fix the problem, unfortunately 
(transfers are still not remotely as fast as they were initially). 
Resetting url-http-open-connections to an empty hash doesn't help either.

I've tried this both at home and at work (different network equipment 
and internet providers, although the routes probably have common 
segments), the behavior is very similar.

Here's a (probably unnecessary) example.

Before:

Proto Recv-Q Send-Q Local Address           Foreign Address 
State       PID/Program name
tcp        0      0 zappa:34964 
ham02s11-in-f36.1e100.net:http ESTABLISHED 31833/src/emacs
tcp        0      0 zappa:43298 
mrs08s04-in-f4.1e100.net:http ESTABLISHED 31833/src/emacs
tcp        0      0 zappa:34954 
ham02s11-in-f36.1e100.net:http TIME_WAIT   -
tcp        0      0 zappa:34958 
ham02s11-in-f36.1e100.net:http ESTABLISHED 31833/src/emacs
tcp        0      0 zappa:34940 
ham02s11-in-f36.1e100.net:http ESTABLISHED 31833/src/emacs
tcp        0      0 zappa:34968 
ham02s11-in-f36.1e100.net:http ESTABLISHED 31833/src/emacs
tcp        0      0 zappa:34974 
ham02s11-in-f36.1e100.net:http ESTABLISHED 31833/src/emacs
tcp        0      0 zappa:34960 
ham02s11-in-f36.1e100.net:http TIME_WAIT   -
tcp        0      0 zappa:34938 
ham02s11-in-f36.1e100.net:http TIME_WAIT   -
tcp        0      0 zappa:34948 
ham02s11-in-f36.1e100.net:http ESTABLISHED 31833/src/emacs
tcp        0      0 zappa:34942 
ham02s11-in-f36.1e100.net:http ESTABLISHED 31833/src/emacs
tcp        0      1 zappa:36306 
ns345059.ip-176-31-225.eu:http LAST_ACK    -
tcp        0      0 zappa:43300 
mrs08s04-in-f4.1e100.net:http ESTABLISHED 31833/src/emacs
tcp        0      0 zappa:34946 
ham02s11-in-f36.1e100.net:http ESTABLISHED 31833/src/emacs
tcp        0      1 zappa:36302 
ns345059.ip-176-31-225.eu:http LAST_ACK    -
tcp        0      0 zappa:34956 
ham02s11-in-f36.1e100.net:http ESTABLISHED 31833/src/emacs
tcp        0      0 zappa:34952 
ham02s11-in-f36.1e100.net:http ESTABLISHED 31833/src/emacs
tcp        0      0 zappa:34944 
ham02s11-in-f36.1e100.net:http ESTABLISHED 31833/src/emacs
tcp        0      1 zappa:36304 
ns345059.ip-176-31-225.eu:http LAST_ACK    -
tcp        0      0 zappa:34950 
ham02s11-in-f36.1e100.net:http ESTABLISHED 31833/src/emacs
tcp        0      0 zappa:34962 
ham02s11-in-f36.1e100.net:http ESTABLISHED 31833/src/emacs
tcp        0      0 zappa:34970 
ham02s11-in-f36.1e100.net:http ESTABLISHED 31833/src/emacs

Right after killing:

Proto Recv-Q Send-Q Local Address           Foreign Address 
State       PID/Program name
tcp        0      0 zappa:34964 
ham02s11-in-f36.1e100.net:http TIME_WAIT   -
tcp        0      0 zappa:43298 
mrs08s04-in-f4.1e100.net:http TIME_WAIT   -
tcp        0      0 zappa:34954 
ham02s11-in-f36.1e100.net:http TIME_WAIT   -
tcp        0      0 zappa:34958 
ham02s11-in-f36.1e100.net:http TIME_WAIT   -
tcp        0      0 zappa:34940 
ham02s11-in-f36.1e100.net:http TIME_WAIT   -
tcp        0      0 zappa:34976 
ham02s11-in-f36.1e100.net:http ESTABLISHED 31833/src/emacs
tcp        0      0 zappa:34968 
ham02s11-in-f36.1e100.net:http TIME_WAIT   -
tcp        0      0 zappa:34974 
ham02s11-in-f36.1e100.net:http TIME_WAIT   -
tcp        0      0 zappa:34960 
ham02s11-in-f36.1e100.net:http TIME_WAIT   -
tcp        0      0 zappa:34938 
ham02s11-in-f36.1e100.net:http TIME_WAIT   -
tcp        0      0 zappa:34948 
ham02s11-in-f36.1e100.net:http TIME_WAIT   -
tcp        0      0 zappa:34980 
ham02s11-in-f36.1e100.net:http ESTABLISHED 31833/src/emacs
tcp        0      0 zappa:34942 
ham02s11-in-f36.1e100.net:http TIME_WAIT   -
tcp        0      0 zappa:43300 
mrs08s04-in-f4.1e100.net:http TIME_WAIT   -
tcp        0      0 zappa:34946 
ham02s11-in-f36.1e100.net:http TIME_WAIT   -
tcp        0      0 zappa:34952 
ham02s11-in-f36.1e100.net:http TIME_WAIT   -
tcp        0      0 zappa:34944 
ham02s11-in-f36.1e100.net:http TIME_WAIT   -
tcp        0      0 zappa:34950 
ham02s11-in-f36.1e100.net:http TIME_WAIT   -
tcp        0      0 zappa:34962 
ham02s11-in-f36.1e100.net:http TIME_WAIT   -
tcp        0      0 zappa:34970 
ham02s11-in-f36.1e100.net:http TIME_WAIT   -

It becomes empty if I wait a little.

bug#34763: 27.0.50; url-retrieve-synchronously misbehaves inside eldoc-documentation-function

[Eli Zaretskii]

> From: Dmitry Gutov <dgutov@yandex.ru>
> Cc: 34763@debbugs.gnu.org
> Date: Wed, 10 Apr 2019 04:44:15 +0300
> 
> >> The list gets shorter over time.
> > 
> > That's why I proposed to kill the processes manually, but you say it
> > didn't help, which is strange.  Does killing processes also make the
> > list shorter?
> 
> It does, yes. If I kill the processes inside Emacs, the connections 
> first become "orphaned", but in a few minutes, they disappear, if I 
> don't create any new ones. That doesn't fix the problem, unfortunately 
> (transfers are still not remotely as fast as they were initially). 
> Resetting url-http-open-connections to an empty hash doesn't help either.

So I guess either Emacs is busy doing something (or making transfers
slower in some other way), or the other end of the connection
(including the transit points) does something to slow us down.  Hmm...

bug#34763: 27.0.50; url-retrieve-synchronously misbehaves inside eldoc-documentation-function

[Dmitry Gutov]

On 10.04.2019 18:46, Eli Zaretskii wrote:

> ...  or the other end of the connection
> (including the transit points) does something to slow us down.  Hmm...

I guess that's possible, but I should reiterate that

bash -c "time curl -v http://www.google.com"

continues to show the same measurements, even when Emacs struggles.

The output is also almost the same if I (setq url-mime-encoding-string 
nil) on Emacs' side (no performance improvement from that either).

bug#34763: 27.0.50; url-retrieve-synchronously misbehaves inside eldoc-documentation-function

[Andreas Schwab]

On Apr 11 2019, Dmitry Gutov <dgutov@yandex.ru> wrote:

> On 10.04.2019 18:46, Eli Zaretskii wrote:
>
>> ...  or the other end of the connection
>> (including the transit points) does something to slow us down.  Hmm...
>
> I guess that's possible, but I should reiterate that
>
> bash -c "time curl -v http://www.google.com"
>
> continues to show the same measurements, even when Emacs struggles.

Does it help to set url-http-attempt-keepalives to nil?

Andreas.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

bug#34763: 27.0.50; url-retrieve-synchronously misbehaves inside eldoc-documentation-function

[Dmitry Gutov]

On 11.04.2019 10:59, Andreas Schwab wrote:

> Does it help to set url-http-attempt-keepalives to nil?

Good question, but no. Setting it to nil immediately made the best case 
worse (as expected), but the performance degraded the same way as I 
continued on typing.

bug#34763: 27.0.50; url-retrieve-synchronously misbehaves inside eldoc-documentation-function

[Dmitry Gutov]

On 11.04.2019 16:08, Dmitry Gutov wrote:
> On 11.04.2019 10:59, Andreas Schwab wrote:
> 
>> Does it help to set url-http-attempt-keepalives to nil?
> 
> Good question, but no. Setting it to nil immediately made the best case 
> worse (as expected), but the performance degraded the same way as I 
> continued on typing.

But speaking of disabling features, disabling cookies via the third 
argument to url-retrieve-synchronously did the trick.

All other symptoms remain (lots of processes and corresponding 
connections), but I can't reproduce the performance problem anymore, at 
least this way. So it seems this part was caused externally after all.

bug#34763: 27.0.50; url-retrieve-synchronously misbehaves inside eldoc-documentation-function

[Eli Zaretskii]

> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Thu, 11 Apr 2019 18:08:53 +0300
> Cc: 34763@debbugs.gnu.org
> 
> But speaking of disabling features, disabling cookies via the third 
> argument to url-retrieve-synchronously did the trick.
> 
> All other symptoms remain (lots of processes and corresponding 
> connections), but I can't reproduce the performance problem anymore, at 
> least this way. So it seems this part was caused externally after all.

OK, thanks.  But we had some changes discussed in this report not yet
pushed, right?

bug#34763: 27.0.50; url-retrieve-synchronously misbehaves inside eldoc-documentation-function

[Dmitry Gutov]

On 11.04.2019 19:22, Eli Zaretskii wrote:

> OK, thanks.  But we had some changes discussed in this report not yet
> pushed, right?

Indeed. I'll test the error handling changes some more.

The fact that we create a lot of processes during all this is also 
worrying, even if is seems to work out okay.

bug#34763: 27.0.50; url-retrieve-synchronously misbehaves inside eldoc-documentation-function

[Dmitry Gutov]

On 11.04.2019 19:22, Eli Zaretskii wrote:

> OK, thanks.  But we had some changes discussed in this report not yet
> pushed, right?

I've pushed the removal of "Transfer interrupted" error because that 
also (very fortuitously) helped with another kind of problem I've been 
seeing: random unexpected jumps of point between positions and windows.

Quit handling in url-retrieve-synchronously remains mysterious still. 
I've put some message calls for debugging, and apparently neither of the 
'delete-process' calls is ever reached these days. At least, I never 
managed to create appropriate conditions.

There's also an old double comment above the first delete-process call, 
apparently waiting for a brave soul to sort this out. (Chong's position 
seems sound to me, FWIW).

bug#34763: 27.0.50; url-retrieve-synchronously misbehaves inside eldoc-documentation-function

[Lars Ingebrigtsen]

Dmitry Gutov <dgutov@yandex.ru> writes:

> Quit handling in url-retrieve-synchronously remains mysterious
> still. I've put some message calls for debugging, and apparently
> neither of the 'delete-process' calls is ever reached these days. At
> least, I never managed to create appropriate conditions.

Indeed; that function is a mess...

But reading through this bug report, it seems as if the bug you
experienced was fixed?  Should this bug report be closed?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#34763: 27.0.50; url-retrieve-synchronously misbehaves inside eldoc-documentation-function

[Dmitry Gutov]

On 15.05.2019 8:16, Lars Ingebrigtsen wrote:

> Indeed; that function is a mess...
> 
> But reading through this bug report, it seems as if the bug you
> experienced was fixed?

Yes, it seems so. I have a much older report about a similar problem 
(predating Emacs 27's eldoc changes), but I can't reproduce anything 
like that anymore.

> Should this bug report be closed?

Here's hoping we'll get around to fixing url-retrieve-synchronously one day.

Closing, thanks all!