Threads vs url-http

Threads vs url-http

[813gan]

[-- Attachment #1: Type: text/plain, Size: 880 bytes --]

Hi.
I attempted to write thread-friendly wrapper around url-http and it appears to hang Emacs.
Am i doing something wrong or it's bug?

(require 'url-http)
(require 'url-parse)
(defun url-retrieve-thread (url)
"Wrapper around url-http that download `URL' without blocking main thread."
(let* ((parsed-url (if (url-p url) url (url-generic-parse-url url)))
(mut (make-mutex))
(cond-var (make-condition-variable mut))
(buf nil)
(cb (lambda (&rest _) (setq buf (current-buffer)) (with-mutex mut (condition-notify cond-var)) )) )
(url-http parsed-url cb nil)
(with-mutex mut
(condition-wait cond-var) )
buf))

Idea was to call it from thread. In most trivial case:
(make-thread (apply 'url-retrieve-thread "http://example.com" nil))

It makes Emacs unresponsive (even C-g does not work).
I tested it on Emacs 28.2 and once of recent development versions (emacs-31.0.50)

Happy Hacking!

[-- Attachment #2: Type: text/html, Size: 2090 bytes --]

Re: Threads vs url-http

[Eli Zaretskii]

> Date: Sun, 29 Sep 2024 00:28:36 +0000
> From: 813gan <813gan@protonmail.com>
> 
> (require 'url-http)
> (require 'url-parse)
> (defun url-retrieve-thread (url)
>   "Wrapper around url-http that download `URL' without blocking main thread."
>   (let* ((parsed-url (if (url-p url) url (url-generic-parse-url url)))
> (mut (make-mutex))
> (cond-var (make-condition-variable mut))
> (buf nil)
> (cb (lambda (&rest _) (setq buf (current-buffer)) (with-mutex mut (condition-notify cond-var)) )) )
>     (url-http parsed-url cb nil)
>     (with-mutex mut
>       (condition-wait cond-var) )
>     buf))
> 
> Idea was to call it from thread. In most trivial case:
> (make-thread (apply 'url-retrieve-thread "http://example.com" nil))

You are waiting on a condvar in the same thread which should notify
the condvar, don't you?  The callback passed to url-http will be
called in the context of the same thread in which url-http was called,
but that thread is meanwhile blocked in the condition-wait you call
after url-http.

AFAIU your intent, you should wait on the condvar in the main thread,
not in the thread which calls url-http.

> It makes Emacs unresponsive (even C-g does not work).

Yes, because it's deadlock: a thread is waiting in a condvar which no
other thread will call.

Re: Threads vs url-http

[813gan]

On Sunday, September 29th, 2024 at 07:26, Eli Zaretskii <eliz@gnu.org> wrote:

> 
> 
> > Date: Sun, 29 Sep 2024 00:28:36 +0000
> 
> > From: 813gan 813gan@protonmail.com
> > 
> > (require 'url-http)
> > (require 'url-parse)
> > (defun url-retrieve-thread (url)
> > "Wrapper around url-http that download `URL' without blocking main thread."
> > (let* ((parsed-url (if (url-p url) url (url-generic-parse-url url)))
> > (mut (make-mutex))
> > (cond-var (make-condition-variable mut))
> > (buf nil)
> > (cb (lambda (&rest _) (setq buf (current-buffer)) (with-mutex mut (condition-notify cond-var)) )) )
> > (url-http parsed-url cb nil)
> > (with-mutex mut
> > (condition-wait cond-var) )
> > buf))
> > 
> > Idea was to call it from thread. In most trivial case:
> > (make-thread (apply 'url-retrieve-thread "http://example.com" nil))
> 
> 
> You are waiting on a condvar in the same thread which should notify
> the condvar, don't you? The callback passed to url-http will be
> called in the context of the same thread in which url-http was called,
> but that thread is meanwhile blocked in the condition-wait you call
> after url-http.
> 
> AFAIU your intent, you should wait on the condvar in the main thread,
> not in the thread which calls url-http.
> 
> > It makes Emacs unresponsive (even C-g does not work).
> 
> 
> Yes, because it's deadlock: a thread is waiting in a condvar which no
> other thread will call.

Yeah. my understanding of parallelism seems to be worst than i excepted.

Other implementation (naive?) follows.

(defun wiki-read--url-retrieve (url)
  "Wrapper around url-http that download `URL' without blocking thread."
  (let* ((parsed-url (if (url-p url) url (url-generic-parse-url url)))
	 (buf nil)
	 (cb (lambda (&rest _) (setq buf (current-buffer)) )) )
    (url-https parsed-url cb nil)
    (while (not buf) (thread-yield))
    buf))
(make-thread (apply 'wiki-read--url-retrieve "https://google.com" nil))

Why thread-yield don't unblock this?

Thanks

Re: Threads vs url-http

[Michael Albinus]

813gan <813gan@protonmail.com> writes:

Hi,

> (defun wiki-read--url-retrieve (url)
>   "Wrapper around url-http that download `URL' without blocking thread."
>   (let* ((parsed-url (if (url-p url) url (url-generic-parse-url url)))
> 	 (buf nil)
> 	 (cb (lambda (&rest _) (setq buf (current-buffer)) )) )
>     (url-https parsed-url cb nil)
>     (while (not buf) (thread-yield))
>     buf))
> (make-thread (apply 'wiki-read--url-retrieve "https://google.com" nil))
>
> Why thread-yield don't unblock this?

Because url-https blocks alrready, I believe.

There's also another trap when using threads with url-https, see
bug#73199. I'm working on a fix.

> Thanks

Best regards, Michael.

Re: Threads vs url-http

[Eli Zaretskii]

> Date: Sun, 29 Sep 2024 23:53:44 +0000
> From: 813gan <813gan@protonmail.com>
> Cc: emacs-devel@gnu.org
> 
> (defun wiki-read--url-retrieve (url)
>   "Wrapper around url-http that download `URL' without blocking thread."
>   (let* ((parsed-url (if (url-p url) url (url-generic-parse-url url)))
> 	 (buf nil)
> 	 (cb (lambda (&rest _) (setq buf (current-buffer)) )) )
>     (url-https parsed-url cb nil)
>     (while (not buf) (thread-yield))
>     buf))
> (make-thread (apply 'wiki-read--url-retrieve "https://google.com" nil))
> 
> Why thread-yield don't unblock this?

make-thread needs a function as its first argument.  But you pass it
the value returned by 'apply', which is not a function at all.  So
what happens here, I think, is this:

  . Emacs invokes 'apply' to evaluate the argument of make-thread
  . 'apply' invokes your wiki-read--url-retrieve function
  . the function runs in the main thread, not in the thread started by
    make-thread (which is not even called, because the code is stuck
    in 'apply', see below)
  . after url-https returns (which it does immediately), the code
    immediately enters the while-loop, where it calls thread-yield
  . because there's only one running thread, thread-yield grabs the
    global lock immediately after releasing it, and the loop keeps
    looping
  . because the program loops continuously, it doesn't let url-https
    read from the network process (since Emacs only does that when it
    is idle)
  . so the output of the network sub-process is never read, and so the
    sub-process never ends, and so the callback is never called, and
    the loop never ends