Re: To fetch URL, extract <title> element?

[Yuri Khan <yuri.v.khan@gmail.com>]

On Thu, 12 Nov 2020 at 01:05, Jean Louis <bugs@gnu.support> wrote:
> > > What is the standard built-in way to fetch the http[s] URL?
> >
> > `url-retrieve' sounds appropriate.
>
> I am trying like this:
>
> (defun wrs-fetch-title (url)
>   (url-retrieve url #'wrs-get-title (list url)))
>
> (defun wrs-get-title (status url)
>   (message-any status))
>
> (wrs-fetch-title "http://localhost")
>
> At least I get status nil, but I do not know how to get the HTML
> text.

Have you read the docstring of ‘url-retrieve’?

    CALLBACK is called when the object has been completely retrieved, with
    the current buffer containing the object, and any MIME headers associated
    with it.[…]

So probably:

(defun wrs-fetch-title (url)
  (url-retrieve url #'wrs-get-title (list url)))

(defun wrs-get-title (status url)
  (goto-char (point-min))
  (search-forward "\n\n")  ; skip HTTP headers
  (if (search-forward-regexp "<title\\(?:\\s+[^>]*\\)?>\\([^<]*\\)</title>"
                             nil 'noerror)
      (message "URL: %s Title: %s" url (match-string 1))))

(wrs-fetch-title "https://gnu.org/")
⇒ URL: https://gnu.org/ Title: The GNU Operating System and the Free
Software Movement


(For demonstration purposes, I’m overlooking error handling and MIME
type checks. In a real program, you ought to first make sure you got a
successful status, then check that the response you got has a
‘Content-Type’ of either ‘text/html’ or ‘application/xhtml+xml’ (with
possible parameters such as ‘charset’), and only then look for
HTML-specific <title>…</title> tags.)