How to get title of web page by url?

How to get title of web page by url?

[filebat Mark]

[-- Attachment #1: Type: text/plain, Size: 353 bytes --]

Hi emacsers

I'm wondering is there any existing lisp code snippet to get title of web
page by url.
Such as, given "http://www.emacswiki.org/emacs/Git", we will get the title
of this web page, which is "EmacsWiki: Git:".

Function of w3m-current-title is quite close, but a standalone lisp function
is much preferred.

-- 
Thanks & Regards

Denny Zhang

[-- Attachment #2: Type: text/html, Size: 471 bytes --]

Re: How to get title of web page by url?

[Thamer Mahmoud]

filebat Mark <filebat.mark@gmail.com> writes:

> Such as, given "http://www.emacswiki.org/emacs/Git", we will get the title
> of this web page, which is "EmacsWiki: Git:".
>
> Function of w3m-current-title is quite close, but a standalone lisp function
> is much preferred.

Using the url.el package,

(defun www-get-page-title (url)
  (with-current-buffer (url-retrieve-synchronously url)
    (goto-char 0)
    (re-search-forward "<title>\\(.*\\)<[/]title>" nil t 1)
    (match-string 1)))

(www-get-page-title "http://www.emacswiki.org/emacs/Git")
=> "EmacsWiki: Git"

hth,
Thamer

Re: How to get title of web page by url?

[filebat Mark]

[-- Attachment #1: Type: text/plain, Size: 1905 bytes --]

Thanks, Thamer. It works.

Below is the code snippet.

Well, I still have an encoding problem.
To get the title of "http://www.baidu.com", the title we get is displayed as
unrecognizable codes.

I have tried to encode it, in the way of "(setq web_title_str
(encode-coding-string  web_title_str 'utf-8-dos))", but it fails.
Since I am a newbie for emacs encoding, can you please help me to point what
the problem is?

;; -------------------------- separator --------------------------
(defun get-page-title()
  "Get title of web page, whose url can be found in current line"
  (interactive)
  ;; Get url from current line
  (copy-region-as-kill (re-search-backward "^") (re-search-forward "$"))
  (setq url (substring-no-properties (current-kill 0)))
  ;; Get title of web page, with the help of functions in url.el
  (with-current-buffer (url-retrieve-synchronously url)
    (goto-char 0)
    (re-search-forward "<title>\\(.*\\)<[/]title>" nil t 1)
    (setq web_title_str (match-string 1)))
    (setq web_title_str (encode-coding-string web_title_str 'utf-8-dos))
  ;; Insert the title in the next line
  (reindent-then-newline-and-indent)
  (insert web_title_str)
  )


On 7/28/10, Thamer Mahmoud <thamer.mahmoud@gmail.com> wrote:
>
> filebat Mark <filebat.mark@gmail.com> writes:
>
> > Such as, given "http://www.emacswiki.org/emacs/Git", we will get the
> title
> > of this web page, which is "EmacsWiki: Git:".
> >
> > Function of w3m-current-title is quite close, but a standalone lisp
> function
> > is much preferred.
>
>
> Using the url.el package,
>
> (defun www-get-page-title (url)
>   (with-current-buffer (url-retrieve-synchronously url)
>     (goto-char 0)
>     (re-search-forward "<title>\\(.*\\)<[/]title>" nil t 1)
>     (match-string 1)))
>
> (www-get-page-title "http://www.emacswiki.org/emacs/Git")
> => "EmacsWiki: Git"
>
> hth,
>
> Thamer
>
>
>


-- 
Thanks & Regards

Denny Zhang

[-- Attachment #2: Type: text/html, Size: 2709 bytes --]

Re: How to get title of web page by url?

[Thamer Mahmoud]

filebat Mark <filebat.mark@gmail.com> writes:

> Thanks, Thamer. It works.
>
> Below is the code snippet.
>
> Well, I still have an encoding problem.
> To get the title of "http://www.baidu.com", the title we get is displayed as
> unrecognizable codes.
>
> I have tried to encode it, in the way of "(setq web_title_str
> (encode-coding-string  web_title_str 'utf-8-dos))", but it fails.

I'm also new to Elisp (well sort of). 

But here is a modified version that should handle both charsets and
newlines (and other issues noticed by Deniz Dogan. Thanks).

(defun www-get-page-title (url)
  (let ((title))
    (with-current-buffer (url-retrieve-synchronously url)
      (goto-char (point-min))
      (re-search-forward "<title>\\([^<]*\\)</title>" nil t 1)
      (setq title (match-string 1))
      (goto-char (point-min))
      (re-search-forward "charset=\\([-0-9a-zA-Z]*\\)" nil t 1)
      (decode-coding-string title (intern (match-string 1))))))

The robustness of this code would still depend on whether the HTML is
well-formed, but it should be good enough I think.

--
Thamer









> Since I am a newbie for emacs encoding, can you please help me to point what
> the problem is?



>
> ;; -------------------------- separator --------------------------
> (defun get-page-title()
>   "Get title of web page, whose url can be found in current line"
>   (interactive)
>   ;; Get url from current line
>   (copy-region-as-kill (re-search-backward "^") (re-search-forward "$"))
>   (setq url (substring-no-properties (current-kill 0)))
>   ;; Get title of web page, with the help of functions in url.el
>   (with-current-buffer (url-retrieve-synchronously url)
>     (goto-char 0)
>     (re-search-forward "<title>\\(.*\\)<[/]title>" nil t 1)
>     (setq web_title_str (match-string 1)))
>     (setq web_title_str (encode-coding-string web_title_str 'utf-8-dos))
>   ;; Insert the title in the next line
>   (reindent-then-newline-and-indent)
>   (insert web_title_str)
>   )
>
>
> On 7/28/10, Thamer Mahmoud <thamer.mahmoud@gmail.com> wrote:
>>
>> filebat Mark <filebat.mark@gmail.com> writes:
>>
>> > Such as, given "http://www.emacswiki.org/emacs/Git", we will get the
>> title
>> > of this web page, which is "EmacsWiki: Git:".
>> >
>> > Function of w3m-current-title is quite close, but a standalone lisp
>> function
>> > is much preferred.
>>
>>
>> Using the url.el package,
>>
>> (defun www-get-page-title (url)
>>   (with-current-buffer (url-retrieve-synchronously url)
>>     (goto-char 0)
>>     (re-search-forward "<title>\\(.*\\)<[/]title>" nil t 1)
>>     (match-string 1)))
>>
>> (www-get-page-title "http://www.emacswiki.org/emacs/Git")
>> => "EmacsWiki: Git"
>>
>> hth,
>>
>> Thamer
>>
>>
>>

Re: How to get title of web page by url?

[Lennart Borgman]

On Wed, Jul 28, 2010 at 5:34 PM, Thamer Mahmoud
<thamer.mahmoud@gmail.com> wrote:
> filebat Mark <filebat.mark@gmail.com> writes:
>
>> Thanks, Thamer. It works.
>>
>> Below is the code snippet.
>>
>> Well, I still have an encoding problem.
>> To get the title of "http://www.baidu.com", the title we get is displayed as
>> unrecognizable codes.
>>
>> I have tried to encode it, in the way of "(setq web_title_str
>> (encode-coding-string  web_title_str 'utf-8-dos))", but it fails.
>
> I'm also new to Elisp (well sort of).
>
> But here is a modified version that should handle both charsets and
> newlines (and other issues noticed by Deniz Dogan. Thanks).
>
> (defun www-get-page-title (url)
>  (let ((title))
>    (with-current-buffer (url-retrieve-synchronously url)
>      (goto-char (point-min))
>      (re-search-forward "<title>\\([^<]*\\)</title>" nil t 1)
>      (setq title (match-string 1))
>      (goto-char (point-min))
>      (re-search-forward "charset=\\([-0-9a-zA-Z]*\\)" nil t 1)
>      (decode-coding-string title (intern (match-string 1))))))
>
> The robustness of this code would still depend on whether the HTML is
> well-formed, but it should be good enough I think.


Have a look at url-copy-file for how to get this correct. (Or
web-vcs-url-copy-file in nXhtml which is a little bit more careful.)

Re: How to get title of web page by url?

[Thamer Mahmoud]

> (defun www-get-page-title (url)
>   (let ((title))
>     (with-current-buffer (url-retrieve-synchronously url)
>       (goto-char (point-min))
>       (re-search-forward "<title>\\([^<]*\\)</title>" nil t 1)
>       (setq title (match-string 1))
>       (goto-char (point-min))
>       (re-search-forward "charset=\\([-0-9a-zA-Z]*\\)" nil t 1)
>       (decode-coding-string title (intern (match-string 1))))))

Just did a test on a wikipedia page, and looks like
`decode-coding-string' doesn't handle upper-case charsets, like UTF-8,
only utf-8.

So the last line should be:

(decode-coding-string title (intern (downcase (match-string 1)))))))

--
Thamer

Re: How to get title of web page by url?

[filebat Mark]

[-- Attachment #1: Type: text/plain, Size: 2223 bytes --]

Thank you very much, Thamer! It serves my need very well.

Though html parser shall be more powerful, grepping the string shall be good
enough for my requirement.
Thank you all for the attention and valuable discussion.

Post the complete lisp function here, if someone else need it.
;; -------------------------- separator --------------------------
(defun get-page-title()
  "Get title of web page, whose url can be found in the current line"
  (interactive)
  ;; Get url from current line
  (copy-region-as-kill (re-search-backward "^") (re-search-forward "$"))
  (setq url (substring-no-properties (current-kill 0)))
  ;; Get title of web page, with the help of functions in url.el
  (with-current-buffer (url-retrieve-synchronously url)
    ;; find title by grep the html code
    (goto-char 0)
    (re-search-forward "<title>\\([^<]*\\)</title>" nil t 1)
    (setq web_title_str (match-string 1))
    ;; find charset by grep the html code
    (goto-char 0)
    (re-search-forward "charset=\\([-0-9a-zA-Z]*\\)" nil t 1)
    ;; downcase the charaset. e.g, UTF-8 is not acceptible for emacs, while
utf-8 is ok.
    (setq coding_charset (downcase (match-string 1)))
    ;; decode the string of title.
    (setq web_title_str (decode-coding-string web_title_str (intern
coding_charset)))
    )
  ;; Insert the title in the next line
  (reindent-then-newline-and-indent)
  (insert web_title_str)
  )



On Thu, Jul 29, 2010 at 2:14 AM, Thamer Mahmoud <thamer.mahmoud@gmail.com>wrote:

>
> > (defun www-get-page-title (url)
> >   (let ((title))
> >     (with-current-buffer (url-retrieve-synchronously url)
> >       (goto-char (point-min))
> >       (re-search-forward "<title>\\([^<]*\\)</title>" nil t 1)
> >       (setq title (match-string 1))
> >       (goto-char (point-min))
> >       (re-search-forward "charset=\\([-0-9a-zA-Z]*\\)" nil t 1)
> >       (decode-coding-string title (intern (match-string 1))))))
>
> Just did a test on a wikipedia page, and looks like
> `decode-coding-string' doesn't handle upper-case charsets, like UTF-8,
> only utf-8.
>
> So the last line should be:
>
> (decode-coding-string title (intern (downcase (match-string 1)))))))
>
> --
> Thamer
>
>
>


-- 
Thanks & Regards

Denny Zhang

[-- Attachment #2: Type: text/html, Size: 2895 bytes --]

Re: How to get title of web page by url?

[Deniz Dogan]

2010/7/28 Thamer Mahmoud <thamer.mahmoud@gmail.com>:
>    (re-search-forward "<title>\\(.*\\)<[/]title>" nil t 1)

Why [/] and not /?

By the way, this will not work in scenarios where the title is spread
out across multiple lines:

<title>
  Hello
</title>

How would you solve this in Emacs Lisp?

-- 
Deniz Dogan

Re: How to get title of web page by url?

[Teemu Likonen]

* 2010-07-28 16:12 (+0200), Deniz Dogan wrote:

> 2010/7/28 Thamer Mahmoud <thamer.mahmoud@gmail.com>:
>>    (re-search-forward "<title>\\(.*\\)<[/]title>" nil t 1)

> By the way, this will not work in scenarios where the title is spread
> out across multiple lines:
>
> <title>
>   Hello
> </title>
>
> How would you solve this in Emacs Lisp?

Regexps can match whitespace too. Just leave out spaces, tabs and
newlines in the beginning and end of title text. Also note that the
title text itself may contain newlines. We should probably replace
newlines with spaces in the matching string.

The real solution for extracting title from a HTML text are not regular
expressions but a specific HTML parser. The Lisp way to write such
parser would be to turn the document (or only the head part) to nested
lists and other s-expressions and then dive into the list to find the
title. Such parsers already exist for Common Lisp but I'm not sure about
Emacs Lisp.

Re: How to get title of web page by url?

[Andreas Röhler]

[ ... ]

> The real solution for extracting title from a HTML text are not regular
> expressions but a specific HTML parser. The Lisp way to write such
> parser would be to turn the document (or only the head part) to nested
> lists and other s-expressions and then dive into the list to find the
> title. Such parsers already exist for Common Lisp but I'm not sure about
> Emacs Lisp.
>
>

beg-end.el

at

http://bazaar.launchpad.net/~a-roehler/s-x-emacs-werkstatt

is an essay for such a parser

see thing-at-point-markup.el too, which serves markup-languages as xml, html

thing-at-point-utils.el offers functions to grasp everything between 
angles - and does count nesting.

try ar-angled-lesser-atpt for example

all this needs

thingatpt-utils-base.el,

where the core routines reside.

Have a look, how the parser mentioned is employed via 
beginning-of-form-base, end-of-form-base from there.


Andreas


Andreas

--
https://code.launchpad.net/~a-roehler/python-mode
https://code.launchpad.net/s-x-emacs-werkstatt/

Re: How to get title of web page by url?

[Andreas Röhler]

>

> http://bazaar.launchpad.net/~a-roehler/s-x-emacs-werkstatt

link don't work, sorry

this does it:

https://code.launchpad.net/s-x-emacs-werkstatt/

Re: How to get title of web page by url?

[Ted Zlatanov]

On Wed, 28 Jul 2010 16:12:52 +0200 Deniz Dogan <deniz.a.m.dogan@gmail.com> wrote: 

DD> 2010/7/28 Thamer Mahmoud <thamer.mahmoud@gmail.com>:
>>    (re-search-forward "<title>\\(.*\\)<[/]title>" nil t 1)

DD> Why [/] and not /?

DD> By the way, this will not work in scenarios where the title is spread
DD> out across multiple lines:

DD> <title>
DD>   Hello
DD> </title>

DD> How would you solve this in Emacs Lisp?

It will break in many other situations.  For example, it doesn't check
that the <title> tag is inside the <head> tag or remove comments.  It's
hard to parse HTML properly.

Ted