eww.el: Patch to cache the parse tree

eww.el: Patch to cache the parse tree

[T.V. Raman]

Hi,

I'd like  to add some code to eww.el so that the parsed document
is cached ( -- this will enable  functionality such as document
filtering etc (see
http://emacspeak.googlecode.com/svn/trunk/lisp/shr-url.el) for
similar functionality that I originally built using bare shr.

Here is a patch against Master:

git diff master
diff --git a/lisp/net/eww.el b/lisp/net/eww.el
index 86e0977..a446a01 100644
--- a/lisp/net/eww.el
+++ b/lisp/net/eww.el
@@ -89,6 +89,9 @@
   :group 'eww)

 (defvar eww-current-url nil)
+(defvar eww-current-dom nil)
+(make-variable-buffer-local 'eww-current-dom)
+
 (defvar eww-current-title ""
   "Title of current page.")
 (defvar eww-history nil)
@@ -208,6 +211,7 @@ word(s) will be searched for via `eww-search-prefix'."
 		  (start end &optional base-url))

 (defun eww-display-html (charset url)
+  (declare (special eww-current-dom))
   (or (fboundp 'libxml-parse-html-region)
       (error "This function requires Emacs to be compiled with libxml2"))
   (unless (eq charset 'utf8)
@@ -219,6 +223,7 @@ word(s) will be searched for via `eww-search-prefix'."
 	  'base (list (cons 'href url))
 	  (libxml-parse-html-region (point) (point-max)))))
     (eww-setup-buffer)
+    (setq eww-current-dom document)
     (let ((inhibit-read-only t)
 	  (after-change-functions nil)
 	  (shr-width nil)
@@ -387,9 +392,11 @@ word(s) will be searched for via `eww-search-prefix'."
   )

 (defun eww-save-history ()
+  (declare (special ew-current-dom))
   (push (list :url eww-current-url
 	      :title eww-current-title
 	      :point (point)
+              :dom eww-current-dom
 	      :text (buffer-string))
 	eww-history))

@@ -427,6 +434,7 @@ word(s) will be searched for via `eww-search-prefix'."
   (let ((inhibit-read-only t))
     (erase-buffer)
     (insert (plist-get elem :text))
+    (setq eww-current-dom (plist-get elem :dom))
     (goto-char (plist-get elem :point))
     (setq eww-current-url (plist-get elem :url)
 	  eww-current-title (plist-get elem :title))))
09:05:43 raman-glaptop net $
-- 

--

eww.el: Patch to cache the parse tree

[T.V. Raman]

Following up to myself:

To see how I am leveraging this patch,  see the sections on DOM
Filtering in
http://emacspeak.googlecode.com/svn/trunk/lisp/emacspeak-eww.el


-- 

-- 


On 11/27/13, T.V. Raman <tv.raman.tv@gmail.com> wrote:
> Hi,
>
> I'd like  to add some code to eww.el so that the parsed document
> is cached ( -- this will enable  functionality such as document
> filtering etc (see
> http://emacspeak.googlecode.com/svn/trunk/lisp/shr-url.el) for
> similar functionality that I originally built using bare shr.
>
> Here is a patch against Master:
>
> git diff master
> diff --git a/lisp/net/eww.el b/lisp/net/eww.el
> index 86e0977..a446a01 100644
> --- a/lisp/net/eww.el
> +++ b/lisp/net/eww.el
> @@ -89,6 +89,9 @@
>    :group 'eww)
>
>  (defvar eww-current-url nil)
> +(defvar eww-current-dom nil)
> +(make-variable-buffer-local 'eww-current-dom)
> +
>  (defvar eww-current-title ""
>    "Title of current page.")
>  (defvar eww-history nil)
> @@ -208,6 +211,7 @@ word(s) will be searched for via `eww-search-prefix'."
>  		  (start end &optional base-url))
>
>  (defun eww-display-html (charset url)
> +  (declare (special eww-current-dom))
>    (or (fboundp 'libxml-parse-html-region)
>        (error "This function requires Emacs to be compiled with libxml2"))
>    (unless (eq charset 'utf8)
> @@ -219,6 +223,7 @@ word(s) will be searched for via `eww-search-prefix'."
>  	  'base (list (cons 'href url))
>  	  (libxml-parse-html-region (point) (point-max)))))
>      (eww-setup-buffer)
> +    (setq eww-current-dom document)
>      (let ((inhibit-read-only t)
>  	  (after-change-functions nil)
>  	  (shr-width nil)
> @@ -387,9 +392,11 @@ word(s) will be searched for via `eww-search-prefix'."
>    )
>
>  (defun eww-save-history ()
> +  (declare (special ew-current-dom))
>    (push (list :url eww-current-url
>  	      :title eww-current-title
>  	      :point (point)
> +              :dom eww-current-dom
>  	      :text (buffer-string))
>  	eww-history))
>
> @@ -427,6 +434,7 @@ word(s) will be searched for via `eww-search-prefix'."
>    (let ((inhibit-read-only t))
>      (erase-buffer)
>      (insert (plist-get elem :text))
> +    (setq eww-current-dom (plist-get elem :dom))
>      (goto-char (plist-get elem :point))
>      (setq eww-current-url (plist-get elem :url)
>  	  eww-current-title (plist-get elem :title))))
> 09:05:43 raman-glaptop net $
> --
>
> --
>

Re: eww.el: Patch to cache the parse tree

[Lars Magne Ingebrigtsen]

"T.V. Raman" <tv.raman.tv@gmail.com> writes:

> I'd like  to add some code to eww.el so that the parsed document
> is cached ( -- this will enable  functionality such as document
> filtering etc (see
> http://emacspeak.googlecode.com/svn/trunk/lisp/shr-url.el) for
> similar functionality that I originally built using bare shr.

I've now applied parts of your patch, and rewrote bits.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

eww.el: Patch to cache the parse tree

[T.V. Raman]

Thanks Lars,

-- 
Best Regards,
--raman


On 12/1/13, Lars Magne Ingebrigtsen <larsi@gnus.org> wrote:
> "T.V. Raman" <tv.raman.tv@gmail.com> writes:
>
>> I'd like  to add some code to eww.el so that the parsed document
>> is cached ( -- this will enable  functionality such as document
>> filtering etc (see
>> http://emacspeak.googlecode.com/svn/trunk/lisp/shr-url.el) for
>> similar functionality that I originally built using bare shr.
>
> I've now applied parts of your patch, and rewrote bits.
>
> --
> (domestic pets only, the antidote for overdose, milk.)
>    bloggy blog: http://lars.ingebrigtsen.no
>

eww.el: Patch to cache the parse tree

[T.V. Raman]

Hi Lars,

One more request: in the call
(libxml-parse-html-region (point) (point-max))

could you please pass in the current url as a third argument so
that libxml knows to handle it as a base URL?

-- 
Best Regards,
--raman


On 12/1/13, Lars Magne Ingebrigtsen <larsi@gnus.org> wrote:
> "T.V. Raman" <tv.raman.tv@gmail.com> writes:
>
>> I'd like  to add some code to eww.el so that the parsed document
>> is cached ( -- this will enable  functionality such as document
>> filtering etc (see
>> http://emacspeak.googlecode.com/svn/trunk/lisp/shr-url.el) for
>> similar functionality that I originally built using bare shr.
>
> I've now applied parts of your patch, and rewrote bits.
>
> --
> (domestic pets only, the antidote for overdose, milk.)
>    bloggy blog: http://lars.ingebrigtsen.no
>

eww: display page source (was: eww.el: Patch to cache the parse tree)

[Ted Zlatanov]

[-- Attachment #1: Type: text/plain, Size: 227 bytes --]

Lars, since you're in a generous mood ;)

Here's a patch to display the current page's HTML source.  Can you see
if it's acceptable?  I used the `eww-current-dom' code as a guide to
where to put the stateful data.

Thanks
Ted


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: eww-display-source.patch --]
[-- Type: text/x-diff, Size: 2605 bytes --]

=== modified file 'lisp/net/eww.el'
--- lisp/net/eww.el	2013-12-03 04:54:17 +0000
+++ lisp/net/eww.el	2013-12-04 16:43:02 +0000
@@ -117,6 +117,7 @@
 
 (defvar eww-current-url nil)
 (defvar eww-current-dom nil)
+(defvar eww-current-source nil)
 (defvar eww-current-title ""
   "Title of current page.")
 (defvar eww-history nil)
@@ -247,6 +248,7 @@
 	     (list
 	      'base (list (cons 'href url))
 	      (libxml-parse-html-region (point) (point-max))))))
+    (setq eww-current-source (buffer-substring (point) (point-max)))
     (eww-setup-buffer)
     (setq eww-current-dom document)
     (let ((inhibit-read-only t)
@@ -375,6 +377,14 @@
   (unless (eq major-mode 'eww-mode)
     (eww-mode)))
 
+(defun eww-view-source ()
+  (let ((buf (get-buffer-create "*eww-source*"))
+        (source eww-current-source))
+    (with-current-buffer buf
+      (delete-region (point-min) (point-max))
+      (insert (or eww-current-source "no source")))
+    (view-buffer buf)))
+
 (defvar eww-mode-map
   (let ((map (make-sparse-keymap)))
     (suppress-keymap map)
@@ -395,6 +405,7 @@
     (define-key map "d" 'eww-download)
     (define-key map "w" 'eww-copy-page-url)
     (define-key map "C" 'url-cookie-list)
+    (define-key map "v" 'eww-view-source)
 
     (define-key map "b" 'eww-add-bookmark)
     (define-key map "B" 'eww-list-bookmarks)
@@ -411,6 +422,7 @@
 	 :active (not (zerop eww-history-position))]
 	["Browse with external browser" eww-browse-with-external-browser t]
 	["Download" eww-download t]
+	["View page source" eww-view-source]
 	["Copy page URL" eww-copy-page-url t]
 	["Add bookmark" eww-add-bookmark t]
 	["List bookmarks" eww-copy-page-url t]
@@ -424,6 +436,7 @@
   ;; FIXME?  This seems a strange default.
   (set (make-local-variable 'eww-current-url) 'author)
   (set (make-local-variable 'eww-current-dom) nil)
+  (set (make-local-variable 'eww-current-source) nil)
   (set (make-local-variable 'browse-url-browser-function) 'eww-browse-url)
   (set (make-local-variable 'after-change-functions) 'eww-process-text-input)
   (set (make-local-variable 'eww-history) nil)
@@ -437,6 +450,7 @@
 	      :title eww-current-title
 	      :point (point)
               :dom eww-current-dom
+              :source eww-current-source
 	      :text (buffer-string))
 	eww-history))
 
@@ -468,6 +482,7 @@
   (let ((inhibit-read-only t))
     (erase-buffer)
     (insert (plist-get elem :text))
+    (setq eww-current-source (plist-get elem :source))
     (setq eww-current-dom (plist-get elem :dom))
     (goto-char (plist-get elem :point))
     (setq eww-current-url (plist-get elem :url)

Re: eww: display page source

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> Lars, since you're in a generous mood ;)
>
> Here's a patch to display the current page's HTML source.  Can you see
> if it's acceptable?  I used the `eww-current-dom' code as a guide to
> where to put the stateful data.

Looks good to me, please apply.  >"?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: eww: display page source

[Ted Zlatanov]

On Thu, 05 Dec 2013 02:37:09 +0100 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Ted Zlatanov <tzz@lifelogs.com> writes:
>> Lars, since you're in a generous mood ;)
>> 
>> Here's a patch to display the current page's HTML source.  Can you see
>> if it's acceptable?  I used the `eww-current-dom' code as a guide to
>> where to put the stateful data.

LMI> Looks good to me, please apply.  >"?

OK; done.  I call `html-mode' if it's loaded to do nice highlighting; I
hope that's OK.

Ted

Re: eww: display page source

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> OK; done.  I call `html-mode' if it's loaded to do nice highlighting; I
> hope that's OK.

Sure; that's nice.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: eww.el: Patch to cache the parse tree

[Lars Magne Ingebrigtsen]

"T.V. Raman" <tv.raman.tv@gmail.com> writes:

> One more request: in the call
> (libxml-parse-html-region (point) (point-max))
>
> could you please pass in the current url as a third argument so
> that libxml knows to handle it as a base URL?

Hm...  doesn't eww do the base URL expansion stuff itself?  I don't
recall why, though.  >"?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

eww.el: Patch to cache the parse tree

[T.V. Raman]

I thought I ran into a case where the base url was unset (dont
remember where) looked at the code, and felt that may be we would
have done better by passing in the base-url. I see that you cons
on base href to the DOM being passed to shr. ...
-- 

-- 


On 12/14/13, Lars Magne Ingebrigtsen <larsi@gnus.org> wrote:
> "T.V. Raman" <tv.raman.tv@gmail.com> writes:
>
>> One more request: in the call
>> (libxml-parse-html-region (point) (point-max))
>>
>> could you please pass in the current url as a third argument so
>> that libxml knows to handle it as a base URL?
>
> Hm...  doesn't eww do the base URL expansion stuff itself?  I don't
> recall why, though.  >"?
>
> --
> (domestic pets only, the antidote for overdose, milk.)
>    bloggy blog: http://lars.ingebrigtsen.no
>

eww.el: Patch to cache the parse tree

[T.V. Raman]

Also, Lars, do you have any interest in incorporating the DOM
filtering code I wrote for Emacspeak users; may be useful for the
mainstream user -- hard for me to tell. You can see the code
here:
http://emacspeak.googlecode.com/svn/trunk/lisp/emacspeak-eww.el


-- 

-- 


On 12/14/13, Lars Magne Ingebrigtsen <larsi@gnus.org> wrote:
> "T.V. Raman" <tv.raman.tv@gmail.com> writes:
>
>> One more request: in the call
>> (libxml-parse-html-region (point) (point-max))
>>
>> could you please pass in the current url as a third argument so
>> that libxml knows to handle it as a base URL?
>
> Hm...  doesn't eww do the base URL expansion stuff itself?  I don't
> recall why, though.  >"?
>
> --
> (domestic pets only, the antidote for overdose, milk.)
>    bloggy blog: http://lars.ingebrigtsen.no
>

Re: eww.el: Patch to cache the parse tree

[Lars Ingebrigtsen]

"T.V. Raman" <tv.raman.tv@gmail.com> writes:

> Also, Lars, do you have any interest in incorporating the DOM
> filtering code I wrote for Emacspeak users; may be useful for the
> mainstream user -- hard for me to tell. You can see the code
> here:
> http://emacspeak.googlecode.com/svn/trunk/lisp/emacspeak-eww.el

This looks useful, but it does look like it would slow down HTML
rendering.

eww is plenty slow already, and is the sort of system where a lot of
good, small improvements may conspire collectively to make it totally
unusable, while each improvement is a good thing individually.

So I think this would be better as a part of Emacspeak.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/