XWidget/WebKit: Getting Web Content From Webkit To Emacs

XWidget/WebKit: Getting Web Content From Webkit To Emacs

[raman]

Now that we have XWidget/Webkit --- I was wondering what it would take
to get content from Webkit into Emacs --- this time in the form of
"real content" rather than "What You See Is All You Have Content".

Some Thoughts:

1. Would it be possible to get the DOM (as an html DOM a la EWW) from
Webkit? That would give us content that Emacs ordinarily cannot get --
e.g. content generated through script evaluation.

2. Ability to send events / scripts from Emacs to WebKit -- I see that
some of this might already be possible from looking at the code in
xwidget.el. It's not clear how Emacs can get at structured content
returned by JS.


3. Function xwidget-webkit-copy-selection-as-kill will hopefully let
me get the rendered text contents into the kill ring -- but how do I
select the contents of a webkit buffer (without using a mouse)?

-- 

-- 

Re: XWidget/WebKit: Getting Web Content From Webkit To Emacs

[joakim]

raman@google.com writes:

> Now that we have XWidget/Webkit --- I was wondering what it would take
> to get content from Webkit into Emacs --- this time in the form of
> "real content" rather than "What You See Is All You Have Content".
>
> Some Thoughts:
>
> 1. Would it be possible to get the DOM (as an html DOM a la EWW) from
> Webkit? That would give us content that Emacs ordinarily cannot get --
> e.g. content generated through script evaluation.

When I wrote the code there were some problems with that area of the
webkit api.

There is a start of an implementation of this functionality in the main
xwidget branch(not the xwidget_branch): xwidget-webkit-dom-dump

I attach the partial implementation below.


> 2. Ability to send events / scripts from Emacs to WebKit -- I see that
> some of this might already be possible from looking at the code in
> xwidget.el. It's not clear how Emacs can get at structured content
> returned by JS.


You can have a look at xwidget-webkit-execute-script-rv.

Sadly there wasn't a good function for returning values from webkit in
their api when I wrote the code. I haven't checked if that situation has
changed recently. The return value is pased by setting the
document.title. Ahem... I'm not sure about the limits of this mechanism,
it has been working for the things I've done anyway.



>
>
> 3. Function xwidget-webkit-copy-selection-as-kill will hopefully let
> me get the rendered text contents into the kill ring -- but how do I
> select the contents of a webkit buffer (without using a mouse)?

You will need to code something that moves the selection about by
sending js code to the webkit through xwidget-webkit-execute-script.

I haven't done something like that yet. 

Initial DOM dump experimental code.

-#ifdef HAVE_XLXW_DOMDUMP
-/* dom dumping is work in progress.
-   2 of the methods used from webkit are deprecated nowm and replacements sought
- */
-static void
-xwidget_webkit_dom_dump (WebKitDOMNode * parent)
-{
-  WebKitDOMNodeList *list;
-  int length;
-  WebKitDOMNode *attribute;
-  WebKitDOMNamedNodeMap *attrs;
-  WebKitDOMNode *child;
-  printf ("node:%d type:%d name:%s content:%s\n", parent,
-          webkit_dom_node_get_node_type (parent),
-          //1 element 3 text 8 comment 2 attribute
-          webkit_dom_node_get_local_name (parent),
-          webkit_dom_node_get_text_content (parent));
-
-  if (webkit_dom_node_has_attributes (parent))
-    {
-      attrs = webkit_dom_node_get_attributes (parent);
-
-      length = webkit_dom_named_node_map_get_length (attrs);
-      for (int i = 0; i < length; i++)
-        {
-          attribute = webkit_dom_named_node_map_item (attrs, i);
-          printf (" attr node:%d type:%d name:%s content:%s\n",
-                  attribute, webkit_dom_node_get_node_type (attribute),
-                  //1 element 3 text 8 comment
-                  webkit_dom_node_get_local_name (attribute),
-                  webkit_dom_node_get_text_content (attribute));
-        }
-    }
-  list = webkit_dom_node_get_child_nodes (parent);
-  length = webkit_dom_node_list_get_length (list);
-
-  for (int i = 0; i < length; i++)
-    {
-      child = webkit_dom_node_list_item (list, i);
-      //if(webkit_dom_node_has_child_nodes(child))
-      xwidget_webkit_dom_dump (child);
-    }
-}
-
-
-DEFUN ("xwidget-webkit-dom-dump",
-       Fxwidget_webkit_dom_dump, Sxwidget_webkit_dom_dump,
-       1, 1, 0, doc:	/*Dump the DOM contained in the webkit
-                          instance in XWIDGET. */
-       )
-  (Lisp_Object xwidget)
-{
-  WEBKIT_FN_INIT ();
-  xwidget_webkit_dom_dump (WEBKIT_DOM_NODE
-                           (webkit_web_view_get_dom_document
-                            (WEBKIT_WEB_VIEW (xw->widget_osr))));
-  return Qnil;
-}
-
-#endif /* HAVE_XLXW_DOMDUMP */

> -- 

-- 
Joakim Verona

Re: XWidget/WebKit: Getting Web Content From Webkit To Emacs

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Could we please not refer to web pages as "content"?  Many pages
contain works (textual or graphical) that deserve appreciation,
but "content" reduces them all to the level of "filler for a box".

See http://gnu.org/philosophy/words-to-avoid.html#Content.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: XWidget/WebKit: Getting Web Content From Webkit To Emacs

[Vasilij Schneidermann]

> Sadly there wasn't a good function for returning values from webkit in
> their api when I wrote the code. I haven't checked if that situation has
> changed recently. The return value is pased by setting the
> document.title. Ahem... I'm not sure about the limits of this mechanism,
> it has been working for the things I've done anyway.


This is still the case for webkit1gtk, webkit2gtk however introduced
the functions webkit_web_view_run_javascript and
webkit_web_view_run_javascript_finish which allow you to evaluate
JavaScript asynchronously and retrieve the return value once it's
done.

Re: XWidget/WebKit: Getting Web Content From Webkit To Emacs

[joakim]

Vasilij Schneidermann <v.schneidermann@gmail.com> writes:

>> Sadly there wasn't a good function for returning values from webkit in
>> their api when I wrote the code. I haven't checked if that situation has
>> changed recently. The return value is pased by setting the
>> document.title. Ahem... I'm not sure about the limits of this mechanism,
>> it has been working for the things I've done anyway.
>
>
> This is still the case for webkit1gtk, webkit2gtk however introduced
> the functions webkit_web_view_run_javascript and
> webkit_web_view_run_javascript_finish which allow you to evaluate
> JavaScript asynchronously and retrieve the return value once it's
> done.

Thanks for this information! I will try it out once things have stabilized.

-- 
Joakim Verona