DOM manipulation functions

DOM manipulation functions

[Lars Magne Ingebrigtsen]

As part of the TLS management thing, it was suggested that Emacs display
visually certificate fingerprints:

http://sebsauvage.net/wiki/doku.php?id=php:vizhash_gd

I've started looking into this, and it seems like this would be really
easy to implement as SVG images -- SVG provides all the primitives
vizhash needs.

SVG is basically an XML DOM, and while constructing these it would be a
lot easier if Emacs had a DOM manipulation library.

So here's my two questions:

1) Would it be OK if I added a DOM manipulation/searching library to
Emacs?

2) Due to stupidity on my part I disregarded Chong's advice on how to do
the DOM manipulation things in shr.el, so we now basically have two DOMs
in Emacs -- the xml.el one and the one that shr uses.

I think it would be a good idea if I rewrote shr.el to just use the
xml.el DOM, like Chong said back then.  This will break any third-party
code that relies on shr internals, though.

So: New dom.el, and rewrite shr/eww to depend on dom.el.

Stefan, does this sound OK to you?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: DOM manipulation functions

[joakim]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> As part of the TLS management thing, it was suggested that Emacs display
> visually certificate fingerprints:
>
> http://sebsauvage.net/wiki/doku.php?id=php:vizhash_gd
>
> I've started looking into this, and it seems like this would be really
> easy to implement as SVG images -- SVG provides all the primitives
> vizhash needs.
>
> SVG is basically an XML DOM, and while constructing these it would be a
> lot easier if Emacs had a DOM manipulation library.
>
> So here's my two questions:
>
> 1) Would it be OK if I added a DOM manipulation/searching library to
> Emacs?
>
> 2) Due to stupidity on my part I disregarded Chong's advice on how to do
> the DOM manipulation things in shr.el, so we now basically have two DOMs
> in Emacs -- the xml.el one and the one that shr uses.
>
> I think it would be a good idea if I rewrote shr.el to just use the
> xml.el DOM, like Chong said back then.  This will break any third-party
> code that relies on shr internals, though.
>
> So: New dom.el, and rewrite shr/eww to depend on dom.el.
>
> Stefan, does this sound OK to you?

It sounds cool, and I would like to use it in the xwidgets branch, if possible.

-- 
Joakim Verona

Re: DOM manipulation functions

[Rüdiger Sonderfeld]

On Sunday 23 November 2014 16:18:24 Lars Magne Ingebrigtsen wrote:
> 2) Due to stupidity on my part I disregarded Chong's advice on how to do
> the DOM manipulation things in shr.el, so we now basically have two DOMs
> in Emacs -- the xml.el one and the one that shr uses.
> 
> I think it would be a good idea if I rewrote shr.el to just use the
> xml.el DOM, like Chong said back then.  This will break any third-party
> code that relies on shr internals, though.

Aren't there three?  I think libxml has its own DOM structure as well or is it 
the same as xml.el?  Because in shr.el the code converts the libxml DOM into 
its own DOM.  But that's just from my memory.

> So: New dom.el, and rewrite shr/eww to depend on dom.el.

I think it would be awesome if we had one unified DOM implementation, which 
could either use libxml (if available) or xml.el (as a fallback) to parse XML.

Regards,
Rüdiger

Re: DOM manipulation functions

[Lars Magne Ingebrigtsen]

Rüdiger Sonderfeld <ruediger@c-plusplus.de> writes:

> Aren't there three?  I think libxml has its own DOM structure as well or is it 
> the same as xml.el?

The libxml DOM is the same as the xml.el DOM.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: DOM manipulation functions

[Steinar Bang]

>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org>:

> Rüdiger Sonderfeld <ruediger@c-plusplus.de> writes:
>> Aren't there three?  I think libxml has its own DOM structure as well or is it 
>> the same as xml.el?

> The libxml DOM is the same as the xml.el DOM.

That's sounds great.  The libxml2 DOM is the fastest DOM(-ish)
implementation I know of (even though it's quite old by now).

Re: DOM manipulation functions

[raman]

I'd love to have the DOM manipulation functionality  to use from
Emacspeak -- so 1+!

Re: DOM manipulation functions

[Leo Liu]

On 2014-11-23 16:59 +0100, Rüdiger Sonderfeld wrote:
> I think it would be awesome if we had one unified DOM implementation,
> which could either use libxml (if available) or xml.el (as a fallback)
> to parse XML.

I also much prefer a unified and solid dom manipulation library in
emacs.

Leo

mapping data formats imported from C libraries (was: DOM manipulation functions)

[Ted Zlatanov]

On Mon, 24 Nov 2014 08:26:47 +0800 Leo Liu <sdl.web@gmail.com> wrote: 

LL> On 2014-11-23 16:59 +0100, Rüdiger Sonderfeld wrote:
>> I think it would be awesome if we had one unified DOM implementation,
>> which could either use libxml (if available) or xml.el (as a fallback)
>> to parse XML.

LL> I also much prefer a unified and solid dom manipulation library in
LL> emacs.

This is going to come up with libjson (see
https://github.com/vincenthz/libjson/blob/master/json.h) vs. json.el as
well. And libyaml probably, though that one may get mapped to whatever
we do for JSON anyway. Do we try to keep the existing data mappings or
fit closer to the library's mappings at the C level?

Ted

Re: DOM manipulation functions

[Nicolas Richard]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
> I think it would be a good idea if I rewrote shr.el to just use the
> xml.el DOM, like Chong said back then.  This will break any third-party
> code that relies on shr internals, though.
>
> So: New dom.el, and rewrite shr/eww to depend on dom.el.

Great idea. I always wondered why shr had its own representation.

Please note that there are various existing implementations that might
be worth looking at, including one already named dom.el
(http://www.emacswiki.org/emacs/dom.el). See
http://www.emacswiki.org/emacs/XmlParser

-- 
Nicolas Richard

Re: mapping data formats imported from C libraries

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> This is going to come up with libjson (see
> https://github.com/vincenthz/libjson/blob/master/json.h) vs. json.el as
> well. And libyaml probably, though that one may get mapped to whatever
> we do for JSON anyway. Do we try to keep the existing data mappings or
> fit closer to the library's mappings at the C level?

I think it makes sense to map all those into the Emacs DOM.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: mapping data formats imported from C libraries

[Ted Zlatanov]

On Tue, 25 Nov 2014 16:54:50 +0100 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Ted Zlatanov <tzz@lifelogs.com> writes:
>> This is going to come up with libjson (see
>> https://github.com/vincenthz/libjson/blob/master/json.h) vs. json.el as
>> well. And libyaml probably, though that one may get mapped to whatever
>> we do for JSON anyway. Do we try to keep the existing data mappings or
>> fit closer to the library's mappings at the C level?

LMI> I think it makes sense to map all those into the Emacs DOM.

You mean json.el's mapping?  I don't think there's an official DOM for that.

Ted

Re: mapping data formats imported from C libraries

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> LMI> I think it makes sense to map all those into the Emacs DOM.
>
> You mean json.el's mapping?  I don't think there's an official DOM for that.

I want there to be One True Emacs DOM, and that's the xml.el/libxml2
DOM.  :-)  JSON maps just fine unto that DOM.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: mapping data formats imported from C libraries

[Ted Zlatanov]

On Tue, 25 Nov 2014 17:32:00 +0100 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Ted Zlatanov <tzz@lifelogs.com> writes:
LMI> I think it makes sense to map all those into the Emacs DOM.
>> 
>> You mean json.el's mapping?  I don't think there's an official DOM for that.

LMI> I want there to be One True Emacs DOM, and that's the xml.el/libxml2
LMI> DOM.  :-)  JSON maps just fine unto that DOM.

Yes, but in a really sucky way. If you look at the libjson and json.el
output, it's much leaner. Mapping JSON up to XML pretty much guarantees
pain when you use it and on the way back.  Do you want to inflict that
on everyone in the name of a unified DOM?

Ted

Re: mapping data formats imported from C libraries

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> Yes, but in a really sucky way. If you look at the libjson and json.el
> output, it's much leaner. Mapping JSON up to XML pretty much guarantees
> pain when you use it and on the way back.  Do you want to inflict that
> on everyone in the name of a unified DOM?

The Emacs DOM is quite non-sucky.  It's just a tree.

(thing ((attrib . value)) (foo) (bar))

JSON maps onto it really nicely.  The attributes are always nil, though.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: mapping data formats imported from C libraries

[Steinar Bang]

>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org>:

> The Emacs DOM is quite non-sucky.  It's just a tree.

> (thing ((attrib . value)) (foo) (bar))

Nice.  How do you get to that tree from libxml2? Is it parsed into the
libxml2 DOM first and then transferred into the lisp structure?

If so, is the libxml2 tree kept in memory as well, or is it free'd?  Or
is the lisp structure a thin wrapper over the libxml2 native tree?

Or is the XML stream parsed by libxml2 directly into the lisp structure?
Ie. the lisp structure is the only in-memory representation of the DOM?

Re: mapping data formats imported from C libraries

[Ted Zlatanov]

On Tue, 25 Nov 2014 18:06:37 +0100 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Ted Zlatanov <tzz@lifelogs.com> writes:
>> Yes, but in a really sucky way. If you look at the libjson and json.el
>> output, it's much leaner. Mapping JSON up to XML pretty much guarantees
>> pain when you use it and on the way back.  Do you want to inflict that
>> on everyone in the name of a unified DOM?

LMI> The Emacs DOM is quite non-sucky.  It's just a tree.

LMI> (thing ((attrib . value)) (foo) (bar))

LMI> JSON maps onto it really nicely.  The attributes are always nil, though.

Yeah, I've seen the mess that libxml2 and json.el produce (not the
coder's fault, it's just a pain to adapt without native types).  It
doesn't help that XML/SGML and Lisp are inbred cousins :)

* it's a pain to distinguish JSON's `false' and `null' while using them
  in code

* in json.el you pick the key-value storage method! yay!

* in json.el you pick the array storage method! yay!

* JSON data is not a tree

My ideal mapping from libjson and libyaml would be:

* unambiguous: you can reconstruct a canonical version of the input
  stream from the parse data, as much as possible (JSON is better at this)

* native: you can iterate over JSON arrays as Lisp lists, and JSON
  key-value mappings as alists.  JSON's `false' and `null' are both
  false in Lisp boolean context.

* clean: no extra cons cells for attributes

* readable: this is huge IMO, to be able to pretty-print the data and
  quickly see what it contains... one of the things I like about JSON
  vs. SGML/XML.

The ideal thing may be a "libjson/libyaml native" data format that can
then be bidirectionally mapped to the DOM you suggest...

Ted

Re: mapping data formats imported from C libraries

[Lars Magne Ingebrigtsen]

Steinar Bang <sb@dod.no> writes:

> Nice.  How do you get to that tree from libxml2? Is it parsed into the
> libxml2 DOM first and then transferred into the lisp structure?

Yes.

> If so, is the libxml2 tree kept in memory as well, or is it free'd?

It's freed.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: mapping data formats imported from C libraries

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> Yeah, I've seen the mess that libxml2 and json.el produce (not the
> coder's fault, it's just a pain to adapt without native types).  It
> doesn't help that XML/SGML and Lisp are inbred cousins :)
>
> * it's a pain to distinguish JSON's `false' and `null' while using them
>   in code

Well, I don't think that's a DOM point.  

> * in json.el you pick the key-value storage method! yay!
>
> * in json.el you pick the array storage method! yay!
>
> * JSON data is not a tree

Sure it is.

{"a": {"b": [1, 2], "c": 4}}

> My ideal mapping from libjson and libyaml would be:
>
> * unambiguous: you can reconstruct a canonical version of the input
>   stream from the parse data, as much as possible (JSON is better at this)

Trivial from the Emacs DOM.

> * native: you can iterate over JSON arrays as Lisp lists, and JSON
>   key-value mappings as alists.  JSON's `false' and `null' are both
>   false in Lisp boolean context.

Outside the scope of the DOM.

> * clean: no extra cons cells for attributes

Doesn't matter.  :-)

> * readable: this is huge IMO, to be able to pretty-print the data and
>   quickly see what it contains... one of the things I like about JSON
>   vs. SGML/XML.

`pp' prints the DOM real purdy like.  I've web scraped bunches and
bunches of pages, and reading the DOM in the *scratch* buffer is really
easy.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: mapping data formats imported from C libraries

[Ted Zlatanov]

On Wed, 26 Nov 2014 16:56:15 +0100 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Ted Zlatanov <tzz@lifelogs.com> writes:
>> * JSON data is not a tree

LMI> Sure it is.

LMI> {"a": {"b": [1, 2], "c": 4}}

I think that and [1, 2, 3] are not trees.  There is no root value.

But as for the rest, OK, I'll give it a try.  I'm just not excited about
it for the reasons I listed.

Ted

Re: DOM manipulation functions

[Lars Magne Ingebrigtsen]

This is now on the trunk.  Please let me know whether something broke in
eww or shr.

I should probably add a manual entry for dom.el...  But where?  The
lispref manual?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: DOM manipulation functions

[Ted Zlatanov]

On Wed, 26 Nov 2014 19:44:13 +0100 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> This is now on the trunk.  Please let me know whether something broke in
LMI> eww or shr.

LMI> I should probably add a manual entry for dom.el...  But where?  The
LMI> lispref manual?

Maybe a new "external data access" section in lispref? It could also
have the general libxml info and I can later add libjson and libyaml
subsections.

Ted

Re: DOM manipulation functions

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> Maybe a new "external data access" section in lispref? It could also
> have the general libxml info and I can later add libjson and libyaml
> subsections.

I added it as a subsection to the node that mentioned the libxml-parse-*
functions, but perhaps all that should be moved...  somewhere...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: mapping data formats imported from C libraries

[Steinar Bang]

>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org>:

>> If so, is the libxml2 tree kept in memory as well, or is it free'd?

> It's freed.

Ok.  That means the superfast libxml2 XPath isn't available.

Re: mapping data formats imported from C libraries

[Lars Magne Ingebrigtsen]

Steinar Bang <sb@dod.no> writes:

> Ok.  That means the superfast libxml2 XPath isn't available.

No, if you have a 46GB XML document, Emacs isn't the ideal way to deal
with that document.  :-)  (Unless you have a 64GB RAM machine.  Then it
should be OK, I think.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no