online conversion support from xsd to rng?

online conversion support from xsd to rng?

[joakim]

It occurred to me that the Nxml mode in Emacs isn't as useful as it could
be. Nxml supports rng schema descriptors(relax ng). Xml files are usually
described by xsd files such as this one:
http://maven.apache.org/xsd/maven-4.0.0.xsd.

So, if I could convert a xsd to a rng on the fly, I could immediately
have function completion and validation on a number of xml documents. 

James Clark(the original author of the nxml mode) wrote a utility called
Trang: http://www.relaxng.org/#conversion

This could presumably be run online somewhere and provide the needed
service for Emacs, more conveniently than having to install the program
locally. (Trang is a free software Java program compilable with GCJ)

This raises a number of questions:

- do we want Emacs to have this type of facility at all? (I would say
yes, provided we solve a number of problems)

- how do we ensure this backend(and future other backends) are free
software?

- what would the backend interface look like? Maybe we could generate
ELPA packages on the fly?


-- 
Joakim Verona

Re: online conversion support from xsd to rng?

[joakim]

joakim@verona.se writes:

> It occurred to me that the Nxml mode in Emacs isn't as useful as it could
> be. Nxml supports rng schema descriptors(relax ng). Xml files are usually
> described by xsd files such as this one:
> http://maven.apache.org/xsd/maven-4.0.0.xsd.
>
> So, if I could convert a xsd to a rng on the fly, I could immediately
> have function completion and validation on a number of xml documents. 
>
> James Clark(the original author of the nxml mode) wrote a utility called
> Trang: http://www.relaxng.org/#conversion
>
> This could presumably be run online somewhere and provide the needed
> service for Emacs, more conveniently than having to install the program
> locally. (Trang is a free software Java program compilable with GCJ)
>
> This raises a number of questions:
>
> - do we want Emacs to have this type of facility at all? (I would say
> yes, provided we solve a number of problems)
>
> - how do we ensure this backend(and future other backends) are free
> software?
>
> - what would the backend interface look like? Maybe we could generate
> ELPA packages on the fly?

I noticed trang doesnt do xsd->rng conversion. That can instead be handled
by a xslt script:
http://xsdtorngconverter.googlecode.com/svn/trunk/xsdtorngconverter/XSDtoRNG.xsl

That seems to strengthen the argument that Emacs usage could be made
easier by providing interfaces to server-side services. Setting up all
this conversion machinery client-side is not trivial. OTOH we must then
solve how to ensure backends are free software, perhaps like specifying
a REST api that provides a GPL compliance symbol, similar to how dynamic
library FFI was solved. 

-- 
Joakim Verona

Re: online conversion support from xsd to rng?

[Chong Yidong]

joakim@verona.se writes:

> This could presumably be run online somewhere and provide the needed
> service for Emacs, more conveniently than having to install the program
> locally. (Trang is a free software Java program compilable with GCJ)
>
> This raises a number of questions:
>
> - do we want Emacs to have this type of facility at all? (I would say
> yes, provided we solve a number of problems)
>
> - how do we ensure this backend(and future other backends) are free
> software?
>
> - what would the backend interface look like? Maybe we could generate
> ELPA packages on the fly?

Why can't we simply include every common schema in etc/schema/?  That
directory haven't been updated (AFAICT) since 23.1, so if you want to
update and/or add to it, please feel free (we just need the
specification to be free; see etc/schema/README).

Re: online conversion support from xsd to rng?

[joakim]

Chong Yidong <cyd@stupidchicken.com> writes:

> joakim@verona.se writes:
>
>> This could presumably be run online somewhere and provide the needed
>> service for Emacs, more conveniently than having to install the program
>> locally. (Trang is a free software Java program compilable with GCJ)
>>
>> This raises a number of questions:
>>
>> - do we want Emacs to have this type of facility at all? (I would say
>> yes, provided we solve a number of problems)
>>
>> - how do we ensure this backend(and future other backends) are free
>> software?
>>
>> - what would the backend interface look like? Maybe we could generate
>> ELPA packages on the fly?
>
> Why can't we simply include every common schema in etc/schema/?  That
> directory haven't been updated (AFAICT) since 23.1, so if you want to
> update and/or add to it, please feel free (we just need the
> specification to be free; see etc/schema/README).

Thats a possibility. We would also need the possibility for packages to
add schemas.

But still, I have the feeling there will be thousands of schemas in the
end. Anyway, Im workin on a small elisp package that will download and
convert schemas to rng localy, but it will require xsltproc and trang to
be installed. 

-- 
Joakim Verona

Re: online conversion support from xsd to rng?

[Thien-Thi Nguyen]

() joakim@verona.se
() Tue, 07 Dec 2010 23:42:05 +0100

   but it will require xsltproc and trang to
   be installed. 

I just noticed:

$ ldd ~/bin/bloody-emacs | grep expat
	libexpat.so.1 => /usr/lib/libexpat.so.1 (0xf6ed5000)

so perhaps you could write C bindings to libexpat and port xsltproc and
trang (or the desired subset of their functionality) to Emacs Lisp.

[Insert customary XML denigrations here.]

Re: online conversion support from xsd to rng?

[joakim]

Thien-Thi Nguyen <ttn@gnuvola.org> writes:

> () joakim@verona.se
> () Tue, 07 Dec 2010 23:42:05 +0100
>
>    but it will require xsltproc and trang to
>    be installed. 
>
> I just noticed:
>
> $ ldd ~/bin/bloody-emacs | grep expat
> 	libexpat.so.1 => /usr/lib/libexpat.so.1 (0xf6ed5000)
>
> so perhaps you could write C bindings to libexpat and port xsltproc and
> trang (or the desired subset of their functionality) to Emacs Lisp.

Hmmm. Intriguing. Hadn't thought of that. Maybe everything we need is
already in there somewhere...

>
> [Insert customary XML denigrations here.]

Its not so bad these days IMHO. More and more packages seem to realize
that xml = json = lisp lists, so many web services export their
interfaces through both xml and json.

-- 
Joakim Verona

Re: online conversion support from xsd to rng?

[Andrew W. Nosenko]

On Wed, Dec 8, 2010 at 03:31, Thien-Thi Nguyen <ttn@gnuvola.org> wrote:
> () joakim@verona.se
> () Tue, 07 Dec 2010 23:42:05 +0100
>
>   but it will require xsltproc and trang to
>   be installed.
>
> I just noticed:
>
> $ ldd ~/bin/bloody-emacs | grep expat
>        libexpat.so.1 => /usr/lib/libexpat.so.1 (0xf6ed5000)
>
> so perhaps you could write C bindings to libexpat and port xsltproc and
> trang (or the desired subset of their functionality) to Emacs Lisp.

You don't need to port the libxslt from the libxml2 to libexpat --
libxml2 already used in emacs

$ ldd emacs | grep libxml2
	libxml2.so.5 => /usr/local/lib/libxml2.so.5 (0x804240000)

Also, while libexpat obtained through X11 dependency, libxml2 linked
in deliberately (thread started at
http://lists.gnu.org/archive/html/emacs-devel/2010-09/msg00179.html).

-- 
Andrew W. Nosenko <andrew.w.nosenko@gmail.com>

Re: online conversion support from xsd to rng?

[Thien-Thi Nguyen]

() "Andrew W. Nosenko" <andrew.w.nosenko@gmail.com>
() Wed, 8 Dec 2010 12:45:26 +0200

   You don't need to port the libxslt from the libxml2 to libexpat --
   libxml2 already used in emacs

   $ ldd emacs | grep libxml2
           libxml2.so.5 => /usr/local/lib/libxml2.so.5 (0x804240000)

I think there are two areas of primary concern:
(a) conversion between XML and Lisp trees (nested lists)
(b) routines for manipulating the tree

Personally (in a Scheme context, which is similar to Emacs Lisp), i
use ‘ttn-do xml2sexp’ for (a), and ad-hoc car/cdr climbing for (b),
avoiding libxslt altogether.  The idea is to flee the angry brackets
into the loving parens immediately.  (I briefly looked at the pattern
matching / transform primitives of XSLT and was horrified.)

Re: online conversion support from xsd to rng?

[Thien-Thi Nguyen]

() joakim@verona.se
() Wed, 08 Dec 2010 09:19:43 +0100

   Maybe everything we need is
   already in there somewhere...

XML in Emacs: like whipping up dinner
by rooting around the kitchen sink...

RE: online conversion support from xsd to rng?

[Drew Adams]

> I think there are two areas of primary concern:
> (a) conversion between XML and Lisp trees (nested lists)
> (b) routines for manipulating the tree

Once you've converted XML to Lisp you lose all XML-level access, transformation,
etc.  IOW, once in Lispland, no XPath, XQuery, XSLT,...  If your processing of
XML also involves, well, XML processing (e.g. XQuery transformations) that is
part of the given, then you've lost that.

IOW, conversion to conses and processing using Lisp can be useful, but depending
on your context there can be a cost.

Re: online conversion support from xsd to rng?

[Štěpán Němec]

"Drew Adams" <drew.adams@oracle.com> writes:

>> I think there are two areas of primary concern:
>> (a) conversion between XML and Lisp trees (nested lists)
>> (b) routines for manipulating the tree
>
> Once you've converted XML to Lisp you lose all XML-level access, transformation,
> etc.  IOW, once in Lispland, no XPath, XQuery, XSLT,...  If your processing of
> XML also involves, well, XML processing (e.g. XQuery transformations) that is
> part of the given, then you've lost that.
>
> IOW, conversion to conses and processing using Lisp can be useful, but depending
> on your context there can be a cost.

This is nonsense. XPath and friends are not bound to the external
representation of the data in any way. They operate on the DOM tree. So
as long as your XML conversion to sexp format is lossless (which it
should be, of course), you lose nothing, only gain the advantages of
saner format (i.e. sexp) both for humans and the machine -- you just
need an XPath etc. implementation for your language.

  Štěpán

Re: online conversion support from xsd to rng?

[Lennart Borgman]

On Fri, Dec 10, 2010 at 1:06 AM, Štěpán Němec <stepnem@gmail.com> wrote:
> "Drew Adams" <drew.adams@oracle.com> writes:
>
>>> I think there are two areas of primary concern:
>>> (a) conversion between XML and Lisp trees (nested lists)
>>> (b) routines for manipulating the tree
>>
>> Once you've converted XML to Lisp you lose all XML-level access, transformation,
>> etc.  IOW, once in Lispland, no XPath, XQuery, XSLT,...  If your processing of
>> XML also involves, well, XML processing (e.g. XQuery transformations) that is
>> part of the given, then you've lost that.
>>
>> IOW, conversion to conses and processing using Lisp can be useful, but depending
>> on your context there can be a cost.
>
> This is nonsense. XPath and friends are not bound to the external
> representation of the data in any way. They operate on the DOM tree. So
> as long as your XML conversion to sexp format is lossless (which it
> should be, of course), you lose nothing, only gain the advantages of
> saner format (i.e. sexp) both for humans and the machine -- you just
> need an XPath etc. implementation for your language.

Is there such an implementation in for example elisp?

RE: online conversion support from xsd to rng?

[Drew Adams]

>>> I think there are two areas of primary concern:
>>> (a) conversion between XML and Lisp trees (nested lists)
>>> (b) routines for manipulating the tree
> >
> > Once you've converted XML to Lisp you lose all XML-level 
> > access, transformation, etc.  IOW, once in Lispland, no
> > XPath, XQuery, XSLT,...  If your processing of
> > XML also involves, well, XML processing (e.g. XQuery 
> > transformations) that is part of the given, then you've lost that.
> >
> > IOW, conversion to conses and processing using Lisp can be 
> > useful, but depending on your context there can be a cost.
> 
> This is nonsense. XPath and friends are not bound to the external
> representation of the data in any way. They operate on the 
> DOM tree.

Precisely - XPath and friends operate on a DOM or streaming SAX events or PSVI
or XQDM or...  They don't operate on conses. Unless those conses map to a DOM,
PSVI, etc.

No one said anything about using an external representation of the data - except
you.

> So as long as your XML conversion to sexp format is lossless (which it
> should be, of course),

Well sure, if you build a Lisp DOM or the equivalent, so that nothing is lost.
That's not the impression I got from the (admittedly meager) description given.

I think we are partly saying the same thing - we are both saying, I guess, that
unless you have a full, lossless representation you are losing out on something.
You are supposing such a representation; I wasn't hearing that.

And even a perfectly faithful representation is not much without faithful
access. By that I mean access faithful to the existing (standard) XML processing
languages - IOW support for them. I suppose (hope) that's what you mean by
saying that you "need an XPath etc. implementation for your language".

> you lose nothing, only gain the advantages of
> saner format (i.e. sexp) both for humans and the machine -- you just
> need an XPath etc. implementation for your language.

Sure. "Etc."  Y'a qu'a...

A Lisp DOM API, for example ("(b) routines for manipulating the tree").
Or a Lisp PSVI. Or an XQuery Data Model. Or all of the above. Etc.

A fairly far cry from "ad-hoc car/cdr climbing for (b)" (depending on what was
meant).

Don't get me wrong. I'm certainly not against the idea. Just pointing out that
Lisp is not XML (for better or for worse), and for XML processing you need a
representation of XML (nodes - I'm not talking text anymore than you are).

And to be really helpful you need interfaces with (support for) XML languages
(e.g. XQuery). That is, *IF* you want to take advantage of existing code in such
languages and support those languages with a Lisp API (yes, Virginia, there is
lots of existing XML processing code).

If not - if you just build your own Lispy access etc. (e.g. car/cdr climbing)
without support for XML languages, then yes, you forego existing code and
applications written in them.

I'm not familiar with `xml2sexp' - maybe that's truly all that's needed. But
this description didn't really inspire confidence:

"The idea is to flee the angry brackets into the loving parens immediately.  (I
briefly looked at the pattern matching / transform primitives of XSLT and was
horrified.)"

That doesn't give me the impression that "you just need an XPath etc.
implementation for your language." It gives me the impression that the idea is
to not use (support) XPath, XQuery, XSLT, DOM, or any of the rest. Hence my
comment that dropping all of that represents a loss. Or, as I said:

> > IOW, conversion to conses and processing using Lisp can be useful,
 
^^^^^^^^^^^^^^^
> > but depending on your context there can be a cost.

It can be useful, but it is not the same thing as supporting XQuery code, XSLT
code, etc.

Re: online conversion support from xsd to rng?

[Stephen J. Turnbull]

[-- Attachment #1: Type: text/plain, Size: 4 bytes --]

Štěp

[-- Attachment #2: Type: text/plain, Size: 3 bytes --]

án 

[-- Attachment #3: Type: text/plain, Size: 904 bytes --]

Němec writes:

 > This is nonsense. XPath and friends are not bound to the external
 > representation of the data in any way. They operate on the DOM tree. So
 > as long as your XML conversion to sexp format is lossless (which it
 > should be, of course), you lose nothing, only gain the advantages of
 > saner format (i.e. sexp) both for humans and the machine -- you just
 > need an XPath etc. implementation for your language.

Sure.  But our language doesn't have a standard lossless format yet,
it doesn't have an XPath implementation at all, and it's not clear to
me why a lossless sexp format would be all that much more readable
than XML (assuming an XML mode designed to be readable, which would
probably be a bigger gain than XML-to-Lisp -- all XML modes I've seen
have been designed for use by those who think XML is readable).

I think you've got a lot of work ahead of you.

Re: online conversion support from xsd to rng?

[Štěpán Němec]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> Štěpán Němec writes:
>
>  > This is nonsense. XPath and friends are not bound to the external
>  > representation of the data in any way. They operate on the DOM tree. So
>  > as long as your XML conversion to sexp format is lossless (which it
>  > should be, of course), you lose nothing, only gain the advantages of
>  > saner format (i.e. sexp) both for humans and the machine -- you just
>  > need an XPath etc. implementation for your language.
>
> Sure.  But our language doesn't have a standard lossless format yet,
> it doesn't have an XPath implementation at all, and it's not clear to
> me why a lossless sexp format would be all that much more readable
> than XML (assuming an XML mode designed to be readable, which would
> probably be a bigger gain than XML-to-Lisp -- all XML modes I've seen
> have been designed for use by those who think XML is readable).
>
> I think you've got a lot of work ahead of you.

I don't enjoy lengthy unfocused discussions/parallel monologues as much
as you (seem to) do, so I'm not going to try to address all of what you
write (sorry). This time I just somehow couldn't resist and replied to
the nonsense Drew wrote, which in hindsight was probably a mistake.

If you're interested in working with XML in Emacs Lisp, have a look at
the relevant Emacs Wiki page:

http://www.emacswiki.org/emacs/CategoryXML

Granted, the situation in Elisp is not nearly as good as in Scheme for
instance [1], but there is also an XPath implementation linked from
here:

http://www.emacswiki.org/emacs/XmlParser

[1] Which is quite understandable, given the relative quality of Elisp
the language (for me personally, having Scheme in Emacs/Emacs on Scheme
would be no less than a redemption), the related "policies", and how
some of (X)Emacs developers spend the limited amount of time available,
right?

http://okmij.org/ftp/Scheme/xml.html

  Štěpán

Re: online conversion support from xsd to rng?

[Stephen J. Turnbull]

You write:

 > I don't enjoy lengthy unfocused discussions/parallel monologues as much
 > as you (seem to) do,

For better or worse, it's what I do best.  The problem with focused
discussions/sequential monologues is just as great though (missing the
forest because you keep bumping into trees).

 > If you're interested in working with XML in Emacs Lisp

At this point, I'm not.  I'm interested in supporting those who want
to work with XML, which is quite a different thing requiring a
different point of view.

 > Granted, the situation in Elisp is not nearly as good as in Scheme for
 > instance [1], but there is also an XPath implementation linked from
 > here:

I'm not particularly interested in "an implementation"; there are many
and most would really suck to maintain if you were doing it as a
service for others, rather than to get your own work done.

Which implementations of XML features for Elisp do you consider
essential for getting some of your work done?  Give it some serious
consideration, give me a list, and I'll see about getting them into
the XEmacs package tree.  I might even lobby for addition to Emacs.