suggestion: "native" orgmode XML export (and import?)

suggestion: "native" orgmode XML export (and import?)

[Ilya Shlyakhter]

I'm not an emacs-lisp programmer, but I'd like to write scripts
(ideally in Python) to generate custom reports from my .org files.
What would help a lot, is if there was a command to export an .org
file to a "native" XML format that would mirror the org file's
structure and all its logical elements (tags, properties, drawers,
dates etc).   I know about the DocBook exporter, but it maps orgmode's
concepts onto DocBook concepts such as articles.   I'm a longtime
orgmode user and it would be much simpler to write a program in terms
of the familiar org concepts (hierarchical entries, tags, properties
etc).
It would also be great if there was a way to import such an XML file
back into org.  Then one could e.g. take Toodledo.com tasks and
transform them into an orgmode file.

There is an orgmode Python reader at
http://www.members.optusnet.com.au/~charles57/GTD/orgnode.html
and I plan to use that for now.  But it doesn't support all orgmode
features, and more importantly it does its own parsing of orgfiles (so
may not keep up with any future changes).   Using orgmode's own
parser, and then exporting the results as XML, would be much more
reliable.

ilya

Re: suggestion: "native" orgmode XML export (and import?)

[Andrew Stribblehill]

[-- Attachment #1.1: Type: text/plain, Size: 1632 bytes --]

I notice the experimental org-export.el contains an internal representation.
It would probably be very easy for your python to parse the lisp
s-expression it uses, if it were exported.

On Aug 6, 2009 3:55 PM, "Ilya Shlyakhter" <ilya_shl@alum.mit.edu> wrote:

I'm not an emacs-lisp programmer, but I'd like to write scripts
(ideally in Python) to generate custom reports from my .org files.
What would help a lot, is if there was a command to export an .org
file to a "native" XML format that would mirror the org file's
structure and all its logical elements (tags, properties, drawers,
dates etc).   I know about the DocBook exporter, but it maps orgmode's
concepts onto DocBook concepts such as articles.   I'm a longtime
orgmode user and it would be much simpler to write a program in terms
of the familiar org concepts (hierarchical entries, tags, properties
etc).
It would also be great if there was a way to import such an XML file
back into org.  Then one could e.g. take Toodledo.com tasks and
transform them into an orgmode file.

There is an orgmode Python reader at
http://www.members.optusnet.com.au/~charles57/GTD/orgnode.html
and I plan to use that for now.  But it doesn't support all orgmode
features, and more importantly it does its own parsing of orgfiles (so
may not keep up with any future changes).   Using orgmode's own
parser, and then exporting the results as XML, would be much more
reliable.

ilya


_______________________________________________
Emacs-orgmode mailing list
Remember: use `Reply All' to send replies to the list.
Emacs-orgmode@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode

[-- Attachment #1.2: Type: text/html, Size: 2162 bytes --]

[-- Attachment #2: Type: text/plain, Size: 204 bytes --]

_______________________________________________
Emacs-orgmode mailing list
Remember: use `Reply All' to send replies to the list.
Emacs-orgmode@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: suggestion: "native" orgmode XML export (and import?)

[Andrew Stribblehill]

[re-adding the mailing list]

No, EXPERIMENTAL/org-export.el. Here's what I did after M-x load-file
/path/to/org-export.el:

(defun org-export-sexp (arg)
  (interactive "p")
  (let ((bufstr (org-export-parse arg)))
    (save-excursion
      (switch-to-buffer (get-buffer-create "*Org export*"))
      (erase-buffer)
      (insert (format "%s" bufstr)))))

An org file such as:
----
Objectives

* Q3 2009
    :PROPERTIES:
    :SCORE:    0.77
    :END:
** Operations

*** [#0] Oncall. Keep the site up
*** [#1] Keep the tubes clear.

** Projects
----

comes out as:
((:level 1
  :heading Q3 2009
  :properties ((SCORE . 0.77) (CATEGORY . proj-sanitised))
  :content :subtree
  ((:level 2
    :heading Operations
    :properties ((CATEGORY . proj-sanitised))
    :content :subtree
    ((:level 3
      :heading Oncall. Keep the site up
      :properties ((PRIORITY . 0) (CATEGORY . proj-sanitised))
      :content  :subtree nil)
     (:level 3
      :heading Keep the tubes clear.
      :properties ((PRIORITY . 1) (CATEGORY . proj-sanitised))
      :content :subtree nil)))
   (:level 2
    :heading Projects
    :properties ((CATEGORY . proj-sanitised))
    :content :subtree nil))))

2009/8/7 Ilya Shlyakhter <ilya_shl@alum.mit.edu>:
> Thanks for the pointer.   Did you mean org-exp.el?   How exactly do I
> get the internal representation?
> thanks,
> ilya
>
> On Thu, Aug 6, 2009 at 11:58 PM, Andrew Stribblehill<ads@wompom.org> wrote:
>> I notice the experimental org-export.el contains an internal representation.
>> It would probably be very easy for your python to parse the lisp
>> s-expression it uses, if it were exported.
>>
>> On Aug 6, 2009 3:55 PM, "Ilya Shlyakhter" <ilya_shl@alum.mit.edu> wrote:
>>
>> I'm not an emacs-lisp programmer, but I'd like to write scripts
>> (ideally in Python) to generate custom reports from my .org files.
>> What would help a lot, is if there was a command to export an .org
>> file to a "native" XML format that would mirror the org file's
>> structure and all its logical elements (tags, properties, drawers,
>> dates etc).   I know about the DocBook exporter, but it maps orgmode's
>> concepts onto DocBook concepts such as articles.   I'm a longtime
>> orgmode user and it would be much simpler to write a program in terms
>> of the familiar org concepts (hierarchical entries, tags, properties
>> etc).
>> It would also be great if there was a way to import such an XML file
>> back into org.  Then one could e.g. take Toodledo.com tasks and
>> transform them into an orgmode file.
>>
>> There is an orgmode Python reader at
>> http://www.members.optusnet.com.au/~charles57/GTD/orgnode.html
>> and I plan to use that for now.  But it doesn't support all orgmode
>> features, and more importantly it does its own parsing of orgfiles (so
>> may not keep up with any future changes).   Using orgmode's own
>> parser, and then exporting the results as XML, would be much more
>> reliable.
>>
>> ilya
>>
>>
>> _______________________________________________
>> Emacs-orgmode mailing list
>> Remember: use `Reply All' to send replies to the list.
>> Emacs-orgmode@gnu.org
>> http://lists.gnu.org/mailman/listinfo/emacs-orgmode
>>
>

Re: suggestion: "native" orgmode XML export (and import?)

[Ilya Shlyakhter]

That's great, thanks!  I should be able to take it from there.
It would be great if at some point this became official, and also
included an XML exporter and specification.

thanks,

ilya

On Fri, Aug 7, 2009 at 11:13 AM, Andrew Stribblehill<ads@wompom.org> wrote:
> [re-adding the mailing list]
>
> No, EXPERIMENTAL/org-export.el. Here's what I did after M-x load-file
> /path/to/org-export.el:
>
> (defun org-export-sexp (arg)
>  (interactive "p")
>  (let ((bufstr (org-export-parse arg)))
>    (save-excursion
>      (switch-to-buffer (get-buffer-create "*Org export*"))
>      (erase-buffer)
>      (insert (format "%s" bufstr)))))
>
> An org file such as:
> ----
> Objectives
>
> * Q3 2009
>    :PROPERTIES:
>    :SCORE:    0.77
>    :END:
> ** Operations
>
> *** [#0] Oncall. Keep the site up
> *** [#1] Keep the tubes clear.
>
> ** Projects
> ----
>
> comes out as:
> ((:level 1
>  :heading Q3 2009
>  :properties ((SCORE . 0.77) (CATEGORY . proj-sanitised))
>  :content :subtree
>  ((:level 2
>    :heading Operations
>    :properties ((CATEGORY . proj-sanitised))
>    :content :subtree
>    ((:level 3
>      :heading Oncall. Keep the site up
>      :properties ((PRIORITY . 0) (CATEGORY . proj-sanitised))
>      :content  :subtree nil)
>     (:level 3
>      :heading Keep the tubes clear.
>      :properties ((PRIORITY . 1) (CATEGORY . proj-sanitised))
>      :content :subtree nil)))
>   (:level 2
>    :heading Projects
>    :properties ((CATEGORY . proj-sanitised))
>    :content :subtree nil))))
>
> 2009/8/7 Ilya Shlyakhter <ilya_shl@alum.mit.edu>:
>> Thanks for the pointer.   Did you mean org-exp.el?   How exactly do I
>> get the internal representation?
>> thanks,
>> ilya
>>
>> On Thu, Aug 6, 2009 at 11:58 PM, Andrew Stribblehill<ads@wompom.org> wrote:
>>> I notice the experimental org-export.el contains an internal representation.
>>> It would probably be very easy for your python to parse the lisp
>>> s-expression it uses, if it were exported.
>>>
>>> On Aug 6, 2009 3:55 PM, "Ilya Shlyakhter" <ilya_shl@alum.mit.edu> wrote:
>>>
>>> I'm not an emacs-lisp programmer, but I'd like to write scripts
>>> (ideally in Python) to generate custom reports from my .org files.
>>> What would help a lot, is if there was a command to export an .org
>>> file to a "native" XML format that would mirror the org file's
>>> structure and all its logical elements (tags, properties, drawers,
>>> dates etc).   I know about the DocBook exporter, but it maps orgmode's
>>> concepts onto DocBook concepts such as articles.   I'm a longtime
>>> orgmode user and it would be much simpler to write a program in terms
>>> of the familiar org concepts (hierarchical entries, tags, properties
>>> etc).
>>> It would also be great if there was a way to import such an XML file
>>> back into org.  Then one could e.g. take Toodledo.com tasks and
>>> transform them into an orgmode file.
>>>
>>> There is an orgmode Python reader at
>>> http://www.members.optusnet.com.au/~charles57/GTD/orgnode.html
>>> and I plan to use that for now.  But it doesn't support all orgmode
>>> features, and more importantly it does its own parsing of orgfiles (so
>>> may not keep up with any future changes).   Using orgmode's own
>>> parser, and then exporting the results as XML, would be much more
>>> reliable.
>>>
>>> ilya
>>>
>>>
>>> _______________________________________________
>>> Emacs-orgmode mailing list
>>> Remember: use `Reply All' to send replies to the list.
>>> Emacs-orgmode@gnu.org
>>> http://lists.gnu.org/mailman/listinfo/emacs-orgmode
>>>
>>
>

Re: suggestion: "native" orgmode XML export (and import?)

[David Bremner]

Ilya Shlyakhter wrote:

>That's great, thanks!  I should be able to take it from there.
>It would be great if at some point this became official, and also
>included an XML exporter and specification.


There is some code around for converting between xml and sexp
representations.  A few minutes with google found me 

 http://nxg.me.uk/dist/lx/

and

 http://ssax.sourceforge.net/

Maybe that helps you at least start searching.

It wouldn't be hard to export xml from org-mode, but importing it
might be a bit more tedious, where sexp's are really a native format.

d

Re: suggestion: "native" orgmode XML export (and import?)

[Bastien]

Ilya Shlyakhter <ilya_shl@alum.mit.edu> writes:

> That's great, thanks!  I should be able to take it from there.
> It would be great if at some point this became official, and also
> included an XML exporter and specification.

FYI, I'll upload a slightly improved version of org-export.el next week,
together with documentation on how to write an exporter.  But the basic
structure of the parsed buffer is the same, you can use it safely.

In the meantime, it would be useful to describe what kind of XML output
do you want, because "XML" does not really describe anything per se.

Best,

-- 
 Bastien

Re: suggestion: "native" orgmode XML export (and import?)

[Ilya Shlyakhter]

> In the meantime, it would be useful to describe what kind of XML output
> do you want, because "XML" does not really describe anything per se.

I'm looking for XML output that would closely mirror the logical
structure of the org file, and that would contain all the information
in the orgfile (since it's easy to ignore the parts you don't need
during XML processing).  So, something like

<orgfile>
   <entry>
      <headline>Tasks</headline>
      <body>Here are the tasks I need to do</body>
      <children>
          <entry>
             <headline>Buy bread</headline>
             <todo-status>DONE</todo-status>
             <tags><tag>food</tag><tag>errands</tag></tags>
             <properties>
                 <property><name>Importance</name><value>1</value></property>

<property><name>Deadline</name><value><date><day>07</day><month>08</month><year>09</year></date></value></property>
             </properties>
           </entry>
        </children>
      </entry>
   </orgfile>

The details of the XML schema can of course change.   But it should
let you process org file data without having to parse any elements of
the org file (ideally, even dates) -- it would all be parsed by
orgmode's native parsing code and put into XML elements.

If there are questions about how to represent specific org elements in
XML I can try to write a more detailed spec.

thanks,

ilya




On Sat, Aug 8, 2009 at 7:48 AM, Bastien<bastienguerry@googlemail.com> wrote:
> Ilya Shlyakhter <ilya_shl@alum.mit.edu> writes:
>
>> That's great, thanks!  I should be able to take it from there.
>> It would be great if at some point this became official, and also
>> included an XML exporter and specification.
>
> FYI, I'll upload a slightly improved version of org-export.el next week,
> together with documentation on how to write an exporter.  But the basic
> structure of the parsed buffer is the same, you can use it safely.
>
> In the meantime, it would be useful to describe what kind of XML output
> do you want, because "XML" does not really describe anything per se.
>
> Best,
>
> --
>  Bastien
>

Re: suggestion: "native" orgmode XML export (and import?)

[B Smith-Mannschott]

On Sat, Aug 8, 2009 at 22:25, Ilya Shlyakhter<ilya_shl@alum.mit.edu> wrote:
>> In the meantime, it would be useful to describe what kind of XML output
>> do you want, because "XML" does not really describe anything per se.
>
> I'm looking for XML output that would closely mirror the logical
> structure of the org file, and that would contain all the information
> in the orgfile (since it's easy to ignore the parts you don't need
> during XML processing).  So, something like
>
> <orgfile>
>   <entry>
>      <headline>Tasks</headline>
>      <body>Here are the tasks I need to do</body>
>      <children>
>          <entry>
>             <headline>Buy bread</headline>
>             <todo-status>DONE</todo-status>
>             <tags><tag>food</tag><tag>errands</tag></tags>
>             <properties>
>                 <property><name>Importance</name><value>1</value></property>
>
> <property><name>Deadline</name><value><date><day>07</day><month>08</month><year>09</year></date></value></property>
>             </properties>
>           </entry>
>        </children>
>      </entry>
>   </orgfile>
>
> The details of the XML schema can of course change.   But it should
> let you process org file data without having to parse any elements of
> the org file (ideally, even dates) -- it would all be parsed by
> orgmode's native parsing code and put into XML elements.
>
> If there are questions about how to represent specific org elements in
> XML I can try to write a more detailed spec.
>

Out of curiosity, how would you want to handle textual content? Pass
it through unchanged with org's wiki-like markup in tact, or somehow
xml-ified?:

*foo* --> *foo*
*foo* --> <strong>foo</strong>

// Ben

Re: suggestion: "native" orgmode XML export (and import?)

[Ilya Shlyakhter]

> Out of curiosity, how would you want to handle textual content? Pass
> it through unchanged with org's wiki-like markup in tact, or somehow
> xml-ified?:
>
> *foo* --> *foo*
> *foo* --> <strong>foo</strong>

Probably the latter, since converting XMLified content to wiki markup
(or any other form) is easy and does not require manual parsing, while
the other direction is harder.
But for a first version, just leaving the markup as text would be fine too.

What would help a lot is, if XML content could be imported as a subtree.
Then you could have "python plugins" that take a subtree (or an entire orgfile)
converted to XML, process it into updated XML, and then that updated
XML gets imported
back into org and replaces the original subtree.   This would make it
easy for users
to implement something like http://tinyurl.com/nskzs8 , and probably
other custom
functions.

On Sun, Aug 9, 2009 at 3:19 PM, B Smith-Mannschott<bsmith.occs@gmail.com> wrote:
> On Sat, Aug 8, 2009 at 22:25, Ilya Shlyakhter<ilya_shl@alum.mit.edu> wrote:
>>> In the meantime, it would be useful to describe what kind of XML output
>>> do you want, because "XML" does not really describe anything per se.
>>
>> I'm looking for XML output that would closely mirror the logical
>> structure of the org file, and that would contain all the information
>> in the orgfile (since it's easy to ignore the parts you don't need
>> during XML processing).  So, something like
>>
>> <orgfile>
>>   <entry>
>>      <headline>Tasks</headline>
>>      <body>Here are the tasks I need to do</body>
>>      <children>
>>          <entry>
>>             <headline>Buy bread</headline>
>>             <todo-status>DONE</todo-status>
>>             <tags><tag>food</tag><tag>errands</tag></tags>
>>             <properties>
>>                 <property><name>Importance</name><value>1</value></property>
>>
>> <property><name>Deadline</name><value><date><day>07</day><month>08</month><year>09</year></date></value></property>
>>             </properties>
>>           </entry>
>>        </children>
>>      </entry>
>>   </orgfile>
>>
>> The details of the XML schema can of course change.   But it should
>> let you process org file data without having to parse any elements of
>> the org file (ideally, even dates) -- it would all be parsed by
>> orgmode's native parsing code and put into XML elements.
>>
>> If there are questions about how to represent specific org elements in
>> XML I can try to write a more detailed spec.
>>
>
> Out of curiosity, how would you want to handle textual content? Pass
> it through unchanged with org's wiki-like markup in tact, or somehow
> xml-ified?:
>
> *foo* --> *foo*
> *foo* --> <strong>foo</strong>
>
> // Ben
>

Re: suggestion: "native" orgmode XML export (and import?)

[Sebastian Rose]

Hi Ilya,


is XHTML not XML enough?

I use the XHTML to extract snippets and their language, tags, sections,
section IDs...

The XHTML export is able to export each and every information an
Org-file can possibly contain.  All textual content and links, IDs
... are converted to valid X(HT)ML.

As XHTML export is already used a lot, why not put more energy into
improving that existing one (specification), instead of another
exporter?

 XHTML => SAX-parser (e.g.) => what ever you want

???



   Sebastian



Ilya Shlyakhter <ilya_shl@alum.mit.edu> writes:
>> In the meantime, it would be useful to describe what kind of XML output
>> do you want, because "XML" does not really describe anything per se.
>
> I'm looking for XML output that would closely mirror the logical
> structure of the org file, and that would contain all the information
> in the orgfile (since it's easy to ignore the parts you don't need
> during XML processing).  So, something like
>
> <orgfile>
>    <entry>
>       <headline>Tasks</headline>
>       <body>Here are the tasks I need to do</body>
>       <children>
>           <entry>
>              <headline>Buy bread</headline>
>              <todo-status>DONE</todo-status>
>              <tags><tag>food</tag><tag>errands</tag></tags>
>              <properties>
>                  <property><name>Importance</name><value>1</value></property>
>
> <property><name>Deadline</name><value><date><day>07</day><month>08</month><year>09</year></date></value></property>
>              </properties>
>            </entry>
>         </children>
>       </entry>
>    </orgfile>
>
> The details of the XML schema can of course change.   But it should
> let you process org file data without having to parse any elements of
> the org file (ideally, even dates) -- it would all be parsed by
> orgmode's native parsing code and put into XML elements.
>
> If there are questions about how to represent specific org elements in
> XML I can try to write a more detailed spec.
>
> thanks,
>
> ilya
>
>
>
>
> On Sat, Aug 8, 2009 at 7:48 AM, Bastien<bastienguerry@googlemail.com> wrote:
>> Ilya Shlyakhter <ilya_shl@alum.mit.edu> writes:
>>
>>> That's great, thanks!  I should be able to take it from there.
>>> It would be great if at some point this became official, and also
>>> included an XML exporter and specification.
>>
>> FYI, I'll upload a slightly improved version of org-export.el next week,
>> together with documentation on how to write an exporter.  But the basic
>> structure of the parsed buffer is the same, you can use it safely.
>>
>> In the meantime, it would be useful to describe what kind of XML output
>> do you want, because "XML" does not really describe anything per se.
>>
>> Best,
>>
>> --
>>  Bastien